IndiaFinBench: An Evaluation Benchmark for Large Language Model Performance on Indian Financial Regulatory Text
The first expert-annotated benchmark for LLM reasoning over Indian financial regulation — where numerical thresholds and amendment chains break strong models.
The discovery in one figure
The paper in five minutes
Large language models are increasingly asked legal and financial questions, but nearly all of their testing uses American and European text. This benchmark tests twelve leading models on India's actual financial rulebook — SEBI and RBI regulations spanning 1992 to 2026 — with 406 questions written and verified by hand. The questions require exactly what real compliance work requires: interpreting rules, calculating with legal thresholds, spotting contradictions, and tracking rules that changed over time. The clearest finding: numbers are the hardest. On numerical reasoning alone, model scores spread by 36 points.
The research question
Financial NLP benchmarks are Western-heavy. How do frontier LLMs handle SEBI/RBI regulatory text with dense amendment chains, jurisdiction-specific terminology, and numerical thresholds?
How it works
406 expert-annotated QA items over 192 SEBI/RBI documents (1992–2026) across regulatory interpretation, numerical reasoning, contradiction detection, and temporal reasoning; 12 contemporary LLMs; model-based and human annotation validation; a deployed hybrid-RAG open-book system with retrieval ablations.
- 01Curate the corpus192 SEBI/RBI regulatory documents, 1992–2026
- 02Write expert QA406 items: interpretation · numerical · contradiction · temporal
- 03Validate annotationsmodel-based and human cross-checks, agreement reported
- 04Evaluate 12 LLMsstrict scoring plus judge-corrected audit (format vs reasoning)
- 05Ship it openpublic dataset, leaderboard, and live retrieval-augmented demo
Experimental results
Strict accuracy spans 70.4–89.7%. Numerical reasoning is the most discriminative axis (35.9 pp spread). Judge-corrected audits separate format non-compliance from true reasoning failure. Hybrid RRF retrieval lifts Recall@5 to 0.785 (+9.7 pp over dense-only).
17492786289.785.583.783.370.4Numerical reasoning alone spreads models by 35.9 points.
- 406expert QA items · 192 documents
- 35.9 ppnumerical-reasoning spread across 12 LLMs
- 0.785Hybrid RRF Recall@5 (+9.7 pp)
Figures from the paper
Figures generated by the paper's own pipeline — reproducible from the repository.
What this changes
Extends deployment-aware evaluation to LLMs and an underrepresented jurisdiction, with the dataset, leaderboard, and live demo all public.
Citation
Manuscript under review — citation will be posted on acceptance. Reach me at rajveerpall04@gmail.com.