IndiaFinBench: An Evaluation Benchmark for Large Language Model Performance on Indian Financial Regulatory Text
A public benchmark of 406 expert-annotated questions on Indian financial regulation, with twelve LLMs evaluated zero-shot.
The paper in five minutes
Financial NLP benchmarks are built almost entirely from Western sources such as SEC filings and US earnings reports. IndiaFinBench is written from regulation issued by SEBI and RBI, text those benchmarks do not cover, and it tests twelve LLMs on four kinds of question: interpreting a rule, doing the arithmetic it implies, spotting a contradiction, and placing a rule in time.
The research question
How well do current LLMs handle Indian financial regulatory text, which existing financial benchmarks do not cover?
How it works
406 expert-annotated question-answer pairs drawn from 192 SEBI and RBI documents span regulatory interpretation (174), numerical reasoning (92), contradiction detection (62) and temporal reasoning (78). Annotation quality is checked by a model-based secondary pass and a 180-item human inter-annotator study over three rounds. Twelve models are evaluated zero-shot.
- 01Build the benchmark406 question-answer pairs from 192 SEBI and RBI documents
- 02Four task typesinterpretation 174 · numerical 92 · contradiction 62 · temporal 78
- 03Validate the labelsa model-based second pass and a 180-item human agreement study
- 04Evaluate twelve modelszero-shot, with 10,000-resample bootstrap significance tests
Experimental results
Accuracy ranges from 70.4% (Gemma 4 E4B) to 89.7% (Gemini 2.5 Flash), and every model beats a non-specialist human baseline of 69.0%. Numerical reasoning separates the models most, with a 35.9-point spread. Bootstrap testing with 10,000 resamples finds three statistically distinct performance tiers.
- 406expert-annotated regulatory questions
- 70.4% to 89.7%zero-shot accuracy across twelve models
- 35.9 ptsspread on numerical reasoning, the most discriminative task
Stated honestlyHuman inter-annotator agreement is lower than the model-based check (kappa 0.645 against 0.918 on contradiction detection), and the human baseline is a single non-specialist reference point.
What this changes
Regulatory text outside Western corpora is where a financial assistant would actually be deployed in India, and it now has a public test set with released model outputs.
Resources
- Benchmark Evaluation
- NLP · LLMs
- Financial ML
Citation
Preprint. The public version is linked above. Reach me at rajveerpall04@gmail.com.