PaperlayerView exact revision on arXiv

Source-linked AI summary

IndicParam: Benchmark to evaluate LLMs on low-resource Indic Languages

Ayush Maheshwari, Kaushal Sharma, Vivek Patel, Aditya Maheshwari

arXiv:2512.00333v3cs.CL

TL;DR

Systematic evaluation remains limited for low- and extremely-low-resource Indic languages, and existing benchmarks provide partial or no support for several understudied languages. IndicParam is a benchmark of 13K+ questions for 11 low- and extremely-low-resource Indic languages, with linguistic or knowledge-based labels, evaluated across 20 LLMs. Across 20 evaluated LLMs, Gemini-2.5 achieves the highest overall accuracy at 58%, while the strongest contemporary systems remain challenging on these languages.

  • Problem

    Systematic evaluation remains limited for low- and extremely-low-resource Indic languages, and existing benchmarks provide partial or no support for several understudied languages.

    Source: §1 · p. 1

  • Method

    IndicParam is a benchmark of 13K+ questions for 11 low- and extremely-low-resource Indic languages, with linguistic or knowledge-based labels, evaluated across 20 LLMs.

    Source: §1 · pp. 2–3

  • Results

    Across 20 evaluated LLMs, Gemini-2.5 achieves the highest overall accuracy at 58%, while the strongest contemporary systems remain challenging on these languages.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    IndicParam fills a key gap in Indic benchmarks and the findings underline the need for more balanced pretraining and adaptation for under-represented Indic languages.

    Source: §5 · p. 9

    The benchmark draws exclusively on UGC-NET linguistics and literature papers and uses zero-shot log-likelihood multiple-choice evaluation, so it may not represent everyday language use or results under richer prompting.

    Source: §6 · p. 9

Abstract

from arXiv · show

While large language models excel on high-resource multilingual tasks, low- and extremely low-resource Indic languages remain severely under-evaluated. We present IndicParam, a human-curated benchmark of over 13,000 multiple-choice questions covering 11 such languages (Nepali, Gujarati, Marathi, Odia as low-resource; Dogri, Maithili, Rajasthani, Sanskrit, Bodo, Santali, Konkani as extremely low-resource) plus Sanskrit-English code-mixed set. We evaluated 20 LLMs, both proprietary and open-weights, which reveals that even the top-performing Gemini-2.5 reaches 58% average accuracy, followed by GPT-5 (45) and DeepSeek-3.2 (43.1). We additionally label each question as knowledge-oriented or purely linguistic to discriminate factual recall from grammatical proficiency. Further, we assess the ability of LLMs to handle diverse question formats-such as list-based matching, assertion-reason pairs, and sequence ordering-alongside conventional multiple-choice questions. IndicParam provides insights into limitations of cross-lingual transfer and establishes a challenging benchmark for Indic languages. The dataset is available at https://huggingface.co/datasets/bharatgenai/IndicParam. Scripts to run benchmark are present at https://github.com/ayushbits/IndicParam.

1 Introduction

Evaluation of low- and extremely-low-resource Indic languages remains limited, especially for understudied languages and code-mixed use. IndicParam introduces a 13K+ question benchmark and evaluates 20 LLMs, with Gemini-2.5 achieving the highest reported average accuracy at 58%.

1 Introduction

  • Existing Indic benchmarks have substantial coverage of major languages but partial or no support for several understudied languages, and rarely cover code-mixed use.The introduction notes that most existing benchmarks concentrate on 11 major languages representing more than 93% of speakers.Source: §1 · pp. 1–2
  • FineWeb2 contains 4.49 billion words for these 11 languages, of which Nepali, Marathi, Gujarati, and Odia account for 4.45 billion.The remaining six languages have only 36 million words collectively, despite the 11 languages representing about 280 million speakers.Source: §1 · p. 1
  • The authors classify Marathi, Gujarati, Nepali, and Odia as low-resource and the remaining seven languages as extremely low-resource based on web-corpus availability.The benchmark also includes a code-mixed Sanskrit-English set.Source: §1 · pp. 1–2
  • 58% average accuracy makes Gemini-2.5 the top-performing evaluated model; GPT-5 follows at 45%.No model exceeds a 58% average score on IndicParam.Source: §1 · p. 2
  • IndicParam provides 13K+ questions across 11 low- and extremely-low-resource Indic languages, plus a code-mixed English-Sanskrit variant.The benchmark also supplies question-type and question-category annotations and supports evaluation of 20 proprietary and open-weight LLMs.Source: §1 · pp. 2–3

2 Related Work

Existing Indic benchmarks largely emphasize widely used languages or tasks adapted from English datasets, leaving extremely low-resource languages with limited coverage. IndicParam adds graduate-level questionnaires in four low-resource and seven extremely low-resource languages to assess language competence and general knowledge.

2 Related Work

  • Indic-GenBench covers 29 Indic languages with reading-comprehension and translation tasks primarily adapted from English benchmarks, while IndicParam uses graduate-level examination questions.Indic-GenBench QA restricts answers to spans supported by the given passage.Source: §2 · p. 3
  • IndicGLUE focuses on supervised NLP tasks for prominent Indic languages, particularly Hindi, while IndicXTREME expands task coverage and language coverage to 11 Indic languages.The supplied description of IndicXTREME notes that many long-tail languages remain outside its scope.Source: §2 · p. 3
  • Recent benchmarks include graduate-level Hindi questionnaires, a 79K-question set across 11 Indian languages, English-language cultural-alignment benchmarks, and multimodal cultural evaluation across 15 Indic languages.These works are ParamBench, MILU, Sanskriti and Pariksha, and Drishtikon, respectively.Source: §2 · p. 3
  • IndicParam targets the limited coverage of extremely low-resource languages with graduate-level questionnaires in four low-resource and seven extremely low-resource languages.The questionnaires assess both language competence and general knowledge.Source: §2 · p. 3

3 IndicParam

IndicParam is a 13,207-question benchmark for language understanding and general knowledge in 11 Indic languages, with a Sanskrit-English code-mixed set. Its construction classifies questions by skill and format, audits text quality, and documents model-evaluation setup.

3 IndicParam

  • IndicParam contains 13,207 questions evaluating language understanding and general knowledge in 11 Indic languages, plus a Sanskrit-English code-mixed question set.Questions were collected from UGC-NET papers and answer keys published between 2012 and 2018.Source: §3 · p. 3

3.1 Question Classification

  • Question classification separates language-understanding questions from general-knowledge questions, with LU shares ranging from under 1% in Gujarati to nearly 28% in Rajasthani.Odia and Sanskrit also have over 20% LU coverage.Source: §3.1 · p. 4

3.2 Question Type Classification

  • The benchmark labels six question formats: MCQ (73%), list matching (9%), ordering (6.5%), assertion and reasoning (6.1%), incorrect-statement identification (4.1%), and blank filling (1.1%).MCQs are less prevalent in Bodo (36.6%), Gujarati (33%), and Dogri (21.7%) than in the benchmark overall.Source: §3.2 · p. 4, Table 3 · p. 4

3.3 Quality control and Verification

  • A random 5% post-verification audit reported WER 0.68% and CER 0.12%, after script-tailored OCR and expert manual correction.The audit sample contained 10,427 words and 61,547 characters.Source: §3.3 · pp. 4–5

3.4 Setup

  • The evaluation covers varied open-weight model families and proprietary systems, with checkpoint, inference, and API-use details provided for the models.The described families include Qwen, Llama, Gemma, Aya, Mistral, DeepSeek, GPT-OSS, and frontier proprietary models.Source: §3.4 · pp. 5–6

4 Results and discussion

IndicParam scores remain moderate even as accuracy rises with model scale, and performance varies across low-resource languages and between knowledge and language-understanding tasks. Gemini-2.5 leads the reported comparisons, while Sanskrit, code-mixed Sanskrit-English, and some language-specific tasks show distinct patterns.

4.1 Overall Model Performance

  • 37.3% for Gemma-3-27B leads the 8B–27B models, while Qwen3-4B and Qwen2.5-3B score 30.8 and 30.1 among models below 8B.Gemma-3-12B scores 35.3 and Mistral-Small-3.1-24B scores 35.1 in the medium-size category.Source: §4.1 · p. 6
  • 58% average accuracy for Gemini-2.5 leads the overall results, followed by GPT-5 (45), DeepSeek-V3.2 (43.1), Claude-Haiku-4.5 (42.7), and Grok-4-Fast (39.6).The reported results show accuracy rising with model scale, while multilingual or Indic-rich fine-tuning also matters.Source: §4.1 · p. 6

4.2 Performance on Low- and Extremely Low-Resource Indic Languages

  • On extremely low-resource languages, Gemini-2.5 averages 59.2%, but most models score below 36% on Bodo and Santali; in the low-resource group, it averages 54.9%.Gemini-2.5 scores 80.3 on Sanskrit-English code-mixed text and 64.5 on Odia; other models score below 50% across the four low-resource languages.Source: §4.2 · pp. 6, 8

4.3 Performance on GK and LU Tasks

  • The evaluation distinguishes General Knowledge questions requiring factual reasoning and culturally grounded world knowledge from Language-Specific Understanding questions probing language proficiency; the latter are a very small share, particularly in Gujarati (0.6%).Source: §4.3 · p. 8

4.3.1 General Knowledge

  • Gemini-2.5 achieves the highest average (58), ahead of GPT-5 (54.8), and leads in Sanskrit (74.4) and Sanskrit–English code-mixed contexts (80.3); medium-sized models (8B–27B) top out at 37.9, while small models (<8B) stay below 32.Source: §4.3.1 · p. 8

4.3.2 Language-Specific Understanding

  • 69.8 is Gemini-2.5’s Language-Specific Understanding average, and linguistic-task performance is higher than General Knowledge on every language except Sanskrit.General Knowledge accuracy correlates more strongly with model scale (r = 0.91) than language understanding does (r = 0.73); the two dimensions correlate at r = 0.79. Gemma-3-27B and Mistral3.1-24B outperform many larger dense models on linguistic tasks, and the paper calls for diverse Indic data, cross-lingual continuation training, and architectures prioritizing morphology and syntax.Source: §4.3.2 · p. 8

5 Conclusion

IndicParam broadens evaluation for low- and extremely low-resource Indic languages, but results show that contemporary LLMs remain limited on the benchmark. The authors conclude that more balanced pretraining and adaptation are needed for under-represented Indic languages.

5 Conclusion

  • IndicParam contains over 13,000 questions across 11 low- and extremely low-resource Indic languages, and only Gemini-2.5 exceeds 45% average accuracy.The authors report that larger closed models and DeepSeek-3.2 consistently outperform open-weight alternatives, while identifying more balanced pretraining and adaptation as needed.Source: §5 · p. 9

6 Limitations

The authors identify limits in both benchmark scope and evaluation setup. The exam-derived questions may not represent everyday or domain-general language use, and zero-shot log-likelihood multiple-choice results may differ under other prompting strategies.

6 Limitations

  • IndicParam uses only UGC-NET linguistics and literature papers, so its academic exam style may not represent everyday or domain-general language use.Evaluation uses zero-shot log-likelihood multiple choice, and results may differ with few-shot or chain-of-thought prompting.Source: §6 · p. 9

7 Ethics

The benchmark uses public resources and native-speaker annotation, with a review process for corrected questions and answer keys. It is released for non-commercial research use.

7 Ethics

  • IndicParam is released for non-commercial research use, and the authors state that no personally identifiable exam-candidate information appears in the benchmark.The code and annotation scripts are licensed under MIT; commercial users are directed to contact UGC-NTA for permissions.Source: §7 · p. 9

A Annotation Setup

  • Official UGC-NET papers and answer keys were collected, converted to machine-readable files, and processed through OCR before tagging and answer-key alignment.The OCR step addressed double-column layouts and PDFs with non-selectable text.Source: A Annotation Setup · p. 11
  • Native-speaker annotators corrected OCR errors, restored characters and diacritics, standardized formatting, and stored each question with explicit metadata fields.Fields include question type, four answer options, correct option, language, subject, year, and a unique question identifier.Source: A Annotation Setup · p. 11
  • Annotators received written guidelines and trial-file feedback, while a senior reviewer checked corrected question-answer pairs against the original exam papers.The two-step review checked OCR corrections, answer-key alignment, and that exactly one option was marked correct.Source: A Annotation Setup · p. 11

A.1 Quality Control and Verification

The authors describe quality-control and verification steps for the benchmark, alongside ethical, intended-use, and licensing considerations.

A.1 Quality Control and Verification

  • The extraction process used script-specific OCR, followed by expert review and manual correction to address recognition errors and structural inconsistencies.Devanagari text was extracted with LLMWhisperer; Surya OCR was used for Gujarati and Odia, which required intensive manual correction.Source: §A.1 · p. 11
  • A randomized 5% post-verification audit found a 0.68% WER and 0.12% CER.The sample comprised 10,427 words and 61,547 characters, with 71 errors.Source: §A.1 · p. 11

B Ethical considerations

  • The dataset uses public exam materials and underwent internal review of provenance, societal impacts, licensing, and responsible use; no formal IRB review was conducted.The authors state that no new human-participant data were collected.Source: B Ethical considerations · pp. 11–12

B.1 Intended use and beneficiaries

  • IndicParam is intended as a graduate-level evaluation resource for researchers, educators, and developers working on educational tools or knowledge-intensive systems.The authors aim for fair, transparent, and reproducible model benchmarking.Source: §B.1 · p. 12

B.2 Licensing and copyright considerations

  • The authors report no explicit redistribution license on the official exam site and release processed question-answer pairs under CC-BY-4.0, requiring attribution to the paper and original source.They describe the transformation as fair dealing for non-commercial research and education and do not redistribute the original PDFs.Source: §B.2 · p. 12

C Question Types-wise results

IndicParam includes conventional multiple-choice items as well as several other question formats, with distributions reported across languages.

C Question Types-wise results

  • The section directs readers to Table 7 for question-type distributions and Table 8 for model performance by question type.MCQs dominate the distribution, while the other formats contribute additional question types.Source: C Question Types-wise results · p. 12, §C.1 · p. 12

C.1 Question Types

  • The benchmark’s question formats include list matching, assertion-reasoning, identifying incorrect statements, ordering, and blank filling in addition to MCQs.Table 7 reports the distribution of these formats across languages.Source: §C.1 · p. 12, C Question Types-wise results · p. 12

C.2 Question type-wise results

Model performance varies across question formats, and evaluation uses a zero-shot prompt. The results identify different strengths across model sizes and categories.

C.2 Question type-wise results

  • Model strengths differ by question format: GPT-5 leads across almost all categories, while Claude-4.5 performs best on incorrect-statement questions.The reported format-specific leaders also vary among smaller, medium-sized, large, and mixture-of-experts models.Source: §C.2 · p. 12

D Zero shot prompting template

E Performance on GK and LU questions

Performance improves with model scale, though gains and variability differ between general-knowledge and language-understanding questions. The section also presents sample questions from the benchmark’s languages.

E Performance on GK and LU questions

F Examples questions in IndicParam

Related papers

Loading 2512.00333v3…