PaperlayerView exact revision on arXiv

Source-linked AI summary

L3Cube-IndicQuest v2: A Large-Scale Multilingual Benchmark for Evaluating Factual Knowledge of Large Language Models Across Indic Languages

Rinit Jain, Tirthraj Mahajan, Advait Joshi, Raviraj Joshi

arXiv:2608.15535v1cs.CLcs.LG

TL;DR

India-specific factual knowledge remains weakly measured across Indic languages and regional domains. L3Cube-IndicQuest v2 addresses this with a multilingual benchmark and finds consistent model rankings across evaluation protocols, with Gemma4 31B outperforming Sarvam 30B in every evaluated Indic language.

  • Problem

    Existing factual benchmarks are English-centric and do not adequately measure India-specific knowledge across regional domains and Indic languages.

  • Method

    The paper constructs and releases a 69,420-pair benchmark across 20 languages using curriculum-grounded data, automated validation, semantic deduplication, and human verification.

  • Results

    All three evaluation protocols produce the same ranking, with Gemma4 31B leading Sarvam 30B in every evaluated language.

  • Takeaways & Limitations

    The results challenge the assumption that Indic-native pre-training alone is sufficient to outperform a capable general-purpose model, while human oversight remains essential.

  • Takeaways & Limitations

    The multilingual evaluation covers only 9 of 19 Indic languages and 2 of 6 models, limiting conclusions about the remaining languages and models.

Abstract

from arXiv · show

We present L3Cube-IndicQuest v2, a large-scale gold-standard multilingual question-answering benchmark for evaluating the India-specific factual knowledge of Large Language Models (LLMs). The benchmark comprises 3,471 curriculum-grounded English question--answer pairs spanning nine domains, curated from educational curricula, competitive examination materials, and domain-specific reference books. We introduce a practical hybrid construction strategy that combines context-grounded LLM-based question generation and validation with semantic deduplication and human verification, enabling scalable creation of benchmark data while preserving annotation quality. The benchmark is translated into 19 Indic languages, yielding a publicly released multilingual dataset of 69,420 question--answer pairs across 20 languages. We evaluate six LLMs under three protocols: LLM-as-a-judge and two deterministic lexical criteria, exact-substring and word-overlap matching. All three produce almost the same model ranking, showing that the results do not depend on the choice of judge. The frontier commercial model leads by a wide margin, and among open-weight models Gemma4 31B outperforms the Indic-specialised Sarvam 30B in every evaluated Indic language.

1 Introduction

The introduction motivates IndicQuest v2 as a response to weak evaluation of India-specific knowledge in English-centric benchmarks and presents a scalable agent–human construction pipeline. It then outlines the benchmark’s multilingual scope, evidence that automated validation is insufficient, and evaluation across three protocols.

  • Motivation: Existing multilingual models often perform poorly on Indian history, law, regional culture, and curriculum-level facts, while Indic languages remain under-represented in training data.Widely used knowledge benchmarks are also predominantly English-centric.
  • Motivation: English-only short-form QA benchmarks do not measure country- or culture-specific knowledge and require deduplication and label checking.SimpleQA established tractable grading with one correct answer, while SimpleQA Verified highlighted quality-control needs.
  • Construction: IndicQuest v2 combines LLM-based generation and validation, semantic deduplication, and human verification to build curriculum-grounded QA data at scale.Agents generate and cross-check candidate pairs, while three annotators review surviving items.
  • Benchmark: 3,471 English question–answer pairs across nine domains expand to 69,420 multilingual pairs spanning 20 languages in the publicly released benchmark.The benchmark targets India-specific factual knowledge and hallucination evaluation.
  • Validation and evaluation: 20–25% of pairs that passed both automated gates were removed by human review, demonstrating the limits of automated validation.Six LLMs are evaluated using an LLM judge, exact-substring matching, and word-overlap matching, with rankings holding across all three protocols.

2 Related Work

IndicQuest v2 extends its predecessor while distinguishing itself from multilingual, regional, and Indic QA benchmarks through open-domain, closed-book factual recall grounded in Indian curricular texts. It also reports deterministic evaluation criteria alongside LLM-as-a-judge scoring to address judge-specific effects.

  • IndicQuest v1: IndicQuest v1 contains 4,000 question–answer pairs across English and 19 Indic languages, covering five domains; English outperformed every Indic language, while Manipuri, Odia, and Urdu were weakest.The v1 comparison used Llama-3.1-405B-Instruct as judge.
  • Multilingual QA Benchmarks: TyDi QA, XQuAD, MLQA, and MEGA provide multilingual or cross-lingual benchmarks, whereas IndicQuest targets open-domain recall without a passage.The cited multilingual benchmarks cover eleven, cross-lingual, or broader multi-dataset settings, while IndicQuest differs in task format.
  • Short-Form Factuality Benchmarks: SimpleQA Verified shows that deduplication, topic balancing, label noise, and redundancy can shift measured accuracy; IndicQuest v2 applies short-answer evaluation to India-specific knowledge in 20 languages.IndicQuest v2 uses the same short-answer design while releasing the benchmark multilingual.
  • Culture- and Region-Specific Evaluation: BLEnD, INCLUDE, MILU, IndicGenBench, and PARIKSHA address cultural, regional, exam-style, generation, or evaluator-agreement dimensions, whereas IndicQuest v2 grounds open-domain factual QA in Indian curricular texts.The comparison distinguishes IndicQuest v2 from everyday cultural knowledge, exam-style understanding, generation, and human–LLM evaluator agreement benchmarks.
  • Indic NLP Datasets and Models: IndicQA, Singh et al.’s closed-form benchmark, and IndicSQuAD evaluate comprehension over supplied passages, whereas IndicQuest v2 measures closed-book recall.IndicQA uses Wikipedia context paragraphs, and IndicSQuAD is an extractive set for nine Indic languages derived from SQuAD.
  • LLM-as-a-Judge: Because judge quirks can affect reported scores, IndicQuest v2 reports two deterministic criteria alongside LLM-as-a-judge evaluation.This builds on prior LLM-as-a-judge work including MT-Bench, Chatbot Arena, AlpacaEval, and surveys of judge failure modes.

3 Dataset Construction

IndicQuest v2 is constructed from formal Indian educational, examination, and domain-specific reference texts rather than web sources, with strict constraints ensuring short, unique, self-contained, India-specific questions. A staged pipeline generates, validates, deduplicates, manually verifies, and translates the resulting English benchmark into 19 Indic languages.

  • Source Selection: IndicQuest v2 draws on NCERT, state-board, CBSE, ICSE, HSC, UPSC, MPSC, legal, constitutional, science-history, and sports-history references.Indian sports history and Indian contributions to science are identified as the most novel domains because they are largely absent from school curricula.
  • Question Constraints: Questions must have one-to-five-word verbatim answers, exactly one defensible answer, self-contained wording, and India-specific content.Ambiguous pairs are discarded rather than repaired, and short answers support plain string-comparison evaluation.
  • Construction Pipeline: A six-stage pipeline extracts passages, generates and scores candidate pairs, semantically deduplicates them, verifies survivors with three annotators, and translates the English set.Deduplication retains the highest-scoring question from each similarity cluster, while singleton clusters pass through unchanged.
  • Verification and Coverage: 3,471 English pairs span nine domains, and expert review removes 20–25% of the set after automated checks.Annotators independently flag factual errors, ambiguity, triviality, or weak India specificity; any flagged pair is removed after discussion.
  • Translation: 19 Indic languages receive translations of the verified pairs using GPT-5.4 Mini via batched API calls.Google Cloud and Gemini Translation were tried first but handled Law and Political Science terminology inconsistently.

4 Evaluation Methodology

The evaluation compares six LLMs spanning model scales, architectures, and training philosophies on questions answered without retrieval or context augmentation. Responses are assessed by an LLM judge and two deterministic lexical criteria, with domain accuracy defined as the fraction of correct verdicts.

  • Model selection: Six LLMs span frontier commercial, general-purpose, open-weight, Indic-native, mid-sized, and community-baseline models.The selection tests multilingual transfer, open-weight competitiveness, Indic-specific pre-training, resource-constrained deployment, and performance-floor behavior.
  • Evaluation protocols: Each model answers every question in the question’s language without retrieval or context augmentation, and Gemma 3 12B returns binary correctness verdicts with justifications.The judge receives the question, gold answer, and model response, while accepting paraphrases and alternate transliterations.
  • Evaluation protocols: Two deterministic criteria score the same 3,471 responses per model: exact substring requires a contiguous gold-answer match, whereas word overlap requires all gold-answer words in any order.Both criteria are case-insensitive; word overlap additionally strips edge punctuation.
  • Evaluation protocols: Neither lexical criterion subsumes the other, and both are stricter than the judge while remaining fully reproducible from the released responses.Substring matching ignores word boundaries, whereas word overlap ignores order but requires exact tokens.
  • Metrics: Domain accuracy is the fraction of correct verdicts among the questions in domain d, while weighted accuracy aggregates domains according to their sizes.The definition applies under whichever evaluation protocol is in use.

5 Results and Analysis

Results show a stable model ranking across evaluation protocols, with Gemma4 31B nearly matching GPT-5.4 Mini and outperforming Sarvam 30B across languages. Performance varies by domain and language, while lexical matching underestimates measured knowledge and curriculum-grounded generation still requires substantial oversight.

  • 5 Results and Analysis: 62.2% places Gemma4 31B above GPT-5.4 Mini at 60.1%, Sarvam 30B at 48.7%, and Gemma2 9B at 41.1% on English.GPT-5.4 Mini overtakes Gemma4 31B in Geography, Political Science, and Sports, while Llama 3.1 8B overtakes Gemma2 9B in Law.
  • 5 Results and Analysis: At most 0.5 pp separates the two lexical criteria on any model, and all three protocols produce almost the same ranking.The results support treating the ranking as a property of the models rather than the selected judge.
  • 5 Results and Analysis: 17–25 pp lower lexical scores form a strict reproducible floor, with the largest shortfalls for Indic answers having multiple accepted spellings.The judge is therefore reported as the primary protocol, while exact-substring and word-overlap matching remain deterministic criteria.
  • 5 Results and Analysis: 13.5 pp separates Gemma4 31B at 62.2% from Sarvam 30B at 48.7%, while Gemma4 leads Sarvam in every language and domain.The gap reaches 17.4–28.3 pp across Indic languages and peaks in Hindi; the comparison does not isolate pre-training strategy from architecture and training differences.
  • 5 Results and Analysis: 66.9% is Gemma4 31B’s Hindi accuracy versus 62.2% in English, whereas Tamil is weakest for both models at 56.9% for Gemma4 and 33.2% for Sarvam.Across languages, Gemma4 ranges from 56.9–66.9%, compared with Sarvam’s 33.2–48.7%; language weaknesses differ beyond Tamil.
  • 5 Results and Analysis: ∼70% makes Commercial Studies the strongest domain, while Sports at ∼41% and Law at ∼46% are weakest across models and languages.The results also report a 20–25% manual removal rate for curriculum-grounded generated pairs, indicating substantial expert oversight requirements.

6 Future Work

Future work will add region-specific question curation and broaden evaluation across all 19 languages, six models, and three protocols. The planned regional split aims to capture deeper, culturally grounded knowledge.

  • 6 Future Work: An 80–20 dataset split would reserve 20% of questions for native regional curation, testing regional depth beyond translating one English pool.For Marathi, native questions would draw on Maharashtra-specific sources to add cultural context.
  • 6 Future Work: Evaluation will extend to all 19 languages and all six models under all three protocols.
  • 6 Future Work: The planned evaluation considers that string matching penalises spelling variants.

7 Conclusion

L3Cube-IndicQuest v2 is a multilingual benchmark for evaluating LLM knowledge in the Indian context, built with automated generation and validation, semantic deduplication, and human verification. Its evaluation protocols agree on the model ranking, while Gemma4 31B trails Sarvam 30B by reported margins that widen across Indic languages.

  • Benchmark contribution: L3Cube-IndicQuest v2 evaluates LLM knowledge in the Indian context using a benchmark constructed with a two-agent Gemini 2.5 Flash pipeline, semantic deduplication, and human verification.The benchmark combines automated question generation and validation with semantic deduplication and human verification.
  • Benchmark contribution: 3,471 English question–answer pairs span nine domains, while 69,420 parallel pairs cover 20 languages.The dataset extends the English benchmark into a large multilingual collection.
  • Evaluation findings: All three evaluation protocols agree on the model ranking, with Gemini 2.5 Flash leading.The consistent ranking indicates that the reported ordering is shared across the evaluated protocols.
  • Evaluation findings: 13.5 pp separates Gemma4 31B and Sarvam 30B overall, widening to 17.4–28.3 pp across Indic languages.The reported gap challenges the assumption that Indic-native pre-training alone is sufficient to outperform a leading model.

Limitations

The study’s multilingual evaluation is limited in scope, and its judging and data-construction procedures introduce unresolved validity and familiarity biases. Human agreement testing is still needed to establish independent judge accuracy.

  • Evaluation scope: The multilingual evaluation covers only 9 of 19 Indic languages, 2 of 6 models, and the judge protocol; other conclusions are limited to English.The study does not evaluate the remaining Indic languages or models under the multilingual protocol.
  • Evaluation validity: Lexical criteria reproduce the ranking but cannot confirm absolute judge accuracy because they are stricter criteria, not independent ground truth.A human agreement study is still needed.
  • Model-related bias: Judge verdicts come from a single Gemma 3 12B model, while Gemini 2.5 Flash serves as both generation and evaluation model, introducing an uncontrolled familiarity bias.The passage identifies shared model roles as a limitation of the evaluation and construction process.

A Evaluation Pipeline

The evaluation pipeline has all six models answer the same question, after which Gemma 3 12B judges each response against the gold-standard answer. It produces a TRUE/FALSE verdict for every response.

  • LLM-as-a-judge evaluation pipeline: Six models respond to the same question in the evaluation pipeline.The shared question ensures each model is evaluated on the same prompt.
  • LLM-as-a-judge evaluation pipeline: Gemma 3 12B evaluates each model response against the gold-standard answer.Gemma 3 12B serves as the evaluator in the pipeline.
  • LLM-as-a-judge evaluation pipeline: The evaluator returns a TRUE/FALSE verdict for each response.Each verdict reflects the evaluation against the gold-standard answer.

B Domain-wise Lexical Results

Table 7 reports domain-wise scores for both deterministic evaluation criteria on the 3,471-question English subset used for Table 4.

  • Domain-wise lexical results: Table 7 presents full domain-wise scores for both deterministic criteria defined in Section 4.3.The scores use the same 3,471-question English subset as Table 4.

C Agent and Evaluation Prompts

The appendix specifies verbatim prompt templates for generating and validating India-specific QA pairs, evaluating models, and judging answer correctness. These prompts enforce strict content, format, language, verification, and semantic-equivalence requirements.

  • C.1 Agent 1: QA Generation Prompt: Agent 1 generates context-rich, unambiguous India-specific QA pairs testing deep knowledge about India.The prompt describes the agent as an expert quizmaster focused exclusively on India-specific trivia and QA creation.
  • C.1 Agent 1: QA Generation Prompt: Agent 1 requires India-specific focus, context-rich questions, absolute answers, complete names, high difficulty, and 1–5-word verbatim answers.It also prohibits text references and emphasizes strict generation constraints.
  • C.1 Agent 1: QA Generation Prompt: The generation prompt enforces grammar, precise terminology, unique knowledge, clear question structures, and quality-over-quantity output behavior.It directs the agent to return an error JSON rather than low-quality question-answer pairs.
  • C.2 Agent 2: Quality Validation Prompt: Agent 2 validates QA triplets against golden-truth standards, scores correctness, relevance, and difficulty, detects duplicates, and verifies facts through multiple Google sources.Validation uses source-text chunks matched by chunk_id and applies ultra-strict rules.
  • C.2 Agent 2: Quality Validation Prompt: A QA triplet is valid only with an overall quality score of at least 7 under the validator’s zero-tolerance requirements.The rules include India-specificity, self-contained context, verbatim 1–5-word answers, grammar, difficulty, truth verification, and duplicate detection.
  • C.3 Model Evaluation Prompt: The model-evaluation prompt requests accurate, concise answers limited to 1–5 words, null for unknown answers, optional reasoning, JSON-only output, and the question’s language.Its input and output are JSON arrays containing question identifiers and answers.
  • C.4 LLM-as-a-Judge Prompt: The LLM judge compares a model answer with the ground truth and accepts semantic equivalence, minor phrasing differences, formatting variation, or extra detail.Clearly wrong, meaning-changing, unrelated, or cross-language answers are judged incorrect.
  • C.4 LLM-as-a-Judge Prompt: The judge returns only a JSON object containing an is_correct boolean and brief reasoning.The prompt supplies the question, ground-truth answer, and given answer as inputs.
Loading 2608.15535v1…