PaperlayerView exact revision on arXiv

Source-linked AI summary

BhashaBench V1: A Comprehensive Benchmark for the Quadrant of Indic Domains

Vijay Devane, Mohd Nauman, Bhargav Patel, Aniket Mahendra Wakchoure, Yogeshkumar Sant, Shyam Pawar, Viraj Thakur, Ananya Godse, Sunil Patra, Neha Maurya, Suraj Racha, Nitish Kamal Singh, Ajay Nagpal, Piyush Sawarkar, Kundeshwar Vijayrao Pundalik, Rohit Saluja, Ganesh Ramakrishnan

arXiv:2510.25409v2cs.CLcs.AI

TL;DR

Existing benchmarks often lack the language coverage and domain specificity needed to evaluate India-centric knowledge systems. BhashaBench V1 provides a bilingual, multi-domain benchmark, and evaluation finds substantial domain and language performance gaps across models.

  • Problem

    Existing benchmarks are often Anglocentric and domain-agnostic, limiting evaluation of India-specific cultural contexts and specialized knowledge.

    Source: §1 · p. 2

  • Method

    BhashaBench V1 is a bilingual benchmark built from curated examination questions across four India-centric domains and more than 90 subdomains.

    Source: §1 · p. 2

  • Results

    Models show domain- and language-specific performance gaps and generally do better in English; GPT-4o achieves 76.49% accuracy in Legal versus 59.74% in Ayurveda.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    BhashaBench V1 provides fine-grained evaluation of LLM capabilities across India-centric knowledge domains and bilingual content.

    Source: §1 · p. 2, §3.1 · p. 4

    Current English-Hindi coverage does not capture India's full linguistic diversity, including regional variation in agriculture, legal terminology, and traditional-medicine nomenclature.

    Source: A Limitations and Biases · p. 20

Abstract

from arXiv · show

The rapid advancement of large language models(LLMs) has intensified the need for domain and culture specific evaluation. Existing benchmarks are largely Anglocentric and domain-agnostic, limiting their applicability to India-centric contexts. To address this gap, we introduce BhashaBench V1, the first domain-specific, multi-task, bilingual benchmark focusing on critical Indic knowledge systems. BhashaBench V1 contains 74,166 meticulously curated question-answer pairs, with 52,494 in English and 21,672 in Hindi, sourced from authentic government and domain-specific exams. It spans four major domains: Agriculture, Legal, Finance, and Ayurveda, comprising 90+ subdomains and covering 500+ topics, enabling fine-grained evaluation. Evaluation of 29+ LLMs reveals significant domain and language specific performance gaps, with especially large disparities in low-resource domains. For instance, GPT-4o achieves 76.49% overall accuracy in Legal but only 59.74% in Ayurveda. Models consistently perform better on English content compared to Hindi across all domains. Subdomain-level analysis shows that areas such as Cyber Law, International Finance perform relatively well, while Panchakarma, Seed Science, and Human Rights remain notably weak. BhashaBench V1 provides a comprehensive dataset for evaluating large language models across India's diverse knowledge domains. It enables assessment of models' ability to integrate domain-specific knowledge with bilingual understanding. All code, benchmarks, and resources are publicly available to support open research.

1 Introduction

Existing benchmarks do not fully capture India-specific cultural and domain expertise, while multilingual and specialized evaluation remains limited. BhashaBench V1 addresses this gap with a bilingual benchmark, and evaluation reveals substantial differences across domains and languages.

1 Introduction

  • Existing benchmarks often emphasize English and may miss India-specific cultural contexts and domain expertise needed for India-centric evaluation.The introduction frames this gap against persistent multilingual and domain-specific performance challenges.Source: §1 · pp. 1–2
  • BhashaBench V1 is a domain-specific, multi-task, bilingual benchmark for Agriculture, Legal, Finance, and Ayurveda, spanning 90+ subdomains and 500+ topics.Its questions were curated from more than 40 government and professional examination papers.Source: §1 · p. 2
  • Across 29+ evaluated models, performance varies by domain and language, with English generally outperforming Hindi.GPT-4o achieves 76.49% accuracy in Legal and 59.74% in Ayurveda; Cyber Law and International Finance are relatively strong, while Panchakarma, Seed Science, and Human Rights remain challenging.Source: §1 · pp. 2–3

2 RELATED WORK

Related work spans general, multilingual, and domain-specific LLMs, alongside specialized benchmarks across several fields. However, existing evaluation remains fragmented across language coverage, task scope, and methodology.

2.1 Exploration of LLMs

  • LLM research includes proprietary, open-source, multilingual, and domain-specific models, while India-focused initiatives include Agri-Param, Ayur-Param, Finance-Param, and Legal-Param.These India-focused efforts use continual pretraining or instruction fine-tuning.Source: §2.1 · p. 3

2.2 Evaluation of LLMs

  • Existing benchmarks range from general-purpose suites to domain-specific resources, but often lack Indic language coverage, broad domain scope, or standardized evaluation methods.Agriculture, finance, legal, and traditional-medicine benchmarks cover differing tasks, while evaluation methods range from accuracy to human judgment.Source: §2.2 · p. 3

3 BhashaBench V1

BhashaBench V1 combines India-specific design principles with examination-based data collection, multilingual processing, and expert validation. Its 74,166-question dataset spans four domains and 91 subdomains, with domain-specific topic and difficulty distributions.

3.1 Design Principles

  • Design principles cover four critical Indian domains, diverse task formats, India-specific reasoning, bilingual evaluation, authentic sources, difficulty levels, and cultural authenticity.The framework spans 90+ subdomains and 500+ topics.Source: §3.1 · p. 4

3.2 Data Collection

  • Collection draws on more than 40 examination types, including regional assessments that incorporate state-specific topics and local knowledge.The source papers are manually curated by subject matter experts for topic tags, language annotation, and answer-key validation.Source: §3.2 · p. 4

3.3 Data Processing

  • Surya OCR was selected for Indic-language and domain-specific examination materials, with reported normalized text similarity of 98.1% for English and 98.9% for Hindi.The passage reports 97.8% average similarity, compared with 88.0% for Tesseract and 96.7% for Google Vision API.Source: §3.3 · p. 4
  • The extraction pipeline uses GPT-OSS-120B to identify questions, parse options, align answer keys, and standardize outputs as JSON with domain metadata.Cleaning filters image-based questions and those with more than four options, verifies language, classifies question formats, and detects duplicates.Source: §3.3 · pp. 4–5
  • Extracted question-answer pairs undergo expert validation for accuracy, cultural context, ambiguity, consistency, and linguistic authenticity.The validation process is intended to preserve idiomatic language and the complexity of the original examination material.Source: §3.3 · p. 5

3.4 Data Analysis

  • The dataset contains 74,166 questions across four domains and 91 subdomains, with 70.8% in English and 29.2% in Hindi.More than 90% of questions use the multiple-choice format.Source: §3.4 · pp. 5–6
  • Agriculture covers 25 subdomains, Finance 30, Ayurveda 16, and Legal 20, with distinct topic concentrations and difficulty profiles.Agronomy leads Agriculture with 5,078 questions; Problem Solving leads Finance with 5,686; Kayachikitsa leads Ayurveda with 3,134; Civil Litigation & Procedure leads Legal with 7,126.Source: §3.4 · p. 5

4 Experimental Setup

The evaluation combines zero-shot scoring for open-source models with API-based generation for closed-source and large proprietary models. Both approaches support systematic comparison across model types.

4 Experimental Setup

  • Open-source models are evaluated zero-shot with the LM-EVALUATION-HARNESS, using log-likelihood to select the highest-probability answer among multiple-choice options.The setup assesses base and instruction-fine-tuned models without task-specific examples.Source: §4 · p. 6
  • Closed-source and large proprietary models are evaluated through their APIs, using generated responses in structured JSON for automated parsing.The paper cites computational constraints and access limitations as reasons for API-based evaluation.Source: §4 · p. 6

5 Results and Discussions

The results show that model performance varies with domain, language, question difficulty, question type, and model scale or architecture. Larger and instruction-tuned models generally perform better, but strengths differ across domains and formats.

5.1 Zero-Shot Performance Across All Domains (EN + HI)

  • 67.25% average accuracy makes Qwen3-235B-A22B-Instruct the strongest model across English and Hindi, ahead of GPT-4O at 66.18%.Table 1 reports zero-shot scores across the four domains in English and Hindi.Source: §5.1 · p. 6, Table 1 · p. 7

5.2 How do models perform in subdomains

  • 91.43% in Research & Statistics and 91.6% in Agricultural Biotechnology are Qwen3-235B-A22B-Instruct’s strongest reported subdomain scores.Finance and Legal have the highest performance ceilings, while Ayurveda is most challenging and top models rarely exceed 80% in specialized areas such as Panchakarma.Source: §5.2 · pp. 6–7, Figure 4 · p. 8, Figure 5 · p. 8

5.3 Performance Analysis Across Question Difficulty Levels

  • In BBA, GPT-4o scores 66.4% on Easy questions and 47.09% on Hard questions, while smaller models score lower across difficulty levels.BBF, BBK, and BBL show the same general pattern of larger and instruction-tuned models outperforming smaller models, particularly on Hard questions.Source: §5.3 · p. 7

5.4 Performance Analysis Across Question Types

  • 71.43% is the top reported score for Rearrange the Sequence in BBL, while performance varies across question types and models.Larger models consistently outperform smaller or non-instruction-tuned models on Match the Column and Reading Comprehension.Source: §5.4 · p. 8

5.5 Performance Analysis of GPT Model Family

  • 71.05% in Finance gives gpt-oss-120b a 16–35 percentage-point improvement over gpt-oss-20b across the four domains.GPT-4o scores higher in Legal at 76.49% and Agriculture at 73.5%; the authors associate gpt-oss-120b’s Finance strength with BBF’s mathematical-reasoning emphasis.Source: §5.5 · p. 9

5.6 Performance Analysis of Small Models

  • 39.97% in BBA for Param-1 and 42.45% in BBK for Qwen2.5-3B illustrate the small models’ complementary domain strengths.Instruction-tuning effects vary by architecture, and the authors report that efficient, targeted models can provide utility in resource-constrained scenarios.Source: §5.6 · p. 9

6 Conclusion

BhashaBench V1 evaluates LLMs on India-centric knowledge across four domains using a broad, bilingual benchmark. Its evaluation finds domain and language performance disparities, and the benchmark and supporting materials are released for open research.

6 Conclusion

  • BhashaBench V1 evaluates LLMs across Agriculture, Legal, Finance, and Ayurveda, covering more than 90 subdomains and 500+ specialized topics.Its questions are curated from authentic government and professional examination papers.Source: §6 · p. 9
  • Models perform better in Legal contexts than in traditional knowledge systems such as Ayurveda, and better on English than Hindi across all domains.The paper presents these results as substantial performance disparities in India-specific contexts.Source: §6 · p. 9
  • The authors release BhashaBench V1 with evaluation code and documentation to foster open research on inclusive, culturally aware language models.They describe the benchmark as foundational for developing models suited to India’s diverse linguistic and knowledge landscape.Source: §6 · p. 9

A Limitations and Biases

The authors identify boundaries in the benchmark’s language, domain, and evaluation coverage, alongside biases tied to sources and examination frameworks. They also discuss potential benefits and risks of applying models in critical domains.

A Limitations and Biases

  • English and Hindi coverage omits India’s other official languages and regional dialects, leaving regional practices and terminology in areas such as agriculture, law, and traditional medicine outside the current evaluation.The benchmark covers four domains but does not encompass the full breadth of India-specific knowledge systems.Source: A Limitations and Biases · p. 20
  • Structured questions from government and professional examinations may not capture all contextual reasoning required in complex domain applications.The authors describe this format as relevant to real-world applications while acknowledging this boundary.Source: A Limitations and Biases · p. 20
  • Source-material, language-resource, and examination-framework biases may underrepresent regional practices and contemporary developments or reflect institutional perspectives in the original assessments.The authors note that Hindi is a relatively lower-resource context than English and say broad coverage mitigates examination-framework bias significantly.Source: A Limitations and Biases · p. 20

B Towards Broader Impact

  • The authors anticipate benefits in access to domain expertise but flag risks including over-reliance in critical decisions, displacement of traditional practitioners, and favoring formal education over experiential knowledge.They call for responsible use with human oversight and transparent disclosure of model limitations in critical domains.Source: B Towards Broader Impact · pp. 20–21
  • The authors state that the data comes from publicly available government and professional examinations, was verified by domain experts, and contains no personally identifiable information.They intend the resource for academic research and education, with safeguards for deployment in critical domains.Source: B Towards Broader Impact · pp. 20–21

C.1 Details of Data Collection and Processing

The appendix outlines a data-collection process using examination materials from authoritative sources across India. It describes expert curation, source coverage, and the collection period.

C.1 Details of Data Collection and Processing

  • The appendix documents BhashaBench V1’s examination-source, processing, and quality-validation methodology.The data collection strategy draws on authoritative national and regional assessment bodies.Source: §C.1 · p. 21, §C.1.1 · p. 21

C.1.1 Examination Source Documentation

  • Collected examination materials span 1995–2025, with Table 2 listing organizations and their examination-year ranges.Subject experts curated papers from official portals, tagging topics and languages and validating answer keys; regional exams contribute state-specific topics and local knowledge systems.Source: §C.1.1 · p. 21, Table 2 · p. 21

C.1.2 Processing Pipeline Architecture

BhashaBench V1’s processing pipeline combines multilingual data acquisition, extraction, filtering, classification, and expert review, using domain-specific prompts and annotation guidelines to support consistency and cultural authenticity. The extraction prompts recover questions and answers from OCR text while preserving context and structuring metadata, with legal and finance prompts addressing domain-specific validation, terminology, OCR correction, and answer detection.

C.1.2 Processing Pipeline Architecture

  • The seven-stage pipeline transforms examination materials into structured data through provenance tracking, OCR and extraction, quality filtering, metadata classification, and expert validation.Acquisition records exam year, conducting body, subject, and language; Surya OCR digitizes documents, GPT-OSS-120B structures content, and filtering checks language, duplicates, and content quality. About 30% of questions lacked explicit subdomain labels, so GPT-OSS-120B classifications were refined using expert-informed taxonomies; reviewers checked factual accuracy and linguistic and cultural nuance.Source: §C.1.2 · pp. 22, 24

C.1.3 Annotation Guidelines

  • Annotation guidelines standardize answer verification across domains and languages while requiring source cross-checks, domain expertise, and preservation of linguistic and cultural context.All extracted question-answer pairs receive expert review for factual accuracy, ambiguities, consistency, and natural language flow; bad and neutral samples are manually corrected.Source: §C.1.3 · p. 25, §C.1.2 · p. 25

C.1.4 Data Processing Prompts

  • The data-processing prompts direct extraction to recover every question, embed shared context, normalize options, detect answers, and preserve self-contained question structure.For question metadata, domain-wise few-shot examples guide question-type and difficulty labeling; difficulty is categorized as Easy, Medium, or Hard, and question formats use six standard categories.Source: §C.1.2 · pp. 24–25, §C.1.4 · pp. 26–27
  • Ayurveda and agriculture prompts specify domain-specific OCR corrections and option recovery, including standardized medical and crop terminology, units, and institution names.The prompts also require preserving context and embedding detected answers in structured question output.Source: §C.1.4 · pp. 26, 28
  • Legal extraction prompts add OCR correction for legal terminology, embedded case or statutory context, exactly four normalized options, and mandatory three-tier structural, legal-coherence, and logical-consistency validation.Questions failing fundamental validation criteria are skipped, while borderline questions may be marked for review if they pass basic validation.Source: §C.1.4 · pp. 30–31
  • Legal extraction prompts require rigorous validation, self-contained questions with embedded context, and accurate preservation of legal terminology, citations, and references.They specify a strict JSON schema and advise skipping questions only after thorough validation failure.Source: §C.1.4 · pp. 31–33
  • Financial prompts prescribe OCR corrections, question-boundary detection, aggressive option recovery, and multi-pass answer detection, with answers embedded in each question.They also direct extraction of partial questions with an [INCOMPLETE] tag, while the extraction step says to take only complete, meaningful questions with options.Source: §C.1.4 · pp. 33–34
  • A key-details prompt requests JSON fields for question type, difficulty level, topic, and subdomain, using domain-specific examples and annotation lists.The prompt frames these as details to extract for each question.Source: §C.1.4 · p. 35

C.2 Detailed Data Analysis of BhashaBench V1

The section organizes BhashaBench V1’s data analysis through domain-specific question-count tables and cross-domain distributions of language, difficulty, and question type.

C.2 Detailed Data Analysis of BhashaBench V1

D More Details on Experiment Setup

The experiment setup uses standardized task formats for lm-eval and API evaluations, with distinct inference procedures for open-source and API-based models.

D.1 Task Formatting Template Used in LM Eval

  • The lm-eval template is applied consistently across task types, including assertion/reasoning, fill-in-the-blanks, MCQs, matching, comprehension, and sequence rearrangement.Source: §D.1 · pp. 38–39

D.2 Task Formatting Template Used in API-Driven Evaluation

  • API evaluations separate the system role instructions from the question-and-options prompt and request only the correct option letter.This structure is intended to keep tasks consistent and responses focused on answer selection.Source: §D.2 · p. 39

D.3 Details of Inference Implementation

  • Open-source models run on eight NVIDIA H200 GPUs through vLLM and lm-eval defaults, while API models use temperature 0 and average three repetitions.API-based evaluations disable web search and external tool calls.Source: §D.3 · p. 39

E.2 Zero-Shot sub-domain wise Performance Across BhashaBench V1 Domains

The section reports zero-shot evaluation through tables covering domains and difficulty levels, question types, and model-family subdomain performance.

E.2 Zero-Shot sub-domain wise Performance Across BhashaBench V1 Domains

  • Table 12 covers zero-shot scores across Ayurveda, Finance, Agriculture, and Legal at Easy, Hard, and Medium difficulty levels.Source: Table 12 · p. 40
  • Tables 14–17 compare subdomain performance for Gemma, Llama, Qwen, and GPT model families across their listed model sizes and variants.The captions specify base and instruction-tuned comparisons for Gemma, Llama, and Qwen; the GPT table compares model sizes.Source: Table 14 · p. 40, Table 15 · p. 43, Table 16 · p. 45
  • Table 13 organizes zero-shot scores by six question types, including assertion/reason, fill-in-the-blanks, MCQ, matching, reading comprehension, and sentence rearrangement.Source: Table 13 · p. 43

Related papers

Loading 2510.25409v2…