Source-linked AI summary
Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages
Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M. Khapra, Anoop Kunchukuttan, Pratyush Kumar
TL;DR
Indic languages are underserved by existing multilingual data, models, and benchmarks despite their large speaker base. The paper builds larger Indic-focused corpora, a human-supervised benchmark, and IndicBERT v2; IndicBERT outperforms strong baselines on 7/9 evaluation tasks. Its data sources and language-identification coverage retain documented limitations.
Problem
Existing multilingual models and benchmarks provide limited data, capacity, and evaluation coverage for low-resource Indic languages.
Method
The paper releases IndicCorp v2, creates the nine-task human-supervised IndicXTREME benchmark, and trains IndicBERT v2 on IndicCorp v2.
Results
IndicBERT outperforms strong baselines on 7/9 IndicXTREME evaluation tasks.
Takeaways & Limitations
The work provides Indic-focused corpora, models, and a crosslingual NLU benchmark spanning many Indic languages.
Takeaways & Limitations
The corpus may contain biases from public-domain news sources, and language-identification tools are unavailable for some very low-resource languages.
Abstract
from arXiv · showhide
Building Natural Language Understanding (NLU) capabilities for Indic languages, which have a collective speaker base of more than one billion speakers is absolutely crucial. In this work, we aim to improve the NLU capabilities of Indic languages by making contributions along 3 important axes (i) monolingual corpora (ii) NLU testsets (iii) multilingual LLMs focusing on Indic languages. Specifically, we curate the largest monolingual corpora, IndicCorp, with 20.9B tokens covering 24 languages from 4 language families - a 2.3x increase over prior work, while supporting 12 additional languages. Next, we create a human-supervised benchmark, IndicXTREME, consisting of nine diverse NLU tasks covering 20 languages. Across languages and tasks, IndicXTREME contains a total of 105 evaluation sets, of which 52 are new contributions to the literature. To the best of our knowledge, this is the first effort towards creating a standard benchmark for Indic languages that aims to test the multilingual zero-shot capabilities of pretrained language models. Finally, we train IndicBERT v2, a state-of-the-art model supporting all the languages. Averaged across languages and tasks, the model achieves an absolute improvement of 2 points over a strong baseline. The data and models are available at https://github.com/AI4Bharat/IndicBERT.
1 Introduction
Indic languages remain underserved by multilingual models because low-resource languages lack data, model capacity, and broad evaluation coverage. The paper addresses these gaps with IndicCorp v2, IndicXTREME, and IndicBERT v2.
- Indic languages receive limited support because large multilingual corpora often lack pretraining data for languages such as Bodo, Dogri, and Kashmiri.
- Low-resource languages receive a poor share of model capacity and vocabulary when trained alongside English and other resource-rich languages.
- 15 of 22 constitutionally recognized Indic languages lack representation in XTREME-R for any task.
- IndicCorp v2 contains 20.9 billion tokens and 1.1 billion sentences across four Indic language families, a 2.3x improvement over IndicCorp v1.
- IndicXTREME provides 105 evaluation sets across nine diverse tasks, including 52 newly created sets manually developed with language experts.
- IndicBERT outperforms strong baselines on 7/9 IndicXTREME evaluation tasks.
2 Related Work
Prior Indic language research relies on sparse or noisy corpora, multilingual models that compete for capacity, and benchmarks with limited Indic coverage. IndicXTREME organizes broader zero-shot evaluation across diverse tasks and languages.
- 2.1 Resources: Wikipedia is high quality but sparsely populated for Indic languages, while CommonCrawl-derived corpora are often noisy and contain offensive content.
- 2.1 Resources: IndicCorp v1 was the first corpus curated exclusively for Indic languages, and this work expands its language coverage and data volume.
- 2.2 Models: Most multilingual pretrained models make Indic languages compete for model capacity with other high-resource languages.
3 IndicXTREME
IndicXTREME is a human-supervised zero-shot benchmark spanning nine diverse tasks and multiple Indic languages, combining existing datasets with newly created evaluation sets.
- IndicXTREME groups nine tasks into five classification, two structure prediction, one question-answering, and one sentence-retrieval task categories.
- IndicQA is a manually curated cloze-style reading-comprehension dataset for evaluating question-answering models in 11 Indic languages.
- IndicXNLI uses automatically translated XNLI data in 11 Indic languages, but only some test-set portions were manually verified because of cost and time constraints.
- FLORES contributes 1,012 manually translated sentences in 18 Indic languages for retrieval evaluation without fine-tuning.
- The benchmark includes manually translated, curated, verified, or post-edited evaluation data across Indic languages.
4 IndicCorp v2
IndicCorp v2 is a large multilingual corpus built from diverse web sources, especially news, with human-verified URLs and post-processing intended to improve cleanliness across Indic languages.
- IndicCorp v2 contains 20.9 billion tokens across 24 languages, including 14.4B tokens from 23 Indic languages and 6.5B Indian English tokens.
- The corpus is 2.3× larger than IndicCorp v1 and includes 1.08 billion tokens from the bottom 11 low-resource languages.
- News articles serve as primary sources, supplemented by language-specific sources discovered through repositories and automated web searches.
- Human annotators verify candidate URLs, and 1–33% of URLs are identified as noisy and discarded across languages.
- Paragraph-level language identification removes text outside the target language, except for Bodo and Dogri, which the libraries do not support.
- A multilingual blacklist created from annotator- and researcher-provided offensive-word lists is used to filter offensive content.
5 IndicBERT v2
IndicBERT v2 is a 278M-parameter BERT model trained to support all 24 IndicCorp languages, using monolingual and parallel data with alternative pretraining objectives.
- IndicBERT v2 has 278M parameters and supports all 24 languages in IndicCorp.
- Training compares Masked Language Modeling and Translation Language Modeling objectives over multilingual corpora.
- MLM training uses monolingual corpora spanning 24 languages, 5 language families, and 13 scripts.
- TLM training uses mined Samanantar parallel data and machine-generated English translations of IndicCorp.
- Underrepresented languages are upsampled with a 0.3 temperature coefficient to address skewed data distributions.
- The vocabulary contains language-identity tokens and 250K tokens for 24 languages across 13 scripts.
6 Experiments
Experiments compare IndicBERT v2 with established multilingual baselines and evaluate zero-shot transfer through task-specific fine-tuning and ablations of parallel data and script choices.
- IndicBERT v2 is compared with IndicBERT v1, mBERT, XLMR, and MuRIL.
- Zero-shot evaluation fine-tunes models on English and tests them on available Indic test sets.
- Four IndicBERT v2 variants test MLM, TLM with Samanantar, TLM with back-translated IndicCorp data, and script conversion to Devanagari.
- The ablations examine how parallel-data availability and quality affect crosslingual performance.
- Table 4 reports language-averaged F1 for structure prediction and question answering, and accuracy for other tasks.
- Table 5 reports absolute performance and improvement over English-development-set tuning for in-language and in-family development sets.
7 Results
IndicBERT v2 models outperform baselines across most IndicXTREME tasks, with results favoring Indic-focused pretraining, larger monolingual corpora, and language-family-aware transfer choices.
- IndicBERT v2 models beat baselines on 7/9 IndicXTREME tasks and perform best on 17/20 languages when averaged across tasks.
- Indic-only models generally perform better because languages do not compete for model capacity, although MuRIL remains stronger on paraphrase detection and NER.
- High-quality parallel data improves the TLM objective, whereas machine-translated data hurts performance.
- The larger IndicCorp v2 corpus improves performance over IndicCorp v1, establishing the utility of expanded monolingual data.
- In-family development sets generally match in-language sets and outperform English validation sets for model selection.
- Extremely low-resource languages show substantial performance drops, and IndicXTREME evaluates only 19 of the 24 languages in IndicCorp.
8 Conclusion
The work contributes IndicCorp v2, IndicBERT v2, and IndicXTREME to support Indic language technology across corpora, models, and crosslingual evaluation.
- The paper releases IndicCorp v2 for 24 Indic languages, IndicBERT v2 pretrained on it, and IndicXTREME for 20 Indic languages.
Limitations
The corpus and benchmark have important scope boundaries: public news sources may carry bias, low-resource language identification remains imperfect, and many extremely low-resource languages are not covered.
- Public news sources may propagate inherent biases into models trained on the corpus.The work does not apply debiasing techniques and leaves them for future work.
- Language identification remains constrained because tools are unavailable for some very low-resource languages, including Bodo, Dogri, and Khasi.The authors use Unicode-based cleaning but acknowledge that source issues may remain.
- The benchmark and models do not represent hundreds of extremely low-resource languages.The authors identify building datasets for these languages as future work.
Ethics Statement
The paper reports that annotation was performed by compensated native speakers and that the released code, models, and dataset use permissive licenses.
- Native-speaker annotators were paid competitive monthly salaries based on qualifications, experience, and national government norms.They were informed that the datasets would be publicly released.
- The released code and models use an MIT License, while the dataset uses a CC-0 License.
- IndicBERT models were trained on 20.9 billion tokens across 24 Indic languages using TPUs and then fine-tuned on NVIDIA A100 GPUs.The authors estimate 5.4 MTCO2e for pretraining and 41.04 kg of CO2e for fine-tuning.
B Data Distribution
The paper describes the data and procedures underlying its Indic-language resources, including corpus construction, benchmark creation, and question-answering annotation.
- Figure 1 presents the upsampled data distribution.
- Baseline language models: The baseline models include mBERT, XLM-R, IndicBERT v1, and MuRIL, with differing language coverage, architectures, and training corpora.XLM-R covers 15 Indic languages, IndicBERT v1 supports 11, and MuRIL is trained on 16 Indic languages.
- IndicQA: Question creation uses translated context splits to reduce lexical overlap, while separate annotators mark answer spans and create unanswerable questions.The second context portion is withheld during answer annotation, making questions derived from it unanswerable.
- Benchmark data: The benchmark includes manually created paraphrase and non-paraphrase examples, product reviews, and a transfer-language experiment for named-entity recognition.These resources rely on multilingual annotation procedures and controlled instructions.
J IndicCorp Data Cleaning
The paper cleans IndicCorp through source filtering, language and script checks, document-length and offensive-content filtering, then evaluates tokenizer fertility, language identification, and downstream trends.
- Source selection: News sources are discovered through directories and multilingual web searches, then manually screened to remove spam websites.
- Language identification: Language identification combines cld3 and langdetect because cld3 does not identify Assamese and Oriya.
- Text cleaning: Sentences are removed when native-script characters comprise less than 0.75 of the total or when punctuation-stripped documents contain fewer than 10 words.
- Text cleaning: Offensive-content filtering removes sentences containing listed offensive words or complete listed phrases.The lists average about 90 words or phrases per language and are manually verified by native speakers.
- Tokenizers: IndicBERT’s tokenizer has consistently lower fertility scores than mBERT, XLM-R, and MuRIL across languages.The exception in Meitei is attributed to a Bengali-versus-Meitei script mismatch between FLORES and IndicCorp.
- Language Identification: IndicBERT identifies all languages except Manipuri with high accuracy in language identification, while Manipuri fails because of script mismatch.
- Data and performance: Increasing pretraining-data size generally improves downstream performance across tasks and languages, except for IndicXParaphrase and with high variance for Naamapadam.The authors hypothesize that noun-phrase representations may explain these exceptions.