PaperlayerView exact revision on arXiv

Source-linked AI summary

IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages

Jay Gala, Pranjal A. Chitale, Raghavan AK, Varun Gumma, Sumanth Doddapaneni, Aswanth Kumar, Janki Nawale, Anupama Sujatha, Ratish Puduppully, Vivek Raghavan, Pratyush Kumar, Mitesh M. Khapra, Raj Dabre, Anoop Kunchukuttan

arXiv:2305.16307v3cs.CL

TL;DR

Machine translation for India’s 22 scheduled languages lacked comprehensive training data, relevant benchmarks, and models supporting the full set. The paper builds BPCC and IN22, trains IndicTrans2, and releases the resulting models and resources openly; it reports broad gains over existing open-source systems and parity with commercial models, while noting limited generalization for several low-resource languages.

  • Problem

    High-quality, accessible MT lacked parallel training data, robust India-relevant benchmarks, and models covering all 22 scheduled Indian languages.

  • Method

    The paper combines large-scale corpus construction and mining, human-translated seed data, diverse benchmark creation, multilingual model training, and open release.

  • Results

    IndicTrans2 consistently outperformed other open-source models across multi-domain benchmarks and performed at par with commercial models, while supporting all 22 scheduled languages.

  • Takeaways & Limitations

    BPCC, IN22, and openly released IndicTrans2 resources broaden access to training, evaluation, and translation systems for India’s scheduled languages.

  • Takeaways & Limitations

    IndicTrans2 has limited generalization for Manipuri (Meitei), Santali, and Sindhi (Devnagari), reflecting scarcity of human-annotated data and limited web content.

Abstract

from arXiv · show

India has a rich linguistic landscape with languages from 4 major language families spoken by over a billion people. 22 of these languages are listed in the Constitution of India (referred to as scheduled languages) are the focus of this work. Given the linguistic diversity, high-quality and accessible Machine Translation (MT) systems are essential in a country like India. Prior to this work, there was (i) no parallel training data spanning all 22 languages, (ii) no robust benchmarks covering all these languages and containing content relevant to India, and (iii) no existing translation models which support all the 22 scheduled languages of India. In this work, we aim to address this gap by focusing on the missing pieces required for enabling wide, easy, and open access to good machine translation systems for all 22 scheduled Indian languages. We identify four key areas of improvement: curating and creating larger training datasets, creating diverse and high-quality benchmarks, training multilingual models, and releasing models with open access. Our first contribution is the release of the Bharat Parallel Corpus Collection (BPCC), the largest publicly available parallel corpora for Indic languages. BPCC contains a total of 230M bitext pairs, of which a total of 126M were newly added, including 644K manually translated sentence pairs created as part of this work. Our second contribution is the release of the first n-way parallel benchmark covering all 22 Indian languages, featuring diverse domains, Indian-origin content, and source-original test sets. Next, we present IndicTrans2, the first model to support all 22 languages, surpassing existing models on multiple existing and new benchmarks created as a part of this work. Lastly, to promote accessibility and collaboration, we release our models and associated data with permissive licenses at https://github.com/AI4Bharat/IndicTrans2.

1 Introduction

The paper addresses limited training data, benchmarks, and model coverage for India’s 22 scheduled languages by building data resources, benchmarks, multilingual models, and open-access releases. Its contributions include BPCC, IN22, IndicTrans2, and related openly released resources.

  • Motivation: India has 1,369 identified mother tongues, with 22 constitutionally scheduled languages spoken as first languages by approximately 97% of the population.English is also widely used in formal communication, while regional-language translation supports communication, inclusion, government, judiciary, and education.
  • Research gap: Existing multilingual models lack good Indian-language coverage, perform poorly on these languages, or lack evaluation on diverse India-origin domains.The paper frames this as a gap in both model capability and relevant evaluation resources.
  • Contributions: BPCC contains ~230M bitext pairs, including ~126M newly added pairs and 644K manually translated English-to-Indic pairs.Its components include human-translated seed data, filtered existing corpora, and newly mined or back-translated data.
  • Contributions: IN22 is the first n-way parallel benchmark covering all 22 Indian languages with source-original English, diverse domains, India-specific content, and conversational data.Its subsets cover India-specific articles across 13 domains and everyday conversations across 16 domains.
  • Contributions: IndicTrans2 is the first translation model supporting all 22 scheduled Indian languages and is reported as competitive with open-source and commercial systems.The work also releases direct Indic-to-Indic and compact variants, together with training data, code, and tools under open-source licensing.

2 Related Work

Prior Indic-language MT work established corpora, benchmarks, and neural modeling approaches, but coverage and scale remained uneven across India’s languages. The paper situates its resources within this progression toward broader multilingual evaluation and training.

  • Indian-language context: India has 1,369 identified mother tongues, but many languages still lack online presence and high-quality NLP technologies.The 22 scheduled languages are the paper’s primary coverage target within this broader linguistic landscape.
  • Datasets: Earlier parallel corpora covered subsets of Indian languages, including 12-language ILCI data and English-aligned collections covering 11 or 12 Indic languages.Creating corpora for all Indic languages was constrained by scarce parallel documents and the cost of large-scale human annotation.
  • Benchmarks: Indic-language MT benchmarks expanded through WMT shared tasks, Mann ki Baat, and FLORES-101, but existing resources did not uniformly cover all scheduled languages or India-specific domains.FLORES-101 provided n-way evaluation for 14 Indic languages, while the cited Mann ki Baat benchmark covered 9.
  • Neural MT: Neural MT progressed from recurrent models to Transformer-based systems using attention and subword modeling to address word-order and data-sparsity challenges.These developments accompanied the growth of large-scale parallel corpora for Indic MT.

3 Creating High-quality Translation Datasets at Scale

The paper constructs human-curated training and evaluation resources through a controlled translation workflow and introduces IN22 as a broad n-way benchmark for English and 22 Indic languages. The resulting resources cover multiple domains, India-centric material, and conversational evaluation.

  • Translation workflow: The dataset-building process combines human infrastructure, seed data, benchmark construction, translation workflows, and quality-control procedures.Shoonya supports the annotation tasks and the resulting BPCC-Human and IN22 datasets.
  • Benchmark motivation: IN22 addresses the need for all-22-language coverage, diverse domains, realistic sentence lengths, and India-relevant content absent from more general-purpose benchmarks.The benchmark is designed around practical Indian-language MT use cases and enables broad multilingual evaluation.
  • Benchmark design: IN22 contains Wikipedia, Web Sources, and Conversation subsets designed to diversify content and evaluate day-to-day conversational translation.The Web subset emphasizes India-centric, less crawlable sources, while the Conversation subset targets chat-like applications.
  • Benchmark scope: 2527 original English sentences were translated into 22 Indic languages from scratch, enabling evaluation in 500+ directions and comparison across 462 inter-Indic directions.Metadata includes domains and raw context sentences for fine-grained quality analysis.
  • Training data: BPCC-Human is a manually translated, multi-domain n-way seed corpus, including 644.3K sentence pairs and more than 10K pairs for Bodo, Dogri, Santali, and Goan Konkani.The translations were produced by qualified professional translators with systematic review rather than crowdsourcing.
  • Training data: BPCC-H-Daily contains 139.7K bitext pairs across 21 Indic languages focused on spoken text and day-to-day service scenarios.Its content includes queries, commands, feedback, and named entities relevant to applications such as payments, delivery, and government services.

4 Mining Training Data at Scale

The paper scales Indic parallel-data construction through corpus mining, filtering, and human-translated seed data. This produces a substantially larger and broader training collection for Indic MT.

  • Mining focuses on 12 major Indic languages, while low-resource languages rely on high-quality human-translated data because monolingual resources and embedding quality are comparatively limited.
  • ~126 million additional sentence pairs produced an aggregated ~230.5 million-pair corpus after deduplication, roughly 5× larger than Samanantar.
  • 121 million sentence pairs were mined from monolingual corpora across 13 Indic languages.
  • 4.35 million sentence pairs were mined from comparable corpora across 17 Indic languages, covering topics including Religion, Education, and Legal content.
  • Filtering discarded about 75% of existing data because of poor alignment quality, reducing major source collections substantially.Costa-jussà et al. (2022) decreased from 448.1 million to ~85 million pairs, while Ramesh et al. (2022) decreased from 49.7 million to 19.4 million.
  • The mining approach represents sentences in a shared LaBSE embedding space and retrieves semantically close matches using nearest-neighbor search.The workflow uses scalable indexing and retrieval over large monolingual collections.

5 Modeling

IndicTrans2 combines curated and mined parallel data with multilingual preprocessing, a large transformer architecture, staged training, and back-translation. The training design emphasizes data quality, domain coverage, and augmentation.

  • Training combines mined corpora, human-annotated collections, filtered existing corpora, and additional human-labeled seed data.
  • Preprocessing includes normalization, deduplication against benchmarks, handling of special cases, script unification, and separate BPE tokenizers for English and Indic languages.
  • The models use an 18-layer encoder and decoder, 1.1B parameters, 1024-dimensional inputs, 8192-dimensional feedforward layers, and 16 attention heads.
  • FLORES-200 is used as the multi-domain validation set and extended with five additional languages not covered in its original development set.
  • Two-stage training first uses the full parallel corpus, then fine-tunes on high-quality human-generated seed data with multi-domain coverage.
  • Back translation augments training with synthetic data approximately 1.75 times the size of the original corpus.Intermediate IndicTrans2 checkpoints generate the back-translated data, which is combined with the original training data.

6 Evaluation

The evaluation compares IndicTrans2 with open and commercial translation systems across IN22, FLORES-200, and several established benchmarks. IN22 supplies broad, India-specific, n-way evaluation, while metric choice reflects coverage and comparability constraints.

  • IndicTrans2 is compared with IndicTrans1, NLLB, M2M-100, Microsoft Azure Translate, Google Translate, and GPT-3.5.
  • GPT-3.5 is evaluated on IN22 in a zero-shot setting, while proprietary-model comparisons are treated as approximate because training information is limited.
  • The models are evaluated on IN22, FLORES-200, WAT 2020/2021, WMT 2014/2019/2020, UFAL, and NTREX.
  • IN22 evaluates translation across all 22 Indic languages in multi-domain, n-way parallel settings with Wikipedia, Web, and conversational subsets.
  • IN22 contains diverse India-centric content, while FLORES-200 provides multi-domain evaluation across 19 Indic languages with source-original English sentences.
  • COMET cannot cover all 22 languages, while BLEU has tokenization and lexical-overlap limitations, motivating careful selection of primary metrics.

7 Results and Discussion

IndicTrans2 consistently improves translation quality across India-centric, general, and multi-domain benchmarks, with especially notable gains for low-resource languages. The study also examines language coverage, data quality, sequential training, pivoting, and distillation trade-offs.

  • Benchmark results: 5.2 points in En-Indic and 6.3 points in Indic-En improve over IndicTrans1 on IN22-Gen.
  • Benchmark results: 1.7 and 4.1 points separate IndicTrans2 from the best commercial and open-source En-Indic systems on IN22-Gen.
  • Language-level results: Low-resource languages receive substantial gains, including Dogri, Manipuri, Kashmiri, and Santali across reported evaluation directions.
  • Benchmark results: 5.8 and 7.4 points improve over IndicTrans1 in En-Indic and Indic-En translation on FLORES-200.
  • Benchmark results: IndicTrans2 consistently outperforms other open-source models across multi-domain benchmarks including IN22, FLORES-200, WAT2020, WAT2021, and NTREX.
  • Language and script coverage: IndicTrans2 extends open-source coverage to all 22 scheduled languages, including reasonable performance for Bodo, Dogri, and Konkani and previously unsupported scripts.
  • Data quality and training: 94M quality-filtered bitext pairs outperform models trained on 384M unfiltered pairs in the data scale-versus-quality ablation.
  • Data quality and training: 2.1 points in En-Indic and 1 point in Indic-En improve on primary benchmarks after fine-tuning with human-annotated seed data.

8 Conclusion

The paper delivers training data, benchmarks, and multilingual models for all 22 scheduled Indian languages. Its open-source releases include large and compact models for compute-rich and compute-scarce settings.

  • BPCC provides approximately 230M bitext pairs covering 22 Indic languages, while IN22 is a multi-domain benchmark for the same language set.
  • The authors train 1.1B-parameter English-centric models and 211M-parameter compact versions, plus direct Indic-Indic models of 1.2B and 350M parameters.
  • BLEU, chrF++, and COMET evaluations find the models comparable to, or better than, publicly available open and commercial systems.
  • The data, benchmarks, and model artifacts are released publicly as a foundation and guide for future Indic and low-resource language research.

9 Limitations and Future Work

The paper identifies limitations spanning data, modeling, benchmarks, and evaluation, especially for low-resource Indic languages. It outlines future work to improve generalization, evaluation breadth, and benchmark validity.

  • Data: Limited high-quality annotated data and web content constrain mining and generalization for Manipuri (Meitei), Santali, and Sindhi (Devnagari).Existing sentence-embedding models can also produce noisy or suboptimal mined pairs.
  • Modeling: A performance gap remains between low-resource and high-resource languages despite IndicTrans2 consistently outperforming baseline systems.Future work includes exploiting language relatedness, parameter-efficient domain adaptation, and extending coverage beyond the 22 scheduled languages.
  • Benchmark: IN22 and FLORES-200 use English-original n-way test sets, making Indic-to-English sources translationese rather than original text.This limits how directly those evaluations represent naturally occurring Indic-language inputs.
  • Evaluation: The preliminary human evaluation covers only 100 IN22-Gen sentences, motivating broader free-form and task-oriented studies.The paper also calls for automatic metrics better calibrated to Indic languages and human judgments.
  • Fairness: Web-collected training data may introduce social biases, requiring systematic study and mitigation for safer accessibility.The paper notes that distilled models can propagate or amplify teacher-model biases.

10 Author Contributions

The project used a distributed contribution structure covering corpus creation, human translation, model development, evaluation, research leadership, and direction.

  • Corpus and Data: Parallel Corpus Collection and Mining was led by Raghavan AK, Jay Gala, and Aswanth Kumar.
  • Corpus and Data: Human Translation was contributed by Pranjal A. Chitale, Jay Gala, Mitesh M. Khapra, Pratyush Kumar, Anoop Kunchukuttan, Janki Nawale, and Anupama Sujatha.
  • Models: Model Training and Distillation were contributed by Pranjal A. Chitale, Raj Dabre, Jay Gala, and Varun Gumma.
  • Evaluation: Model Evaluation involved Pranjal A. Chitale, Raj Dabre, Sumanth Doddapaneni, Jay Gala, Varun Gumma, Anoop Kunchukuttan, and Ratish Puduppully.
  • Leadership: Research Leads were Raj Dabre, Mitesh M. Khapra, Pratyush Kumar, and Anoop Kunchukuttan, while project direction included Vivek Raghavan.

The Team Behind the Scenes

The project involved operations, engineering, language experts, translators, and evaluation researchers across the 22 scheduled languages. Additional experiments examined cross-lingual transfer and comparisons with GPT3.5.

  • Project Team: AI4Bharat and partner teams included operations, frontend, backend, DevOps, architecture, and full-stack contributors.
  • Language Experts: Language experts and translators are listed for Assamese, Bengali, Bodo, Dogri, Gujarati, Hindi, Kannada, Kashmiri, Konkani, Maithili, Malayalam, Marathi, Manipuri, Nepali, Odia, Punjabi, Sanskrit, Santali, Sindhi, Tamil, and Telugu.
  • Additional Experiments: The study evaluates zero-shot Indic-to-English transfer for additional low-resource languages using top-3 related languages selected by similarity.
  • Additional Experiments: IndicTrans2 models outperform GPT3.5 on IN22-Gen and IN22-Conv in both translation directions, with average gains of 12.2 and 14.5 points.

B.5 Results on WAT2020 & WAT2021

The appendix reports WAT2020 and WAT2021 evaluations alongside other Indic-language benchmarks and multiple automatic metrics. These benchmarks cover selected Indic languages, English, and specified domains or shared-task settings.

  • WAT Benchmarks: WAT2020 and WAT2021 provide benchmark evaluations for Indic-English translation, with WAT2021 covering 10 Indic languages and English.
  • WAT Benchmarks: WAT2020 includes eight Indic languages in the news domain, while related Hindi-English data cover IT and WikiNews domains.
  • Metrics: The appendix reports chrF++, COMET, and BLEU results for WAT2020 and WAT2021.
  • Additional Benchmarks: Additional benchmark tables cover WMT shared tasks, UFAL, NTREX, IN22, and FLORES evaluation sets.

C Human Evaluation

The human evaluation uses calibrated XSTS ratings to compare IndicTrans2 with other multilingual MT systems, while examining resource-level trends and metric correlations. Results place IndicTrans2 ahead of Google and NLLB, comparable to Azure, but the limited evaluation sample warrants caution.

  • Evaluation methodology: The evaluation samples 100 En-Indic sentence pairs from Wikipedia and Web sources, covering diverse sentence lengths.The sample contains 50 examples from each source subset in IN22-Gen.
  • Evaluation methodology: XSTS uses reference-free human ratings focused primarily on adequacy, making it suitable for comparing translation quality in low-resource languages.Raters assess meaning preservation without reference translations on a 1–5 scale.
  • Overall results: IndicTrans2 significantly outperforms Google and NLLB 54B and performs comparably with Azure in the overall human evaluation.The comparison uses ANOVA with post-hoc Tukey HSD testing at p ≤0.05.
  • Resource-level trends: IndicTrans2 performs especially strongly for low-resource languages such as Konkani, Sanskrit, and Nepali, while most supported languages approach a 4 XSTS rating.High-resource languages including Hindi, Bengali, and Telugu are also represented in the distribution analysis.
  • Calibration: Calibration adjusts XSTS scores across languages, with notable post-calibration decreases for Punjabi and Sanskrit and improvements for Malayalam, Tamil, and Urdu.Calibration is intended to improve reliability and comparability across languages and models.
  • Metric correlation: Across languages, XSTS shows moderate correlation with BLEU but weaker correlation with chrF++, indicating that automatic metrics capture different quality aspects.Reported Spearman correlations are 0.49 with BLEU and 0.12 with chrF++ across all languages.

E Additional details about IN22 Benchmark

IN22 is designed as a diverse, India-centered benchmark spanning Wikipedia, government and digital-library web sources, and constructed conversations. Its creation combines controlled source selection, contextual translation, human review, and quality safeguards across all 22 languages.

  • Benchmark composition: IN22 contains Wikipedia, Web Sources, and Conversation subsets designed to represent varied source and domain characteristics.The benchmark’s domain diversity is summarized separately for its subsets.
  • Wikipedia subset: Wikipedia sentences are selected across categories, filtered by length and FLORES-200 overlap, and presented with a three-sentence context window.Sentences shorter than 6 or longer than 80 words are discarded.
  • Web Sources subset: Web Sources draw on Indian government websites and digital libraries to add India-centric, multi-domain material underrepresented in earlier benchmarks.PDF documents are used to discover sentences that may not occur in common web crawls.
  • Conversation subset: Conversation items begin from predefined prompts and scenarios, are enacted by two annotators, and are translated into 22 Indic languages with conversational context.The prompt establishes the topic, while the scenario supplies the specific interaction setting.
  • Quality-controlled workflow: Source selection and verification enforce domain coverage, length distribution, licensing, validity, metadata quality, and translatability before translation.Annotators classify candidates using tags including clean, difficult vocabulary, context incomplete, ambiguous, and profane.
  • Quality-controlled workflow: Translators receive context, guidelines, optional MT assistance for training data, terminology resources, and pivot translations for selected low-resource languages.Dedicated reviewers subsequently correct grammatical, syntactic, and dialectal problems.
  • Model access: The released IndicTrans2 models support translation across 22 Indic languages, including 25 language-script combinations to and from English.They also support Indic-to-Indic translation through a pivot-based approach.

J.3 Data, Metrics, Limitations, and Recommendations

This model-card section documents IndicTrans2’s training, fine-tuning, evaluation, metrics, intended uses, and open release. It identifies unsupported additional languages as a scope boundary and frames future expansion or fine-tuning as next steps.

  • Data: IndicTrans2 is trained with parallel corpora and augmented with synthetic data generated from intermediate models.The section distinguishes the training dataset from the fine-tuning datasets BPCC-H-Wiki and NLLB-Seed.
  • Evaluation: Evaluation uses FLORES-200 and IN22, with generation and evaluation procedures also applied to the compared baselines.The model is evaluated using the same procedure as the baselines.
  • Metrics: chrF++, BLEU, and COMET are reported, with chrF++ used as the primary metric alongside a small XSTS human evaluation.The XSTS evaluation covers only a small portion of the IN22 Combined evaluation set.
  • Limitations and recommendations: Additional languages are not currently supported, leaving expansion or minimal fine-tuning as future work.The stated scope boundary concerns languages beyond the 22 scheduled Indic languages.
  • Access: The model is released with its training data, source code, and tools under a permissive license to support replication and further improvement.The model card specifies the MIT license and identifies the open-source repository as a resource.
  • Intended use: IndicTrans2 is intended for research and commercial machine translation across 22 Indic languages, with support for single or batch translation and pivot-based Indic-to-Indic translation.The model card lists researchers, individuals, and organizations among intended users.

K.3 Data, Metrics, Limitations, and Recommendations

The IndicTrans2-M2M and IndicTrans2-Dist model cards describe data, evaluation, metrics, model configurations, intended use, and licensing. They note that the distilled models lack XSTS human evaluation and that additional languages remain unsupported.

  • Data and evaluation: IndicTrans2-M2M and IndicTrans2-Dist use parallel corpora for training and are evaluated on FLORES-200 and IN22.The evaluation procedure is shared with the compared baselines.
  • Metrics: chrF++, BLEU, and COMET are used for evaluation, with chrF++ designated as the primary metric.The metric description applies to both M2M and distilled models.
  • Limitations: IndicTrans2-Dist models do not receive an XSTS human evaluation, limiting direct human-evaluation evidence for the distilled variants.The model card explicitly records this omission among the models’ limitations.
  • Recommendations: Additional languages remain unsupported, with expansion or minimal fine-tuning identified as future work for the M2M and distilled models.The recommendation applies to improving existing supported languages as well as adding languages.
  • Model configuration: IndicTrans2-Dist consists of 18-layer encoder-decoder models with 211M parameters for English-to-Indic and Indic-to-English translation.The two models cover the opposite English–Indic directions.
  • Intended use: The models support research and commercial translation across 22 Indic languages, including 25 language-script combinations to and from English, plus pivot-based Indic translation.The intended users include researchers, individuals, and organizations.

L.3 Data, Metrics, Limitations, and Recommendations

The paper outlines the datasets, evaluation setup, known limitations, and future directions for IndicTrans2-Dist. Evaluation uses multiple metrics with chrF++ as primary, while coverage and human-evaluation gaps remain.

  • Data: IndicTrans2-Dist training uses BPCC-H-Wiki and NLLB-Seed for fine-tuning.These datasets are described as the fine-tuning data for the distilled models.
  • Evaluation: Evaluation considers FLORES-200, IN22, and baseline systems under the same generation and evaluation procedure.The models and compared baselines follow the procedure described for generation and evaluation.
  • Metrics: IndicTrans2-Dist is evaluated with chrF++, BLEU, and COMET, using chrF++ as the primary metric.
  • Limitations: The authors describe known caveats and do not conduct an XSTS human evaluation for IndicTrans2-Dist.
  • Recommendations: Future work could extend IndicTrans2-Dist to additional languages or improve supported languages through minimal fine-tuning.The paper notes that some languages are currently unsupported.

M.1 Dataset Description

BPCC is a publicly accessible corpus combining mined and human-translated data for all 22 scheduled Indic languages, while IN22 provides a multi-domain, n-way benchmark. The resources support training and evaluation but acknowledge privacy, bias, and licensing considerations.

  • Coverage and access: The dataset covers 22 scheduled Indic languages across 25 language-script combinations and is publicly accessible through stated usage directions.
  • BPCC: BPCC contains approximately 230M bitext pairs, including approximately 126M newly mined pairs and 2.2M human gold-standard English–Indic pairs.The human component includes Wikipedia and day-to-day-use subsets, including 644K and 139K sentence pairs respectively.
  • IN22: IN22 is a comprehensive multi-domain, n-way benchmark covering 22 Indic languages through IN22-Wiki, IN22-Web, and IN22-Conv.The sources include Wikipedia, government PDFs, open-source books, and an in-house conversation benchmark.
  • Uses: The resources are intended for training machine translation models and evaluating them with the IN22 benchmark.Initial model and baseline results are made available through the open-source repository.
  • Risks: Large-scale web mining may unintentionally include personal or sensitive information, while biases are not explicitly examined.The authors invite reports of potential information leakage and identify bias investigation as future work.
Loading 2305.16307v3…