Source-linked AI summary
Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages
Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj, Mayank Jobanputra, Raghavan AK, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Mahalakshmi J, Divyanshu Kakwani, Navneet Kumar, Aswin Pradeep, Srihari Nagaraj, Kumar Deepak, Vivek Raghavan, Anoop Kunchukuttan, Pratyush Kumar, Mitesh Shantadevi Khapra
TL;DR
Resource-poor Indic languages lack the parallel data needed for strong neural machine translation, despite their linguistic diversity and large speaker base. Samanantar combines public corpora with web mining using multilingual alignment, OCR, and efficient retrieval, and the resulting IndicTrans models outperform existing systems on many benchmarks. The corpus also receives high human semantic-similarity ratings, while its authors identify noisy existing sources and weaker alignment for some low-resource languages as areas for improvement.
Problem
Resource-poor languages, particularly related low-resource Indic languages, have limited parallel and monolingual corpora and evaluation resources for neural machine translation.
Method
The paper builds Samanantar by combining existing public corpora with web-mined data using multilingual sentence representations, OCR, and approximate nearest-neighbor search.
Results
IndicTrans trained on Samanantar outperforms nearly all existing models across most evaluated languages and directions, while included sentence pairs receive high human semantic-similarity scores.
Takeaways & Limitations
Samanantar provides a large publicly available resource and multilingual models for advancing Indic-language machine translation and multilingual NLP research.
Takeaways & Limitations
The authors identify the need to improve LaBSE representations for low-resource languages and longer sentences, and note that some existing sources are extremely noisy.
Abstract
from arXiv · showhide
We present Samanantar, the largest publicly available parallel corpora collection for Indic languages. The collection contains a total of 49.7 million sentence pairs between English and 11 Indic languages (from two language families). Specifically, we compile 12.4 million sentence pairs from existing, publicly-available parallel corpora, and additionally mine 37.4 million sentence pairs from the web, resulting in a 4x increase. We mine the parallel sentences from the web by combining many corpora, tools, and methods: (a) web-crawled monolingual corpora, (b) document OCR for extracting sentences from scanned documents, (c) multilingual representation models for aligning sentences, and (d) approximate nearest neighbor search for searching in a large collection of sentences. Human evaluation of samples from the newly mined corpora validate the high quality of the parallel sentences across 11 languages. Further, we extract 83.4 million sentence pairs between all 55 Indic language pairs from the English-centric parallel corpus using English as the pivot language. We trained multilingual NMT models spanning all these languages on Samanantar, which outperform existing models and baselines on publicly available benchmarks, such as FLORES, establishing the utility of Samanantar. Our data and models are available publicly at https://ai4bharat.iitm.ac.in/samanantar and we hope they will help advance research in NMT and multilingual NLP for Indic languages.
1 Introduction
Indic languages remain underserved by neural MT because they lack sufficient parallel and monolingual data, despite a large multilingual speaker base. Samanantar addresses this gap by combining existing and newly mined data, then training IndicTrans, which outperforms existing systems across many Indic-language evaluations.
- Motivation: Limited parallel and monolingual data constrain neural MT improvements for resource-poor languages, especially when related languages are collectively low-resource.Transfer learning can help, but limited in-language data makes it difficult, particularly for groups of related low-resource languages.
- Motivation: India and the wider subcontinent contain linguistically diverse, closely related languages spoken by over 1 billion people, creating demand for publicly available translation systems.The paper frames large-scale parallel-data creation as the practical response, while noting that manual translation would be slow and expensive.
- Contribution: 49.7M English–Indic sentence pairs comprise Samanantar, including 37.4M newly mined pairs and 12.4M compiled from existing sources.The collection covers English and 11 Indic languages, with language-wise totals ranging from 141K English–Assamese pairs to 10.1M English–Hindi pairs.
- Evaluation: IndicTrans trained on Samanantar performs better than existing commercial, open-source, and existing-data systems across multiple test sets spanning 10 Indic languages.The evaluation compares against Google, Microsoft, OPUS-MT, mBART50, CVIT-ILMulti, and models trained on all existing parallel sources.
- Contribution: The paper contributes Samanantar, IndicTrans, and human judgments on crosslingual textual similarity for about 9,566 sentence pairs.These resources are intended for public release to support research on Indic-language translation and multilingual NLP.
2 Samanantar: A Parallel Corpus for Indic Languages
Samanantar combines existing public resources with web-mined data from machine-readable, scanned-document, and monolingual sources to expand English–Indic and Indic–Indic parallel coverage. Its mining pipeline uses multilingual embeddings, OCR, and approximate nearest-neighbor search, producing a substantial increase in publicly available Indic-language sentence pairs.
- Corpus construction: Samanantar covers English paired with 11 Indic languages and adds 55 Indic-language pairs obtained by pivoting through English.The corpus is built by collating existing public parallel sources and mining additional parallel sentences from three web-source types.
- Existing sources: 12.4M parallel sentences come from existing sources after deduplication, including OPUS and non-OPUS corpora.The authors note that no publicly available MT system had previously been trained on all these existing sources together.
- Existing sources: Existing sources were retained without additional cleaning despite severe noise in some resources, including JW300 sentence pairs that were not translations.The authors leave data filtering and sampling strategies for future work.
- Web mining: Machine-readable comparable corpora are extracted from multilingual news and educational sources, then sentence-aligned within matching date or video contexts.Custom extractors, subtitle collection, sentence tokenization, and metadata-based clustering support this extraction process.
- Web mining: LaBSE cosine similarity selects candidate translation pairs, after which LAS thresholds, deduplication, minimum English length, and language identification filter the data.A threshold of 0.75 was found to work well across different sources.
- Document mining: OCR enables extraction from scanned or legacy-encoded government and legislative PDFs that standard PDF parsers cannot process.The paper uses public sources from Tamil Nadu, Bangladesh, West Bengal, Andhra Pradesh, and Telangana.
- Monolingual mining: IndicCorp mining uses LaBSE embeddings and FAISS indexing to retrieve nearest English sentences efficiently from a large monolingual collection.FAISS clusters and searches dense vectors using normalized inner product, equivalent to cosine similarity.
3 Analysis of the Quality of the Mined Parallel Corpus
Human evaluation finds that Samanantar’s mined sentence pairs generally have high semantic similarity, while revealing weaker LaBSE alignment for low-resource languages and longer sentences. Error analysis further shows high extraction accuracy, especially for definitely accepted pairs, but identifies residual alignment mistakes.
- Annotation Task and Setup: 9,566 sentence pairs across 11 Indic languages were sampled for human cross-lingual semantic similarity evaluation.The sample was stratified across definite accept, marginal accept, and reject sets, with annotations from 38 annotators.
- Annotation Results and Discussion: 4.27 mean STS and 5 median STS for ‘All accept’ pairs indicate high semantic similarity on the 0–5 scale.‘Definite accept’ pairs averaged 4.63, compared with 3.89 for ‘Marginal accept’ and 2.94 for ‘Reject’.
- Annotation Results and Discussion: 0.37 Spearman correlation between LAS and STS shows a moderately positive relationship between LaBSE alignment and human similarity judgments.The lower correlation for Assamese and Odia points to further alignment challenges for the smallest-resource languages.
- Annotation Results and Discussion: -0.35 Spearman correlation between sentence length and LAS indicates that longer sentences are less likely to receive high LaBSE alignment.Sentence length was almost uncorrelated with STS, with a Spearman coefficient of -0.04.
- Annotation Results and Discussion: 79.5% extraction accuracy overall and 90.1% for the Definitely Accept bucket show that LAS-based filtering can yield accurate parallel corpora.Marginally Accept and Definitely Accept pairs mainly contained minor quantity or special-word alignment errors, whereas Marginally Reject pairs could be non-parallel.
- Annotation Results and Discussion: The annotation task validated the selected thresholds while identifying needs for better LaBSE alignment in low-resource languages and longer sentences.The released dataset and human judgments are intended to support cross-lingual semantic similarity evaluation between English and Indic languages.
4 IndicTrans: Multilingual, single Indic script models
IndicTrans uses multilingual models spanning Indic languages, with a shared script representation and language tags to support transfer across languages. Training uses Samanantar data with strict overlap removal and standard Transformer training and decoding settings.
- IndicTrans trains One-to-Many models for English-to-Indic translation and Many-to-One models for Indic-to-English translation across the Indic languages.
- Data Representation: Representing Indic-language data in a single script increases lexical sharing, reduces subword-vocabulary fragmentation, and permits a smaller vocabulary.
- Source sentences begin with a special source-language token, while target-language tokens identify the desired output language in multilingual translation.
- Training Data: Training removes pairs whose English or corresponding Indic sentences overlap with validation or test data across the joint model’s language pairs.
- Network & Training: The models use six-layer Transformer encoders and decoders, 1536-dimensional embeddings, 16 attention heads, and 4096-dimensional feedforward layers.
- Decoding: Decoding uses beam search with beam size 5 and a length penalty of 1.
5 Experimental Setup
The experiments evaluate Samanantar-trained translation systems against commercial, public, and existing-data baselines across diverse Indic translation benchmarks. BLEU with reproducible SacreBLEU settings is used, while training and evaluation avoid data overlap.
- Benchmarks: The benchmarks include WAT2020, WAT2021, WMT, UFAL Entam, FLORES, and an additional PMIndia Assamese-English test set.
- Evaluation: BLEU scores measure translation quality, using direction-specific tokenization and SacreBLEU signatures to support consistency and reproducibility.
- Models: The evaluation compares Samanantar-trained systems with commercial systems, public NMT systems, and models trained on all previously available parallel data.
- Models: OPUS-MT and mBART50 provide publicly available comparison systems, while Transformer and mT5 models are trained or fine-tuned on prior parallel-data sources.
- Data Hygiene: All models trained or fine-tuned in the study exclude overlaps between training and test or validation sets.
6 Results and Discussion
IndicTrans trained on Samanantar generally outperforms existing systems across Indic-English and English-Indic benchmarks, including independently created FLORES evaluations. Gains are especially pronounced for low-resource languages, though commercial systems remain stronger on some FLORES comparisons.
- IndicTrans trained on Samanantar outperforms nearly all existing models in both Indic-English and English-Indic directions across the reported benchmarks.The stated exception is languages in the WMT and UFAL en-ta benchmark.
- IndicTrans trained on Samanantar clearly outperforms IndicTrans trained only on existing resources on the FLORES devtest benchmark.
- The experiments report results for Indic-English and English-Indic translation using BLEU-score tables, with Table 6 covering multiple test sets.
- On FLORES, Samanantar-trained models remain a few points behind commercial systems, whose training data better capture FLORES domain and distribution.
- Performance gains are higher for low-resource languages such as Assamese, Odia, and Kannada, especially in the Indic-English direction.The discussion attributes these gains to multilingual training and transfer from related, better-resourced languages.
7 Conclusion
Samanantar provides large-scale parallel data for Indic languages, including web-mined and English-pivoted sentence pairs, and supports IndicTrans, which outperforms existing models. The authors also identify representation learning, training objectives, and multilingual pretraining as areas for further exploration.
- 37.4M parallel sentences are mined using web-crawled monolingual corpora, multilingual representations, approximate nearest neighbor search, and optical character recognition.
- 9,566 human judgments indicate that the mined sentence pairs are high quality.
- IndicTrans trained on Samanantar outperforms existing models across a wide variety of benchmarks for Indic languages.
- Further work should improve low-resource-language representations, data-aware training schedules and objectives, and multilingual-model pretraining.
- Samanantar, IndicTrans, and a manually annotated cross-lingual similarity dataset are intended to support further research in Indic NMT and multilingual NLP.