Source-linked AI summary
PMIndia -- A Collection of Parallel Corpora of Languages of India
Barry Haddow, Faheem Kirefu
TL;DR
Parallel corpora are scarce for many South Asian languages, limiting available material for machine translation. The paper presents PMIndia, an English-parallel corpus of Indian-language news articles, and assesses automatic sentence alignment. The intersection of two aligners yields 79% valid alignments under the conservative measure and 94% under the liberal measure.
Problem
South Asian languages are poorly supported by parallel corpora, and almost all are considered under-resourced for machine translation.
Source: §1 · p. 1Method
The paper builds PMIndia from news updates on the Prime Minister of India’s website and reports initial NMT experiments using generally low-resource settings.
Source: §1 · p. 1, §3.1 · p. 2, §4 · p. 4Results
79% of alignments from the intersection of both aligners were judged valid, rising to 94% under the more liberal measure.
Source: §3.2 · p. 4Takeaways & Limitations
PMIndia is released as an English-parallel corpus covering 13 Indian languages under a CC-BY-4.0 licence.
Source: §1 · p. 1The corpus includes languages spoken in India, and the alignment assessment does not distinguish which languages are sources versus translationese.
Source: §1 · p. 1, §3.2 · p. 3
Abstract
from arXiv · showhide
Parallel text is required for building high-quality machine translation (MT) systems, as well as for other multilingual NLP applications. For many South Asian languages, such data is in short supply. In this paper, we described a new publicly available corpus (PMIndia) consisting of parallel sentences which pair 13 major languages of India with English. The corpus includes up to 56000 sentences for each language pair. We explain how the corpus was constructed, including an assessment of two different automatic sentence alignment methods, and present some initial NMT results on the corpus.
1 Introduction
South Asian languages are poorly served by parallel corpora, and existing coverage for many is limited or domain-specific. PMIndia introduces a publicly available English-parallel corpus built from Indian-language news updates.
1 Introduction
- South Asian languages are poorly supported by parallel corpora, and almost all are considered under-resourced for machine translation.The largest known South Asian parallel corpus cited here is the IIT Bombay English–Hindi corpus, with about 1.5 million sentence pairs from various sources.Source: §1 · p. 1
- Existing coverage is limited for many South Asian languages, with some available corpora restricted to religious texts, localization, or noisy, domain-specific material.The passage also names WikiMatrix and JW300 as newer multilingual corpora.Source: §1 · p. 1
- PMIndia pairs English with 13 Indian languages using news updates from the Prime Minister of India’s website, and the corpus is released under CC-BY-4.0.The site’s pages are available in English and Hindi, with translations into other languages to varying degrees.Source: §1 · p. 1
2 Languages of India
India’s languages mostly belong to the Indo-Aryan and Dravidian families, and their writing systems share features that matter for text processing.
2 Languages of India
- Most Indian languages fall into the Indo-Aryan or Dravidian families; the Constitution lists 22 scheduled languages, all but Sanskrit with more than two million speakers.Seven scheduled languages are reported among the world’s 20 most spoken languages, according to the cited Ethnologue source.Source: §2 · pp. 1–2
- Many Indian languages use Brahmic abugida scripts, where consonant-vowel sequences form units and a letter may require up to three Unicode characters.Naive processing can separate a consonant from its vowel diacritic; transliteration between Indian scripts is described as feasible because phonemic inventories are mostly similar, except for Tamil’s reduced inventory.Source: §2 · p. 2
3 Corpus Preparation
The authors collected and processed PMIndia’s multilingual articles, then aligned their English and Indian-language sentences with hunalign and, where available, Vecalign. Human evaluation found high validity in the released intersection, while alignment coverage and agreement varied by language pair.
3.1 Crawling and Extraction
- The authors scraped the PMIndia news archive and used article hyperlinks to align documents, checking translated titles to distinguish Indian-language pages from English fallbacks.Article counts by language are reported in Table 1.Source: §3.1 · p. 2, Table 1 · p. 2
- They removed untranslated embedded tweets before extraction and extended the Moses splitter for Indian scripts, language-specific non-breaking prefixes, and itemised lists.Early alignment experiments showed that inconsistent list handling caused errors.Source: §3.1 · p. 2
3.2 Sentence Alignment
- The corpus retains only 1–1 alignments; where Vecalign is available, the release uses the hunalign–Vecalign intersection, otherwise it uses hunalign alone.Vecalign was applicable only where LASER alignments were available; Table 2 reports pair counts by method and release version.Source: §3.2 · pp. 2–3, Table 2 · p. 3
- Agreement between the aligners was mostly around 80% and symmetric, with better agreement for English–Hindi, possibly reflecting better resources or fewer indirect translations.The authors note that they did not analyze which languages were sources and which were translationese.Source: §3.2 · p. 3
- Among aligner-only samples, 41 hunalign pairs versus 26 Vecalign pairs were valid translations, while free translations numbered 2 and 13, respectively.The released corpus uses the intersection where both aligners are available.Source: §3.2 · pp. 3–4
- Among 100 sampled sentence pairs, 79% of intersection alignments were judged valid, rising to 94% under the liberal criterion; estimated precision was 70%/88% for Vecalign and 72%/88% for hunalign (conservative/liberal).Source: Table 4 · p. 4, §3.2 · p. 4
4 Machine Translation Experiments
The experiments train NMT systems in both directions for every released language pair, using held-out development and test sets. Reported BLEU is especially low into Dravidian languages, while scores into English are nearly always higher.
4 Machine Translation Experiments
- The systems were trained in both directions for every released language pair, with 1,000 parallel sentences each reserved for development and test.The data was normalized and tokenized; Indian-language data used the Indic NLP toolkit and omitted truecasing.Source: §4 · p. 4
- The evaluation used a shallow Marian RNN with generally low-resource-appropriate parameters, rather than language-pair-specific hyperparameter tuning.The authors note that tuning each pair individually could produce better results.Source: §4 · p. 4
- Table 5 reports BLEU scores for English-to-Indian-language and reverse systems, with 1,000 sentences reserved each for development and test.Scores were evaluated on tokenized text using Moses multi-bleu.perl.Source: Table 5 · p. 4, §4 · p. 4
- BLEU scores are very low into Indian languages, especially Dravidian languages, while scores into English are nearly always higher.The authors suggest the lower scores into Indian languages mainly reflect language difficulties rather than dataset problems, while noting that BLEU is less reliable for agglutinative Dravidian languages.Source: §4 · pp. 4–5
- Assamese translations perform poorly with datasets under 10,000 sentences, whereas Manipuri and Urdu score much higher despite dataset size.The authors suggest those results may reflect more domain-specific translation samples.Source: §4 · pp. 4–5
5 Conclusions
The conclusion presents PMIndia as a corpus pairing English with 13 low-resource South Asian languages, drawn from the Indian prime minister’s website. It proposes a future multiparallel release, motivated by the scarcity of such corpora and reported BLEU gains from exploiting them in one-to-many low-resource settings.
5 Conclusions
- PMIndia pairs English with 13 low-resource South Asian languages using text from the Indian prime minister’s website.The current corpus is English-centric.Source: §5 · p. 5
- The authors intend to release a multiparallel version instead of the current English-centric corpus.They note that multiparallel corpora are even scarcer for South Asian languages, and that exploiting them in one-to-many low-resource settings yields considerable BLEU gains.Source: §5 · p. 5