Source-linked AI summary
The IIT Bombay English-Hindi Parallel Corpus
Anoop Kunchukuttan, Pratik Mehta, Pushpak Bhattacharyya
TL;DR
Hindi’s widespread use across India and its role in administration motivate substantial English-Hindi parallel-data resources. The paper compiles and aligns corpora, reports baseline machine-translation results, and releases the corpus for research use.
Problem
Hindi’s widespread use in India, including administrative and business contexts, motivates the development of English-Hindi parallel-data resources.
Method
The corpus combines existing sources with newly collected data, including government websites, and uses sentence alignment to extract parallel sentences from comparable HTML data.
Results
1,492,827 training segments were assembled, and baseline phrase-based SMT and NMT systems were trained and evaluated on the corpus.
Takeaways & Limitations
The corpus is available for research and non-commercial use, with sub-corpora separately extractable for applications such as domain adaptation.
Takeaways & Limitations
Parallel-sentence extraction from the comparable corpus achieved 88.6% precision in a manual check of 300 sentences.
Abstract
from arXiv · showhide
We present the IIT Bombay English-Hindi Parallel Corpus. The corpus is a compilation of parallel corpora previously available in the public domain as well as new parallel corpora we collected. The corpus contains 1.49 million parallel segments, of which 694k segments were not previously available in the public domain. The corpus has been pre-processed for machine translation, and we report baseline phrase-based SMT and NMT translation results on this corpus. This corpus has been used in two editions of shared tasks at the Workshop on Asian Language Translation (2016 and 2017). The corpus is freely available for non-commercial research. To the best of our knowledge, this is the largest publicly available English-Hindi parallel corpus.
1. Introduction
Hindi’s widespread use across India and beyond creates substantial demand for digital communication and English–Hindi machine translation.
- Hindi is a major language and lingua franca in India, with recognized regional-language status in several countries.The passage describes Hindi’s broad geographic and institutional use.
- Hindi’s large speaker population and English’s comparatively smaller native-speaker base create demand for communication between the languages.The passage reports 422 million native Hindi speakers, more than 500 million total speakers, and around 125 million English speakers in India.
- The paper identifies substantial potential for English–Hindi machine translation because digital communication in Hindi must interface with English.
2. Dataset
The IIT Bombay corpus combines diverse English–Hindi resources, including newly collected and domain-specific data, with alignment and dataset statistics documented for research use.
- Corpus Details: The corpus compiles resources from OPUS, HindEn, TED, and CFILT corpora spanning sentences, phrases, dictionary entries, applications, and domains.
- Corpus Details: New components include government-web data, terminology-dictionary translations, bilingual WordNet entries, and judicial-domain translations.Judicial data includes translations by experienced in-house translators and graduate students, with the latter judged good quality by annotators.
- Corpus Details: Gyaan-Nidhi is an English–Indian-language multilingual corpus in HTML format, so it is not sentence aligned in its original form.The authors extracted parallel corpora using Moore’s sentence-alignment technique and manually checked 300 extracted sentences.
- Corpus Details: Table 1 identifies newly released corpora not previously available in the public domain with an asterisk.
- Corpus Details: 88.6% precision was obtained for parallel sentences extracted from the comparable corpus in a manual check of 300 sentences.The alignment method combines sentence-length and word-correspondence models without language- or corpus-specific knowledge.
- Corpus Statistics: 1,492,827 training segments, 520 development segments, and 2,507 test segments comprise the reported datasets.The test and development sets use newswire sentences from the WMT 2014 English–Hindi shared task; word-type OOV rates are 11.4% for Hindi and 6.7% for English.
3. Baseline Systems
The paper establishes phrase-based SMT and subword-level attention NMT baselines for English–Hindi and Hindi–English translation and evaluates them with BLEU and METEOR variants.
- Baseline Systems: The authors trained PBSMT and NMT systems in both English–Hindi and Hindi–English directions using off-the-shelf machine-translation toolkits.
- SMT Setup: PBSMT used Moses with phrase extraction, lexicalized reordering, Batch MIRA tuning, and 5-gram Kneser–Ney language models.Additional Hindi and English monolingual corpora were used for language-model training.
- NMT Setup: NMT used a subword-level encoder–decoder with attention and separate English and Hindi BPE vocabularies.The architecture used one 512-unit GRU encoder layer and one 512-unit GRU decoder layer.
- Results: Table 3 reports the results for the baseline systems.
- Results: Hindi-target evaluation used METEOR-Indic, which supports synonym and stem matching through IndoWordNet synsets and a trie-based stemmer.The systems were evaluated using BLEU and METEOR, with METEOR-Indic customized for Hindi.
5. Conclusion and Future Work
The paper presents version 1.0 of the IIT Bombay English–Hindi Parallel Corpus and benchmark SMT and NMT results, while outlining future expansion and stronger baselines.
- The corpus version 1.0 is accompanied by benchmark baseline SMT and NMT results.
- The corpus supported the WAT 2016 and 2017 shared tasks, and its HindiEn component was used in the WMT 2014 shared task.
- The corpus is available under a Creative Commons licence.
- Future work includes adding Government of India website sources, pre-ordering for PBSMT, and synthetic backtranslated corpora for NMT.