Source-linked AI summary
The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English
Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, Marc'Aurelio Ranzato
TL;DR
Low-resource MT lacks both parallel data and reliable public benchmarks, especially for linguistically distant language pairs. The paper introduces Wikipedia-based FLORES benchmarks for Nepali–English and Sinhala–English, constructed with translation-quality checks, and finds that current methods perform poorly while semi-supervised and multilingual approaches lead the tested variants.
Problem
Low-resource MT has scarce parallel data and few reliable public benchmarks, making progress difficult to evaluate on real low-resource language pairs.
Method
The paper builds Nepali–English and Sinhala–English FLORES benchmarks from Wikipedia translations and applies automatic and human quality checks.
Results
Current state-of-the-art methods perform rather poorly on FLORES, while semi-supervised and multilingual neural methods outperform the other tested variants and training settings.
Takeaways & Limitations
FLORES provides freely accessible evaluation data for low-resource MT and supports fairer comparison of methods on actual low-resource languages.
Takeaways & Limitations
Existing parallel data for Nepali–English and Sinhala–English is small, domain-limited, and supplemented mainly by publicly available monolingual data.
Abstract
from arXiv · showhide
For machine translation, a vast majority of language pairs in the world are considered low-resource because they have little parallel data available. Besides the technical challenges of learning with limited supervision, it is difficult to evaluate methods trained on low-resource language pairs because of the lack of freely and publicly available benchmarks. In this work, we introduce the FLoRes evaluation datasets for Nepali-English and Sinhala-English, based on sentences translated from Wikipedia. Compared to English, these are languages with very different morphology and syntax, for which little out-of-domain parallel data is available and for which relatively large amounts of monolingual data are freely available. We describe our process to collect and cross-check the quality of translations, and we report baseline performance using several learning settings: fully supervised, weakly supervised, semi-supervised, and fully unsupervised. Our experiments demonstrate that current state-of-the-art methods perform rather poorly on this benchmark, posing a challenge to the research community working on low-resource MT. Data and code to reproduce our experiments are available at https://github.com/facebookresearch/flores.
1 Introduction
Low-resource MT suffers from scarce parallel data and lacks reliable public benchmarks, while evaluation-set construction is costly and difficult to quality-check. FLORES addresses this gap with Nepali–English and Sinhala–English benchmarks that challenge current methods.
- Motivation: Low-resource MT systems perform poorly because parallel training data is scarce, despite the potential importance of these languages.
- Motivation: Reliable public benchmarks are needed to track progress, but constructing them is expensive because fluent bilingual translators and quality-assurance resources are limited.
- Contribution: FLORES introduces Nepali–English and Sinhala–English evaluation benchmarks built from professionally translated Wikipedia sentences.The released tune, development, and test sets contain thousands of sentences for each language pair.
- Results: Current state-of-the-art methods obtain very low BLEU scores on FLORES even when using parallel, monolingual, or filtered Paracrawl data.The results indicate that the benchmarks remain challenging across multiple training-resource settings.
2 Related Work
Prior low-resource MT research develops ways to exploit monolingual, noisy, and related-language data, but publicly available benchmarks remain limited and often unrealistic. FLORES responds by evaluating methods on translations originating in low-resource languages.
- Modeling approaches: Low-resource MT methods increasingly exploit monolingual data through back-translation, language-model integration, and fully unsupervised learning.
- Modeling approaches: Weakly supervised methods augment scarce parallel training data with sentences mined from noisy comparable corpora such as Paracrawl.
- Evaluation resources: Existing low-resource datasets are often inaccessible or derived from English text, limiting their relevance to text originating in low-resource languages.
- Dataset construction: Professional translations are generally preferred to crowd-sourced translations, motivating quality checks for professional low-resource-language data.
- Evaluation resources: Simulating low-resource learning by restricting high-resource European language pairs can obscure practical effects because real low-resource languages may be distant and their corpora non-comparable.
3 Methodology & Resulting Datasets
FLORES constructs Nepali–English and Sinhala–English evaluation sets from Wikipedia, filters translations through automatic and human checks, and organizes balanced validation and test data. The resulting benchmarks cover diverse topics and preserve substantial bilingual evaluation coverage.
- 3.1 Document selection: FLORES selects Wikipedia sentences, translates them professionally in relevant directions, and requests two independent translations per string.
- 3.2 Quality checks: Automatic filters remove translations with suspected fluency, copying, transliteration, disagreement, or vocabulary problems before human review.
- 3.2 Quality checks: Human assessment rates translation quality and English fluency, retaining only translations that pass the manual quality checks.
- 3.3 Resulting Datasets: The benchmark provides tune, validation, and blind test sets for hyperparameter tuning, development generalization, and final evaluation.
- 3.3 Resulting Datasets: The test sets contain 2,924 Nepali–English and 2,905 Sinhala–English sentence pairs, with balanced original-language subsets.Each test set includes 850 sentences originally in English and 850 originally in the corresponding low-resource language.
- 3.3 Resulting Datasets: The datasets span diverse topics, increasing difficulty because models must operate across domains rather than specialize narrowly.
4 Experiments
The experiments evaluate PBSMT and NMT across supervised, unsupervised, semi-supervised, weakly supervised, and multilingual settings on the new benchmarks. The language pairs remain difficult, while back-translation, Hindi data, and filtered noisy parallel data provide the strongest improvements.
- Experimental setup: The study compares Moses PBSMT with Fairseq Transformer NMT using four learning settings and reports detokenized SacreBLEU for translation into English.The settings are supervised, unsupervised, semi-supervised back-translation, and weakly supervised training with filtered Paracrawl.
- Results: PBSMT performs worse than supervised NMT, scoring 2.5, 4.4, 1.6, and 5.0 BLEU across the four translation directions.The scores correspond to English–Nepali, Nepali–English, English–Sinhala, and Sinhala–English, respectively.
- Results: Supervised NMT baselines score below 8 BLEU, while unsupervised NMT approaches obtain scores close to 0 on these distant language pairs.The reported failure of unsupervised NMT is attributed to poor initialization of word embeddings.
- Results: 15.1 BLEU is achieved for both Nepali–English and Sinhala–English with semi-supervised back-translation, nearly doubling scores from 7.6 and 7.2, respectively.The gains are +7.5 BLEU points for Nepali–English and +7.9 BLEU points for Sinhala–English.
- Results: English–Hindi parallel data improves Nepali translation across settings, including a 6.5-BLEU gain for supervised Nepali–English and an 18.8-BLEU result with multilingual pretrained architecture.The semi-supervised Nepali–English gain is 6.4 BLEU points; multilingual unsupervised training reaches 3.9 BLEU for Nepali–English and 2.5 for English–Nepali.
- Results: LASER-filtered Paracrawl improves over unfiltered noisy data by 5.5 BLEU points for Nepali–English and 7.3 for Sinhala–English.Adding Paracrawl Clean to the initial parallel data adds 2.0 and 3.7 BLEU points for the same directions.
5 Discussion
The analysis finds that benchmark difficulty is largely unrelated to reference fluency or document identity, but source origin and domain mismatch affect translation quality. Nepali-origin documents are harder, and in-domain performance substantially exceeds out-of-domain performance.
- Dataset analysis: Translation quality varies by only 10 BLEU points between the easiest and hardest document groups, suggesting that document sampling did not create an extreme difficulty imbalance.The analysis reports similar translation quality across document ids.
- Dataset analysis: Reference fluency does not correlate with machine translation quality, so more fluent human translations are not easier for the system to translate.This finding is measured using sentence-level BLEU and AMT fluency scores.
- Domain drift: In-domain Sinhala–English performance is 10–16 BLEU points higher than out-of-domain performance, indicating substantial domain mismatch effects.The comparison uses held-out Open Subtitles sentences versus the Wikipedia devtest set.
6 Conclusions
The paper addresses low-resource MT through freely released FLORES benchmarks for Nepali–English and Sinhala–English, where public evaluation data are scarce. Experiments find current approaches perform poorly, while the benchmarks support fairer comparison and further research.
- FLORES provides freely released evaluation benchmarks for Nepali–English and Sinhala–English, addressing the lack of public low-resource MT benchmarks.
- Nepali and Sinhala differ substantially from English in syntax and morphology, while offering little public parallel data alongside substantial monolingual resources.
- Current state-of-the-art approaches perform rather poorly on the new benchmarks, while semi-supervised and especially multilingual neural methods outperform the other evaluated settings.
A List of Topics
Table 6 organizes dev, devtest, and test sentences by the Wikipedia topics of their source documents.
- Table 6 reports topic distributions for sentences in the dev, devtest, and test sets according to their source Wikipedia documents.
B Statistics of automatic filtering and manual filtering
The dataset quality process combines automatic and manual filtering, using averaged translation-quality and fluency scores alongside adequacy and fluency checks.
- Translations with averaged quality scores below 70 are removed after three raters assess each sentence on a 0–100 scale.
- Translations with averaged fluency scores below 3 are removed after five raters assess English fluency on a 1–5 scale.
- Table 7 summarizes filtering failures, with automatic checks preceding two human checks for adequacy and fluency.
C List of Wikpedia Documents
Table 8 lists the Wikipedia documents used in the datasets, their domains, names or English translations, and corresponding topics.
- Table 8 links each Wikipedia document to its domain, document name or English translation, and corresponding topic.
D Examples from devtest
This section presents example sentences from the En-Ne, Ne-En, En-Si, and Si-En devtest sets, alongside hypotheses generated by the paper’s semi-supervised model.
- One example sentence describes a companion being removed from substance after education or because of family disease conditions.
- Table 9 covers example sentences from the En-Ne, Ne-En, En-Si, and Si-En devtest sets.
- The displayed system hypotheses were generated with the semi-supervised model using beam search decoding.