The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English

• Neural MT performs poorly when parallel training data is scarce, despite near-human performance on some adequately resourced language pairs; improving low-resource translation could benefit a large fraction of the world’s population.Source: §1 · p. 1
• Low-resource MT needs methods that use monolingual, noisy comparable, and related-language parallel data, alongside reliable public benchmarks for tracking translation quality.Source: §1 · p. 1
• Creating low-resource evaluation sets is costly and time-consuming because fluent bilingual translators are scarce, quality assurance is underdeveloped, and cultural context can be difficult for non-native reviewers to verify.Source: §1 · pp. 1–2
• The benchmarks contain professionally translated Wikipedia sentences for Nepali–English and Sinhala–English, with tune, development, and test sets for each pair.Nepali–English has 2,559 tune, 2,835 development, and 2,924 test sentences; Sinhala–English has 2,898 tune, 2,766 development, and 2,905 test sentences.Source: §1 · p. 2
• Current state-of-the-art methods achieve very low BLEU on these benchmarks even using all available parallel data plus monolingual data or filtered Paracrawl1, making them useful test beds.Source: §1 · p. 2

2 Related Work

• Low-resource MT approaches use monolingual data for semisupervised back-translation (Sennrich et al., 2015), decoder language models (Stahlberg et al., 2018), and unsupervised learning (Lample et al., 2018b; Artetxe et al., 2018), or augment supervision with mined parallel sentences.Distant languages also motivate language-independent lexical representations for transfer learning (Gu et al., 2018).Source: §2 · p. 2
• LORELEI translations are restricted to program participants, while Asian Language Treebank datasets originate in English and may not generalize to text from low-resource languages.Source: §2 · p. 2
• Crowd-sourced translations have generally lower quality than professional translations; this work investigates quality checks to filter professional low-resource translations into a high-quality benchmark.Earlier work extracted crowd-sourced translations using automatic methods.Source: §2 · p. 2
• Few low-resource datasets are public; simulations that limit English–French training data enable comparison but may not reflect distant languages or noncomparable corpora (Lample et al., 2018a; Neubig and Hu, 2018).Such simulations can leave the practical implications of methods unclear.Source: §2 · p. 2

3.1 Document selection

• Wikipedia sentences were selected from the top 25 documents per source language using language identification and length filters, then 2,500 per language were sampled for professional translation.English sentences were translated into Nepali and Sinhala, while Nepali and Sinhala sentences were translated into English; each string was translated twice by different translators.Source: §3.1 · p. 3

3.2 Quality checks

• Automatic filters removed translations with English perplexity above 3000.0, paired char-BLEU below 15 or above 90, at least 33% transliterated words, or excessive copying or out-of-vocabulary words.The filters aimed to retain fluent translations distinct from their source, similar but not identical to each other, and not transliterations; perplexity and out-of-vocabulary criteria applied to English translations only.Source: §3.2 · pp. 3–4
• Human reviewers rejected translations averaging below 70/100, while five annotators separately rated English fluency and only scores above 3/5 were retained.Three raters assessed translation quality; scores spanning more than 30 points triggered a fourth-rater tie-break, and translations failing human checks were discarded.Source: §3.2 · pp. 3–4

3.3 Resulting Datasets

• Dev, devtest, and test serve tuning, development validation, and blind final evaluation; devtest and test mix translation directions at an approximate 50–50 ratio.Sentences were grouped by document before assignment to minimize the number of documents per set; the dev set was not guaranteed to be balanced.Source: §3.3 · p. 4
• The test sets contain 2,905 Sinhala–English and 2,924 Nepali–English sentence pairs, with 850 source sentences originally in each language per pair.The Sinhala–English set has 1,465 pairs originally in English and 1,440 in Sinhala; the Nepali–English set has 1,462 pairs originally in each language.Source: Table 1 · p. 4, §3.3 · p. 4
• The datasets span diverse topics, including general subjects, history, law, and sports, requiring domain-independent models.The paper notes that this topic richness increases the difficulty of the evaluation sets.Source: §3.3 · p. 4

4.1 Training Data

• Training data included small parallel corpora from OPUS, Global Voices, Bible translations, and Paracrawl; Nepali multilingual experiments also used about 5 million Hindi sentences and 1.5 million English–Hindi parallel sentences.Clean Paracrawl was LASER-filtered to retain sentence pairs with 1 million English tokens; a random-filtered version with the same token count served as a comparison.Source: §4.1 · p. 5, Table 2 · p. 5

4.2 Training Settings

• The study covered supervised, unsupervised, back-translation-based semi-supervised, and LASER-filtered weakly supervised settings, plus joint Hindi–Nepali training for Nepali.Semi-supervised training used up to two iterative back-translation rounds; Nepali multilingual variants also included joint supervised and unsupervised approaches.Source: §4.2 · pp. 5–6

4.3 Models & Architectures

• Models included Moses phrase-based MT and Transformer NMT; supervised Transformers used 5 encoder and decoder layers, while semi-supervised Transformers used 6.Multilingual models shared encoders for Hindi/Nepali-to-English and decoders for English-to-Hindi/Nepali; hyperparameters were cross-validated on the dev set.Source: §4.3 · p. 6

4.4 Preprocessing and Evaluation

• Nepali and Sinhala were tokenized with the Indic NLP Library; NMT used 5K joint source-target BPE symbols, and evaluation used SacreBLEU into English and tokenized BLEU into Nepali or Sinhala.Training pairs exceeding 250 source or target BPE tokens were removed.Source: §4.4 · p. 6

4.5 Results

• Supervised NMT scored below 8 BLEU, while unsupervised NMT was near 0 because the embedding-training monolingual corpora lacked sufficient overlap and comparability.Supervised PBSMT scored 2.5 on English–Nepali, 4.4 on Nepali–English, 1.6 on English–Sinhala, and 5.0 on Sinhala–English.Source: §4.5 · pp. 6–7, Table 3 · p. 7
• Back-translation nearly doubled Nepali–English BLEU from 7.6 to 15.1 (+7.5 points) and Sinhala–English from 7.2 to 15.1 (+7.9 points).It also increased English–Nepali by +2.5 BLEU points and English–Sinhala by +5.3 BLEU points.Source: §4.5 · p. 7, Table 3 · p. 7
• Hindi data improved Nepali–English by 6.5 BLEU points in supervised training and 6.4 in semi-supervised training; pretrained multilingual unsupervised scores reached 18.8 for Nepali–English and 8.3 for English–Nepali.Without the prescribed Lample and Conneau (2019) architecture pretraining, multilingual unsupervised training reached 3.9 BLEU for Nepali–English and 2.5 for English–Nepali.Source: §4.5 · p. 7, Table 3 · p. 7
• LASER-filtered Paracrawl improved over unfiltered Paracrawl by +5.5 BLEU points for Nepali–English and +7.3 for Sinhala–English; adding Clean Paracrawl yielded gains of +2.0 and +3.7, respectively.Unfiltered or randomly filtered Paracrawl scored close to 0 BLEU; the weakly supervised baseline improved on the supervised baseline in all four directions.Source: §4.5 · p. 7, Table 4 · p. 7

5 Discussion

• Reference fluency does not correlate with translation quality, while document difficulty is similar across ids, with a 10-BLEU-point gap between easiest and hardest documents.The authors interpret the document pattern as evidence that random sampling was adequate and no single Wikipedia document produces much harder sentences.Source: §5 · p. 7, Figure 1 · p. 8
• Under supervised MT, original Nepali sentences score 4.9 BLEU, versus 9.1 BLEU for Nepali translationese, suggesting the parallel corpus is closer to English Wikipedia.The authors report that the same origin-related pattern appears in the semi-supervised analysis and that findings on other language directions are similar.Source: §5 · p. 7, Figure 1 · p. 8

5.1 Domain drift

• 10–16 BLEU points higher in-domain translation quality suggests domain mismatch, and semi-supervised Wikipedia back-translation yields larger BLEU gains out-of-domain than in-domain.Open Subtitles test sentences average 6 words, compared with 16 words in FLORES devtest, so sentence-length sensitivity may also contribute to the in-domain advantage.Source: §5.1 · p. 8, Table 5 · p. 8

6 Conclusions

• FLORES benchmarks for Nepali–English and Sinhala–English are freely released for research on low-resource MT, where limited parallel data and a shortage of public evaluation benchmarks pose challenges.The language pairs have syntax and morphology very different from English, alongside available monolingual data, related-language parallel data, and Paracrawl data.Source: §6 · p. 8
• Current state-of-the-art approaches perform rather poorly on FLORES, while semi-supervised and particularly multilingual neural methods outperform the other evaluated variants and training settings.Dataset analysis found no evidence of poor construction quality, but domain mismatch between training and test data partly explains the low BLEU scores.Source: §6 · p. 8

A List of Topics

• Table 6 reports topic distributions for sentences in the dev, devtest, and test sets according to their source Wikipedia documents.Source: Table 6 · p. 11

B Statistics of automatic filtering and manual filtering

• Translations averaging below 70 on a 0–100 quality rating from three raters are removed.The rating guidelines range from completely incorrect translations to perfect translations.Source: Figure 2 · p. 12
• English translations averaging below 3 on a 1–5 fluency rating from five raters are removed.Scores of 1–2 indicate unfluent text, 3 is neutral, and 4–5 indicate fluent text.Source: Figure 3 · p. 12
• Poor translations are first automatically filtered and returned for rework; those that pass receive human adequacy and fluency checks.The reported percentage failing manual checks is conditional on having passed automatic filtering.Source: Table 7 · p. 12

C List of Wikpedia Documents

• Table 8 lists Wikipedia documents by domain, with document names or English translations and corresponding topics; + marks pages removed or unavailable at submission.Document names hyperlink to the original pages.Source: Table 8 · p. 13

• Table 9 shows devtest examples for En-Ne, Ne-En, En-Si, and Si-En, with system hypotheses from the semi-supervised model decoded using beam search.Source: Table 9 · p. 14