Source-linked AI summary
The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MT
Jörg Tiedemann
TL;DR
Machine translation benchmarks have limited realistic coverage of low-resource languages and often simulate scarcity by artificially removing data. This paper releases a broad, systematically labeled collection and open tools for realistic multilingual evaluation, finding that related-language data can improve multilingual performance while realistic zero-shot results remain poor. The benchmark is intended to support continued expansion of test coverage and community experimentation.
Problem
Existing low-resource and zero-shot evaluations often rely on artificially reduced setups, while broader realistic coverage and suitable test data are needed for many languages.
Method
The paper compiles a comprehensive benchmark with realistic training, development, and test data, standardized language and script labels, open preparation and training tools, and multilingual evaluation settings.
Results
Related languages can substantially improve multilingual translation performance, while realistic zero-shot translation remains very poor despite transfer from overlapping related languages.
Takeaways & Limitations
The collection supports systematic studies of multilingual transfer, realistic low-resource translation, and continued benchmark extension through new language data and test translations.
Takeaways & Limitations
The release leaves careful analysis of observed model patterns and qualitative studies for future work.
Abstract
from arXiv · showhide
This paper describes the development of a new benchmark for machine translation that provides training and test data for thousands of language pairs covering over 500 languages and tools for creating state-of-the-art translation models from that collection. The main goal is to trigger the development of open translation tools and models with a much broader coverage of the World's languages. Using the package it is possible to work on realistic low-resource scenarios avoiding artificially reduced setups that are common when demonstrating zero-shot or few-shot learning. For the first time, this package provides a comprehensive collection of diverse data sets in hundreds of languages with systematic language and script annotation and data splits to extend the narrow coverage of existing benchmarks. Together with the data release, we also provide a growing number of pre-trained baseline models for individual language pairs and selected language groups.
1 Introduction
The challenge targets realistic machine translation for many languages by using naturally skewed data rather than artificially reduced or zero-shot setups. It also supports experiments across low-, medium-, and high-resource conditions and transfer methods.
- The challenge aims to encourage open machine translation development for many languages in real-world conditions.
- Artificially removing language pairs from high-resource or multi-parallel data does not realistically represent low-resource or zero-shot translation.
- All available collection data are retained, producing diverse and skewed conditions in which some low-resource languages have only noisy or unrelated training data.
- Zero-shot evaluation is used only when no data are available in any of the sub-corpora.
- The benchmark includes low-, medium-, and high-resource settings and supports transfer-learning and pivot-based experiments across language pairs.
2 Data releases
The release combines large-scale OPUS training data with Tatoeba development and test data, systematic language and script labels, cleaning tools, and multilingual test resources. It preserves alternative translations and adds monolingual data for unsupervised and augmentation methods.
- The release covers 2,961 language pairs and 555 languages with over 500GB of compressed data.
- The package cleans bitexts by removing encoding violations, de-escaping special characters, and filtering noise with automatic language identification.
- Training files retain corpus-origin labels, allowing users to restrict training to subsets for domain matching or noise reduction.
- 2.1 Language labels and scripts: Language labels use ISO-639-3 codes with automatically assigned ISO-15924 script extensions, and released tools support ID conversion and script detection.
- 2.2 Multiple reference translations: All alternative translations are retained, creating implicitly multi-reference test sets with incorrect normalization rather than discarding relevant data.
- Monolingual Wikimedia data are provided for unsupervised translation and augmentation methods such as back-translation.
3 The translation challenge
The translation challenge organizes data by available training size and emphasizes realistic evaluation coverage for low-resource languages. It separates validation and test data while documenting naturally occurring overlap with training data.
- The benchmark focuses on low-resource language coverage and translation quality while also providing higher-resource data for modeling and knowledge transfer.
- The collection defines high-resource settings as 298 language pairs with at least one million training examples.
- Medium-resource settings contain 97 language pairs with more than 100,000 and fewer than one million training examples.
- Low-resource settings contain 87 language pairs with fewer than 100,000 training examples, including 24 below 10,000.
- Zero-shot translation covers 40 language pairs with no training data in the current collection.
- 522 selected language pairs have at least 200 test sentences, although sets below 1,000 sentences may be unreliable as proper test sets.
- Test and validation sets are strictly disjoint, but their average overlap with released training data is about 5.5% for each.
4 The data challenge
The challenge aims to expand low-resource MT benchmarks through community contributions and regularly updated test and validation data, while preserving fair comparisons and identifying coverage gaps.
- New translations contributed through Tatoeba will expand benchmark coverage for low-resource language pairs.The dataset will be updated with new language pairs and larger test sets for existing pairs.
- Old benchmark revisions will remain available so results from earlier models remain replicable when new test sets are released.
- 2,375 language pairs have fewer than 1,000 examples, while 246 languages have English training data but no Tatoeba test data.
- Users are encouraged to contribute translations for underrepresented data sets and add test sets beyond Tatoeba, including WMT news sets where available.
5 How to participate
The challenge defines rules for comparable and unconstrained participation, encourages open model release, and provides models intended for reproducible reuse in research and applications.
- The data release provides a straightforward setup for MT development, with a leaderboard and models intended for reuse and replicability.Released models can support applications, downstream research, fine-tuning, and domain adaptation.
- Participation rules: Development and test data must not be used for training, although development data may support validation and early stopping.
- Participation rules: Official constrained models may train only on the provided training data, with language-pair combinations and training-data backtranslation allowed.
- Participation rules: Unconstrained models may use additional resources or pretrained models but must be reported separately and exclude Tatoeba test data.
- Participation rules: Official leaderboard entries require openly available models with usage instructions to support replicability and reuse.
6 Baseline Models
The release includes Transformer baseline models trained with Marian-NMT for hundreds of language pairs and evaluated on the Tatoeba benchmark.
- Baseline models use six-layer encoder and decoder Transformers with eight attention heads per layer, following Marian-NMT recommendations and OPUS-MT procedures.
- Models for over 400 language pairs are currently available, using SentencePiece, a shared vocabulary of up to 65,000 items, and language-label tokens when needed.
- The training strategy uses V100 GPUs and early stopping after ten iterations of declining validation perplexity.
- Table 1 reports Tatoeba translation scores for selected medium-size language pairs involving English, showing only models above 10 BLEU.
7 Multilingual Models
The multilingual experiments test whether related-language transfer improves Belorussian–English translation across increasingly broad language groups, while revealing performance and training limits at massive scale.
- Adding related languages produces a large performance increase, with broader Slavic coverage outperforming the East Slavic model.The Slavic model covers 13 source languages, compared with 5 for the East Slavic model.
- The Indo-European model performs well despite its much broader language coverage, whereas the all-language model does not achieve the same performance.
- Training costs become extremely high and slow at larger scales, and the authors did not start the four-million-sentence model.
8 Zero-shot and few-shot translation
The benchmark exposes realistic zero-shot and few-shot translation conditions, where multilingual models benefit from related-language overlap but still perform poorly in extremely low-resource settings.
- Zero-shot translation: Awadhi-English is evaluated as a zero-shot task because no Awadhi training data is available.Awadhi is an Eastern Hindi language in the Indo-Iranian branch of Indo-European.
- Zero-shot translation: The global tatoeba-zero model performs poorly, while Indo-European-to-English models perform relatively well through overlap with related languages.The observed performance does not come from Awadhi data during training.
- Few-shot translation: The collection provides 3,613 English-Faroese training examples for a realistic few-shot translation task.Results use multilingual models translating from English into language groups that include Scandinavian languages.
- Few-shot translation: The naive tatoeba-zero model is worst, whereas the English-Germanic model performs best, although performance remains very low for real-world applications.The comparison illustrates the difficulty of extremely low-resource translation.
9 Comparison to the WMT news task
The paper compares Tatoeba baseline models with WMT 2019 English-German news translation without task-specific optimization, finding high out-of-domain quality but weaker multilingual performance in high-resource settings.
- Evaluation setup: The comparison uses English-German WMT 2019 test data and baseline models from the Tatoeba release.BLEU scores are compared with the best score available in the referenced matrix, retrieved October 4, 2020.
- Results: The models achieve high quality on a domain for which they were not optimized.The experiments omit optimization, in-domain back-translation, and news-domain fine-tuning.
- Results: Multilingual models significantly lag behind bilingual models in high-resource cases.The comparison is intended to illustrate released baseline models rather than provide state-of-the-art news systems.
- Results: Except for the move from West Germanic to Germanic languages in the German-English case, each increase in language coverage reduces performance.These multilingual models were not fine-tuned for translation from and to German.
10 Conclusions
The paper releases a broad, annotated machine-translation benchmark and open tooling for realistic low-resource and zero-shot research across hundreds of languages.
- Conclusions: The benchmark covers roughly 3,000 language pairs and over 500 languages and language variants.It provides training and test data for realistic low-resource and zero-shot machine translation.
- Conclusions: The release includes standardized language and script annotations, source information, baseline models, and open-source preparation and training tools.The authors invite contributions to expand the data set, test sets, models, and language coverage.