TICO-19: the Translation Initiative for Covid-19

1 Introduction

• Health authorities’ communication is essential for protecting at-risk populations, yet preventive guidance and technical information about COVID-19 have been difficult to relay.Examples include social distancing, protective equipment, symptoms, testing, and treatments.Source: §1 · p. 1
• Official health communications were mostly limited to major languages, leaving a language gap amid misinformation and needs exceeding NGOs’ translation capacity.TWB had translated over 3.5 million words with more than 80 non-profits for over 100 language pairs in its COVID-19 response.Source: §1 · p. 1
• Because medical information is sensitive, automatically produced translations need to be of the highest possible quality.The initiative hopes its resources will support translation during crises and research on specialized-content MT for low-resource languages.Source: §1 · pp. 1–2
• TICO-19 provides translation memories and technical glossaries, a multilingual medical-domain MT benchmark including very-low-resource languages, and monolingual and bilingual resources for MT and NLP research.The resources are intended to help localizers work consistently and accurately and to track MT quality.Source: §1 · p. 2

2 The Value of Translation Technologies in Crisis Scenarios

• Crisis-response translation has included Haitian Kreyòl SMS triage, translation memories in under-resourced languages, and MT deployments such as Urdu after the Pakistan earthquake.Mission 4636 received as many as 5,000 assistance texts per hour after the Haitian earthquake; translators translated messages and geolocated them for aid-agency triage.Source: §2 · p. 2
• Public MT use in crisis and relief settings is not tracked, so its extent is unknown despite evidence that at least 5 percent of Haitian Kreyòl Microsoft Translator traffic after the earthquake was relief-related.Publicly available apps may work offline when phone infrastructure is unavailable or destroyed.Source: §2 · p. 3
• Crisis-affected communities may access local-language content through SMS, chatbots, or print, while their questions and feedback can help refine content.Relief agencies also need access to local-language SMS and other communications to distribute aid effectively and equitably.Source: §2 · p. 3
• In crisis response, MT can help disseminate essential communications quickly without waiting for human translators, particularly where professional translators are scarce.Domain-specific MT can assist translators with terminology and standardize concepts.Source: §2 · p. 3
• Because medical texts are specialized, useful MT requires high accuracy, benchmark sets, and basic monolingual and parallel training resources.Translation memories in more languages can also help localizers.Source: §2 · p. 3

3 The TICO-19 Translation Benchmark

• The benchmark was designed around diversity, relevance to relief organizations’ language requests, and stringent translation quality assurance.Its English source data combine material from varied public sources and domains.Source: §3 · p. 3, §3.1 · p. 3

3.1 COVID-19 source data

• The source data cover scientific PubMed articles, medical phrases from the CMU English–Haitian Creole dataset, Wikipedia articles, and news, travel, and government or organization material.The CMU conversational phrases were filtered for COVID-19-related keywords.Source: §3.1 · pp. 3–4

3.2 Languages

• The benchmark translates English data into 35 languages, including nine pivot languages and languages prioritized for humanitarian relevance, many of them under-resourced.Sentence-aligned test and development documents support any language pairing among the represented languages.Source: §3.2 · pp. 4–5

3.3 Quality Assurance

• Translations undergo professional editing and arbitration, with a selected fraction independently checked; all language sets were reworked until rated above 95%.The second QA sample comprised 558 sentences, including 329 PubMed sentences; the process was skipped for a handful of languages, such as Dinka and Nuer, where creating translations was itself a challenge.Source: §3.3 · pp. 5–6
• The release includes sampled data with error annotations and corrections, supporting quality-estimation and post-editing research for under-resourced languages.The annotations classify error type and severity; the sample contains at most 558 sentences in each translation direction.Source: §3.3 · p. 6

4 Translator Resources

• TICO-19 converted its translations into TMX translation memories for English–X pairings and selected other pairings relevant to local needs.Examples include French–Congolese Swahili, Farsi–Dari, and Kurdish Kurmanji–Sorani; the files and test and development data are publicly available.Source: §4 · p. 6
• Facebook supplied 364 COVID-19-related terms translated in 92 languages or locales, while Google supplied English-to-100-language terms and about 30,000 terms overall.Google’s set includes 1,300 terms from 27 languages translated into English.Source: §4 · p. 6
• Translators without Borders contributed translations in hard-to-source languages and COVID-19 sources from humanitarian partners, with the augmented dataset slated for its Gamayun portal.Source: §4 · p. 6

5 MT Developer Resources

• The CMU team collected COVID-19-related monolingual data in multiple languages.Some of these data may have different licenses and may not be appropriate for commercial system development.Source: §5 · p. 6

5.1 Monolingual

• COVID-19-related Wikipedia data span 37 languages, from under 1K sentences for some languages to 8K sentences for Spanish.The passage also reports 120K tokens for Hungarian and 160K tokens for Spanish.Source: §5.1 · p. 6
• COVID-19 news articles were collected from BBC World Service, Voice of America, and Deutsche Welle in 22, 31, and 29 languages, respectively.Source: §5.1 · p. 7

5.2 Parallel

• Parallel data scraped from public-service and health sources amount to only a few hundred sentences per direction, insufficient for system development.The collections may still serve as a smaller evaluation set or for terminology extraction.Source: §5.2 · p. 7

Baseline Results

• The baselines use OPUS-MT and other pretrained systems, OPUS-trained models for several directions, and TED-trained multilingual models for English with Hindi, Malay, and Urdu.The reported tables cover English directions, selected French directions, and test-domain breakdowns.Source: §6 · p. 7

Discussion

• No pretrained MT systems or public parallel data were available to build baselines for Dari, Pashto, Tigrinya, Nigerian Fulfulde, Kurdish Sorani, Myanmar, Oromo, Dinka, Nuer, and isiZulu.The authors identify this gap as a need for substantial data collection for under-represented languages.Source: §6 · p. 7
• BLEU scores are 25–49 for Spanish, French, Portuguese, Russian, Chinese, and Indonesian, while Lingala, Luganda, Marathi, and Urdu outputs are described as markedly worse.The passage does not provide the values for the latter languages.Source: §6 · p. 7
• Existing systems can score below 10 BLEU, a quality level the authors describe as unusable for real-world deployment by translators or end-users.Source: §6 · p. 8
• BLEU is generally higher on Wikipedia and news than on PubMed, with an English-to-French difference exceeding 14 points and smaller differences of 6 for Indonesian-English and 4 for Russian-English.The authors caution that these domain test sets differ.Source: §6 · p. 8
• The authors suggest domain adaptation, terminology integration, multilingual NMT, model transfer, and better low-resource-language corpus coverage as future improvement directions.They note that reported systems use general-domain or out-of-domain training data.Source: §6 · p. 8

7 Conclusion

• Efficient and accurate translation for most of the world’s languages, particularly vulnerable ones, remains an unmet need.Source: §7 · p. 8
• The initiative addresses only a fraction of the needs and languages worldwide.Source: §7 · p. 8
• The authors hope the released MT resources have immediate impact and the benchmark prepares academic and industrial researchers for future crises requiring translation technology.Source: §7 · p. 8

A Source Documents

• The listed documents are the original English-language documents that make up the benchmark.Source: A Source Documents · p. 13
• The passage establishes the table’s purpose but gives no details about individual source documents.Source: A Source Documents · p. 13, Table 7 · p. 13
• Table 7 is identified as the list of source documents for the translation benchmark.Source: Table 7 · p. 13

• The quality-assurance set contains 558 sentences from almost all corpus subdomains.Source: B Quality Assurance Documents · p. 14
• It includes 40 conversational-data sentences and one PubMed document.Source: B Quality Assurance Documents · p. 14
• It also includes one complete Wikipedia article and two Wikinews documents.Source: B Quality Assurance Documents · p. 14

• Average quality scores exceeded 95% across all translation directions.Source: C Expected Quality per Language · p. 14
• Evaluations of sampled documents are presented as a proxy for the overall translation quality of the dataset.Source: C Expected Quality per Language · p. 14
• Table 8 reports initial QA scores and whether rework was required, with final scores after corrections when available.An N/A final score means that the final QA score was temporarily unavailable.Source: Table 8 · p. 14

• Facebook terminologies provide English-to-129-language/locale translations for terms including COVID-19, disease, symptoms, and prevention.Source: D Terminology Terms · pp. 14–15
• Google terminologies cover English-to-100 languages/locales with varying term counts, plus 27 additional directions not detailed in the paper.Source: D Terminology Terms · p. 15
• The listed Google subset is intended to be common across all English-to-X terminologies.Source: D Terminology Terms · p. 15
• That Google subset includes terms about quarantine, symptoms, transmission, prevention, and vaccines.Source: D Terminology Terms · pp. 16–17