Source-linked AI summary
MTNT: A Testbed for Machine Translation of Noisy Text
Paul Michel, Graham Neubig
TL;DR
MT systems struggle with naturally occurring noisy text, while publicly available parallel benchmarks for evaluating this problem are lacking. The paper introduces MTNT, a multilingual Reddit-based dataset with professional translations, and shows that standard models remain challenged even after simple domain adaptation. MTNT is intended as a benchmark for developing noise-robust translation methods and evaluations.
Problem
Publicly available parallel corpora with naturally occurring noisy inputs and translations are lacking for evaluating noise-robust MT.
Method
The paper constructs MTNT from noisy Reddit comments and professional translations across English, French, and Japanese, with parallel and monolingual data.
Results
MTNT contains more noise than existing MT test sets, and simple fine-tuning improves BLEU by 3.17 to 7.96 points but does not overcome failures on very noisy sentences.
Takeaways & Limitations
MTNT provides a standard benchmark for studying models, datasets, and evaluation metrics tailored to noisy-text translation.
Takeaways & Limitations
Noisy-text translation includes lexical variation and grammatical inconsistency that simple domain adaptation alone does not resolve.
Abstract
from arXiv · showhide
Noisy or non-standard input text can cause disastrous mistranslations in most modern Machine Translation (MT) systems, and there has been growing research interest in creating noise-robust MT systems. However, as of yet there are no publicly available parallel corpora of with naturally occurring noisy inputs and translations, and thus previous work has resorted to evaluating on synthetically created datasets. In this paper, we propose a benchmark dataset for Machine Translation of Noisy Text (MTNT), consisting of noisy comments on Reddit (www.reddit.com) and professionally sourced translations. We commissioned translations of English comments into French and Japanese, as well as French and Japanese comments into English, on the order of 7k-37k sentences per language pair. We qualitatively and quantitatively examine the types of noise included in this dataset, then demonstrate that existing MT models fail badly on a number of noise-related phenomena, even after performing adaptation on a small training set of in-domain data. This indicates that this dataset can provide an attractive testbed for methods tailored to handling noisy text in MT. The data is publicly available at www.cs.cmu.edu/~pmichel1/mtnt/.
1 Introduction
The paper introduces MTNT, a benchmark built from naturally noisy Reddit comments and professional translations, because existing MT systems remain vulnerable to Internet-style noise and no standard open benchmark exists. It covers English, French, and Japanese in four translation directions and provides training, validation, test, and monolingual data.
- Contribution: The benchmark combines social-media noise such as abbreviations, typographical errors, obfuscated profanity, inconsistent capitalization, slang, and emojis.These forms are illustrated through a deliberately noisy example sentence.
- Motivation: Existing MT systems remain vulnerable to noisy input, producing erroneous translations that can be misinterpreted or offensive.The example translation mishandles abbreviations, profanity, slang, and other non-standard forms.
- Motivation: Previous work on character-level noise does not cover the broader idiosyncrasies of Internet language.The paper identifies social-media phenomena beyond character scrambling or simulated spelling errors.
- Contribution: MTNT addresses the lack of a standard open benchmark for testing MT robustness to naturally occurring noisy text.The dataset is designed specifically for noise and other deviations from standard MT training data.
- Contribution: MTNT contains professionally translated Reddit comments for English-French, French-English, English-Japanese, and Japanese-English.The release includes approximately 1,000 test samples and 6k–36k training samples per language pair, plus monolingual data.
2 Noise and Input Variations in Language on the Internet
The paper treats Internet language variation as a broad set of noise phenomena spanning vocabulary, spelling, grammar, style, scripts, and community-specific usage. It argues that noisy-text translation is not merely ordinary domain adaptation because noise introduces lexical variation and grammatical inconsistency.
- Scope: The meaning and manifestation of noise vary across languages and content types.The paper notes that even the notion of a typo differs for logographic writing systems.
- Noise phenomena: Spoken language and Internet slang create nonstandard forms such as “wanna,” “tbh,” “mdr,” and Japanese equivalents.These forms reflect reductions, abbreviations, and community-specific expressions.
- Noise phenomena: Other variations include incorrectly capitalized proper nouns, regional dialects, and code switching across languages.The examples span Reddit capitalization, multiple English, French, and Japanese dialects, and mixed-language expressions.
- Noise phenomena: Internet text can contain jargon, emojis, masked profanities, spelling errors, omissions, insertions, repetitions, and grammatical errors.The examples include Reddit-specific terms, Unicode characters, altered spellings, and nonstandard constructions.
- Relation to adaptation: Noisy-text translation differs from ordinary domain adaptation through lexical variation and grammatical inconsistency, so it should be treated as a separate problem.The paper reports that adaptation experiments still leave many noise-related errors.
3 Collection Procedure
The collection procedure gathers noisy Reddit comments in English, French, and Japanese, filters them against clean contrast corpora, and obtains professional translations. It produces parallel train, validation, and test data alongside monolingual corpora.
- Data sources: Reddit was selected because its content is likely to contain noise and includes language-specific communities for English, French, and Japanese.The collection uses the public Reddit API and selected subreddits for French and Japanese.
- Identifying noisy comments: Potentially noisy comments were identified by contrasting them with clean MT corpora, using out-of-vocabulary words for English and French and low-probability scores from subword language models.The OOV filter was skipped for Japanese because the reference corpus was not word-segmented.
- Identifying noisy comments: Pre-filtering removes URLs, automated comments, and comments classified as belonging to another language, while retaining natural code-switching, proper nouns, and typos.The language-identification threshold is p(lang | comment) > 0.5 for another language.
- Normalization: Comments are normalized only for noise detection, with Markdown stripped and language-specific tokenization or lowercasing applied; released comments remain unpreprocessed.The same normalization procedure is applied to the contrast corpora.
- Monolingual corpora: Unused comments are released as monolingual corpora for semi-supervised in-domain training and noisy-text language modeling.These corpora are drawn from comments left after constructing the parallel train and test sets.
4 Dataset Analysis
MTNT contains more noise than established MT benchmarks on nearly every measured dimension, with spelling and grammar errors especially prominent in English and French. Compared with synthetic noise, it also covers more varied, naturally occurring phenomena.
- Noise counts are measured per 100 words using spell checkers, custom profanity and emoji lists, and regular expressions.The analysis is restricted to MTNT test sets because some tests are impractical at larger scale.
- Spelling errors dominate noise in English and French, followed by grammar errors, while spelling errors are much less prevalent in Japanese.
- MTNT contains more noise than established benchmarks on all metrics except profanities, which are more frequent in JESC.The comparison supports MTNT as a benchmark for noisy or non-standard translation.
- MTNT includes less systematic spelling noise and more varied phenomena, including emojis and profanities, than synthetic noisy test sets.This makes its noise more representative of naturally occurring text.
5 Machine Translation Experiments
Standard NMT models are evaluated on conventional and MTNT test sets using standardized BLEU and targeted n-gram analyses. Fine-tuning improves BLEU, but translations of very noisy sentences remain inadequate.
- Model Description: The baseline systems use two-layer bidirectional LSTM encoders and decoders, multilayer perceptron attention, tied embeddings, dropout, and BPE subwords.Models are implemented in DyNet with XNMT and trained using Adam.
- Results: The experiments evaluate models trained on standard English-French and English-Japanese corpora against MTNT and other test sets using sacreBLEU.Japanese hypotheses and references are processed with KyTea before BLEU computation.
- Analysis: English outputs under-generate contracted negatives and auxiliaries, while French outputs over-generate formal forms and under-generate informal contractions.
- Analysis: Japanese outputs under-generate informal forms, and ja-en can become incoherent when the system encounters unseen sentences.
- Fine-Tuning: Fine-tuning increases BLEU by 3.17 to 7.96 points depending on the language pair, yet does not solve noisy-text translation.Examples remain inadequate after adaptation, including one incoherent output.
6 Language Modeling Experiments
The paper evaluates character-level language models on MTNT’s monolingual data to establish baselines for modeling noisy English, French, and Japanese text.
- Character-level language models are trained on MTNT monolingual data and evaluated on the source sides of English-French, French-English, and Japanese-English test sets.
- The comparison includes a Kneser-Ney smoothed 6-gram model and an AWD-LSTM, with results reported in bits per character.
- These language-modeling results are intended as baselines for future noisy-text modeling work in the three languages.
7 Related work
Prior work addresses noisy user-generated text through datasets, annotations, adaptation, and synthetic or manually designed evaluation. MTNT extends this work with an open test set of naturally occurring noisy text spanning diverse phenomena.
- User-generated social-media text poses challenges because it contains linguistic phenomena largely absent from news corpora.
- Prior efforts include in-domain datasets, annotations, domain adaptation, and recent studies of neural MT robustness.
- Existing MT evaluation resources include manually annotated sentences and automatically generated test sets targeting specific linguistic divergences or errors.
- MTNT distinguishes itself by providing an open test set of naturally occurring text exhibiting a wide range of phenomena.
8 Conclusion
MTNT provides bidirectional parallel and monolingual data for evaluating machine-translation robustness to naturally occurring Internet noise. The authors find that standard models remain challenged despite simple domain adaptation, motivating specialized models, datasets, and evaluation metrics.
- MTNT tests robustness to noise encountered in natural language on the Internet.
- The dataset contributes parallel training and test data in both directions for English ↔French and English ↔Japanese, plus monolingual data in all three languages.
- MTNT contains more noise than existing machine-translation test sets and challenges models trained on standard corpora.
- Simple domain adaptation alone does not overcome the dataset’s challenges.
- The benchmark is intended to foster models, datasets, and evaluation metrics tailored to noisy-text translation.