Source-linked AI summary
On the Impact of Various Types of Noise on Neural Machine Translation
Huda Khayrallah, Philipp Koehn
TL;DR
The paper asks how different kinds of noise in parallel training data affect neural machine translation. It creates five synthetic noise types and compares their effects on neural and statistical systems, finding that neural models are generally more harmed and can learn excessive copying from untranslated target data.
Problem
The paper investigates which kinds of noisy parallel training data harm neural machine translation, challenging the assumption that adding more data is beneficial.
Method
The authors create five types of synthetic noise, add them to an existing parallel corpus at controlled proportions, and assess neural and statistical translation systems.
Results
Neural systems are harmed more than statistical systems by almost all tested noise types, and copied source segments can induce excessive copying behavior.
Takeaways & Limitations
The findings should inform future work on cleaning parallel corpora, especially because untranslated copies can have a catastrophic effect on neural translation quality.
Takeaways & Limitations
The real-world noise analysis is based on manual examination whose results depend on how crawling and extraction are performed, and disfluency judgments are subjective.
Abstract
from arXiv · showhide
We examine how various types of noise in the parallel training data impact the quality of neural machine translation systems. We create five types of artificial noise and analyze how they degrade performance in neural and statistical machine translation. We find that neural models are generally more harmed by noise than statistical models. For one especially egregious type of noise they learn to just copy the input sentence.
1 Introduction
The paper examines how noisy parallel training data challenges neural machine translation, despite the usual benefits of adding data. It compares neural and statistical systems across noise types, finding neural systems generally more vulnerable and sometimes prone to copying the input.
- Adding equally sized noisy web-crawled data increased SMT by 1.2 BLEU but degraded NMT by 9.9 BLEU.
- The study asks which kinds of noise harm neural machine translation models.
- The authors explore several noise types by adding synthetic noise to an existing parallel corpus.
- Neural systems are harmed more than statistical systems by almost all tested noise types.
- Copied source-language segments can cause neural systems to learn and excessively apply copying behavior.
2 Related Work
Prior work largely addresses noise filtering, detection, or task-specific data selection, often in statistical machine translation. This paper differs by studying harmful training-data noise across domains and its effects on neural and statistical systems.
- Earlier studies filter noisy parallel corpora using outlier detection, classifiers, or trust-based phrase-translation weighting.
- More recent work improves NMT by removing sentence pairs with semantic differences identified through textual entailment and length features.
- Other research detects machine-translated web content using watermarks, word choices, or limited reordering.
- The paper focuses on noisy training data from web-crawled corpora, unlike work targeting human-comprehensible orthographic errors.
- Its focus differs from task-specific data selection because the studied noise is harmful across domains, whereas selection discards data irrelevant to a target domain.
3 Real-World Noise
The authors manually inspect 200 ParaCrawl sentence pairs to characterize real-world noise, while noting that observed frequencies depend on crawling and extraction procedures. Misalignment is the largest category, alongside wrong-language, untranslated, malformed, and short segments.
- The manual study examined 200 sentence pairs, but its results depend strongly on crawling and extraction procedures.
- Misaligned German–English sentence pairs are the largest observed error category, accounting for 41%.
- Wrong-language content totals 23%, including other languages, German–German pairs, and English–English pairs.
- Untranslated pairs account for 4%, while random bytes, markup, Javascript, and very short segments form additional categories.
- The authors exclude disfluent language from the error count because judging disfluency is subjective.
- Near-identical content is difficult to assess because completely identical sentences may still be useful, while intermediate cases have unclear utility.
4 Types of Noise
The paper defines five naturally occurring noise types and simulates them in parallel data to measure their effect without first solving noise detection. The types cover alignment, word order, language, translation status, and segment length.
- The authors simulate five noise types in artificial data to study their impact while avoiding the problem of detecting each type.
- Misaligned sentences: Misaligned sentences are created by disrupting sentence alignment, reflecting faulty document or sentence matching.
- Misordered words: Misordered-word noise randomly reorders words from original source or target sentences.
- Wrong language: Wrong-language noise adds French–English or German–French data to a German–English corpus on the bad source or target side.
- Untranslated sentences: Untranslated-sentence noise copies source or target sentences into the opposite side of original parallel data.
- Short segments: Short-segment noise uses parallel sentences with maximum lengths of 2 or 5 tokens, modeling dictionary-like additions.
5 Experimental Setup
The experiments compare RNN-based neural and phrase-based statistical machine translation systems on German-to-English data while adding several types of noisy parallel data at controlled proportions. They use standard WMT datasets, evaluate on newstest2017, and include naturally occurring ParaCrawl noise.
- Data: Experiments translate German to English using about 83 million clean training tokens per language from WMT datasets.Europarl, News Commentary, and the Rapid EU Press Release corpus provide the clean training data.
- Evaluation: The evaluation uses newstest2015 for SMT tuning, newstest2016 for NMT development, and newstest2017 for reporting results.
- Noise construction: Artificial noise is created by perturbing the clean corpus, pairing identical sentences, substituting wrong-language data, or extracting short bilingual segments.Misaligned sentence and misordered word noise perturb the clean corpus; untranslated-sentence noise uses identical sentence pairs, while wrong-language noise uses EU Bookstore data and short segments come from OPUS corpora.
- Noise proportions: Noisy corpora are added at 5%, 10%, 20%, 50%, and 100% of the clean-corpus size, with noisy target text included in the SMT language model.The setup reflects adding potentially noisy data to an existing clean corpus; naturally occurring noise is also evaluated using raw deduplicated 2016 ParaCrawl data.
6 Impact on Translation Quality
Noise generally harms neural machine translation more than statistical machine translation, with untranslated target sentences causing the most severe degradation and inducing copying behavior. Other noise types have smaller or uneven effects, while wrong-language data can remain comparatively benign under domain mismatch.
- Noise types: -1.9 BLEU for NMT versus -0.6 for SMT under mismatched sentences; source-side word disorder and wrong-language targets likewise harm NMT more.The reported degradations are -1.7 versus -0.3 for misordered source words and -2.2 versus -0.6 for wrong-language target data.
- Untranslated target sentences: A 5% addition of untranslated target sentences lowers NMT from 27.2 to 17.6 BLEU, while 100% lowers it to 3.2.The corresponding SMT degradation at 100% is only 2.9 BLEU, from 24.0 to 21.1.
- Copied output: Learning curves initially improve on noisy data before overfitting to the copied portion of the training set.The development-set BLEU curve uses beam search, whereas training optimizes cross-entropy with a perfect prefix.
- Wrong-language noise: Wrong-language data is surprisingly robust in both systems when German–French data is added to German–English data in equal amounts.Only 6.7% of NMT outputs are mostly French at the 100% noise level, attributed to domain differences between noisy and clean training data.
7 Conclusion
The paper finds that neural machine translation is less robust to most tested parallel-data noise than statistical machine translation, with copied source segments causing excessive copying.
- Neural machine translation is less robust to many types of parallel-data noise than statistical machine translation.
- Five noise types were defined in parallel data, motivated by a study of raw web crawl data.
- Untranslated copies of the source can cause neural machine translation to learn excessive input-copying behavior.