Source-linked AI summary
How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs
Rico Sennrich
TL;DR
The paper addresses the difficulty of evaluating whether machine translation systems learn specific linguistic phenomena. It introduces contrastive translation-pair evaluation and LingEval97, finding that character-level decoders outperform subword models on unknown names but perform worse on morphosyntactic agreement.
Problem
Evaluating how well machine translation systems learn specific linguistic phenomena has been difficult because BLEU and coarse automatic error analysis provide limited linguistic insight.
Method
The authors compare the model scores of human reference translations with minimally modified contrastive translations containing specific errors.
Results
Character-level decoders outperform subword models at processing unknown names but perform worse at modelling morphosyntactic agreement over long distances.
Takeaways & Limitations
Contrastive translation pairs reveal differences in learned linguistic structures that similar BLEU scores do not distinguish.
Takeaways & Limitations
The evaluation uses automatically generated errors and reports human ceilings below 100% because some references contain errors or cases are undecidable.
Abstract
from arXiv · showhide
Analysing translation quality in regards to specific linguistic phenomena has historically been difficult and time-consuming. Neural machine translation has the attractive property that it can produce scores for arbitrary translations, and we propose a novel method to assess how well NMT systems model specific linguistic phenomena such as agreement over long distances, the production of novel words, and the faithful translation of polarity. The core idea is that we measure whether a reference translation is more probable under a NMT model than a contrastive translation which introduces a specific type of error. We present LingEval97, a large-scale data set of 97000 contrastive translation pairs based on the WMT English->German translation task, with errors automatically created with simple rules. We report results for a number of systems, and find that recently introduced character-level NMT systems perform better at transliteration than models with byte-pair encoding (BPE) segmentation, but perform more poorly at morphosyntactic agreement, and translating discontiguous units of meaning.
1 Introduction
The paper addresses the difficulty of evaluating specific linguistic phenomena in machine translation, especially whether character-level NMT produces coherent and grammatical sentences. It introduces contrastive evaluation to reveal strengths and weaknesses that BLEU cannot distinguish.
- Motivation: BLEU and conventional automatic error analysis provide limited linguistic insight into specific translation phenomena.BLEU relies on short n-gram precision and is unsuitable for measuring global coherence or grammaticality.
- Motivation: Character-level NMT is evaluated on whether it produces coherent and grammatical sentences, a question not answered by prior BLEU-based comparisons.Similar BLEU scores do not establish comparable grammaticality or global coherence.
- Approach: NMT models can score arbitrary sentence pairs, enabling comparison between a human reference and an otherwise identical translation containing a specific error.The model is tested on whether it assigns higher probability to the reference than to the contrastive example.
- Scope: The proposed analysis targets agreement over long distances, discontiguous verb-particle constructions, unseen-word transliteration, and polarity preservation.These phenomena have previously been challenging for machine translation.
- Contribution: The paper reports results for NMT systems using different subword units to identify their respective strengths and weaknesses.The evaluation is designed to provide a more nuanced comparison than aggregate translation quality scores.
2 Contrastive Translation Pairs
LingEval97 constructs English→German contrastive pairs by minimally modifying correct references to introduce targeted translation errors. Its automatically generated categories cover agreement, separable particles, polarity, and transliteration, with safeguards and human ceilings addressing construction uncertainty.
- Dataset construction: LingEval97 contains approximately 97 000 contrastive translation pairs created from WMT English→German test sets.Each pair contains a correct reference and a minimally modified translation with one introduced error.
- Evaluation: Model accuracy is the proportion of pairs for which the reference receives a higher score than the contrastive translation.This directly tests whether the model prefers the error-free variant.
- Error categories: The test set introduces noun-phrase and subject-verb agreement errors by changing determiner gender or verb number.These categories target German morphosyntactic agreement.
- Error categories: It tests discontiguous semantic units by replacing a separable verb particle unseen with that verb in training data.The altered particle is intended to disrupt a verb-particle construction.
- Error categories: Polarity errors are created by inserting or deleting nicht, swapping ein with kein, or altering the negation prefix un-.These changes can reverse the meaning of a translation while modifying only a small string segment.
- Error categories: For unseen names, adjacent characters are swapped as a proxy for testing copying or transliteration ability.Because English and German use the same script, the paper treats copying and transliteration similarly for these models.
- Construction safeguards: Automatic processing uses corpus statistics, syntactic analysis, and finite-state morphology, with high-precision rules restricting some transformations.For example, determiner gender is changed only when case and number are unambiguous.
- Construction safeguards: Human ceiling scores account for generated examples that may not constitute genuine errors, as well as flawed or undecidable references.The ceiling is estimated from 20 contrastive pairs per category where the best system fails.
3 Evaluation
The evaluation compares NMT systems using contrastive translation pairs and linguistic error categories, alongside BLEU baselines. Character-level systems show different strengths from BPE systems, particularly for transliteration, agreement, and polarity.
- Evaluation questions and systems: The evaluation asks whether longer character-level sequences harm grammaticality and how subword models process unseen words, especially names.The study trains BPE-to-BPE, BPE-to-char, and char-to-char systems on WMT15 EN→DE data.
- BLEU baseline: BLEU scores are relatively close across the trained systems and provide no insight into their different strengths and weaknesses.The study reports case-sensitive BLEU using both detokenized and tokenized evaluation scripts.
- Contrastive-pair results: Character-decoder models make fewer transliteration errors than BPE-to-BPE, but perform worse on separable verb particles and agreement, especially over longer distances.For distances over 15 words, subject-verb agreement accuracy is 90.7%, 85.2%, and 82.3% for BPE-to-BPE, BPE-to-char, and char-to-char, respectively.
- Contrastive-pair results: Polarity errors primarily involve deleting negation markers, and the analysis identifies polarity as a substantial translation problem.The polarity results are reported in Table 5.
- Contrastive-pair results: Changing verb number can alter meaning without causing disfluency, making such cases challenging for all models.Accuracy is 87–87.2% for single models and 90% for the WMT16 submission system on this error type.
4 Conclusion
LingEval97 reveals a trade-off: character-level decoders handle unknown names better, while subword models better capture morphosyntactic agreement. The evaluation also identifies negation deletion as especially challenging and supports testing alternative architectures.
- LingEval97: 97 000 contrastive translation pairs comprise LingEval97, which assesses neural MT on challenging linguistic phenomena.The pairs introduce specific errors into contrastive translations to evaluate model preferences.
- Main findings: Character-level decoders outperform subword models on unknown-name processing but perform worse on morphosyntactic agreement over long distances.The comparison identifies complementary strengths and weaknesses across subword choices.
- Main findings: Deletion of negation markers is the most challenging tested error type.
- Future use: Contrastive-pair evaluation can assess alternative architectures and extend to new language pairs and error types.