Source-linked AI summary
A Multifaceted Evaluation of Neural versus Phrase-Based Machine Translation for 9 Language Directions
Antonio Toral, Víctor M. Sánchez-Cartagena
TL;DR
The paper addresses limited understanding of how neural machine translation differs from phrase-based translation beyond overall performance scores. It compares state-of-the-art systems across nine language directions and multiple dimensions, finding broader advantages for NMT alongside a weakness on very long sentences.
Problem
Overall system scores provide limited detail about how NMT and PBMT differ in fluency, reordering, sentence-length effects, and error categories.
Method
The study conducts a multifaceted comparison of state-of-the-art NMT and PBMT systems across nine language directions.
Results
NMT outputs are considerably different and more fluent, have more accurate word order, and perform better on inflection and reordering across the evaluated directions.
Takeaways & Limitations
NMT’s strengths generalize across multiple language directions, but its advantage reverses for very long sentences, where PBMT performs better.
Takeaways & Limitations
The conclusion that NMT introduces more word-order changes than PBMT is limited because only one direction uses a hierarchical PBMT system.
Abstract
from arXiv · showhide
We aim to shed light on the strengths and weaknesses of the newly introduced neural machine translation paradigm. To that end, we conduct a multifaceted evaluation in which we compare outputs produced by state-of-the-art neural machine translation and phrase-based machine translation systems for 9 language directions across a number of dimensions. Specifically, we measure the similarity of the outputs, their fluency and amount of reordering, the effect of sentence length and performance across different error categories. We find out that translations produced by neural machine translation systems are considerably different, more fluent and more accurate in terms of word order compared to those produced by phrase-based systems. Neural machine translation systems are also more accurate at producing inflected forms, but they perform poorly when translating very long sentences.
1 Introduction
The paper evaluates whether neural machine translation (NMT) differs from phrase-based machine translation (PBMT) across output similarity, fluency, reordering, sentence length, and error categories. It broadens prior analysis through a multilingual, multifaceted study of nine language directions and publicly available state-of-the-art systems.
- Background: NMT jointly trains translation components and represents translation units as numeric vectors, whereas PBMT trains separate models combined in a loglinear scheme.The paper also associates NMT with stronger generalization and recurrent modeling of long-distance phenomena.
- Research questions: The study asks whether NMT differs from PBMT in output variability, fluency, monotonicity, word order, sentence-length effects, and error categories.The error categories are inflectional, reordering, and lexical errors.
- Novelty: Compared with prior work on one language direction and transcribed speeches, this study covers nine directions and news stories.The prior study considered one direction, whereas this study covers nine; the content also differs from transcribed speeches to news stories.
- Novelty: The evaluation uses larger test sets of 1 999 to 3 000 sentences, but only single independent references rather than independent and post-edited references.These choices distinguish the study from the comparison work described in the introduction.
- Evaluation scope: Analyses include output similarity, fluency, reordering, word order, sentence length, and error-category performance.The paper explicitly identifies these as the dimensions investigated in subsequent experiments.
- Evaluation scope: The systems are selected from the best PBMT and NMT submissions to WMT16, supporting reproducibility, state-of-the-art coverage, and comparisons across six language families.The outputs are publicly available, and the experiments cover six languages from Germanic, Slavic, Romance, and Finno-Ugric families.
2 Experimental Setup
The experiments compare the best constrained PBMT and NMT systems submitted to WMT16 across nine English-centered language directions. The setup uses published systems and BLEU-based selection, with some language-specific coverage and evaluation constraints.
- Systems and languages: Experiments use the best constrained PBMT and NMT systems submitted to WMT16’s news translation task for nine language directions.The directions pair English with Czech, German, Finnish, Romanian, and Russian, with Finnish covered only from English.
- System selection: When systems are tied in human evaluation, the selection uses the highest BLEU score, and BLEU ties are resolved with the best TER score.This rule applies when systems belong to the same human-evaluation equivalence class.
- Evaluation data: Some experiments use fewer languages because the required tools are unavailable for certain languages.This is an explicit scope limitation on experimental coverage.
- Systems and languages: The selected systems are encoder-decoder NMT models with attention operating on subword units, while PBMT submissions may include neural features that are not full NMT systems.The paper describes lexical or unsupervised segmentation procedures for the NMT directions.
3 Output Similarity
The output-similarity analysis tests whether NMT systems produce outputs that differ from PBMT outputs by comparing pairwise chrF overlap among submitted systems. Across English-to-target directions, within-paradigm overlaps are higher than cross-paradigm overlaps, indicating substantially different NMT outputs and greater NMT variability.
- Method: The analysis compares pairwise chrF overlap among top NMT and PBMT outputs to assess how different the paradigms’ translations are.The analysis is restricted to directions out of English because the opposite directions had at most one NMT submission.
- Method: Table 3 reports average overlaps between pairs of NMT and PBMT outputs for each English-to-target language direction.Higher overlap values indicate greater output similarity.
- Results: Across all directions, PBMT-PBMT overlaps are highest, NMT-NMT overlaps are next, and PBMT-NMT overlaps are lowest.This ordering shows that outputs cluster more strongly within paradigms than across paradigms.
- Method: The overlap metric is chrF, which the paper uses because it correlates better with human judgments for morphologically rich languages such as Finnish.For languages with simpler morphology such as English, its correlation is described as comparable to BLEU.
- Results: NMT produces considerably different outputs from PBMT and exhibits higher inter-system variability than PBMT.Pairs of NMT systems overlap less than pairs of PBMT systems despite the NMT systems sharing an encoder-decoder-with-attention architecture.
4 Fluency
The fluency experiment evaluates NMT and PBMT outputs using perplexity from neural language models trained on News Crawl data. NMT outputs are generally more fluent, while Finnish is the sole exception and the proxy has an acknowledged model-alignment caveat.
- Method: Fluency is estimated from output perplexity under neural language models trained on News Crawl 2015 data.The models use a projection layer, an LSTM layer, and a tanh layer; training data comprise a random sample of 4 million sentences.
- Results: For every language direction except Finnish, PBMT outputs have higher perplexity than NMT outputs.Perplexity on PBMT output is slightly lower only for translation into Finnish, possibly because PBMT was reranked with a neural language model.
- Results: −10.45% is the average relative perplexity difference between PBMT and NMT outputs across language directions.The paper uses this result to characterize the overall fluency difference.
- Results: NMT outputs are generally more fluent than PBMT outputs, extending prior manual fluency findings for directions into English to directions out of English.The paper also notes that neural language models outperform n-gram models when assessing fluency of real text.
- Caveat: The fluency proxy may be questioned because the neural language models resemble the NMT decoder architecture.The authors address this caveat using prior manual evaluation and evidence favoring neural over n-gram language models.
5 Reordering
The evaluation measures system reordering with word alignments and finds that NMT generally changes word order more than PBMT while producing order closer to the reference.
- Method: The study estimates reordering by comparing word alignments with monotone and reference alignments using Kendall’s tau distance.Word alignments are obtained for the source, reference, PBMT output, and NMT output, with larger distances indicating more similar alignments in Table 5.
- Reordering amount: NMT generally introduces more sentence-structure changes than PBMT, except for EN→DE and EN→FI.The exceptions are attributed to hierarchical PBMT for EN→DE and neural-language-model reranking for EN→FI.
- Reordering amount: Both MT systems introduce less reordering than the reference translation.
- Reference similarity: Across 9 directions, NMT reordering is closer to the reference in 5, PBMT is closer in 1, and differences are insignificant in 3.
6 Sentence Length
NMT outperforms PBMT on shorter sentences but loses its advantage on longer ones, with relative improvement declining as sentence length increases across language directions.
- Performance by length: For EN→FI, NMT outperforms PBMT through 36–40-word sentences, while PBMT wins for longer sentences.PBMT remains fairly stable as length increases, whereas NMT performance clearly decreases.
- Method: Sentence subsets range from 1–5 words through 46–50 words, with a final category for sentences longer than 50 words.
- Performance by length: Across all 9 language directions, NMT’s relative chrF1 improvement over PBMT has a strong negative Pearson correlation of -0.79 with sentence length.
- Performance by length: Pearson correlations are negative for every language direction except DE→EN.
7 Error Categories
NMT reduces inflection and reordering errors relative to PBMT, while lexical-error differences are smaller and vary across language directions.
- Method: Inflection and reordering errors are identified from word-level comparisons using full and base forms, edit distance, WER, and position-independent error rates.
- Error definitions: The analysis groups missing-word, extra-word, and lexical-choice errors into one lexical-errors category because they are difficult to disambiguate.
- Results: NMT reduces inflection errors by 14.6% out of English and 7.91% into English on average.
- Results: NMT reduces reordering errors by 12.82% out of English and 11.94% into English on average.
- Results: Lexical-error differences are smaller and mixed: NMT reduces errors by 0.59% out of English but increases them by 2.17% for Romanian.
- Results: Into English, NMT reduces lexical errors by 1.35% on average and outperforms PBMT for every direction except RU→EN.
8 Conclusions
Across 9 language directions, the evaluation finds that NMT outputs are more fluent, reorder words more accurately, and handle inflection better than PBMT, while performance declines on very long sentences. Some analyses, especially error-category analyses, remain intentionally superficial.
- NMT outputs are considerably different from PBMT outputs and show greater variability across NMT systems.
- NMT outputs are more fluent for language directions both into and out of English.
- NMT reorderings are closer to reference reorderings than PBMT reorderings for most language pairs, despite differing from pure and hierarchical PBMT in reordering extent.
- For most examined languages, NMT’s improvement over PBMT decreases as sentence length increases, and PBMT outperforms NMT on very long sentences.The authors attribute this difference partly to shorter sentences in prior evaluation data and partly to subword-based NMT systems having longer effective input sequences.
- NMT performs better than PBMT on inflection and reordering across all language directions, whereas lexical-error differences are smaller and inconsistent.NMT outperforms PBMT on lexical errors in 7 directions, while PBMT is better in the remaining 2.
- Error-category analyses were conducted only superficially, focusing on conclusions that generalize across language directions.The paper publicly releases all data to support deeper language-specific studies.