Source-linked AI summary
Beyond Error Propagation in Neural Machine Translation: Characteristics of Language Also Matter
Lijun Wu, Xu Tan, Di He, Fei Tian, Tao Qin, Jianhuang Lai, Tie-Yan Liu
TL;DR
Neural machine translation often produces lower accuracy in later parts of left-to-right translations, a pattern commonly linked to error propagation. The paper analyzes this pattern using decoding-direction comparisons, teacher forcing, and language-branching statistics. It finds that error propagation contributes, but language branching better corresponds to which part is more accurate.
Problem
Prior work commonly attributed accuracy drop—the lower accuracy of right-part words in left-to-right translations—to error propagation, while other possible causes had received less study.
Source: §1 · p. 1Method
The paper compares decoding directions, uses teacher forcing during inference, and analyzes n-gram and dependency statistics in relation to language branching.
Source: §1 · p. 2, §5.1 · p. 5Results
Takeaways & Limitations
The findings suggest left-to-right NMT models fit better for right-branching languages such as English, and right-to-left models fit better for left-branching languages such as Japanese.
Source: §7 · p. 8The n-gram statistics exclude sentences shorter than four words and assume even-length sentences, removing the middle word otherwise.
Source: §5.1 · pp. 5–6
Abstract
from arXiv · showhide
Neural machine translation usually adopts autoregressive models and suffers from exposure bias as well as the consequent error propagation problem. Many previous works have discussed the relationship between error propagation and the \emph{accuracy drop} (i.e., the left part of the translated sentence is often better than its right part in left-to-right decoding models) problem. In this paper, we conduct a series of analyses to deeply understand this problem and get several interesting findings. (1) The role of error propagation on accuracy drop is overstated in the literature, although it indeed contributes to the accuracy drop problem. (2) Characteristics of a language play a more important role in causing the accuracy drop: the left part of the translation result in a right-branching language (e.g., English) is more likely to be more accurate than its right part, while the right part is more accurate for a left-branching language (e.g., Japanese). Our discoveries are confirmed on different model structures including Transformer and RNN, and in other sequence generation tasks such as text summarization.
1 Introduction
The paper investigates whether error propagation fully explains accuracy drop, the observed difference between left- and right-part translation accuracy. Its experiments find that propagation contributes but does not fully explain the gap, while language branching correlates with which part is more accurate.
- Accuracy drop refers to the lower accuracy of right-part words than left-part words in left-to-right NMT translations.The paper investigates its causes rather than proposing a direct solution.Source: §1 · p. 1
- In both left-to-right and right-to-left models, the right part is less accurate, so error propagation alone cannot explain accuracy drop.The experiments cover German-to-English, English-to-German, and English-to-Chinese.Source: §1 · p. 1
- With teacher forcing, later parts improve regardless of decoding direction, confirming error propagation contributes; the right part nevertheless remains less accurate.Teacher forcing feeds ground-truth preceding words during inference, eliminating exposure bias and error propagation.Source: §1 · p. 2
- Right-branching target languages tend to have higher accuracy in the left part, while left-branching languages tend to have higher accuracy in the right part, regardless of decoding direction.The authors report that n-gram frequency and conditional probabilities, as well as dependency counts, show the same directional pattern.Source: §1 · p. 2
- The authors suggest matching decoding direction to branching: left-to-right for right-branching languages and right-to-left for left-branching languages.English and Japanese are given as examples, respectively.Source: §1 · p. 2
2 Related Work
Related work distinguishes exposure bias from error propagation and describes methods aimed at these issues and accuracy drop. The paper identifies language characteristics as another cause that earlier accuracy-drop work had not deeply studied.
2.1 Exposure Bias and Error Propagation
- Exposure bias is the training–inference discrepancy between teacher-forced training and autoregressive generation; error propagation is the resulting accumulation of errors along a generated sequence.The paper notes that these distinct concepts are often discussed together.Source: §2.1 · p. 2
- Prior work used scheduled sampling, reinforcement learning, and minimum risk training to address exposure bias and limitations of maximum-likelihood training.Scheduled sampling was proposed to eliminate exposure bias and resulting error propagation.Source: §2.1 · p. 2
2.2 Tackling Accuracy Drop
- Earlier accuracy-drop approaches mainly attributed the problem to error propagation and used agreement regularization or two-pass decoding to improve translation quality.Agreement regularization combines left-to-right and right-to-left models; two-pass decoding refines generated sequences.Source: §2.2 · pp. 2–3
- The paper addresses a gap in prior work by studying causes beyond error propagation, including language characteristics.Its findings state that accuracy drop is not only caused by error propagation.Source: §2.2 · p. 3
3 Error Propagation and Accuracy Drop
The analyses find that error propagation contributes to accuracy drop, but does not fully explain it. Teacher-forcing comparisons show that accuracy improves more in later-generated regions, while an accuracy gap can remain even when preceding words are supplied from the reference.
3.1 Error Propagation is Not the Only Cause
- The experiments compare left- and right-to-left Transformer translations on three language tasks, scoring equal-sized sentence halves with BLEU.The tasks are German-English, English-German, and English-Chinese; the analysis also includes teacher forcing during inference.Source: §3.1 · p. 3
- Accuracy remains higher in the left half for both left-to-right and right-to-left decoding, so error propagation is not the only cause of accuracy drop.The opposite-direction result contradicts the expectation that later-generated words should be less accurate because of accumulated errors.Source: §3.1 · pp. 3–4
3.2 The Influence of Error Propagation
- Teacher forcing improves both halves, showing exposure bias; for En-Zh, later-generated halves improve more than earlier ones in both decoding directions.For left-to-right En-Zh, the right-half improvement is 2.64 BLEU versus 1.70 for the left; for right-to-left En-Zh, the left-half improvement is 2.82 versus 1.77 for the right.Source: §3.2 · p. 4, Table 3 · p. 4
- With teacher forcing, En-De left-to-right BLEU remains higher for the left half (9.43) than the right half (8.36), indicating other causes contribute.The authors report similar results for other language pairs and models.Source: §3.2 · p. 4, Table 3 · p. 4
4 Language Branching Matters
The paper relates accuracy differences between sentence halves to language branching. English is right-branching and Japanese is left-branching; the observed accuracy pattern reverses between them.
4 Language Branching Matters
- For En-Jp, the right half is more accurate than the left regardless of decoding direction or teacher forcing, opposite to English, German, and Chinese.The comparison motivates examining differences in language structure.Source: §4 · pp. 4–5, Table 4 · p. 5
- Branching describes sentence parse-tree shape: right-branching languages are typically head-initial, while left-branching languages are head-final.English is described as typically right-branching and Japanese as almost fully left-branching; German and Chinese are less conclusive or mixed cases.Source: §4 · p. 5
- The authors propose that right-branching structure can make the left half clearer and more accurate, while additional modifiers follow in the right half.This is offered as an explanation for accuracy drop based on sentence structure.Source: §4 · p. 5
5 Correlation between Language Branching and Accuracy Drop
The section relates language branching to accuracy differences between the left and right halves of translations using n-gram and dependency statistics. Both analyses find patterns consistent with higher accuracy in the half containing more predictable or internally connected language structure.
5 Correlation between Language Branching and Accuracy Drop
- The section examines n-gram frequency and conditional probability alongside dependency statistics, finding that both correlate with left-versus-right accuracy differences.The authors frame these statistics as language-structure characteristics that relate to how easily each half can be predicted.Source: §5 · p. 5, §5.1 · p. 5
5.1 N-gram Statistics
- For right-branching languages, left-half n-gram frequency and conditional probability are higher; for left-branching Japanese, the pattern reverses, matching the observed accuracy differences.The authors interpret these statistics as indicating clearer patterns in the more accurate half.Source: §5.1 · p. 6
- Figure 1 shows accuracy drop increasing as the gap in n-gram statistics shifts from negative to positive.The plotted accuracy drop uses teacher-forced left-to-right BLEU scores and the 3-gram frequency and conditional-probability gaps.Source: §5.1 · pp. 6–7, Figure 1 · p. 7
5.2 Dependency Statistics
- English sentences have more dependencies within the left half, while Japanese sentences have more within the right half, consistent with their respective accuracy patterns.The analysis counts dependencies in each half of English and Japanese training-corpus sentences.Source: §5.2 · p. 7
6 Extended Analyses and Discussions
The extended analyses test whether findings generalize to another left-branching language, another model structure, and abstractive summarization. They report distinct accuracy patterns across these settings.
6.1 More Languages on Left-Branching
- English-Turkish shows a higher-accuracy right half than left half, but its language-branching influence is weaker than error propagation.With teacher forcing, the right half becomes more accurate; in normal left-to-right translation, the left half is more accurate.Source: Table 8 · p. 8, §6.1 · p. 8
6.2 Other Model Structures
- With a GRU-based RNN, De-En left halves remain more accurate than right halves in both decoding directions.The authors report that this is consistent with their Transformer observations.Source: Table 9 · p. 8, §6.2 · p. 8
6.3 Other Sequence Generation Tasks
- In abstractive summarization, English summaries have more accurate left halves than right halves in both decoding directions.The authors relate this pattern to English being right-branching.Source: Table 10 · p. 8, §6.3 · p. 8
7 Conclusion
The conclusion argues that language branching correlates with accuracy drop, while error propagation alone cannot explain it. It recommends matching decoding direction to branching, and identifies broader language and task coverage for future work.
7 Conclusion
- Language branching correlates with accuracy drop, and error propagation alone cannot explain the problem.The authors say n-gram and dependency statistics support this correlation.Source: §7 · p. 8
- The authors suggest left-to-right models for right-branching languages and right-to-left models for left-branching languages.Future work will study more languages, including those without obvious branching, and branching effects in other natural-language tasks.Source: §7 · p. 8
A.1 NMT Datasets
The appendix describes five translation datasets used for the experiments and notes their dataset sizes and text preprocessing.
A.1 NMT Datasets
- The experiments use five translation tasks: De-En, En-De, En-Zh, En-Jp, and En-Tr.The datasets are IWSLT 2014 De-En and En-Tr, WMT 2014 En-De, WMT 2017 En-Zh, and ASPEC En-Jp.Source: §A.1 · p. 11
- Training-set sizes range from about 153k parallel sentences for De-En to nearly 24M for En-Zh.The other reported training-set sizes are about 4.5M for En-De, 1.5M for En-Jp, and about 350k for En-Tr.Source: §A.1 · p. 11
- The first three tasks use byte-pair encoded subwords, En-Jp uses word-level text, and En-Tr uses Zemberek morphological segmentation.Source: §A.1 · p. 11
A.2 NMT Models
The paper uses Transformer models for its translation experiments and GRU-based RNN models for translation and summarization. It also describes decoding and training choices for these models.
A.2 NMT Models
- Transformer settings vary by task: small for De-En and En-Tr, base v1 for En-De and En-Jp, and big for En-Zh.Right-to-left training reverses target sentences; decoding uses greedy search.Source: §A.2 · p. 11
- The GRU-based RNNs use a bidirectional encoder and a unidirectional decoder.The De-En model has 256-dimensional embeddings and hidden states; summarization uses 512-dimensional embeddings and a 1024-dimensional hidden state.Source: §A.2 · p. 11
- The RNN models are trained with Adadelta at learning rate 1.0.Source: §A.2 · p. 11
A.3 Dependency Parsing Results
The appendix provides English and Japanese dependency-parsing examples and describes counting dependencies whose words both fall in the same sentence half.
A.3 Dependency Parsing Results
- English dependency parsing uses Stanford Parser with NLTK, illustrated on an English sentence and its dependency tuples.The example sentence is “and the great indicator of that, of course, is language loss.”Source: §A.3 · p. 11
- The count includes dependency tuples whose two words both belong to the left half or both belong to the right half.For example, “indicator” depends on “the,” and both are in the left half.Source: §A.3 · p. 11
- Japanese parsing uses J.DepP, and the final punctuation token is excluded when counting words.The supplied Japanese example ends with “。”; that token is removed from the count.Source: §A.3 · pp. 11–12