PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging the Gap between Training and Inference for Neural Machine Translation

Wen Zhang, Yang Feng, Fandong Meng, Di You, Qun Liu

arXiv:1906.02448v2cs.CLcs.LGstat.ML

TL;DR

NMT training uses ground-truth context while inference relies on model-generated context, creating exposure bias and penalizing reasonable alternative translations. The paper samples both contexts using word- or sentence-level oracle optimization, improving performance across language pairs and model architectures.

  • Problem

    NMT training and inference use different context distributions, causing exposure bias, while word-level matching can overcorrect reasonable alternative translations.

  • Method

    The approach samples ground-truth or predicted context words during training, selecting predicted sequences with word-level or sentence-level optimization.

  • Results

    +2.36 BLEU points on average over four test datasets, with improvements also reported for Transformer and English→German translation tasks.

  • Takeaways & Limitations

    The oracle-based training strategy improves NMT performance across different language pairs and model architectures.

  • Takeaways & Limitations

    Sentence-level oracle training requires force decoding because naive beam search cannot guarantee equal sequence lengths.

Abstract

from arXiv · show

Neural Machine Translation (NMT) generates target words sequentially in the way of predicting the next word conditioned on the context words. At training time, it predicts with the ground truth words as context while at inference it has to generate the entire sequence from scratch. This discrepancy of the fed context leads to error accumulation among the way. Furthermore, word-level training requires strict matching between the generated sequence and the ground truth sequence which leads to overcorrection over different but reasonable translations. In this paper, we address these issues by sampling context words not only from the ground truth sequence but also from the predicted sequence by the model during training, where the predicted sequence is selected with a sentence-level optimum. Experiment results on Chinese->English and WMT'14 English->German translation tasks demonstrate that our approach can achieve significant improvements on multiple datasets.

1 Introduction

NMT training conditions on ground-truth context, whereas inference feeds back the model’s own generated words, creating exposure bias and accumulating errors. The paper motivates training with both ground-truth and predicted context to better match inference conditions and address strict word-level matching.

  • Training–inference discrepancy: NMT encoder-decoder models predict each target word conditioned on preceding context words, using ground-truth words as context during training.The introduction identifies RNN-, CNN-, and attention-based models within this framework.
  • Training–inference discrepancy: At inference, the model generates the entire sequence independently and feeds its previous predictions back as context.Thus, training uses the data distribution while inference uses the model distribution.
  • Exposure bias: This distributional mismatch, known as exposure bias, creates a training–inference gap and causes errors to accumulate as target sequences grow.The passage explicitly attributes the accumulating errors to exposure bias.
  • Proposed remedy: A proposed remedy is to train NMT models with both ground-truth and predicted words as context, matching the conditions encountered during inference.This motivation is inspired by DATA AS DEMONSTRATOR (DAD).
  • Training objective: Word-level cross-entropy training requires strict pairwise matching between predicted and ground-truth sequences.The passage presents this requirement as a motivation for seeking alternatives.

2 RNN-based NMT Model

The paper presents an RNN-based NMT model as a representative setting for its method. The model uses a bidirectional GRU encoder, attention-based source information extraction, and a GRU-based decoder that predicts target-word probabilities.

  • Encoder: A bidirectional GRU encoder produces forward and backward hidden-state sequences, concatenated into each source-word annotation.The annotation of source word x_i is h_i = [forward h_i; backward h_i].
  • Attention: The attention mechanism evaluates and normalizes relevance between the target word at step j and each source word.It extracts source information through a source context vector.
  • Attention: The source context vector is computed as a weighted sum of all source annotations.
  • Decoder: A decoder variant of GRU unrolls target information and computes the target hidden state at each step.
  • Decoder: The decoder produces a probability distribution over the target vocabulary conditioned on the previous ground-truth word embedding, source context vector, and hidden state.A linear transformation maps the decoder representation so each target word corresponds to one output dimension.

3 Approach

The method alternates between ground-truth and previously predicted words as context with a certain probability, training the model to handle test-time conditions. It selects oracle words either greedily at the word level or through a sentence-level optimum, which supports n-gram matching with the ground-truth sequence.

  • 3 Approach: The framework feeds either ground-truth words or previous predicted words as context with a certain probability.This trains the model to handle situations that appear during testing and potentially reduces the training–inference gap.
  • 3 Approach: Oracle words are selected using either word-level greedy search or a sentence-level optimum.The two selection strategies operate at different granularities: individual words versus an oracle sequence.
  • 3 Approach: The sentence-level oracle provides an option for n-gram matching with the ground-truth sequence.The supplied passage introduces this matching option as part of sentence-level oracle selection.

1. Select an oracle word yoracle

The method selects oracle context words or sentences to expose NMT training to predicted contexts while retaining flexible, sentence-level translation choices. It samples between ground-truth and oracle words with a probability that decays over training.

  • Sampling with Decay: At each decoding step, the method samples the ground-truth word with probability p or the oracle word with probability 1−p.This replaces the usual context word with a mixture of ground-truth and predicted contexts during training.
  • Oracle Word Selection: Oracle words simulate decoding context and can be selected by word-level greedy search or sentence-level beam search reranked with BLEU.Sentence-level oracles allow more flexible translation through n-gram matching.
  • Oracle Word Selection: Word-level oracles choose the highest-probability word, while Gumbel-Max noise provides a more robust categorical sampling procedure.The resulting distribution uses (oj−1 + η) /τ before softmax, and the 1-best word is selected; the noise does not affect the training loss.
  • Oracle Word Selection: Sentence-level oracles select the highest-BLEU candidate from k-best beam-search translations evaluated against the ground-truth sequence.Force decoding ensures every candidate has |y∗| words followed by EOS before BLEU reranking.
  • Sampling with Decay: The ground-truth sampling probability starts at p=1 and decreases progressively as training epochs advance, transitioning training toward oracle contexts.A fixed p would either slow convergence early or insufficiently expose the model to inference-time contexts late in training.

4 Related Work

Related work addresses exposure bias in NMT through training contexts that include model predictions and through sentence-level objectives. These approaches aim to reduce the mismatch between training and inference and increase flexibility in generation.

  • Word-level training: DAD initialized examples with adjacent ground-truth words, then added predicted words paired with the next ground-truth words during training.Bengio et al. further developed this direction by sampling context from previous ground-truth and predicted words.
  • Sentence-level training: Sentence-level training uses metrics such as BLEU to provide greater flexibility for generation and mitigate exposure bias.Ranzato et al. introduced MIXER, which directly optimized the sentence-level BLEU used at inference.

5 Experiments · 5.1 Settings · 5.2 Systems

Experiments evaluate the proposed method on Chinese→English and English→German translation tasks using task-specific datasets, vocabulary settings, and comparison systems. OR-NMT extends RNNsearch with oracle mechanisms and Gumbel noise to recover from overcorrection.

  • 5 Experiments: Experiments cover NIST Chinese→English and WMT’14 English→German translation tasks.These are the two translation settings used for evaluation.
  • 5.1 Settings: The Zh→En training data contains 1.25M sentence pairs from LDC corpora.MT02 is the validation set, while MT03–MT06 are test sets containing 919, 1788, 1082, and 1664 sentences respectively.
  • 5.1 Settings: The En→De experiments use the WMT’14 corpus containing 4.5M sentence pairs.The passage also states that newstest2013 is used as the validation set, but the excerpt ends before further details.
  • 5.1 Settings: Zh→En limits both source and target vocabularies to the most frequent 30K words, covering approximately 97.7% and 99.3% of corpus words.For En→De, sentences use BPE with 37k merging operations and vocabularies of 39418 and 40274 tokens.
  • 5.2 Systems: RNNsearch is an improved model whose decoder uses two GRUs and an attention mechanism.Its implementation follows the model described in Section 2.
  • 5.2 Systems: SS-NMT applies scheduled sampling to RNNsearch using the same decay scheme as Equation 15.MIXER instead uses BLEU as its sentence-level metric and obtains average rewards through an offline method with a 1-layer linear regressor.
  • 5.2 Systems: OR-NMT augments RNNsearch with word-level and sentence-level oracles plus Gumbel noises to improve overcorrection recovery.Sentence-level oracle selection uses beam size 3, τ=0.5, and µ=12.

5.3 Results on Zh→En Translation

This section verifies the method on NIST Chinese-to-English translation datasets using two baseline models.

  • 5.3 Results on Zh→En Translation: The method is evaluated on NIST Zh→En translation datasets.The experiments focus on Chinese-to-English translation.
  • 5.3 Results on Zh→En Translation: The verification uses two baseline models.
  • 5.3 Results on Zh→En Translation: The experiments are presented as results for Zh→En translation.

Results on the RNNsearch · Results on the Transformer · 5.4 Factor Analysis

OR-NMT mitigates exposure bias and overcorrection by incorporating predicted context during training, improving both RNNsearch and Transformer systems. Factor analysis shows that word-level and sentence-level oracle strategies contribute measurable BLEU gains, with the sentence-level oracle performing better.

  • Results on the RNNsearch: The RNNsearch baseline outperforms a previous shallow RNN-based system with coverage modeling and achieves competitive performance against related methods.The cited comparisons are Tu et al. (2016), Shen et al. (2016), and Zhang et al. (2017).
  • Results on the RNNsearch: OR-NMT outperforms RNNsearch and competitive exposure-bias baselines SS-NMT and MIXER by incorporating sentence-level optimum predictions during training.The compared methods include scheduled sampling (SS-NMT) and sentence-level training (MIXER).
  • Results on the Transformer: On the stronger Transformer, the word-level method improves the base model by +0.54 BLEU points on average, while the sentence-level method adds +1.0 BLEU points.Both improvements are reported relative to the Transformer base model or preceding method as described.
  • 5.4 Factor Analysis: Factor analysis evaluates the word-level oracle, sentence-level oracle, and Gumbel noise for oracle selection to assess strategies addressing overcorrection.The experiments are reported in Table 2 for Zh→En translation.
  • 5.4 Factor Analysis: +1.21 BLEU points result from using only the word-level oracle, indicating that predicted context words can mitigate exposure bias.This result is reported on Zh→En translation using average BLEU scores on MT03∼06 datasets.
  • 5.4 Factor Analysis: +0.62 BLEU points are gained with the sentence-level oracle, which performs better than the word-level oracle in BLEU.The paper conjectures that greater flexibility for word generation may explain the sentence-level oracle’s superiority.
  • 5.4 Factor Analysis: Training loss curves and validation-set BLEU trends are presented for RNNsearch with word-level and sentence-level oracle factors.Figure 4 compares training loss, while Figure 5 shows validation-set BLEU trends on Zh→En translation.

5.5 About Convergence · 5.6 About Length

The convergence analysis compares training behavior and shows that sentence-level oracle sampling with Gumbel noise produces the strongest BLEU performance, while the length analysis finds improvements over the baseline across all source-length bins. RNNsearch reaches its best result at the seventh epoch, although its training loss continues decreasing and requires longer convergence.

  • 5.5 About Convergence: RNNsearch converges quickly and achieves its best result at the 7-th epoch, although its training loss continues decreasing and needs longer to converge.
  • 5.5 About Convergence: The convergence comparison evaluates RNNsearch, word-level oracle without noise, and sentence-level oracle with noise using training loss curves.
  • 5.5 About Convergence: Sentence-level oracle sampling with noise (τ=0.5) obtains the best model on the MT03 test set.
  • 5.5 About Convergence: Without noise, the system converges to a lower BLEU score, whereas repeated use of its own results causes overfitting and quick convergence.
  • 5.5 About Convergence: The method benefits from sentence-level sampling and Gumbel noise during training.
  • 5.6 About Length: Translations are grouped into source-length bins, and BLEU scores are tested separately for each bin on the MT03 test set.
  • 5.6 About Length: Our approach achieves big improvements over the baseline system in all source-length bins, especially in (10,20], (40,50], and (70,80].

5.7 Effect on Exposure Bias · 5.8 Results on En→De Translation

The proposed model improves predicted probabilities for most gold words and outperforms RNNsearch on Zh→En, supporting its effectiveness against exposure bias. On WMT’14 En→De, it significantly improves both RNNSearch and Transformer baselines, demonstrating cross-language effectiveness.

  • 5.7 Effect on Exposure Bias: The SO model without noise was trained from the pre-trained RNNsearch model.The paper identifies this setup using the red dashed lines in Figures 5 and 6.
  • 5.7 Effect on Exposure Bias: Performance was compared across different source-sentence lengths on the Zh→En MT03 test set.Figure 7 reports the length-based performance comparison.
  • 5.7 Effect on Exposure Bias: 65.0% of gold words received higher probabilities from the proposed model than from the baseline.Among 28,266 reference gold words, N=18,391 had higher predicted probabilities.
  • 5.8 Results on En→De Translation: On WMT’14 En→De, the proposed method significantly outperformed the competitive baseline and related approaches.The results are reported as case-sensitive BLEU scores in Table 3; ‡ denotes significance at p<0.01 versus RNNsearch and Transformer.

6 Conclusion

The paper addresses the training–inference discrepancy in end-to-end NMT by sampling each word’s context from either the ground-truth word or the model’s previous prediction. It also defines predicted words as oracle words, which can be generated using word-level or sentence-level optimization.

  • 6 Conclusion: NMT uses ground-truth words as context during training but previously generated words during inference, creating a discrepancy.The model generates translations word by word in both settings, but the source of contextual words differs.
  • 6 Conclusion: The method mitigates this discrepancy by sampling either the ground-truth word or the previous predicted word as context when predicting each word.This sampling scheme exposes the model to predicted context during training.
  • 6 Conclusion: The predicted words used as alternative context are referred to as oracle words and can be generated with word-level or sentence-level optimization.The passage identifies both optimization granularities for generating oracle words.
Loading 1906.02448v2…