Source-linked AI summary
Sequence Level Training with Recurrent Neural Networks
Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, Wojciech Zaremba
TL;DR
Standard text-generation training uses ground-truth inputs and a non-sequence-level objective, while testing requires generating from the model’s own predictions. MIXER addresses this mismatch through incremental sequence-level training with REINFORCE and cross-entropy, improving greedy generation across three tasks and competing with beam search.
Problem
Training exposes models to ground-truth words, but generation feeds back their own predictions, creating exposure bias and potentially large deviations from the target sequence.
Method
MIXER gradually shifts a cross-entropy-trained model toward its own predictions while using a hybrid cross-entropy–REINFORCE loss to optimize sequence-level rewards.
Results
MIXER improves generation over XENT by 1 to 3 points across all three tasks, outperforming strong baselines with greedy search and competing with beam search.
Takeaways & Limitations
Sequence-level training with an incremental transition from cross-entropy to REINFORCE enables successful reinforcement-learning-based text generation despite large action spaces.
Takeaways & Limitations
REINFORCE-based training remains challenging with large action spaces because random-policy exploration is unlikely to improve within a reasonable time.
Abstract
from arXiv · showhide
Many natural language processing applications use language models to generate text. These models are typically trained to predict the next word in a sequence, given the previous words and some context such as an image. However, at test time the model is expected to generate the entire sequence from scratch. This discrepancy makes generation brittle, as errors may accumulate along the way. We address this issue by proposing a novel sequence level training algorithm that directly optimizes the metric used at test time, such as BLEU or ROUGE. On three different tasks, our approach outperforms several strong baselines for greedy generation. The method is also competitive when these baselines employ beam search, while being several times faster.
1 INTRODUCTION
Text generation models face exposure bias because training uses ground-truth inputs while testing feeds back model predictions, and word-level losses do not directly optimize sequence metrics. The paper proposes MIXER, an incremental hybrid training algorithm, and evaluates it across three tasks.
- Motivation: Exposure bias arises because models train on ground-truth words but generate by feeding their own predictions back as inputs.Errors can accumulate because the training and test-time input distributions differ.
- Motivation: Word-level cross-entropy training does not directly optimize the sequence-level metrics used to evaluate generated text.The paper targets metrics such as BLEU and ROUGE through sequence-level training.
- MIXER: MIXER combines REINFORCE with cross-entropy and incrementally shifts the model from ground-truth inputs toward its own predictions.It begins from a cross-entropy-trained policy rather than a random policy, helping address the large action space of text generation.
- Evaluation: MIXER is evaluated against strong baselines on Text Summarization, Machine Translation, and Image Captioning.The comparison includes cross-entropy-trained RNNs, DAD, and the proposed End-to-End BackProp model.
- Results: MIXER with greedy search is more accurate than the baselines on all three tasks and more accurate than cross-entropy with beam search.Greedy MIXER is at least 10 times faster than a cross-entropy model using a beam of size 10.
2 RELATED WORK
Related work addresses sequence generation through beam search, imitation learning, and reinforcement learning. These approaches differ in how they handle model-generated inputs, optimization, and sequence-level objectives.
- Beam Search: Beam search explores multiple alternative generation paths and typically improves generation by one or two BLEU points.Its computational cost increases with the number of active paths in the beam.
- Beam Search: Beam search makes generation at least k times slower when the beam contains k active paths.This motivates methods that improve greedy generation without requiring extensive inference-time search.
- Imitation Learning: SEARN and related imitation-learning methods let models use their own predictions during training before learning actions selected by a search procedure.For text generation, computing an oracle for the optimal next word after predicted words is generally intractable.
- Imitation Learning: DAD addresses the oracle issue by using the kth action taken by the optimal policy as the target action at step k.This provides a text-generation approach related to the paper’s use of model predictions during training.
3 MODELS
The section reviews recurrent text-generation models and training objectives, highlighting the mismatch between word-level training and sequence generation at test time. It then introduces sequence-level training with MIXER, which uses model predictions and REINFORCE to optimize sequence rewards.
- Model setup: RNNs generate next-word distributions from the current word, hidden state, and optionally a context vector.The model can use Elman RNNs or LSTMs, with context encoded for the generation task.
- Word-level training: XENT trains each next-word prediction on ground-truth prefixes, optimizing local decisions rather than the whole generated sequence.Generation then proceeds greedily, one word at a time, from the model's previous predictions.
- Word-level training: Exposure bias arises because training supplies ground-truth words while test-time generation feeds the model's own potentially incorrect predictions back as inputs.This mismatch can cause the generated sequence to deviate substantially from the target sequence.
- Alternative baselines: DAD mixes ground-truth words and model predictions during training, but keeps ground-truth targets, does not back-propagate through sampled predictions, and remains word-level.When inputs diverge from the reference, its targets may no longer align with the generated sequence.
- Sequence-level training: MIXER combines incremental learning with a hybrid XENT–REINFORCE loss, beginning from a cross-entropy-trained policy before using model predictions and sequence-level rewards.REINFORCE supports nondifferentiable rewards such as BLEU and ROUGE-2, while the incremental schedule helps address text generation's large action space.
4 EXPERIMENTS
Experiments on summarization, translation, and image captioning evaluate MIXER against strong baselines using task-specific metrics and greedy or beam generation. MIXER improves over XENT across tasks, while metric alignment and decoding strategy affect performance and speed.
- Tasks and evaluation: Experiments cover text summarization, German-English translation, and MSCOCO image captioning.The evaluation uses ROUGE for summarization and BLEU for translation and captioning.
- Greedy-generation results: MIXER improves generation over XENT by 1 to 3 points across all three tasks.MIXER produces the best generations among the compared methods, while DAD is usually better than XENT but worse than MIXER.
- Metric alignment: Metric choice matters: summarization BLEU is 8.16 for XENT, 5.80 for MIXER trained with ROUGE, and 9.32 for MIXER trained with BLEU.The corresponding ROUGE scores are 15.1 for MIXER trained with BLEU and 16.22 for MIXER optimizing ROUGE.
- Beam search: Beam search improves every method, but its benefit depends strongly on the task.Figure 6 compares test scores as the number of beam hypotheses k increases, including MIXER with greedy generation and k = 10.
- Beam search: MIXER’s greedy performance exceeds what baselines achieve with beam search in two of three tasks and is several times faster.MIXER relies only on greedy search, whereas beam search explores multiple hypotheses.
- Training considerations: REINFORCE did not converge for these applications, and hybrid XENT-REINFORCE without incremental learning was insufficient from a random policy.The authors report that exploration from a random policy had little chance of success.
5 CONCLUSIONS
MIXER addresses exposure bias and sequence-level objective mismatch by combining incremental learning with reinforcement learning initialized from cross-entropy training. It outperforms strong baselines with greedy generation and remains competitive with beam search, while leaving estimation and training-search improvements open.
- Motivation: MIXER targets exposure bias and losses that do not operate at the sequence level.These are identified as two major deficiencies in current text-generation training.
- Approach: MIXER initializes reinforcement learning from a cross-entropy-trained policy and gradually increases exposure to the model’s own predictions.This incremental-learning framework replaces a random starting policy.
- Conclusions: MIXER outperforms three strong baselines with greedy generation and is very competitive with beam search.The approach is described as agnostic to the underlying model and reward-function form.
- Future work: Future work includes better average-reward estimation and more comprehensive search methods during training.Poor reward estimates can slow convergence of REINFORCE and MIXER, while the current algorithm uses a single training sample.
6 SUPPLEMENTARY MATERIAL
The supplementary material details the attentive encoder and beam-search procedure, and provides qualitative and metric-based comparisons of MIXER with alternative systems.
- Qualitative Comparison: Qualitative examples compare greedy generations with ground-truth summaries across several news contexts, including cases containing tokenizer-produced <unk> tokens.The examples include outputs from XENT, DAD, E2E, and MIXER.
- Evaluation: Figure 8 compares relative BLEU and ROUGE-2 gains against XENT for DAD, E2E, and MIXER under matched and mismatched training metrics.The figure also reports MIXER trained with ROUGE-2 when evaluating BLEU, and vice versa.
- The Attentive Encoder: The attentive encoder computes the conditioning vector c_t from the source sentence and current hidden state h_t.The source sentence is represented as word and position embeddings, while aggregate vectors are padded at both boundaries before producing c_t.
- Beam Search: Beam search maintains the k highest-scoring partial sequences, with k = 1 reducing to greedy left-to-right search.At each step, the algorithm expands retained sequences using their k most likely next-word candidates.
- Updates: The updated version reports both ROUGE-2 and BLEU for summarization and uses LSTMs for machine translation and image captioning.These changes demonstrate MIXER's use with different evaluation metrics and underlying parametric models.