Source-linked AI summary
Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, Noam Shazeer
TL;DR
Recurrent sequence models train with true previous tokens but infer using their own predictions, allowing early errors to accumulate in unfamiliar states. The paper introduces Scheduled Sampling, which gradually replaces true tokens with sampled model outputs during training. Experiments report improved performance across sequence prediction tasks, including a first-place result in the 2015 MSCOCO image captioning challenge.
Problem
Training uses true previous tokens while inference uses generated tokens, creating a mismatch that can amplify early errors and move the model into unfamiliar states.
Method
Scheduled Sampling uses curriculum learning to gradually replace true previous tokens with model-generated tokens during training.
Results
Experiments on several sequence prediction tasks yielded performance improvements, and the approach supported a first-place finish in the 2015 MSCOCO image captioning challenge.
Takeaways & Limitations
Training models on their own mistakes makes them more robust to the training–inference mismatch during sequence generation.
Takeaways & Limitations
Speech-recognition results are not directly comparable to typical hybrid models because the experiments omit HMM decoding and its associated smoothing and language models.
Abstract
from arXiv · showhide
Recurrent Neural Networks can be trained to produce sequences of tokens given some input, as exemplified by recent results in machine translation and image captioning. The current approach to training them consists of maximizing the likelihood of each token in the sequence given the current (recurrent) state and the previous token. At inference, the unknown previous token is then replaced by a token generated by the model itself. This discrepancy between training and inference can yield errors that can accumulate quickly along the generated sequence. We propose a curriculum learning strategy to gently change the training process from a fully guided scheme using the true previous token, towards a less guided scheme which mostly uses the generated token instead. Experiments on several sequence prediction tasks show that this approach yields significant improvements. Moreover, it was used successfully in our winning entry to the MSCOCO image captioning challenge, 2015.
1 Introduction
Sequence models are trained with true previous tokens but must generate using their own predictions at inference, creating a mismatch that can amplify errors. The paper proposes curriculum learning to bridge this gap by gradually exposing models to generated tokens.
- Sequence prediction tasks include machine translation and image captioning, where recurrent networks generate variable-length token sequences from sequence or non-sequence inputs.
- Training maximizes each target token’s likelihood conditioned on the current state and previous target token, whereas inference uses a model-generated previous token.
- Early generation mistakes can be fed back into the model and rapidly amplified because resulting states may be unseen during training.
- The proposed curriculum gradually changes training from using known previous tokens to handling the model’s own mistakes, improving robustness during inference.
- The paper is organized around the proposed approach, related work, experiments, and conclusions.
2 Proposed Approach
The approach trains recurrent sequence models by replacing true previous tokens with model predictions according to a curriculum. It formalizes sequence likelihood, inference, beam search, and schedules that progressively reduce teacher guidance.
- 2.1 Model: The model represents variable-length target sequences as tokens from a fixed dictionary, using an end-of-sequence token to terminate generation.
- 2.1 Model: The recurrent network computes sequence likelihood by conditioning each token on the input, recurrent state, and previous output token.
- 2.2 Training: Training uses mini-batch stochastic gradient descent to maximize the log likelihood of correct target sequences across training pairs.
- 2.3 Inference: During inference, the model generates one token at a time using its previous prediction because the true previous token is unavailable.
- 2.3 Inference: Beam search maintains multiple candidate sequences, but continuous recurrent states prevent efficient path factorization and keep the candidate set small.
- 2.3 Inference: Inference errors can move the model into unfamiliar states and cause cumulative bad decisions along the sequence.
- 2.4 Bridging the Gap with Scheduled Sampling: Scheduled Sampling flips a coin for each token, using the true previous token with probability ϵ_i and a model estimate with probability 1 − ϵ_i.
- 2.4 Bridging the Gap with Scheduled Sampling: The curriculum decreases ϵ_i over training, beginning with more truth-guided inputs for convergence and ending with more model-generated inputs to match inference.
3 Related Work
Prior work addresses the training–inference mismatch through policy-based retraining, beam-search alignment, or dynamic target adaptation. The proposed approach differs by using a single online model in supervised stochastic-gradient training, avoiding costly batch retraining and difficulties with recurrent neural network state search.
- Policy-based approaches: SEARN addresses compounding errors by retraining models according to the current policy across meta-iterations.Its setting involves sequential decisions where early mistakes can produce poor global performance.
- Online training: The proposed method is completely online: one model is trained while its policy gradually evolves, unlike SEARN’s batch approach.The paper states this makes training much faster than the related batch procedure.
- Beam-search alignment: Beam-search approaches align training and inference by using a training beam to compare the model’s estimate with the guided solution under a ranking loss.This line of work targets parsing tasks whose outputs are trees.
- Beam-search alignment: Beam search is difficult to use efficiently with recurrent neural networks because their multidimensional continuous state sequences cannot be factored easily.The paper notes this difficulty affects both training and inference.
- Dynamic target adaptation: Dynamic-oracle online parsing adapts targets to the model’s decisions, but uses a perceptron and a fixed probability of choosing the truth during training.These differences distinguish it from the recurrent, probability-scheduled approach considered here.
4 Experiments
Experiments evaluate scheduled sampling on image captioning, constituency parsing, and speech recognition. Across these tasks, the reported results indicate benefits from training with sampled previous tokens, while always sampling or sequence-level sampling can perform poorly in some settings.
- 4.1 Image Captioning: Scheduled sampling improved image-captioning performance across the reported MSCOCO development-set metrics, while Always Sampling performed very poorly.The evaluation used 75k training images and a separate 5k-image development set, with five captions per image.
- 4.2 Constituency Parsing: The parsing model used a one-layer 512-cell LSTM with 512-dimensional word embeddings and attention over input-sequence LSTM states.The input dictionary contained around 90k words, and the target dictionary contained 128 symbols.
- 4.3 Speech Recognition: The speech-recognition model used aligned acoustic features and target sequences, with recurrent computation initialized by a start-of-sequence token.The recurrent state used the previous target token after the first time step; the experiments used TIMIT4 data and Kaldi-generated targets.
- 4.3 Speech Recognition: In speech recognition, sampling-trained models decoded better than the ground-truth-fed baseline even though the baseline had better next-step prediction.Testing used beam-search decoding, while next-step error was measured with ground-truth inputs on validation data.
5 Conclusion
The paper introduces curriculum learning to shift sequence-model training from true previous tokens toward model-generated tokens. Experiments report performance improvements without longer training times, while identifying differentiable sampling and improved sampling strategies as future directions.
- The proposed curriculum gradually changes training from conditioning on known previous tokens to conditioning on tokens generated by the model.This exposes the model to its own mistakes during training, matching inference more closely.
- Experiments on several sequence prediction tasks yield performance improvements without incurring longer training times.
- Future work includes back-propagating errors through sampling decisions and exploring sampling strategies conditioned on model confidence.