Source-linked AI summary
Classical Structured Prediction Losses for Sequence to Sequence Learning
Sergey Edunov, Myle Ott, Michael Auli, David Grangier, Marc'Aurelio Ranzato
TL;DR
Neural sequence-to-sequence models are often trained with token-level objectives despite generating complete sequences, motivating sequence-level training. The paper applies classical structured-prediction losses to neural models and finds them highly competitive, including state-of-the-art results across reported tasks. Training with current-model candidates and combining sequence- and token-level losses are important for best performance, though candidate regeneration substantially slows training.
Problem
Token-level training is inconsistent with sequence-level generation, motivating sequence-level objectives for neural sequence-to-sequence models.
Method
The paper applies classical sequence-level likelihood, margin, expected-risk, and hybrid objectives to neural sequence-to-sequence models using beam-search or sampled candidates.
Results
Classical structured-prediction losses are competitive with recent methods, outperform beam search optimization in a comparable setup, and achieve 41.5 BLEU on WMT’14 English-French.
Takeaways & Limitations
Best performance requires combining sequence-level with token-level losses and training on candidates decoded by the current model.
Takeaways & Limitations
Generating candidates for each batch slows training substantially, and regenerating them less frequently is left to future work.
Abstract
from arXiv · showhide
There has been much recent work on training neural attention models at the sequence-level using either reinforcement learning-style methods or by optimizing the beam. In this paper, we survey a range of classical objective functions that have been widely used to train linear models for structured prediction and apply them to neural sequence to sequence models. Our experiments show that these losses can perform surprisingly well by slightly outperforming beam search optimization in a like for like setup. We also report new state of the art results on both IWSLT'14 German-English translation as well as Gigaword abstractive summarization. On the larger WMT'14 English-French translation task, sequence-level training achieves 41.5 BLEU which is on par with the state of the art.
1 Introduction
The paper applies classical structured-prediction objectives to neural sequence-to-sequence models, addressing the mismatch between token-level training and sequence-level generation. These losses perform competitively across translation and summarization tasks.
- Motivation: Sequence-to-sequence models are commonly trained token-by-token but generate complete sequences at test time.This mismatch motivates sequence-level training methods such as REINFORCE, actor-critic, and beam search optimization.
- Approach: The paper revisits classical structured-prediction objectives and applies sequence-level likelihood, margin, expected-risk, and combined losses to neural models.The study also compares combinations of global sequence losses with token-level likelihood.
- Findings: The surveyed sequence-level losses perform similarly and outperform beam search optimization in a comparable setup.The comparison covers neural sequence-to-sequence training on the reported tasks.
- Findings: The experiments achieve the best reported accuracy at the time on IWSLT’14 German-English translation and Gigaword abstractive summarization.The paper also evaluates risk minimization on the larger WMT’14 English-French task.
2 Sequence to Sequence Learning
The model uses an attention-based encoder-decoder architecture, primarily implemented with gated convolutional networks. Its objectives are model agnostic, and fast generation supports sequence-level training.
- Architecture: The model follows an encoder-decoder architecture with soft attention, using convolutional rather than recurrent networks in most experiments.The convolutional design is motivated by computational and accuracy considerations.
- Architecture: The objective functions are model agnostic and apply equally to recurrent and convolutional models.The paper demonstrates this applicability with recurrent LSTM models in comparison to prior work.
- Architecture: The encoder maps source sentence x to states z, while the decoder generates output sequence u left to right using prior decoder information and attention context.At each step, attention context is computed as a weighted sum of encoder states.
- Architecture: Gated convolutional networks enable fast generation, which is essential because sequence-level losses generate model outputs during training.Encoder and decoder blocks use fixed-width convolutions, with causal convolutions in the decoder.
3 Objective Functions
The paper compares token-level, sequence-level, and hybrid objective functions. Sequence-level objectives can optimize task metrics directly but require candidate generation and operate on a tractable subset of outputs.
- Objective-function scope: The compared losses operate over individual tokens, entire sequences, or combinations of tokens and sequences.Figure 1 provides an overview of these objective functions.
- Objective-function scope: Token-level objectives have been the predominant focus of prior sequence-to-sequence research.These losses compute the objective additively over individual tokens.
Token Negative Log Likelihood (TokNLL)
Token-level likelihood minimizes the negative log likelihood of the reference tokens and serves as the baseline objective for comparison.
- Definition: TokNLL minimizes the negative log likelihood of individual reference tokens.The reference sequence is denoted t = (t1, . . . , tn).
- Role: TokNLL is the most common loss function optimized in related work and provides the comparison baseline.
Token NLL with Label Smoothing (TokLS)
Token-level likelihood can make models overly confident, so TokLS regularizes predictions by smoothing the target distribution toward a prior. Sequence-level objectives instead require candidate generation but can directly optimize metrics such as BLEU or ROUGE.
- TokLS: Label smoothing regularizes likelihood training by reducing the model’s confidence in its predictions.It addresses the risk that extreme zero-or-one training predictions may hurt generalization.
- TokLS: Label smoothing is equivalent to adding DKL(f∥p(ti|t1, . . . , ti−1, x)) to the token negative log-likelihood.The prior distribution f is independent of the current input x.
- TokLS: TokLS replaces a one-hot target with q(u) = 1 − ϵ and q(u′) = ϵ for u′ ≠ u.The parameter ϵ controls how much probability mass is distributed away from the observed word.
- Sequence-level objectives: Sequence-level objectives generate and score multiple candidates, enabling direct optimization of BLEU or ROUGE at higher computational cost.Because the full output space is intractable, losses are computed over a model-generated subset U(x).
Sequence Negative Log Likelihood (SeqNLL)
SeqNLL applies likelihood training to whole output sequences rather than individual tokens, using length normalization to avoid favoring shorter outputs. Its pseudo reference is selected from generated candidates according to BLEU or ROUGE against the gold reference.
- SeqNLL: SeqNLL minimizes the negative log-likelihood of an entire output sequence rather than individual tokens.The sequence log-likelihood is formed from the token log probabilities.
- SeqNLL: Length normalization divides the sum of token log probabilities by the number of tokens to avoid bias toward shorter sequences.This produces a length-normalized sequence likelihood.
- Pseudo reference: The pseudo reference is the candidate with the highest BLEU or ROUGE score against the gold reference.Candidates are drawn from the model’s output set, and sentence-level BLEU uses smoothed initial counts except for unigrams.
Expected Risk Minimization (Risk)
Risk minimizes expected task-specific cost across candidate sequences, using metrics such as BLEU or ROUGE rather than concentrating on one output. It differs from REINFORCE by considering multiple candidates and using sequence-level expected cost, while the paper also evaluates complementary margin and combined objectives.
- Expected Risk Minimization (Risk): Risk minimizes expected task-specific cost over candidate sequences, using costs such as 1−BLEU(t, u).Unlike SeqNLL, it can increase scores for several low-cost candidates rather than focusing on one sequence.
- Expected Risk Minimization (Risk): A gold reference can produce degenerate solutions, so the method may instead use a pseudo-reference selected from candidate outputs.The pseudo-reference is chosen as the candidate with the highest BLEU score; using the gold reference can assign low probabilities to nearly all outputs.
- Expected Risk Minimization (Risk): Risk differs from REINFORCE by evaluating multiple sequences and estimating one sequence-level expected cost rather than relying on a single sample and word-specific baselines.Both objectives optimize an expected cost or reward, but their expectation estimates and gradient baselines differ.
- Margin Objectives: MaxMargin enforces a score margin between the highest-scoring candidate and a reference, with a validation-tuned scale β applied to the cost difference.The paper uses unnormalized pre-softmax model scores and replaces the human reference with a pseudo-reference in this setting.
- Margin Objectives: MultiMargin extends MaxMargin by enforcing margins between every candidate sequence and a reference, while SoftmaxMargin adds task costs inside SeqNLL’s exponential.SoftmaxMargin penalizes high-cost outputs proportionally to their costs.
- Combined Objectives: Weighted combines token-level and sequence-level losses, whereas Constrained selects one loss per input to retain token-level accuracy while optimizing at the sequence level.Constrained uses the sequence loss when the current token loss is at least as good as a fixed converged token-level baseline.
4 Candidate Generation Strategies
Because the full output space is intractable, training uses a generated subset of candidate sequences. Beam search supplies high-probability candidates, while sampling supplies more diverse alternatives; candidate refresh timing trades accuracy against speed and staleness.
- Candidate Sets: Sequence-level objectives operate on an intractable output space, so training uses K generated candidate sequences instead of enumerating all outputs.The candidate set is U(x) = {u1, . . . , uK}.
- Search Strategies: Beam search maintains the top-K candidates at each step, whereas sampling independently draws K sequences from the model’s conditional distribution.Beam search emphasizes high-probability candidates; sampling introduces greater candidate diversity.
- Online and Offline Generation: Online generation refreshes candidates whenever an input is encountered, while offline generation creates them before training and keeps them static.Offline generation is faster because pre-generated hypotheses support efficient batched computation, but its candidates can become stale.
- Candidate Set Design: Adding the reference translation to the candidate set can destabilize training by causing low probabilities across most outputs and degrading model-generated candidates.Accordingly, the paper excludes the reference translation from candidate sets.
5 Experimental Setup
The experiments evaluate sequence-level objectives on German-English translation, English-French translation, and Gigaword summarization under defined data, preprocessing, and model settings. The setup includes both a smaller comparable translation task and a much larger WMT task, with a convolutional encoder-decoder architecture used in most experiments.
- Translation: IWSLT’14 German-English uses 160K sentence pairs, BPE with 14,000 types, and case-insensitive BLEU evaluation.The test set concatenates the available IWSLT 2014 test and development sets.
- Translation: WMT’14 English-French retains 35.5M sentence pairs after length and source-target ratio filtering, using 40K BPE types.Results are reported on newstest2014 and a held-out validation set of 26,658 sentence pairs.
- Training Configuration: Sequence-level models are initialized from token-level models after 200 baseline epochs and then trained for another 10 to 20 epochs.The learning rate is annealed by factors of 10 until it falls below 10^-4.
- Training Configuration: Sequence-level scores and probabilities are length-normalized, training candidates are limited to fewer than 200 tokens, and experiments generally use 16 candidates per example.Ablation experiments use five candidates for faster turnaround.
- Summarization: Gigaword summarization uses 3.8M training examples, 190K validation examples, and a 2,000-pair test set evaluated with ROUGE-1, ROUGE-2, and ROUGE-L F1.The preprocessing matches the setup of Rush et al. (2015).
6 Results
Across translation and summarization experiments, sequence-level objectives consistently improve or remain competitive with token-level baselines and beam search optimization. Results also show practical trade-offs involving objective combination, initialization, candidate generation, and task scale.
- Sequence-level losses: All sequence-level losses outperform token-level losses on IWSLT’14 German-English translation, with Risk improving the baseline by up to 0.61 BLEU.The comparison reports averages and standard deviations over five runs.
- Objective combinations: Weighted combinations of TokLS and Risk outperform pure Risk, with α = 0.3 performing best among the tested combination strategies.The weighted strategy is used in the remaining experiments.
- Initialization: Initializing weighted Risk with TokLS yields 0.7-0.8 better BLEU than initialization with TokNLL.The authors attribute this advantage to regularization from label smoothing.
- Candidate generation: Offline candidate generation has 26 times higher throughput than online generation, although offline generation gives lower accuracy because candidates become stale.Online regeneration is slower because it requires incremental inference during training.
- Candidate generation: Beam search performs better than sampling for every candidate-set size considered, with 16 candidates balancing efficiency and accuracy.Beam search emphasizes high-probability candidates, whereas sampling produces more diverse candidates.
- Comparison with beam search optimization: Risk improves BLEU by +2.75 over the reimplemented baseline, slightly exceeding Beam Search Optimization’s reported +2.33 BLEU improvement.The comparison uses unnormalized beam search with k = 5.
- WMT’14 English-French: On WMT’14 English-French, sequence-level training adds +0.37 BLEU to a strong model, while Risk reaches 41.5 BLEU when trained on news-commentary data.Adding decoder self-attention produces a smaller +0.2 BLEU gain from Risk.
- Abstractive summarization: On Gigaword, TokLS outperforms prior approaches on ROUGE-2 and ROUGE-L, while Risk further improves the strong baseline.Weighted training exceeds Risk-only training on validation RG-1, RG-2, and RG-L.
7 Conclusion
The paper compares classical structured-prediction losses on a strong neural sequence-to-sequence model and finds them highly competitive. Best performance requires combining sequence- and token-level losses and training on candidates decoded by the current model, yielding gains on translation and summarization benchmarks.
- The paper presents a comprehensive comparison of classical structured-prediction losses applied to a strong neural sequence-to-sequence model.
- Combining sequence-level and token-level losses is necessary for best performance.
- Training on candidates decoded with the current model is also necessary for best performance.
- Classical expected risk can slightly outperform beam search optimization in a like-for-like setup.