Source-linked AI summary
Calibrating Sequence likelihood Improves Conditional Language Generation
Yao Zhao, Misha Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, Peter J. Liu
TL;DR
MLE-trained conditional language models often lack calibrated sequence likelihoods for ranking generated quality because training data provides sparse target supervision. SLiC calibrates candidate likelihoods against target similarity in the model’s latent space, producing strong results across generation tasks while eliminating decoding heuristics and retaining benefits as models scale.
Problem
Sparse single-target supervision leaves MLE-trained models without direct comparison signals for ranking plausible generated sequences by quality.
Method
SLiC decodes candidates from a fine-tuned model and continues training to align their sequence likelihoods with target similarity in the model’s latent space.
Results
SLiC exceeds or matches state-of-the-art results across summarization, question answering, question generation, and data-to-text generation, while eliminating decoding heuristics.
Takeaways & Limitations
Calibration benefits persist as models scale, and smaller calibrated models might outperform larger counterparts under the same inference compute budget.
Abstract
from arXiv · showhide
Conditional language models are predominantly trained with maximum likelihood estimation (MLE), giving probability mass to sparsely observed target sequences. While MLE trained models assign high probability to plausible sequences given the context, the model probabilities often do not accurately rank-order generated sequences by quality. This has been empirically observed in beam search decoding as output quality degrading with large beam sizes, and decoding strategies benefiting from heuristics such as length normalization and repetition-blocking. In this work, we introduce sequence likelihood calibration (SLiC) where the likelihood of model generated sequences are calibrated to better align with reference sequences in the model's latent space. With SLiC, decoding heuristics become unnecessary and decoding candidates' quality significantly improves regardless of the decoding method. Furthermore, SLiC shows no sign of diminishing returns with model scale, and presents alternative ways to improve quality with limited training and inference budgets. With SLiC, we exceed or match SOTA results on a wide range of generation tasks spanning abstractive summarization, question generation, abstractive question answering and data-to-text generation, even with modest-sized models.
1 INTRODUCTION
Conditional language generation commonly uses MLE, but sparse single-target supervision leaves sequence likelihood poorly calibrated for ranking generated quality. SLiC addresses this by calibrating candidate likelihoods to latent-space similarity, improving generation across tasks and model scales while removing decoding heuristics.
- Conditional generation covers summarization, question answering, question generation, and data-to-text tasks, commonly using pretrained encoder-decoder models fine-tuned downstream.
- MLE factorizes sequence probability into autoregressive token predictions, training models on observed target sequences.The sequence probability is modeled as a product of conditional next-token probabilities.
- Single-target training data leaves models without direct supervision for comparing plausible sequences, weakening alignment between sequence likelihood and quality.Prior work reports low correlation between sequence probability and quality, with exposure bias further increasing likelihood-estimation noise.
- SLiC decodes candidates from a fine-tuned model, then continues training to align their likelihoods with target similarity in the model’s latent space.The approach extends pretraining and fine-tuning with an offline calibration stage and avoids a separate reranking model.
- SLiC exceeds or matches state-of-the-art results across summarization, question answering, question generation, and data-to-text generation tasks.
- SLiC removes decoding heuristics, retains significant benefits as models scale, and may let smaller calibrated models outperform larger models under equal inference budgets.
2 CALIBRATING SEQUENCE LIKELIHOOD
SLiC adds a calibration stage that trains model likelihoods to rank decoded candidates according to their similarity to the target, while retaining the fine-tuned model’s behavior through token-level regularization.
- Calibration stage: SLiC decodes candidates from a fine-tuned model, then continues training with calibration and regularization losses.The calibration stage extends the usual pretraining and fine-tuning paradigm.
- Similarity function: Candidate similarity is computed from decoder hidden-state representations using cosine matching over spans of 1, 2, 4, and 8 tokens.The scoring function aggregates span-level similarities with an F-measure-based function.
- Calibration loss: The calibration objective aligns candidate sequence likelihoods with their similarity to the target sequence.It supports pairwise ranking, margin, list-wise ranking, and expected reward loss types.
- Regularization loss: Regularization uses token-level cross-entropy or KL divergence to prevent substantial deviation from the fine-tuned MLE objective.Cross-entropy is the standard fine-tuning MLE objective, while KL divergence matches token-level probability distributions.
- Candidate decoding methods: SLiC can use beam search, diverse beam search, or nucleus sampling to generate candidates.These methods respectively target likelihood optimization, diverse outputs, or diverse candidates restricted to high-probability tokens.
3 EXPERIMENTS
Experiments show that sequence likelihood calibration consistently improves conditional generation across tasks, decoding methods, model sizes, and inference budgets. Calibrated models eliminate common decoding heuristics and can match or exceed strong baselines with smaller models.
- Experimental Setup: Experiments cover summarization, generative question answering, question generation, and data-to-text tasks using PEGASUS models from 50M to 2B parameters.Final results use PEGASUS2B on eight datasets, while ablations, benefits analyses, and scaling experiments use four summarization datasets.
- Ablation Studies: All calibration loss types significantly improve over fine-tuned-only models, with rank loss performing best among the tested objectives.The reported ordering is rank, margin, list rank, then reward.
- Ablation Studies: About 85% of the calibration gain remains without regularization, while cross-entropy and KL-divergence regularization perform similarly.Regularization losses constrain deviation from the fine-tuned objective; the recommended recipe uses KL divergence.
- Ablation Studies: The optimal calibration decoding method is dataset-dependent, but the worst method achieves 90% of the best method’s gains and beam search has the highest average quality.Differences between candidate-generation methods are small, and the paper recommends beam search for calibration candidates.
- Benefits of Calibrated Sequence Likelihood: Calibrated models improve monotonically as decoding candidates increase, unlike fine-tuned-only models, and do not require length normalization or repetition blocking.Calibrated models also show much lower repetition rates, while fine-tuned-only models can degrade or develop more repetitions under heuristic decoding.
- Scaling Properties of Calibrated Models: Calibration benefits persist from 50M to 2B parameters, and smaller calibrated models can outperform larger models under equal inference budgets by decoding more candidates.Returns from additional candidates usually diminish after ten candidates, but candidate decoding can improve quality and efficiency.
- Final Results: Calibrated models consistently improve over fine-tuned-only models and exceed or match state-of-the-art results across all evaluated datasets.On XSUM, SAMSum, WebNLG-en, and CommonGen, calibrated 2B models are ten to a hundred times smaller than the cited state-of-the-art models.
4 RELATED WORKS
Prior work addresses sequence-generation quality through reinforcement learning, reranking, sequence-level objectives, and semantic smoothing. These approaches differ in whether they optimize external rewards, rank candidates separately, or incorporate sequence-level supervision into generation.
- Reinforcement Learning: Reinforcement-learning approaches optimize sequence-level rewards, including ROUGE or human-preference reward models, but direct ROUGE optimization can reduce fluency.Mixed token-level and sequence-level objectives were reported to improve readability in one cited approach.
- Candidate Reranking: Two-stage systems generate candidate sequences and rank them with separate quality-estimation or reranking models.Examples include diverse-beam candidate generation with a RoBERTa ranker and sequence-to-sequence rerankers.
- Sequence-Level Objectives: Multi-task and sequence-level learning combine token-level generation with candidate-ranking or structured-prediction objectives to improve translation and summarization.BRIO determines ranking order using external metrics such as ROUGE and BERTScore, while other work combines sequence-level and token-level losses.
- Contrastive and Smoothing Methods: Contrastive and semantic label-smoothing methods adjust likelihoods for generated or semantically related sequences relative to gold references.These methods decrease likelihood for model-generated silver summaries or smooth over relevant sequences similar to the target.
5 CONCLUSION
The paper presents SLiC as a third training stage that calibrates candidate likelihoods using latent-space similarity to targets. It reports heuristic-free decoding, persistent scaling benefits, and state-of-the-art or competitive results across eight generation datasets.
- Conclusion: SLiC adds a third stage after pretraining and fine-tuning, decoding candidates and aligning their likelihoods with target similarity in the model’s latent space.The recommended recipe selects checkpoints by perplexity, uses beam-search candidates, rank loss, and KL-divergence regularization.
- Conclusion: SLiC eliminates decoding heuristics, retains benefits as models scale, and may let smaller calibrated models outperform larger models under equal inference budgets.The conclusion specifically identifies beam-size optimization, length normalization, and repetition prevention as unnecessary for calibrated models.
- Conclusion: Calibrated PEGASUS2B models exceed or match state-of-the-art results on eight datasets spanning summarization, question answering, question generation, and data-to-text generation.The conclusion frames these results as spanning multiple conditional language-generation task families.
A.1 DATASETS AND TASKS
The evaluation spans diverse datasets and tasks covering summarization, question generation, question answering, and data-to-text generation.
- Summarization: The summarization evaluation uses CNN/DailyMail, XSUM, RedditTIFU-long, and SAMSum.These datasets vary in domain, style, abstractiveness, and summary length.
- Question generation: SQuAD QG generates questions from passage–answer pairs, using 75,722 training, 10,570 validation, and 11,877 test examples.
- Question answering: MSMARCO NLG generates well-formed answers from input queries and answering passages.The NLGEN subset contains 182,669 queries with well-formed answers.
- Data-to-text: WebNLG-en verbalizes RDF-triple sets into natural-language texts, with 16,095 inputs and 38,872 data–text pairs.
- Data-to-text: CommonGen generates coherent sentences describing input sets of common concepts.The dataset contains 35,141 concept sets split into training, validation, and test sets.
- Dataset statistics: Table 4 summarizes the statistics of the datasets used in the evaluation.
B MODEL ARCHITECTURE
The section reports model sizes and configurations used in the experiments.
- Model configurations: Model sizes and their configurations are reported in Table 5.
- Model configurations: The section provides configuration information alongside model-size reporting.
- Model configurations: Table 5 presents the model-size information for the evaluated systems.
C ABLATION STUDY
The ablation-study section reports the experimental settings used to analyze SLiC methods.
- Ablation settings: SLiC methods used in the ablation study are reported in Table 6.
- Ablation settings: The ablation analysis uses reported SLiC method configurations as its experimental basis.
- Ablation settings: Table 6 presents the experimental settings for the ablation studies.
D DECODING METHODS
The section describes decoding and length-normalization settings for calibrated models, with evaluation across multiple candidate counts and ROUGE curves.
- Decoding evaluation: At evaluation time, calibrated models are decoded with 1, 2, 5, 10, and 20 candidates.
- Decoding evaluation: ROUGE 1/2/L values for the decoding curves are reported in Table 8.
- Length normalization: Length-normalization experiments select the brevity penalty α using validation-set ROUGE for fine-tuned models or disable it.
- Computation: The FLOPs estimate accounts for half-length effective decoder attention context and omits smaller decoding-method computation costs.
G SCALING
The scaling experiments evaluate SLiC with multiple decoding candidate counts and report ROUGE scaling curves. The corresponding experimental settings and ROUGE metrics are organized in Tables 10 and 11.
- Experimental setup: Scaling experiments evaluate models with 1, 2, 5, 10, and possibly 15 or 20 decoding candidates.The passage describes the candidate counts used at evaluation time, with uncertainty for the largest settings.
- Experimental setup: Table 10 contains the experimental settings for the scaling experiments.
- Scaling results: Table 11 reports ROUGE R1, R2, and RL numbers for the scaling curve shown in Figure 3.
H FINAL RESULTS
The final-results experiments use SLiC settings selected according to subsection 3.3 and report a length-normalization study. Hyperparameter choices include fixed β and lr × λ values, with the best learning rate tuned on validation data.
- Final-results setup: The final-results SLiC configuration is selected according to subsection 3.3.
- Hyperparameters: SLiC uses learning rate lr, ranking constant β, and regularization strength λ as its three hyperparameters.
- Hyperparameters: β is fixed at 10 and lr × λ is fixed at 1e−5, while lr is tuned on the validation set.
- Final-results setup: The experiments include a length-normalization study, with its settings reported in Table 12.
- Hyperparameters: Table 13 reports the learning rates used for the final results.