PaperlayerView exact revision on arXiv

Source-linked AI summary

Text Generation by Learning from Demonstrations

Richard Yuanzhe Pang, He He

arXiv:2009.07839v2cs.CLcs.LG

TL;DR

Text generation by autoregressive MLE faces mismatches between likelihood-based training and quality evaluation, and between gold histories in training and model-generated histories at inference. The paper proposes GOLD, an offline RL method that learns from reference demonstrations using importance weighting. Across summarization, question generation, and machine translation, GOLD outperforms MLE and RL fine-tuning by task metrics and human-rated quality.

  • Problem

    Autoregressive MLE trains on likelihood while generation is evaluated for output quality, and its gold-prefix training histories differ from model-generated inference histories.

    Source: §1 · p. 1

  • Method

    GOLD frames text generation as offline RL from reference demonstrations and uses importance weighting with rewards that estimate human likelihood.

    Source: §1 · pp. 1–2

  • Results

    GOLD outperforms MLE and RL fine-tuning on summarization, question generation, and machine translation by task metrics and human-rated quality.

    Source: §1 · p. 2

  • Takeaways & Limitations

    GOLD models are less sensitive to decoding algorithms and alleviate exposure bias, with output quality degrading little as generation length increases.

    Source: §1 · p. 2

    The paper focuses on generating a single high-quality output and leaves generating outputs that are both diverse and high-quality to future work.

    Source: §4.4 · p. 8

Abstract

from arXiv · show

Current approaches to text generation largely rely on autoregressive models and maximum likelihood estimation. This paradigm leads to (i) diverse but low-quality samples due to mismatched learning objective and evaluation metric (likelihood vs. quality) and (ii) exposure bias due to mismatched history distributions (gold vs. model-generated). To alleviate these problems, we frame text generation as an offline reinforcement learning (RL) problem with expert demonstrations (i.e., the reference), where the goal is to maximize quality given model-generated histories. We propose GOLD (generation by off-policy learning from demonstrations): an easy-to-optimize algorithm that learns from the demonstrations by importance weighting. Intuitively, GOLD upweights confident tokens and downweights unconfident ones in the reference during training, avoiding optimization issues faced by prior RL approaches that rely on online data collection. According to both automatic and human evaluation, models trained by GOLD outperform those trained by MLE and policy gradient on summarization, question generation, and machine translation. Further, our models are less sensitive to decoding algorithms and alleviate exposure bias.

1 INTRODUCTION

The introduction identifies mismatches between likelihood-based training and quality-based evaluation, and between gold histories in training and model-generated histories at inference. It proposes GOLD, which learns from reference demonstrations through off-policy importance weighting, and reports gains across three generation tasks.

1 INTRODUCTION

  • Autoregressive MLE training uses negative log-likelihood while evaluation judges output quality, and model misspecification can give probability mass to both high- and low-quality sequences.The authors say this makes careful decoding-algorithm selection necessary in practice.Source: §1 · p. 1
  • Training conditions on gold prefixes but inference conditions on model-generated histories, creating exposure bias that can compound errors and contribute to repetition and hallucination.The cited prior work is Ranzato et al. (2016), Bengio et al. (2015), Ross et al. (2011), Holtzman et al. (2020), and Wang & Sennrich (2020).Source: §1 · p. 1
  • GOLD frames generation as offline RL from reference sequences and uses importance weighting to give more weight to examples with higher model probability.Its reward functions estimate how likely a human would have generated a sequence.Source: §1 · pp. 1–2
  • GOLD outperforms MLE and RL fine-tuning on news summarization, question generation, and machine translation by task metrics and human-rated quality.The paper also reports less sensitivity to decoding algorithms and reduced exposure bias, with output quality degrading little as generation length increases.Source: §1 · p. 2

2 FROM MLE TO RL FRAMEWORK

This section contrasts MLE's likelihood-based training with an evaluation target based on human-perceived output quality. It then formulates generation as sequential decision-making, where the policy seeks expected reward over model-induced trajectories.

2 FROM MLE TO RL FRAMEWORK

  • MLE models a sequence as tokens generated sequentially from a conditional distribution given the context and preceding tokens.At inference, tokens are generated sequentially according to the model distribution.Source: §2 · p. 2
  • The evaluation target is perceptual quality: how likely a human would have generated the output given its context.The section notes that practical task quality may involve fluency, correctness, and interestingness.Source: §2 · p. 2
  • MLE encourages high recall by assigning probability to human-generated sequences, whereas evaluation favors high precision across model outputs.Because human likelihood is unknown and its expectation is difficult to estimate, the paper develops an RL training objective to approximate the evaluation target.Source: §2 · p. 2
  • The RL formulation treats each generated token as an action and the context-plus-prefix as a state, with the generation model serving as the policy.A sequence is represented as a trajectory, and the objective is expected reward under trajectories induced by the policy; oracle human-likelihood rewards would match the desired evaluation metric.Source: §2 · p. 2

3 APPROACH

The approach uses off-policy policy gradients with demonstrations and reward estimates to train generation policies without relying on online exploration. GOLD approximates importance weights and designs rewards from estimated human likelihood, with implementation choices intended to stabilize training.

3.1 OFF-POLICY POLICY GRADIENT

  • Policy-gradient training can get stuck in zero-reward regions, while MLE initialization or interleaved MLE updates can bias parameters toward MLE and often yield marginal gains.The return is estimated from future rewards, and the expectation is estimated with Monte Carlo trajectories from the policy.Source: §3.1 · p. 3
  • Offline learning uses static demonstrations without further environment interaction, a fit for text generation because token transitions are known and exploration has limited benefit without a reliable reward for novel outputs.The paper argues that rewards are difficult to estimate for possible generations beyond the demonstrations.Source: §3.1 · p. 3
  • In offline learning, policy expectations must be estimated from trajectories generated by a different behavior policy, motivating importance sampling.The per-action approximation is described as sensitive to optimization hyperparameters and slower to converge; an empirical demonstration distribution is used so the behavior-policy denominator is constant and can be ignored.Source: §3.1 · p. 3
  • The approximate gradient upweights actions with high return and actions preferred by the current policy, focusing learning on easier examples to improve precision.The per-action weighting approximation is biased, but the text reports it can reduce variance and work when behavior and target policies are close.Source: §3.1 · p. 3

3.2 REWARD

  • The δ-reward treats all demonstrations as equally good and assigns zero reward to other outputs, with reward delivered at a demonstrated trajectory's terminal state.This is a sequence-level reward design.Source: §3.2 · p. 4
  • Estimated human likelihood is approximated by pMLE on demonstrations, and the paper defines reward functions based on that estimate.The product-based reward gives high sequence reward only when every word has high likelihood; a summed-probability alternative allows later high-reward actions to compensate for earlier bad decisions.Source: §3.2 · p. 4

3.3 THE GOLD ALGORITHM

  • GOLD initializes the policy from MLE, periodically synchronizes a weighting policy, and lower-bounds importance weights to avoid drastic changes.It truncates future trajectories after five steps, subtracts a baseline to reduce variance, and lower-bounds pMLE rewards; the authors say GOLD training usually takes less time than MLE given an existing pMLE.Source: §3.3 · p. 4

8 Return: πθ

GOLD outperforms MLE on reported automatic metrics across standard and transformer settings. The analysis describes its precision–recall trade-off and examines human judgments, decoding, exposure bias, on-policy training, and diversity.

8 Return: πθ

  • The experiments cover question generation, summarization, extreme summarization, and German-to-English translation, with automatic metrics and human ratings reported for NQG and summarization.The models include standard baselines and GOLD variants, with policy-gradient training used as an on-policy comparison.Source: §8 · pp. 4–5

4.2 RESULTS AND ANALYSIS

  • GOLD scores better than MLE across standard-model tasks and transformer tasks, including pretrained-model settings.Table 1 reports standard-model test results, and Table 3 covers transformer models.Source: §4.2 · p. 5, Table 1 · p. 5, Table 3 · p. 6
  • Workers preferred GOLD-trained outputs over MLE-trained outputs more often in the reported human comparisons.Table 5 compares GOLD-trained and MLE-trained BART outputs for the same source.Source: §4.2 · p. 6, Table 5 · p. 7
  • GOLD’s token-loss distribution concentrates on near-zero losses with a long high-loss tail, consistent with higher precision at the expense of reference recall.The authors attribute its better metric scores to focusing on easy-to-learn tokens.Source: §4.2 · pp. 6–7, Figure 1 · p. 6
  • GOLD performs similarly with argmax decoding and top-k sampling, whereas MLE suffers significantly from sampling.The authors relate GOLD’s reduced decoding dependence to its high-precision models.Source: Table 2 · p. 5, §4.2 · p. 7
  • On NQG, MLE’s human-rated output quality drops beyond 14 words, while GOLD-s stays relatively stable across lengths.The authors conclude GOLD suffers less from exposure bias; gold-prefix NLL also varies little as length increases.Source: §4.2 · p. 7, Figure 2 · p. 7

4.3 COMPARISON WITH ON-POLICY TRAINING

  • On-policy fine-tuning marginally improves MLE and GOLD, but MLE with policy gradient remains worse than GOLD and the overall benefit is unclear.The comparison uses BLEU and ROUGE-2 rewards for NQG and CNN/DM, respectively.Source: Table 4 · p. 6, §4.3 · p. 8

4.4 DISCUSSION ON GENERATION DIVERSITY

  • GOLD trades recall for precision and may limit diverse high-quality outputs in creative or open-ended settings, while measured linguistic complexity remained comparable to MLE.The discussion frames generating a single high-quality output as reasonable for most conditional text-generation tasks and leaves jointly diverse, high-quality generation for future work.Source: §4.4 · pp. 8–9

5 RELATED WORK

The related work addresses exposure bias, sequence-level reward optimization, and high-precision generation, while highlighting the costs and optimization challenges of existing approaches. GOLD differs by using demonstrations and generic reward functions.

5 RELATED WORK

  • Exposure-bias methods based on global sequence scores can be complex and costly, while learning-to-search requires annotations or costs for non-reference sequences.The cited work describes these requirements as potential barriers when such information is unavailable for text generation.Source: §5 · p. 9
  • Policy-gradient methods optimize sequence-level metrics, but discriminator-reward approaches often underperform MLE due to optimization challenges.Off-policy RL has also been used in dialogue, where online user interaction is expensive.Source: §5 · p. 9
  • Prior high-quality-generation work selects outputs during decoding or discourages implausible samples during training; this paper instead proposes a general learning framework for mismatched objectives.Examples include negative sampling, self-training on high-quality samples, and confidence-oriented decoding with calibration.Source: §5 · p. 9

6 CONCLUSION

The conclusion presents off-policy learning as an efficient way to address objective and history mismatches in text generation. The appendix describes four evaluated tasks, their dataset splits, and preprocessing resources.

6 CONCLUSION

  • The paper concludes that efficient off-policy learning addresses MLE’s objective and history discrepancies, with optimization advantages like MLE.The authors suggest that techniques such as proximity constraints could be integrated to improve performance further.Source: §6 · p. 9

A.1 TASKS AND DATASETS

  • Evaluations cover question generation, extractive and abstractive summarization, and German-to-English translation, with distinct output requirements across these tasks.CNN/DM targets tend to be extractive, XSum targets highly abstractive, and the translation task is described as having a smaller high-quality output space.Source: §A.1 · p. 15
  • Train/dev/test splits are 86229/8913/8919 for NQG, 287227/13368/11490 for CNN/DM, 204045/11332/11334 for XSum, and 160239/7283/6750 for IWSLT14 De-En.Source: §A.1 · p. 15
  • The appendix identifies preprocessing instructions for NQG, summarization, and translation datasets and points readers to the project codebase.Source: §A.1 · p. 15

A.2 MODEL ARCHITECTURES

The experiments use standard sequence-to-sequence architectures and transformer models, including pretrained BART for several tasks and a separate transformer for translation.

A.2 MODEL ARCHITECTURES

  • Experiments use two architecture sets: standard models and transformers.Source: §A.2 · p. 15
  • Standard models are NQG++ with GRU for question generation and a pointer-generator LSTM for summarization, each using two encoder and decoder layers.Source: §A.2 · p. 15
  • BART is tested on NQG, CNN/DM, and XSum; IWSLT14 De-En uses a moderate-size transformer with 512-dimensional embeddings and six encoder and decoder layers.The authors use pretrained BART to test whether their methods work with stronger-capability models.Source: §A.2 · p. 15

A.3 MORE ON REPRODUCIBILITY

The appendix documents implementation settings, model scale, runtime, and approximation choices. Its Q-value experiments compare future-trajectory strategies and report similar results for longer horizons, with fixed-step approaches shortening training time.

A.3 MORE ON REPRODUCIBILITY

  • The appendix provides the project code location and reports that hyperparameters were tuned over a small set of values, so more careful tuning may slightly improve performance.Source: §A.3 · pp. 15–16
  • Standard architectures use a 5e-4 learning rate; BART-based NQG, CNN/DM, and XSum use 2e-5, while IWSLT14 De-En uses 3e-4.For standard models, the pMLE lower bound is 0.1 for NQG, 0.01 for CNN/DM GOLD-p, and 0 for CNN/DM GOLD-s; transformer settings use 0.01 for GOLD-p and 0.1 for GOLD-s. Synchronization intervals are 1,500 steps for NQG; for CNN/DM, 1,500 for GOLD-δ and GOLD-p and 5,000 for GOLD-s; transformer intervals are 1,000 for NQG and 5,000 for CNN/DM, XSum, and IWSLT14 De-En.Source: §A.3 · pp. 15–16
  • For standard models, the appendix reports 10372565 parameters for NQG++ and 19965705 for pointer-generator; transformer models have 406290432 parameters for BART and 39469056 for IWSLT14 De-En.Source: §A.3 · p. 16
  • Reported per-epoch training times range from about 3 minutes for IWSLT14 De-En to 11 hours for CNN/DM transformers on P40 GPUs.The appendix notes that V100 GPUs would likely reduce transformer training time per epoch.Source: §A.3 · p. 16

A.4 MORE DISCUSSION ON APPROXIMATIONS

  • The future trajectory is truncated after five steps, bounding current-plus-future steps at six and effectively using a 0.83 discount factor.Source: §A.4 · p. 16
  • Using the entire future trajectory on NQG gave similar performance but longer training than the five-future-step approach; fixed horizons of 5, 7, or 10 steps gave similar, slightly better results than 1, 2, or 3.Fixed-step approaches also allowed constant baselines and shorter training time.Source: §A.4 · p. 17

A.5 DETAILS ON ON-POLICY EXPERIMENTS

The appendix reports tests of on-policy baselines and notes that better tuning could strengthen GOLD+PG models. It also compares transformer summarization with lead-3 baselines.

A.5 DETAILS ON ON-POLICY EXPERIMENTS

  • On-policy baselines used REINFORCE with sequence-level BLEU or ROUGE-2 rewards; MLE initialization with interpolated policy-gradient losses was more stable than random initialization with MIXER.The random-initialization MIXER approach did not improve on the interpolated setup under the tested architecture.Source: §A.5 · p. 17
  • MIXER with a learned baseline yielded only slight NQG improvement (BLEU 14.71), while GOLD avoids learning baselines that can affect on-policy performance.Source: §A.5 · p. 17
  • GOLD+PG experiments used only constant baselines, and the authors note that better baseline tuning could potentially improve performance.Source: §A.5 · p. 17

B.1 LEAD-3 BASELINES FOR SUMMARIZATION

  • Transformer models beat lead-3 on CNN/DM and XSum by a large margin; lead-3 uses each source's first three sentences as its summary.Source: §B.1 · p. 17

B.2 PERFORMANCE WITH TRANSFORMER ARCHITECTURES

Transformer experiments report strong automatic-metric results, including CNN/DM comparisons with prior systems and a BLEU gain on IWSLT14 De-En. Analyses also find more stable quality with longer GOLD generations, and the appendix provides example outputs.

B.2 PERFORMANCE WITH TRANSFORMER ARCHITECTURES

  • CNN/DM ROUGE-1/2/L reached 45.40/22.01/42.25 with GOLD-p and 44.82/22.09/41.81 with GOLD-s.Both exceeded PEGASUS (44.17/21.47/41.11) and ProphetNet (44.20/21.17/41.30) on automatic metrics; on XSum, GOLD exceeded PEGASUS trained on C4 (45.20/22.06/36.99) but trailed its Huge-News result.Source: §B.2 · p. 17
  • GOLD-s improved IWSLT14 De-En BLEU by 0.81 points and outperformed approaches without knowledge distillation or data augmentation, as far as the authors were aware.Source: §B.2 · p. 17

B.3 MORE ON EXPOSURE BIAS

  • MLE BLEU falls with generation length while GOLD-s stays relatively stable; gold-history token accuracy also stays length-stable, supporting exposure bias as the main source of MLE's decline.Figure 3 compares sentence-level BLEU by length, while Figure 4 reports token prediction accuracy conditioned on gold histories.Source: §B.3 · p. 18, Figure 3 · p. 18, Figure 4 · p. 19

B.4 EXAMPLES

C.1 PAIRWISE COMPARISON

Human pairwise comparisons favored GOLD-s over MLE in standard-model NQG evaluations. The section also describes how workers compared outputs and how NQG generations were rated for quality.

C.1 PAIRWISE COMPARISON

  • NQG workers compared generated questions given the source paragraph and queried words; summarization workers judged closeness to the reference because reading full articles was infeasible.Source: §C.1 · p. 21
  • In 183 standard-model NQG comparisons, workers preferred GOLD-s in 44.3% of pairs, MLE in 23.0%, and tied 32.8%.Source: §C.1 · p. 21

C.2 NQG RATING

  • NQG ratings sampled 736 annotations, with at least 30 sentences per two-length bucket for each model.Workers rated correctness and fluency/coherence on a four-point scale.Source: §C.2 · pp. 21–22

Related papers

Loading 2009.07839v2…