PaperlayerView exact revision on arXiv

Source-linked AI summary

Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation

Kushal Arora, Layla El Asri, Hareesh Bahuleyan, Jackie Chi Kit Cheung

arXiv:2204.01171v3cs.CLcs.AIcs.LG

TL;DR

Language generation models exhibit repetition, incoherence, and hallucinations, and exposure bias has been proposed as a possible source. The paper analyzes this issue through imitation learning and finds error accumulation under teacher forcing, while showing that perplexity does not capture it and that lower accumulation accompanies better generation quality.

  • Problem

    Generation models exhibit repetition, incoherence, and hallucinations, while the proposed link between these behaviors and training–generation mismatch remains a question to examine.

    Source: §1 · p. 1

  • Method

    The paper uses an imitation-learning perspective to formalize exposure bias as error accumulation and relate teacher forcing to behavior cloning.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Teacher-forced models accumulate errors, perplexity fails to capture that accumulation, and decoding techniques associated with better generation quality also show lower error accumulation.

    Source: §1 · p. 2, §9 · p. 9

  • Takeaways & Limitations

    Excess error accumulation due to exposure bias might be a factor affecting language-generation quality.

    Source: §9 · p. 9

    Entropy rate captures the model’s average per-step generalization gap but fails to capture error accumulation from reducing sequential decision-making to supervised learning.

    Source: §7.2 · p. 6

Abstract

from arXiv · show

Current language generation models suffer from issues such as repetition, incoherence, and hallucinations. An often-repeated hypothesis is that this brittleness of generation models is caused by the training and the generation procedure mismatch, also referred to as exposure bias. In this paper, we verify this hypothesis by analyzing exposure bias from an imitation learning perspective. We show that exposure bias leads to an accumulation of errors, analyze why perplexity fails to capture this accumulation, and empirically show that this accumulation results in poor generation quality. Source code to reproduce these experiments is available at https://github.com/kushalarora/quantifying_exposure_bias

1 Introduction

Language models have advanced across generation tasks, yet open-ended generation still exhibits degenerate behaviors often attributed to a mismatch between training and generation. The paper formalizes exposure bias through error accumulation and uses that framing to analyze its relationship to generation quality.

1 Introduction

  • Open-ended settings such as text completion and dialog modeling still show repetition, low diversity, dullness, and incoherence despite models’ success on conditional generation tasks.The passage attributes these behaviors to a mismatch between maximum-likelihood training and generation, citing Welleck et al. (2019), Choi et al. (2020), and Li et al. (2016).Source: §1 · p. 1
  • Exposure bias means training uses ground-truth contexts while generation conditions on the model’s own previous tokens, creating a context-distribution mismatch.The passage associates this mismatch with Ranzato et al. (2016) and Bengio et al. (2015).Source: §1 · p. 1
  • Prior work speculated that exposure-bias errors could compound into incoherence, low vocabulary diversity, hallucinations, and repetition.The cited speculation includes Welleck et al. (2019), Choi et al. (2020), and Li et al. (2016).Source: §1 · p. 1
  • The paper analyzes exposure bias through imitation learning, showing that behavior cloning is equivalent to teacher forcing under a particular loss and borrowing its error-accumulation bound.This formalizes exposure bias in terms of accumulated errors and supports analysis of its impact on generation quality.Source: §1 · p. 2
  • Teacher-forced models accumulate errors; analytically and empirically, perplexity misses this accumulation, while lower exposure bias correlates with better generation quality.The paper uses a quantifiable definition of exposure bias to examine these relationships.Source: §1 · p. 2

2 Language Generation Formulation

The section formulates language generation as producing a variable-length sequence from a language model, optionally conditioned on a source. It also describes maximum-likelihood training as learning next-token distributions from corpus contexts.

2 Language Generation Formulation

  • Language generation produces a variable-length token sequence from a language model, either unconditionally or conditioned on a source, using a decoding algorithm.The formulation specifies a finite vocabulary.Source: §2 · p. 2
  • Language modeling learns a parameterized model that approximates an oracle, while maximum-likelihood training factorizes sequence probability and minimizes corpus negative log-likelihood.The training loss is defined over the corpus, whose size is measured in tokens.Source: §2 · p. 2
  • In the greedy strategy, each next token is the model’s most probable prediction given the current context, and generation stops at the maximum length or an EOS token.The chosen token is appended to form the context for the next prediction.Source: §2 · p. 2

3 An Imitation Learning Perspective of Language Generation

The paper casts language generation as sequential decision-making and language modeling as imitation learning. This framing connects teacher forcing to behavior cloning and provides a basis for analyzing error accumulation.

3 An Imitation Learning Perspective of Language Generation

  • Language generation fits sequential decision-making: the model is a policy, contexts are states, next-token predictions are actions, and concatenation supplies the transition.During generation, earlier predictions affect later ones, so errors can cascade into incoherent sequences.Source: §3 · pp. 2–3
  • Imitation learning trains a policy to reproduce an oracle using either oracle access or data generated by the oracle.In this framing, the language model is the policy and language generation is an imitation-learning instance.Source: §3 · p. 3
  • Behavior cloning reduces sequential decision-making to supervised multiclass classification using training state-action pairs assumed to be independently and identically distributed.The section identifies teacher forcing with behavior cloning under a particular loss function.Source: §3 · pp. 2–3
  • Under the paper’s per-step loss definition, behavior-cloning loss is equivalent to teacher-forcing loss up to a constant, so minimizing either yields the same model.The equivalence depends on defining the oracle’s per-step loss as zero.Source: §3 · p. 3

4 Exposure Bias and Error Accumulation

The section applies imitation-learning regret analysis to exposure bias in language generation. It explains how context drift can compound errors and gives bounds whose worst-case growth is quadratic in sequence length.

4 Exposure Bias and Error Accumulation

  • Exposure bias describes the mismatch between maximum-likelihood training on ground-truth contexts and generation on the model’s own prior tokens, which was hypothesized to accumulate errors.The passage attributes this definition and hypothesis to Ranzato et al. (2016).Source: §4 · p. 4
  • A generated token can make the next context unfamiliar, increasing the chance of further faulty predictions as context drifts from what the oracle would produce.This self-reinforcing drift is described as an accumulation of errors.Source: §4 · p. 4
  • The regret bound’s upper end assumes worst-case linear error growth at each step, yielding quadratic accumulated growth in maximum sequence length T.Its lower bound assumes no error accumulation and an expected error of ϵ at each step.Source: §4 · p. 4
  • The section borrows imitation-learning regret bounds to quantify language-generation error accumulation due to exposure bias.The bound is applied to a model and decoding method up to a specified sequence length.Source: §4 · p. 4

5 Quantifying Error Accumulation due to Exposure Bias

The section defines two metrics for quantifying error accumulation due to exposure bias: one tracks its growth with sequence length, and the other measures excess errors attributable to exposure bias. These metrics support comparisons across models, training algorithms, and decoding strategies.

5 Quantifying Error Accumulation due to Exposure Bias

  • AccErr≤(l) should grow super-linearly with sequence length l if exposure bias leads to error accumulation.The metric captures error growth with sequence length.Source: §5 · p. 5
  • %ExAccErr≤(l) measures the percentage of the model’s excess errors attributable to exposure bias.It uses the lower bound on regret—the minimum errors (ε per step) without error accumulation—as its reference.Source: §5 · p. 5
  • %ExAccErr≤(l) allows comparisons of excess errors across models, training algorithms, and decoding strategies.Perfect mitigation of exposure bias yields zero excess accumulated errors.Source: §5 · p. 5
  • The paper uses these definitions to examine whether error accumulation occurs, why perplexity fails to capture it, and how lower exposure bias correlates with better generation quality.The reported quality characteristics are greater coherence and vocabulary diversity, and less repetition.Source: §5 · p. 5

6 Study Setup: Open-ended Generation

The study uses open-ended text completion to analyze error accumulation from exposure bias and its correlation with generation quality. Its experiments use GPT-2 with Wikitext-103 prompts and a fine-tuned GPT-2 model as an approximate oracle.

6 Study Setup: Open-ended Generation

  • Text completion is presented as a standard setup for measuring open-ended generation quality and as a generalization of applications including story generation, contextual completion, and dialog modeling.The cited examples include Fan et al. (2018), Radford et al. (2019), and Zhang et al. (2018).Source: §6 · p. 5
  • Text completion is used to study exposure-bias error accumulation and its correlation with generation quality.The setup treats a text passage or prefix as input and generates a continuation using a language model and decoding algorithm.Source: §6 · p. 5
  • The study uses GPT2-117M as its evaluation language model and GPT-2 fine-tuned on Wikitext-103 as an approximate oracle.Prompts come from the Wikitext-103 training split.Source: §6 · p. 5
  • Experiments use 20k prompts, each drawn from the first 50 tokens of a Wikitext-103 training sequence of length 512.Completions are generated up to length 512 or the end-of-sequence token.Source: §6 · p. 5

7 Results

Across text-completion experiments, errors accumulate with sequence length, while perplexity fails to capture that accumulation. Sampling-based decoding with lower exposure bias is associated with better generation quality than deterministic decoding.

7.1 Error Accumulation in Language Generation is Real!

  • Accumulated errors grow near-quadratically with sequence length, while excess errors grow near-linearly; greedy decoding produces over 70% more errors.Source: §7.1 · p. 5, Figure 1 · p. 6

7.3 Error Accumulation Impacts Generation Quality

  • Sampling-based decoding yields more diverse, coherent text and lower exposure bias than deterministic decoding; temperature 1.2 produces the least repetition and excess errors.Greedy and beam search fare poorly on exposure-bias and generation-quality measures.Source: Table 1 · p. 6, §7.3 · p. 8

7.2 Perplexity is Not Enough

  • Perplexity correlates with average per-step error at ρ = 0.9997 but correlates less with length-normalized regret at ρ = 0.4003.Source: §7.2 · p. 6, Figure 2 · p. 7
  • GPT-2’s low zero-shot perplexity of 37.50 on Wikitext-103 coexists with repetitive, incoherent completions under greedy or beam search.The paper presents this as an example of perplexity’s inability to capture error accumulation.Source: §7.2 · p. 7

7.3 Error Accumulation Impacts Generation Quality

  • Because decoding choice does not affect average per-step error, the authors link improved generation quality to reduced exposure-bias accumulation.They hypothesize that stochastic sampling helps avoid divergent contexts, while noting that broader analysis is left for future work.Source: §7.3 · p. 8

8 Related Work

Prior work proposed links between exposure bias and generation failures, but its effects lacked a clear formalization and direct empirical support. This paper analyzes exposure bias through error accumulation and presents evidence that it may help explain degeneration in open-ended generation.

Non-MLE Training Methods

  • Earlier approaches aimed to mitigate exposure bias, but they neither formally analyzed it nor provided empirical evidence that they reduced its effects.The paper proposes analyzing exposure bias through error accumulation, a definition it says can be adapted to evaluate new training and modeling approaches.Source: §8 · p. 8

Smarter Decoding Methods

  • Smarter decoding research proposed that exposure bias may explain degeneration, while this paper examines error accumulation as a possible explanation and argues perplexity misses it.The cited passage describes this hypothesis as not yet having received serious treatment.Source: §8 · p. 8

Analyzing Exposure Bias

  • Prior work linked exposure bias to distribution and domain generalization gaps, but did not formally analyze it or directly connect it empirically to those gaps.The passage notes that evidence for such a connection could inform more robust generation models.Source: §8 · p. 8
  • Earlier empirical evaluations disagreed: two validated exposure bias, while He et al. (2019) reported self-recovery that negated its impact.All three analyses used an empirical definition based on Ranzato et al. (2016).Source: §8 · pp. 8–9
  • The paper offers a principled, theoretically grounded analysis and evidence that exposure bias may explain degeneration in open-ended generation under deterministic decoding.The paper frames this as addressing a gap in prior empirical and theoretical treatment.Source: §8 · p. 9

9 Discussion

The analysis predicts and experimentally validates super-linear error accumulation from exposure bias. It argues that perplexity does not capture this accumulation and finds that better-generation techniques coincide with lower accumulated error; the analysis may inform smarter training and decoding algorithms.

9 Discussion

  • Experiments validate the bound’s prediction that exposure bias produces super-linear growth in generation errors.The paper presents the bound as derived from an imitation-learning analysis.Source: §9 · p. 9
  • Perplexity is insufficient to capture error accumulation, and techniques that improve generation quality also show lower accumulation.The paper hypothesizes that excess error accumulation may affect generation quality.Source: §9 · p. 9
  • The authors suggest their principled analysis may inform smarter training and decoding algorithms aimed at reducing error accumulation and improving robustness.They describe this as a potential direction rather than a demonstrated outcome.Source: §9 · p. 9

Related papers

Loading 2204.01171v3…