PaperlayerView exact revision on arXiv

Source-linked AI summary

Locally Typical Sampling

Clara Meister, Tiago Pimentel, Gian Wiher, Ryan Cotterell

arXiv:2202.00666v6cs.CLcs.AI

TL;DR

Probabilistic language generators can produce incoherent or repetitive text despite models performing well under standard metrics. The paper proposes locally typical sampling, which favors words with information content near their context-conditioned expectation; evaluations find competitive human-rated quality and fewer degenerate repetitions than nucleus and top-k sampling.

  • Problem

    Probabilistic language generators can produce incoherent, repetitive, or otherwise undesirable text even when their models perform well on held-out text, leaving a gap between modeling metrics and generation quality.

    Source: §1 · p. 1

  • Method

    Locally typical sampling constrains generation to words whose information content is close to the expected information content given prior context.

    Source: §1 · p. 2

  • Results

    Compared with other stochastic decoding strategies, locally typical sampling yields comparable or better human-rated quality, closer alignment with several human-text quantitative properties, and fewer degenerate repetitions than nucleus and top-k sampling.

    Source: §1 · p. 2, §7 · p. 13

  • Takeaways & Limitations

    The evaluations support per-word information-content typicality as a criterion for generating text with competitive quality and fewer degenerate repetitions.

    Source: §1 · p. 2

    The optimization formulation for the locally typical truncation set can exclude continuations whose log-probabilities lie closest to conditional entropy, and the authors are working on a revised formulation.

    Source: §5.2 · p. 9

Abstract

from arXiv · show

Today's probabilistic language generators fall short when it comes to producing coherent and fluent text despite the fact that the underlying models perform well under standard metrics, e.g., perplexity. This discrepancy has puzzled the language generation community for the last few years. In this work, we posit that the abstraction of natural language generation as a discrete stochastic process--which allows for an information-theoretic analysis--can provide new insights into the behavior of probabilistic language generators, e.g., why high-probability texts can be dull or repetitive. Humans use language as a means of communicating information, aiming to do so in a simultaneously efficient and error-minimizing manner; in fact, psycholinguistics research suggests humans choose each word in a string with this subconscious goal in mind. We formally define the set of strings that meet this criterion: those for which each word has an information content close to the expected information content, i.e., the conditional entropy of our model. We then propose a simple and efficient procedure for enforcing this criterion when generating from probabilistic models, which we call locally typical sampling. Automatic and human evaluations show that, in comparison to nucleus and top-k sampling, locally typical sampling offers competitive performance (in both abstractive summarization and story generation) in terms of quality while consistently reducing degenerate repetitions.

1 Introduction

Probabilistic language models can assign high probability to held-out text yet still generate undesirable, repetitive, or incoherent text. The paper proposes locally typical sampling, which constrains each word’s information content relative to its context-conditioned expectation and yields competitive quality with fewer degenerate repetitions.

1 Introduction

  • High-probability strings can be undesirable despite models placing high probability on held-out text, while maximization-based and stochastic decoding each retain distinct problems.Maximization can produce generic or degenerate text; stochastic methods can produce nonsensical text, and both can enter repetitive loops.Source: §1 · p. 1
  • The paper frames language as communication in which people aim to convey information efficiently and robustly, motivating an information-theoretic account of generation.Prior work on uniform information density empirically observed this property in human language.Source: §1 · p. 2
  • Locally typical sampling restricts generation to words whose information content is close to the expected information content given prior context.The proposal is motivated by the hypothesis that natural text has this per-word property.Source: §1 · p. 2
  • Compared with nucleus and top-k sampling, locally typical sampling reduces degenerate repetitions and produces text generally closer in quality to human text.The reported repetition measure is on par with human text.Source: §1 · p. 2

2 Two Views of Language Modeling

The paper develops an information-theoretic view of language models by distinguishing distributions over complete strings from sequential language processes. This process view supports the paper’s sampling algorithm and is accompanied by formal conditions and modeling assumptions.

2 Two Views of Language Modeling

  • A language model can be viewed either as a distribution over full strings or as a discrete stochastic process, and the latter view is used to derive a sampling algorithm.The paper formally compares the two views.Source: §2 · p. 2

2.1 A Single String-Valued Random Variable

  • Under the traditional view, a language model is a probability distribution over all strings y ∈ V*, represented as one V*-valued random variable.Its factorization into conditional probabilities uses local normalization and an end-of-string symbol.Source: §2.1 · pp. 2–3

2.2 A Discrete Stochastic Process

  • A language process is a sequence of word-valued random variables; it is more general than a language model, which must be tight to avoid leaking probability mass to infinite strings.A language process can continue generating after EOS, unlike the traditional string-valued model.Source: §2.2 · pp. 3–4

2.3 Other Useful Properties

  • The discussion identifies stationarity, ergodicity, and Markov behavior as useful process properties, noting that recurrent neural language models tend to learn distributions with empirically bounded memory.The cited estimate for recurrent models’ memory is roughly 200 words.Source: §2.3 · p. 4

2.4 Estimating a Language Model from Data

  • The standard parameter-estimation method maximizes training-corpus log-likelihood, equivalently minimizing cross-entropy against the empirical distribution.The estimation setup assumes training strings end in EOS.Source: §2.4 · pp. 4–5

3 Information-Theoretic Properties of Language Processes

This section uses information-theoretic properties of language processes to define typicality, then motivates local typicality as a more restrictive per-symbol criterion for strings humans typically produce.

3 Information-Theoretic Properties of Language Processes

  • Viewing language models as discrete stochastic processes provides an information-theoretic basis for analyzing their properties.The section reviews standard information-theoretic definitions before introducing local typicality.Source: §3 · p. 5, §3.1 · p. 5

3.1 Typicality

  • As sequence length grows, the typical set captures nearly all probability mass, although the most likely sequence may lie outside it.These typical-set properties are stated for stationary, ergodic processes.Source: §3.1 · p. 5
  • The ordinary typical set does not ensure human-typical language when the model is imperfect and its process is non-ergodic with EOS absorbing.The paper introduces local typicality to address this mismatch.Source: §3.1 · p. 6

3.2 Local Typicality

  • Local typicality restricts sequences to those where each symbol’s probability is near its context-specific conditional entropy.The paper motivates this per-symbol criterion as useful for describing strings humans typically produce.Source: §3.2 · p. 6

4 Local Typicality in Natural Language

The paper frames language as information transmission and proposes that human-like text keeps each word’s information content near its context-specific expected value. Human-text distributions support this pattern, motivating explicit enforcement when generating with imperfect models.

4.1 Properties of Human Communication

  • Human language users avoid words with very high or low information content while balancing efficient communication and risk of miscommunication.Research also reports a trade-off between information content and speech rate.Source: §4.1 · pp. 6–7

4.2 An Information-Theoretic Formalization

  • The paper hypothesizes that human-produced strings are locally typical, with each word’s information content close to its expected value under the human process.The expected value is the conditional entropy given prior context.Source: §4.2 · p. 7
  • Human-generated text has per-word information-content deviations concentrated near zero, consistent with conditional entropy as the expected value.Figure 1 shows deviation distributions for three tasks; the prose describes them as peaked and centered around approximately zero.Source: §4.2 · p. 7, Figure 1 · p. 7
  • Because language models may fail to capture this human-language property, the authors hypothesize that explicitly enforcing local typicality can patch that shortcoming.The passage offers modeling deficiencies, including poor tail modeling, as possible sources of the mismatch.Source: §4.2 · p. 7

5 Sampling from a Language Process

The section treats language generation as sampling from a word-by-word stochastic process and motivates locally typical sampling as an information-theoretic alternative to existing decoding strategies. The method selects continuations whose information content is near the model’s conditional entropy.

5 Sampling from a Language Process

  • Ancestral sampling draws each word successively from the model; truncated sampling instead restricts each step to a context-dependent subset of the vocabulary.Top-k keeps the k most probable tokens, while nucleus sampling keeps the smallest subset whose cumulative probability exceeds η.Source: §5 · pp. 7–8

5.1 Shortcomings of Existing Algorithms

  • Stochastic decoding can produce lower-probability strings than maximization-based decoding yet outperform it on qualitative metrics.The section proposes an information-theoretic explanation: low-information text can be dull, while repeated substrings convey less information as they recur.Source: §5.1 · p. 8
  • The authors identify the locally typical set as a criterion for human-like messages and motivate a decoding strategy that explicitly enforces it.They caution that directly sampling from the model’s typical set is not a good idea because the model differs from the true natural-language distribution.Source: §5.1 · p. 8

5.2 Locally Typical Sampling

  • Locally typical sampling truncates the vocabulary to words whose negative log-probability lies within a range of the model’s conditional entropy.The range is determined by τ, which controls the probability mass considered.Source: §5.2 · pp. 8–9
  • When conditional entropy is low, locally typical sampling may select the same high-probability words as nucleus sampling.Unlike Mirostat’s fixed target information content, it uses conditional entropy dynamically at each word.Source: §5.2 · p. 9
  • The truncation distribution can be constructed in O(|V| log |V|) time using standard sorting, with an O(|V|) selection alternative noted.Computing conditional entropy and taking words until their cumulative probability exceeds τ are each O(|V|) operations.Source: §5.2 · p. 9
  • An erratum notes that the stated optimization can exclude continuations closest to conditional entropy, unlike the subsequent greedy-algorithm specification.The authors say they are working on a new formulation.Source: §5.2 · p. 9

6 Experiments

The experiments compare locally typical sampling with other decoding strategies on abstractive summarization and story generation using automatic metrics and human ratings. Overall, locally typical sampling performs on par with or better than other sampling techniques, with repetition robust to its hyperparameter choice.

6 Experiments

  • The experiments assess two tasks against nucleus, top-k, temperature, beam search, and Mirostat using automatic metrics and human ratings.The setup uses GPT-2 fine-tuned on WRITINGPROMPTS for story generation and BART fine-tuned on CNN/DAILYMAIL for summarization; all reported metrics use the respective test sets.Source: §6 · p. 9, §6.1 · p. 10

6.1 Setup

  • Automatic evaluation includes perplexity under the generating and an independent language model, MAUVE, REP, Zipf’s coefficient, and n-gram diversity.The setup also reports perplexity and Zipf’s coefficient relative to reference-text measurements.Source: §6.1 · p. 10
  • Human ratings aggregate fluency and relevance for summarization, and fluency, coherence, and interestingness for story generation, using five annotators.The study uses 200 examples per decoding strategy and task, with median scores across raters as final scores.Source: §6.1 · p. 11

6.2 Results

  • Locally typical sampling performs on par with or better than other sampling techniques, with human ratings closest to the reference among stochastic strategies; beam search does better in summarization by a small margin.The reported evaluations cover story generation and abstractive summarization; the tables present automatic metrics and human ratings for the respective tasks.Source: Table 1 · p. 10, §6.2 · pp. 11–12, Table 2 · p. 11
  • REP is less sensitive to τ than to k and η, and most tested τ values yield story-generation REP on par with human text.The figure plots REP for different k and τ/η values, with reference text and Mirostat or beam-search lines as comparators.Source: Figure 2 · p. 11, §6.2 · p. 12
  • In selected examples, locally typical sampling produces a comprehensive, coherent summary and, in story generation, arguably the most fluent story among the decoding strategies.The compared generations use test-set ID 1 for each task and hyperparameters selected for best human scores.Source: §6.2 · p. 12, Table 3 · p. 12

7 Conclusion

The paper frames language generation as an information-theoretic process and proposes locally typical sampling. In two generation tasks, it reports comparable or better human-rated quality and text properties closer to human text than other stochastic strategies.

7 Conclusion

  • The paper models language generation as a discrete stochastic process and uses information theory to characterize expected samples.It proposes locally typical sampling, which constrains choices to words meeting a per-word information-content criterion.Source: §7 · pp. 12–13
  • The conclusion identifies Table 4 as sample generations from GPT-2 large fine-tuned on WritingPrompts, with decoder settings chosen using human-evaluation performance.The caption says the samples correspond to test-set ID 1; full stories appear in the appendix.Source: Table 4 · p. 13
  • Locally typical sampling produces text of comparable or better quality than other stochastic decoding strategies according to human ratings.The conclusion also reports that several quantitative properties of typically sampled text more closely align with human text.Source: §7 · p. 13

Ethical Concerns

The paper notes both worker compensation in its human evaluation and a potential misuse risk: generated text could be made more human-like and convincing for malicious purposes.

Ethical Concerns

  • Crowdworkers were guaranteed a minimum wage of $15 per hour for each task.Source: Ethical Concerns · p. 14
  • The authors warn that the work could help ill-intentioned users make machine-generated text more human-like and convincing.They note that generated text may contain malicious content by design or as a byproduct of training data or algorithms.Source: Ethical Concerns · p. 14

A Human Evaluation Setup

Human ratings were collected through Amazon Mechanical Turk for story generation and abstractive summarization. Workers reviewed task criteria and examples, rated outputs on a five-point scale, and received task-based compensation.

A Human Evaluation Setup

  • Amazon Mechanical Turk was used to collect human ratings of generated text.The passage begins to specify that the study used MTurk workers, but does not complete the accompanying qualification.Source: A Human Evaluation Setup · p. 20
  • Each HIT presented either a story prompt or a news article to summarize, then showed outputs from different decoders and a reference in random order.Workers saw rating criteria, descriptions, and examples before scoring each output.Source: A Human Evaluation Setup · p. 20
  • Workers rated each criterion from 1 to 5, and the median across raters was used as the final score for each output and criterion.The passage points to Table 5 for score statistics but supplies no cell values.Source: A Human Evaluation Setup · p. 20, Table 5 · p. 21
  • Workers were paid $1.50 per summarization HIT and $2 per story-generation HIT, rates reported as exceeding $15 per hour.Source: A Human Evaluation Setup · p. 20
  • Worker instructions required confirmation that they had read the instructions and examples; the listed criteria also include native English speaker status.Source: A Human Evaluation Setup · pp. 20–21

Story Generation (WritingPrompts)

The supplied WritingPrompts material includes a story prompt and sample story passages. Table 6 is described as containing full generations, including stories from the GPT-2 large fine-tuned model.

Story Generation (WritingPrompts)

Loading 2202.00666v6…