PaperlayerView exact revision on arXiv

Source-linked AI summary

On Decoding Strategies for Neural Text Generators

Gian Wiher, Clara Meister, Ryan Cotterell

arXiv:2203.15721v1cs.CLcs.AI

TL;DR

Decoding strategies are often assessed on a single task, although their properties may not transfer across language-generation tasks. This paper compares strategies across tasks using human and automatic evaluation, finding that trade-offs and strategy preferences vary by task. It cautions practitioners to account for task-specific behavior when choosing or developing decoding strategies.

  • Problem

    Decoding strategies are commonly evaluated on a single generation task, despite task differences that make cross-task performance uncertain.

    Source: §1 · p. 1

  • Method

    The paper compares decoding strategies across language-generation tasks and evaluates generated-text attributes using human and automatic evaluations.

    Source: §1 · p. 2, §7 · p. 12

  • Results

    Quality-probability and quality-diversity relationships vary by task; directed tasks show a quality-diversity trade-off, while open-ended tasks show an almost co-dependent relationship.

    Source: §6 · p. 12

  • Takeaways & Limitations

    Practitioners should account for task-specific characteristics when choosing or developing decoding strategies.

    Source: §6 · p. 12

    The study does not thoroughly search decoder hyperparameters, instead using widely used settings intended to represent variants practitioners are likely to use.

    Source: §6 · p. 12

Abstract

from arXiv · show

When generating text from probabilistic models, the chosen decoding strategy has a profound effect on the resulting text. Yet the properties elicited by various decoding strategies do not always transfer across natural language generation tasks. For example, while mode-seeking methods like beam search perform remarkably well for machine translation, they have been observed to lead to incoherent and repetitive text in story generation. Despite such observations, the effectiveness of decoding strategies is often assessed with respect to only a single task. This work -- in contrast -- provides a comprehensive analysis of the interaction between language generation tasks and decoding strategies. Specifically, we measure changes in attributes of generated text as a function of both decoding strategy and task using human and automatic evaluation. Our results reveal both previously-observed and surprising findings. For example, the nature of the diversity-quality trade-off in language generation is very task-specific; the length bias often attributed to beam search is not constant across tasks.

1 Introduction

Decoding strategy effectiveness may depend on the generation task, yet empirical comparisons have typically focused on one task. This work compares strategies across tasks and finds that key trade-offs and preferences vary by task.

1 Introduction

  • Decoding strategies are usually evaluated on a single task, despite qualitative and quantitative differences among generation tasks.High-probability strings can be favorable in machine translation but disfavored in story generation.Source: §1 · p. 1
  • The paper provides a comprehensive cross-task comparison of decoding strategies, grouping methods such as deterministic and stochastic to analyze their effects on text properties.The comparison aims to assess how strategy attributes relate to quantifiable properties of generated text.Source: §1 · p. 2
  • Quality-diversity and quality-probability trade-offs manifest in task-specific ways, with a distinct quality-diversity trade-off appearing only in a subset of tasks.The authors question whether these observations reflect one phenomenon or several related phenomena.Source: §1 · p. 2
  • Directed generation tasks appear to favor deterministic strategies, while story generation shows the inverse preference.The strength of this group-level preference also varies across tasks.Source: §1 · p. 2
  • The authors position their findings as guidance for practitioners and a caution against task-agnostic comparisons of decoding strategies.They suggest that strategies may outperform competitors only on specific tasks.Source: §1 · p. 2

2 Probabilistic Language Generators

The paper frames language generators as probability distributions over possible output strings, optionally conditioned on an input. It describes neural models that assign sequence probabilities through local token distributions and learn weights by minimizing a corpus-based loss.

2 Probabilistic Language Generators

  • A language-generation model defines a probability distribution over possible output strings, optionally conditioned on an input.The output space consists of strings constructed from the vocabulary; BOS and EOS are reserved beginning- and end-of-sentence tokens.Source: §2 · p. 2
  • Neural generators use local normalization, with each token distribution conditioned on the preceding tokens.The paper describes these models as typically parameterized by encoder-decoder architectures with attention and trainable weights.Source: §2 · p. 2
  • Model weights are learned by minimizing a corpus-based loss, typically the examples' negative log-likelihood under the model.The stated objective is to assign high probability to corpus examples.Source: §2 · p. 2

3 The Decoding Problem

Decoding searches for an output string under a model and decision rules; exact maximum a posteriori decoding is often infeasible, and the most probable string may not be high-quality or human-like. The section describes deterministic and stochastic alternatives, from local token choices to sampling and minimum-risk selection.

3 The Decoding Problem

  • Decoding seeks a string under a model and decision rules, but exact MAP decoding is often infeasible and its mode may not be human-like or high-quality.The output space can be exponentially large, while the most probable string may be a poor-quality choice.Source: §3 · pp. 2–3

3.1 Deterministic Algorithms

  • Greedy search chooses the most probable token at each step, whereas beam search retains k high-probability paths and selects a final string by score.Greedy choices are only locally optimal and have no formal guarantee of finding the global decoding optimum.Source: §3.1 · p. 3
  • Diverse beam search modifies beam scoring to encourage diversity between subgroups, using a dissimilarity measure between strings.The algorithm divides the beam into subgroups and adds an inner iteration at each step to maximize intergroup diversity.Source: §3.1 · p. 3

3.2 Stochastic Algorithms

  • Stochastic alternatives include ancestral sampling, top-k sampling from a truncated token distribution, and nucleus sampling with a dynamically sized token set.Nucleus sampling retains the smallest set of tokens whose cumulative probability exceeds a chosen threshold.Source: §3.2 · pp. 3–4
  • Bayes Minimum Risk selects the string minimizing expected risk, defined here as negative utility against a reference, and can be estimated with Monte Carlo sampling.In practice, the search can be limited to the sampled set.Source: §3.2 · p. 4

4 Experimental Setup

The study compares decoding strategies across open-ended and directed generation tasks using transformer models, then evaluates generated text for quality and diversity. Quality evaluation combines automatic metrics with human ratings, while diversity is measured across sets of generations.

4 Experimental Setup

  • The comparison spans open-ended and directed tasks, using state-of-the-art transformer models and open-sourced versions for reproducibility.The setup defines each task as an input space, output space, and model distribution over outputs for each input.Source: §4 · p. 4

4.1 Metrics

  • Quality and diversity are treated as distinct metrics, with their relative importance depending on the generation task.The paper contrasts translation quality priorities with conversational agents’ need for diverse topics.Source: §4.1 · p. 5

4.1.1 Evaluation of Quality

  • Human quality ratings use five annotators for 200 examples per decoding strategy, with task-specific criteria and an eight-point Likert scale.Human annotations are omitted for MT because crowd-worker evaluations have shown no significant gain over automatic metrics amid large evaluation variation.Source: §4.1.1 · pp. 5–6, Table 3 · p. 6

4.1.2 Evaluation of Diversity

  • Diversity is measured mainly across sets of generations per input, using metrics such as self-BLEU and n-gram diversity.Beam methods use K equal to beam size, sampling uses K = 10, and unconditional generation uses random disjoint subsets of size 10.Source: §4.1.2 · p. 6

5 Results

Across the four tasks, decoding outcomes vary by task: quality, diversity, and their trade-offs do not follow one universal pattern. Directed-task quality often favors beam methods, while story generation exposes mode-seeking failures and task-specific length effects.

5.1 Quality

  • Human ratings favor beam variants on directed tasks; automatic metrics also rank beam methods best and sometimes let MBR match their performance.Greedy decoding is on par with beam methods for Diag, while stochastic top-p and top-k often fall short of beam quality.Source: §5.1 · pp. 6–7, Table 4 · p. 8

5.2 Diversity

  • Across tasks, ancestral sampling produces the most diverse outputs; restricting its search space reduces diversity, especially for directed tasks.DBS increases diversity over beam-based methods but remains less diverse than stochastic methods; strategy choice matters much more for diversity in SG than in Diag.Source: §5.2 · pp. 7–8, Figure 3 · p. 7

5.3 Quantitative Trade-offs in NLG Tasks

  • All directed tasks favor mode-seeking strategies, with that preference strengthening as tasks become more semantically constrained; story generation reverses the pattern.The most highly ranked deterministic output is below the worst stochastic output in nearly all SG cases, while MT shows the opposite.Source: Figure 4 · p. 7, §5.3 · pp. 8–9
  • Quality–diversity trade-offs appear in AS and MT, but open-ended tasks show interdependence; Diag falls outside the directed-task pattern.The relationships differ by task, and deterministic and stochastic methods separate sharply along each task’s diversity–quality trend.Source: §5.3 · p. 9, Figure 5 · p. 9

5.4 Eliciting Quantitative and Qualitative Metrics

  • No decoder consistently correlates most strongly with quality: mode-seeking correlations vary between open-ended and directed tasks, while stochastic correlations are more consistent.For diversity, ancestral sampling remains most diverse at the set level, including relative to DBS; sequence-level diversity has little cross-task consistency.Source: §5.4 · pp. 9–10, Figure 6 · p. 10
  • Mode-seeking strategies produce substantial repetition in SG, but repetition stays below 1% in other tasks; length effects also depend on task and decoder.Beam search yields short SG strings while DBS and greedy decoding yield inordinately long ones; MT lengths generally resemble references except under ancestral sampling.Source: §5.4 · pp. 10–11, Figure 7 · p. 11

6 Discussion

The discussion emphasizes that decoder behavior and property trade-offs change across tasks, so strategy choices should account for task-specific patterns. Its comparisons use widely used hyperparameter settings rather than a thorough search.

6 Discussion

  • High probability aligns with high quality in MT but is nearly inversely related in SG; quality–diversity trade-offs occur in directed tasks, unlike open-ended tasks.The authors argue these task-specific properties should inform both the choice and development of decoding strategies.Source: §6 · p. 12
  • On directed tasks, mode-seeking methods perform competitively for quality; top-p and top-k improve quality over ancestral sampling while sacrificing some diversity.Set-level diversity is among the more consistent decoder properties across tasks, according to the discussion.Source: §6 · p. 12
  • The study does not thoroughly search decoder hyperparameters, instead using widely used settings intended to reflect practitioners’ defaults.Because decoder behavior depends on settings such as k and p, the authors identify this choice as a limitation.Source: §6 · p. 12

7 Conclusion

The conclusion emphasizes that generated-text attributes depend on both decoding strategy and task, as shown by human and automatic evaluations. It cautions practitioners against choosing a strategy based on results from other tasks and releases resources to support further analysis.

7 Conclusion

  • Generated-text attributes change with both decoding strategy and task, according to human and automatic evaluations.Source: §7 · p. 12
  • The results confirm task-specific diversity–quality trade-offs while revealing previously unobserved trends in decoding strategies and language-generation tasks.Source: §7 · p. 12
  • Practitioners should take care when choosing a decoding strategy based on findings from alternate tasks, since strategies may be optimized for specific tasks.Source: §7 · p. 12
  • The authors release their evaluation framework and generations to support further analysis, including ablations that separate task effects from design choices.Source: §7 · p. 12

Related papers

Loading 2203.15721v1…