PaperlayerView exact revision on arXiv

Source-linked AI summary

A Thorough Examination of Decoding Methods in the Era of LLMs

Chufan Shi, Haoran Yang, Deng Cai, Zhisong Zhang, Yifan Wang, Yujiu Yang, Wai Lam

arXiv:2402.06925v3cs.CL

TL;DR

Existing decoding research provides limited evidence for choosing methods across general-purpose LLM tasks and models. This paper conducts a broad evaluation of decoding methods across performance, robustness, and efficiency dimensions. It finds that the best method depends on the task, model, and deployment setting, with practical selection requiring attention to these trade-offs.

  • Problem

    Prior decoding studies often focus on narrow aspects, limited task sets, or task-specific models, leaving method choice in general-purpose LLMs insufficiently characterized.

  • Method

    The paper comprehensively evaluates deterministic and stochastic decoding methods across tasks, models, deployment environments, hyperparameter settings, and quantization conditions.

  • Results

    Decoding performance depends on task, model, alignment, and quantization; closed-ended tasks generally favor deterministic methods, whereas open-ended tasks favor stochastic methods.

  • Takeaways & Limitations

    Decoding method selection should account for the target task, model, and priority among performance, robustness, and speed rather than follow a single universal rule.

  • Takeaways & Limitations

    The study does not exhaustively cover all hyperparameters, future models or tasks, or combinations of multiple decoding methods.

Abstract

from arXiv · show

Decoding methods play an indispensable role in converting language models from next-token predictors into practical task solvers. Prior research on decoding methods, primarily focusing on task-specific models, may not extend to the current era of general-purpose large language models (LLMs). Moreover, the recent influx of decoding strategies has further complicated this landscape. This paper provides a comprehensive and multifaceted analysis of various decoding methods within the context of LLMs, evaluating their performance, robustness to hyperparameter changes, and decoding speeds across a wide range of tasks, models, and deployment environments. Our findings reveal that decoding method performance is notably task-dependent and influenced by factors such as alignment, model size, and quantization. Intriguingly, sensitivity analysis exposes that certain methods achieve superior performance at the cost of extensive hyperparameter tuning, highlighting the trade-off between attaining optimal results and the practicality of implementation in varying contexts.

1 Introduction

Decoding methods determine how LLMs become practical task solvers, but prior evidence often centers on narrow tasks or task-specific models. This study therefore evaluates decoding across tasks, models, and deployment environments.

  • Decoding methods bridge next-token prediction and practical text generation in LLMs.
  • Prior studies often examine narrow aspects or limited task sets, while earlier comparative work focused on task-specific language models.
  • The paper asks how practitioners should choose decoding methods in the era of LLMs.
  • The study compares decoding methods across performance, robustness, and speed over diverse tasks, models, and deployment environments.
  • Optimal decoding depends on the task, model, and priority, with closed-ended tasks generally favoring deterministic methods and open-ended tasks favoring stochastic methods.

2 Decoding Methods

The paper covers deterministic and stochastic decoding methods that transform next-token distributions into generated sequences. It describes their operational mechanisms and evaluates commonly used hyperparameter ranges.

  • Decoding methods derive generated token sequences from each prefix’s next-token probability distribution.
  • The experiments search method-specific hyperparameters across ranges guided by relevant literature and common practice.
  • Deterministic Methods: Deterministic methods include greedy, beam, diverse beam, contrastive, contrastive decoding, and frustratingly simple decoding.
  • Deterministic Methods: Greedy Search selects the highest-probability token, whereas Beam Search retains the k most probable sequences at each step.
  • Stochastic Methods: Stochastic methods include temperature, top-p, top-k, η, Mirostat, and typical sampling.

3 Evaluation Setup

The evaluation spans coding, math, summarization, translation, commonsense reasoning, factual knowledge, instruction following, and open-ended generation. It uses task-specific datasets, metrics, and multiple LLM families including aligned and unaligned models.

  • Tasks and Metrics: Coding is evaluated on HumanEval and MBPP using pass@1 accuracy.
  • Tasks and Metrics: Math problem solving uses GSM8K and reports accuracy, while summarization uses CNN/DM and XSUM with RougeL.
  • Tasks and Metrics: Translation uses four WMT22 directions with BLEU, and commonsense reasoning uses CommonsenseQA and StrategyQA with accuracy.
  • Tasks and Metrics: Factual knowledge uses FActScore, instruction following uses AlpacaEval pairwise win rate, and open-ended generation uses MAUVE on Book, Wikinews, and Wikitext.
  • Models: Experiments primarily use aligned and unaligned Llama2 models, with additional tests on MPT, CodeLlama, Qwen, Mistral, Deepseek-MoE, and Llama3 families.

4 Experimental Results

The experiments evaluate decoding methods across task performance, model alignment, hyperparameter sensitivity, decoding speed, and stochastic self-consistency. Results show that method choice is task- and model-dependent, while some methods offer strong performance with greater robustness or efficiency.

  • Performance Analysis: Deterministic methods generally outperform stochastic methods on unaligned Llama2-7B closed-ended tasks, while stochastic methods perform better for open-ended generation.BS, FSD-d, and FSD rank among the top three on 8, 7, and 7 of 11 datasets, respectively; greedy, BS, and DBS have lower MAUVE on open-ended tasks.
  • Performance Analysis: Aligned Llama2-7B-Chat shows narrower performance differences across decoding methods than unaligned Llama2-7B.On MBPP, the range is 7.80%–21.20% for Llama2-7B versus 16.00%–21.60% for Llama2-7B-Chat; RDP measures relative performance variation across methods.
  • Performance Analysis: Deterministic methods lead aligned-model factuality and instruction following, with beam search reaching 47.80% FActScore while mirostat and top-k sampling reach 44.06% and 44.11%.The results associate deterministic decoding with stronger factuality and instruction-following performance on the aligned model.
  • Hyperparameter Sensitivity: Fixed hyperparameters reduce temperature-sampling ANP by 11.59% on Llama2-7B and 3.90% on Llama2-7B-Chat, whereas FSD/FSD-d and BS/DBS remain comparatively robust.CD also decreases by 9.42% on Llama2-7B and 3.35% on Llama2-7B-Chat when hyperparameters are fixed.
  • Decoding Speed: Contrastive search is slowest, reaching 2.00x greedy-search latency, while FSD and FSD-d run as fast as greedy search across generation lengths.Contrastive decoding is about 1.4x slower than greedy search but maintains a constant latency ratio across lengths; stochastic methods are omitted because their latency is close to greedy search.

5 Further Analysis

Further analysis shows that decoding behavior changes with sampling diversity, model scale, and quantization. Self-consistency can improve stochastic decoding, while larger models reduce method differences and quantization can alter robustness.

  • 5.1 Self-Consistency: Sampling more generations consistently improves stochastic decoding performance, and nearly all methods surpass the strongest deterministic baselines at 20 samples.The deterministic baselines are 17.74% for diverse beam search with Llama2-7B and 28.81% for beam search with Llama2-7B-Chat.
  • 5.1 Self-Consistency: Hyperparameter tuning with greater randomness or a larger candidate pool further boosts self-consistency performance.
  • 5.2 Scaling Model Size: As model parameters increase, relative deviation percentage decreases, reducing performance differences among decoding methods.
  • 5.2 Scaling Model Size: η sampling rises from 9.40% on Llama2-7B to 21.60% on Llama2-13B and 38.80% on Llama2-70B for MBPP.
  • 5.3 Quantization: Quantized models have larger relative deviation percentages than FP16 models, indicating reduced robustness across decoding methods.
  • 5.3 Quantization: On GSM8K, η sampling decreases 13.65% under INT8 quantization, while typical sampling improves 6.14% under INT4 quantization.Typical and η sampling are more affected because their computations involve numerically unstable calculations.

6 Conclusion

The study comprehensively evaluates traditional and contemporary decoding methods across tasks, models, and settings. It finds that decoding choice remains crucial because methods offer different advantages in different scenarios.

  • The study analyzes decoding methods across efficacy, robustness, efficiency, and universality in varied tasks, models, and settings.
  • Decoding method choice remains crucial because different methods provide different advantages in different scenarios.

Limitations

The study has scope and coverage limitations concerning evolving models and tasks, hyperparameter exploration, and combinations of decoding methods.

  • New models or tasks may display distinct decoding behavior because the study cannot cover the continually evolving LLM landscape.
  • The hyperparameter sensitivity analysis covers commonly used configurations but is not exhaustive.
  • The paper does not investigate combining multiple decoding methods, such as temperature sampling with repetition penalties.

A.1 Deterministic Methods

The paper reviews deterministic and stochastic decoding methods, describing how they select, score, or filter candidate tokens and sequences. These methods differ in search breadth, diversity control, auxiliary-model contrasts, and randomness.

  • Greedy Search: Greedy Search selects the highest-probability token from the full vocabulary at each time step.Its local token choice does not consider the global sequence score and can get stuck in local optima.
  • Beam Search: Beam Search maintains the k most probable sequences at each time step, with k controlling beam width.The experiments test beam sizes 4 and 8.
  • Diverse Beam Search: Diverse Beam Search divides k sequences into G groups and penalizes similarity with sequences selected by other groups.The experiments configure (k, G) pairs of (4,2), (4,4), (8,2), and (8,4), with diversity penalty λ set to 1.
  • Contrastive Decoding: Contrastive Decoding contrasts expert and amateur-model logits, using candidate tokens selected from the expert distribution.The expert and amateur models share a tokenizer, and TinyLlama-1.1B is used as the amateur model with α=0.1 while β is searched across five values.
  • Frustratingly Simple Decoding: Frustratingly Simple Decoding contrasts the LLM with an auxiliary anti-LM and selects the top-k candidate with the largest FSD score.FSD and FSD-d differ in whether the anti-LM is vectorized or discrete n-gram based.

B Evaluation Benchmarks

The evaluation spans code generation, mathematical reasoning, summarization, translation, commonsense reasoning, factual knowledge, instruction following, and open-ended generation. It combines established datasets with task-specific prompts and automatic metrics.

  • Code Generation: HumanEval and MBPP evaluate Python code generation from docstrings or textual descriptions using automated test cases.HumanEval contains 164 problems, while MBPP contains 500 problems.
  • Mathematical Reasoning: GSM8K evaluates mathematical reasoning with 1,319 grade-school word problems and reports 8-shot pass@1 accuracy.Aligned models use 0-shot prompts, whereas unaligned models use 8-shot prompts.
  • Summarization: CNN/DailyMail and XSUM evaluate summarization across largely extractive and largely abstractive reference summaries using ROUGE-L.Their test splits contain 11.3K and 11.5K examples, respectively.
  • Translation: WMT22 evaluates translation for De⇒En, En⇒De, Zh⇒En, and En⇒Zh with SacreBLEU BLEU scores.The corresponding test-set sizes are 1,984, 2,037, 1,875, and 2,037 samples.
  • Commonsense Reasoning: CommonsenseQA and StrategyQA assess commonsense reasoning involving prior knowledge or multi-hop strategies.CommonsenseQA has 1.22K validation instances, while StrategyQA uses 2.29K test instances in an open-domain setting.
  • Factual Knowledge: FActScore measures factual accuracy in biographies by decomposing generations into atomic statements and comparing them with retrieved knowledge.The evaluation covers biographies of 500 specific individuals.
  • Instruction Following: Alpaca-eval measures instruction-following quality through pairwise win rate against Text-Davinci-003 across 805 prompts.The prompts come from Vicuna, Self-instruct, Open Assistant, Koala, and HH_RLHF.
  • Open-Ended Generation: Book, Wikinews, and Wikitext evaluate open-ended generation with MAUVE, using 500 randomly selected cases from each domain.The source collections contain 1,947, 2,000, and 1,314 prompts, respectively.

D Different Foundation Models

The study extends decoding comparisons across foundation models with different architectures, training data, and aligned versions. It reports recurring patterns while showing that method-specific rankings can still vary, and excludes CD and DoLa from this analysis.

  • Model Coverage: The foundation-model analysis includes CodeLlama-7b, Qwen-7B, MPT-7b, Mistral-7B, deepseek-moe-16b-base, Llama-3-8B, aligned variants, and vicuna-7b-v1.5.These models differ in pre-training data, architecture, and related characteristics.
  • Cross-Model Findings: For unaligned models, deterministic methods generally outperform stochastic methods on all tasks except open-ended text generation.This pattern is reported across the additional foundation models.
  • Cross-Model Findings: Aligned models are less dependent on decoding methods than unaligned models, while temperature sampling generally performs best among stochastic methods for unaligned models.The reported patterns are described as consistent with results from the main evaluation.
  • Cross-Model Findings: Decoding methods can change performance rankings between foundation models: CodeLlama outperforms Qwen by 7.60% on MBPP with top-k sampling but has a different ranking under another method.The passage uses this comparison to illustrate method-dependent model rankings.
  • Scope Boundary: CD and DoLa are excluded because finding an amateur model for each foundation model and selecting appropriate comparison layers are challenging.This is an explicit scope boundary of the foundation-model analysis.
  • Generation Diversity: For open-ended diversity, the best stochastic methods do not necessarily have higher diversity than the best deterministic methods.FSD has the highest diversity score for Llama2-7B, while CD has the highest score for Llama2-Chat.

G Analyses of COMET Score on WMT tasks.

The WMT analysis reports COMET results for translation and finds that, for unaligned models, deterministic decoding generally outperforms stochastic decoding. The broader evaluation also emphasizes that decoding choices affect generated content and may interact with ethical risks.

  • COMET results: For unaligned models, deterministic methods generally outperform stochastic methods on WMT translation according to COMET.The results are reported for translation tasks in Table 33.
  • Ethical implications: Decoding-method choices can significantly affect generated content, potentially amplifying or mitigating biases in underlying models.The paper connects variability across tasks and models with the need for transparency about system limitations and potential biases.
  • Deployment considerations: The study frames decoding selection as a deployment consideration because performance, robustness, and speed trade-offs matter across real-world contexts.The societal-impact discussion highlights informed decisions and possible applications in healthcare, education, and public services.

K Future Work

Future work should test decoding methods across broader tasks, languages, dataset types, user-centered settings, and multimodal models. The proposed extensions target generalizability, real-world utility, emerging capabilities, and modalities beyond current LLM-focused evaluations.

  • Holistic Evaluations Across Diverse Contexts: Broader evaluations across tasks, languages, and dataset types are needed to deepen understanding of the findings’ generalizability.Suggested extensions include temporal knowledge graph completion, text-to-SQL, low-resource languages, and under-represented dialects.
  • User-Centric Evaluation Metrics: New metrics incorporating user feedback and live deployment scenarios could better reflect user satisfaction and real-world efficacy.This future direction is presented as a way to understand the practical utility of decoding methods more directly.
  • Extending to New Tasks: Future validation should examine emerging tasks and attributes such as honesty, including open-ended human-AI collaboration scenarios.The paper identifies honesty and collaboration settings as examples of areas requiring further study as LLMs evolve.
  • Extending to Large Multimodal Models: Decoding research should extend beyond LLMs to large multimodal models and modalities including text-to-image, multimodal question answering, reasoning, and code generation.The paper presents these multimodal applications as requiring future investigation.
Loading 2402.06925v3…