Source-linked AI summary
Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Models
Haoran Wang, Xiongxiao Xu, Philip S. Yu, Kai Shu
TL;DR
Aligning LLM and LVLM outputs with user intent remains challenging, while most existing solutions operate during training. This survey reviews inference-time decoding methods, organizes recent work into three paradigms, and highlights their effectiveness, efficiency, challenges, and applications.
Problem
Aligning LLM and LVLM outputs during inference remains challenging, while prompt engineering is task-specific, expertise-dependent, and sensitive to prompt variations.
Method
The survey systematically reviews decoding methods for LLMs and LVLMs and organizes recent approaches into contrastive, guided, and parallel decoding paradigms.
Results
The survey reports that advanced decoding strategies can mitigate hallucination, improve safety, visual grounding, reasoning, robustness, interpretability, and generation efficiency.
Takeaways & Limitations
Decoding methods offer a cost-effective way to enhance and extend LLM capabilities across complex generation tasks.
Takeaways & Limitations
Important gaps remain in privacy, bias, copyright, high-stakes applications, and balancing accuracy with efficiency in parallel decoding.
Abstract
from arXiv · showhide
Large language models (LLMs) and large vision-language models (LVLMs) have demonstrated impressive generative capabilities, yet ensuring their outputs align with user intent is still challenging. While most existing approaches address this issue at the training stage, inference-time approaches like decoding methods offer a more efficient and scalable solution. Decoding methods control model generation by guiding token-level selection, performing sequence-level generation, or generating tokens in parallel to accelerate the process. In this survey, we identify three emerging paradigms from recent works on decoding methods for LLMs and LVLMs, provide a systematic review of these methods, highlight ongoing challenges, and discuss potential future research directions. Our goal is to underscore the efficiency and effectiveness of decoding methods and offer a practical view of their applications. Paper lists and more resources on decoding methods for LLMs and LVLMs can be found at https://github.com/wang2226/Awesome-LLM-Decoding.
1. INTRODUCTION
As LLMs and LVLMs scale, advanced decoding methods have emerged as inference-time approaches for controlling generated outputs and addressing limitations of prompt engineering. Recent work shows that decoding can improve generation quality, model reliability, and interpretability across multiple capabilities.
- Scaling model size and training data has greatly improved the generative capabilities of LLMs and LVLMs.
- Advanced decoding strategies can mitigate hallucination, improve safety, enhance visual grounding and reasoning, and increase robustness to noisy context.
- Decoding-process modifications can also provide interpretability of language and vision-language models.
- The field has shifted from supervised learning to pre-train-and-fine-tune, and more recently toward pre-train, prompt, and predict.Prompt engineering became prominent within the latter paradigm.
- Prompt engineering is computationally efficient but task-specific, requires expert knowledge, and remains vulnerable to prompt sensitivity.Even slight changes in prompt format can produce considerable performance variations.
2. INFERENCE METHODS TO IMPROVE GENERATION
Inference-time methods address the gap between model training objectives and user expectations by controlling generation without retraining. The survey categorizes these approaches into prompting, latent space manipulation, and decoding algorithms.
- Inference-time methods improve generation in real time without retraining or altering the underlying model, and they can be applied across model sizes.
- These methods are typically model-agnostic, often applicable to decoder-only transformers, and some decoding methods provide better interpretability.
- Prompting: Prompt engineering guides generation by crafting task-specific prompts with clear instructions or examples.
- Latent space manipulation: Latent space manipulation controls generation by modifying internal structures such as attention heads or activation-layer steering vectors, without retraining or fine-tuning.
- Decoding algorithms: Decoding algorithms dynamically adjust output logits or probability distributions during decoding to guide text toward desired attributes and requirements.
3. PRELIMINARIESOFDECODINGSTRATE-GIES · 3.1 Deterministic Methods
The section frames decoding as selecting probable token sequences during autoregressive generation and reviews deterministic methods as foundational approaches. It explains that deterministic selection can cause repetitive, unnatural text, motivating methods that preserve higher-probability alternatives or suppress repetition.
- 3. PRELIMINARIESOFDECODINGSTRATE-GIES: Autoregressive generation predicts each next token from preceding tokens, with sequence length ending when EOS is generated or a maximum timestep is reached.The decoding objective is selecting the most probable sequence under the model’s probability distribution.
- 3. PRELIMINARIESOFDECODINGSTRATE-GIES: Classical token-level strategies—including greedy search, beam search, and sampling—form foundational building blocks for modern decoding approaches.The section introduces these methods while outlining general text-generation objectives.
- 3.1 Deterministic Methods: Deterministic methods select the continuation with the LM’s highest probability but often produce repetitive, overly predictable, and unnatural language.This model degeneration reduces variety and makes outputs less reflective of natural human expression.
- 3.1.1 Greedy Search: Greedy search repeatedly selects the next word with the highest probability until EOS or a maximum timestep T.Its local choices can overlook high-probability words reachable after a lower-probability word, so it does not guarantee a global optimum.
- 3.1.2 Beam Search: Beam search maintains a fixed number of likely hypotheses at each timestep and ultimately selects the hypothesis with the highest overall probability.It reduces the risk of missing high-probability sequences and consistently finds higher-probability outputs than greedy search, but does not guarantee the most likely output.
- 3.1.2 Beam Search: Beam search performs well when the desired generation length is relatively predictable, such as in machine translation.Its broader hypothesis tracking addresses a limitation of locally optimal greedy decisions.
- 3.1.3 Contrastive Search: Contrastive search mitigates string repetition by penalizing previously generated token sequences.It can suppress repetitions more effectively than beam search while using a comparable amount of computational resources.
3.2 Stochastic Methods · 3.3 Speculative Decoding
Stochastic methods introduce randomness during decoding to produce more diverse, natural text than deterministic methods. Speculative decoding improves inference efficiency by decoding multiple tokens simultaneously rather than generating strictly token by token.
- 3.2 Stochastic Methods: Stochastic decoding addresses deterministic methods’ minimal token-probability variance, which can produce predictable and repetitive text.
- 3.2 Stochastic Methods: Common stochastic techniques include top-k sampling, top-p sampling, and temperature scaling.
- 3.2.1 Top-k Sampling: Top-k sampling selects the k most probable next tokens, renormalizes their probabilities, and randomly samples one token iteratively.
- 3.2.2 Top-p (Nucleus) Sampling: Top-p sampling selects the smallest dynamic token set whose cumulative probability reaches a predefined threshold p, addressing fixed-K limitations.
- 3.2.3 Temperature Sampling: Temperature sampling applies parameter t after the transformer’s final layer to control probability-distribution sharpness and randomness.
- 3.2.3 Temperature Sampling: Selecting the optimal temperature typically requires ad-hoc experimentation tailored to the application.
- 3.3 Speculative Decoding: Speculative decoding accelerates LLM inference by enabling simultaneous decoding of multiple tokens per step, mitigating autoregressive latency.
4. DECODING PARADIGMS
Recent decoding research extends beyond token-level diversity and fluency toward sequence-level control, structured guidance, and generation efficiency, yielding three paradigms: contrastive, guided, and parallel decoding.
- The survey identifies contrastive decoding as one of three recent paradigms for LLMs and LVLMs.These paradigms extend earlier token-centric decoding methods.
- The survey identifies guided decoding as a paradigm for structured guidance in recent decoding methods.This reflects a shift beyond primarily token-level diversity and fluency.
- The survey identifies parallel decoding as a paradigm focused on generation efficiency.The three paradigms are contrastive decoding, guided decoding, and parallel decoding.
4.1 Contrastive Decoding
Contrastive decoding improves generation by contrasting positive and negative examples during decoding, operating at both token and layer levels. Methods are broadly divided into token-wise CD, which contrasts probability distributions, and layer-wise CD, which contrasts representations from different model layers.
- 4.1 Contrastive Decoding: Contrastive decoding enhances output quality by contrasting positive and negative examples during generation at both token and layer levels.This provides more control over the quality of the generated sequence than greedy or sampling-based decoding.
- 4.1 Contrastive Decoding: Contrastive decoding methods are broadly classified as token-wise CD or layer-wise CD according to where contrastive examples are used.The classification distinguishes token-level probability contrasts from layer-level contrasts.
- 4.1 Contrastive Decoding: Token-wise CD contrasts token-level probability distributions produced by pairs of contrasting examples.An expert model supplies logits z+ for the user-desired direction, while a weak or base model supplies logits z− as a baseline, with α regulating the adjustment.
- 4.1 Contrastive Decoding: Layer-wise CD contrasts logits from different transformer layers projected onto the vocabulary space to derive the next-token distribution.This approach leverages observations that different layers encode different information and that factual knowledge can be localized in specific layers.
4.2 Guided Decoding
Guided decoding searches for next tokens by adjusting the model’s distribution according to a guidance function and control condition. It comprises classifier-guided and heuristic-guided approaches for steering generation toward desired attributes or constraints.
- Definition and taxonomy: Guided decoding searches for the next token that maximizes a guidance-adjusted objective.The guidance function adjusts the model’s distribution according to specified criteria.
- Definition and taxonomy: Guidance functions adjust the model’s distribution based on criteria represented by a control condition.The survey classifies these strategies into classifier-guided and heuristic-guided decoding.
- Classifier-guided decoding: Classifier-guided decoding uses an external classifier to control specific attributes during text generation.Classifiers may be reward models, pre-trained models, or APIs; PPLM combines a pre-trained language model with attribute classifiers.
- Heuristic-guided decoding: Heuristic-guided decoding steers token search using constraints, lookahead results, or value functions.FUDGE predicts attribute probabilities for evolving sequences and modifies logits to align generation with desired attributes.
4.3 Parallel Decoding
Parallel decoding accelerates generation by producing multiple future-token candidates concurrently, then using the target model to verify and aggregate them. This draft-then-verify paradigm includes greedy and sampling variants.
- Parallel Decoding: Parallel decoding generates multiple candidate sequences or future tokens simultaneously, unlike standard sequential decoding.The method selects the most likely candidate according to predefined criteria.
- Parallel Decoding: The target LLM verifies and aggregates drafted tokens in parallel to speed up inference.This verification follows the initial concurrent drafting process.
- Parallel Decoding: The draft-then-verify paradigm is classified into greedy and sampling approaches.The draft tokens are produced by a separate draft model M before verification.
- Parallel Decoding: Blockwise decoding predicts multiple time steps simultaneously and rolls back to the longest valid prefix selected by a scoring model.Shallow Aggressive Decoding uses a shallow decoder to decode as many tokens as possible for efficient online grammatical error correction.
5. DECODING APPLICATIONS
Section 5 organizes decoding methods by application, emphasizing their use for model alignment and task-specific generation. The reviewed applications include mitigating hallucinations, enhancing safety and reasoning, and improving retrieval-augmented generation and code generation.
- Model Alignment: Decoding strategies improve model alignment by mitigating hallucinations, enhancing safety, and strengthening reasoning during inference.They dynamically tailor output generation to meet user expectations while adhering to ethical standards.
- Hallucination Mitigation: Decoding methods mitigate hallucinations as model-agnostic, interpretable inference-time tools that reduce reliance on language priors and strengthen visual input dependence.DeCoRe dynamically contrasts output tokens to address hallucinations in LVLMs.
- Safety Enhancement: Safety-oriented decoding selects tokens from safety-optimized distributions or directly corrects harmful queries to generate safer responses.SafeInfer uses context-adaptive safety-guided decoding, while Root Defense Strategy corrects harmful queries rather than rejecting them outright.
- Reasoning Enhancement: Contrastive and distillation-based decoding enhances LLM reasoning by preventing abstract reasoning errors and avoiding input copying during chain-of-thought generation.DCD uses Contrastive Chain-of-Thought Prompting with dropout and quantization, while top-k alternatives can reveal chain-of-thought paths.
- Task-Specific Generation: Decoding strategies improve retrieval-augmented generation and code generation by refining decoding, optimizing architectures, and incorporating external resources.Entropy-based decoding, adaptive contrastive decoding, and Nonparametric Decoding target truthful, contextually relevant, and capacity-enhanced retrieval outputs.
5.3 Improve Generation Efficiency
Decoding methods improve generation efficiency across text, image, and video domains. Recent strategies reduce autoregressive computation through adaptive layer skipping, parallel decoding, speculative methods, and non-autoregressive generation.
- Decoding methods boost generation efficiency across text, image, and video domains.
- Text generation: Hierarchical Skip Decoding adaptively skips decoding layers based on sequence length, reducing computational overhead without additional trainable parameters.
- Text-to-image generation: Speculative Jacobi Decoding accelerates autoregressive text-to-image generation through training-free probabilistic parallel decoding.
- Text-to-image generation: Iterative mask-predict methods generate image tokens in parallel and refine predictions using partially observed tokens to improve convergence speed and output quality.
- Video generation: PDVC reformulates dense caption generation as set prediction tasks, while GLOBER uses non-autoregressive video decoding with normalized frame indexes for temporal information.
- Healthcare: Alternate Contrastive Decoding improves medical information extraction by contrasting output distributions from sub-task models after separating identification and classification.
6. DISCUSSIONS
The discussion identifies persistent limitations in decoding methods, including dependence on manual design, scalability constraints, insufficient theoretical grounding, incomplete alignment coverage, and overlooked security risks. It also highlights combining decoding paradigms as a direction for improving generation quality and efficiency.
- Limitations and future directions: Contrastive decoding depends heavily on selecting effective contrastive examples or layers, motivating dynamic and universally applicable construction methods.These methods enhance control over text generation in LLMs and LVLMs.
- Limitations and future directions: Guided decoding can rely on test cases and small search spaces, while TS-LLMs face computational overhead from node expansion and value evaluation in larger scenarios.TS-LLMs perform across reasoning, planning, alignment, and decision-making tasks on trees with depth 64.
- Limitations and future directions: Decoding research needs stronger theoretical foundations to explain model behavior, alongside careful hyperparameter tuning for optimization.Some decoding methods provide insights into intrinsic reasoning abilities, but performance can depend substantially on tuning.
- Limitations and future directions: Combining contrastive, guided, and parallel decoding could jointly improve generation quality and efficiency, as demonstrated by Speculative Decoding with contrastive decoding.The cited combination achieved improved generation quality and inference speedup.
- Limitations and future directions: Although decoding methods increasingly address toxicity, truthfulness, and safety, important alignment gaps remain for privacy, bias, and copyright concerns.Recent work has shifted from text-generation quality toward aligning generated outputs.
- Limitations and future directions: Decoding mechanisms remain overlooked as security risks because methods designed for safe responses can also be exploited to generate malicious outputs.This concern complements broader research on data leakage prevention and jailbreak defense.
7. CONCLUSION
The survey reviews three primary decoding paradigms and their applications in LLMs and LVLMs, emphasizing decoding methods as effective, efficient, and cost-effective approaches for complex generation tasks and extending model capabilities.
- 7. CONCLUSION: The survey comprehensively reviews three primary decoding paradigms and their diverse applications in LLMs and LVLMs.It presents these methods as addressing complex generation tasks with demonstrated effectiveness and efficiency.
- 7. CONCLUSION: Decoding methods offer a cost-effective way to enhance and extend LLM capabilities.The survey calls for further discussion and research in this area.