Source-linked AI summary
Accelerating Diffusion LLMs via Adaptive Parallel Decoding
Daniel Israel, Guy Van den Broeck, Aditya Grover
TL;DR
Autoregressive decoding limits LLM generation speed, while practical dLLM parallelism currently trades quality for throughput. APD adaptively selects parallel token subsets using a multiplicative mixture of dLLM marginals and a smaller autoregressive model’s joint probabilities. With KV caching, limited masked inputs, and three tunable parameters, APD achieves higher throughput with minimal quality degradation across downstream benchmarks.
Problem
Autoregressive decoding is sequential, while current dLLMs cannot match autoregressive speed through parallel generation without reducing quality.
Method
APD dynamically modulates parallel token sampling using a multiplicative mixture of dLLM marginal probabilities and a smaller autoregressive model’s joint probabilities.
Results
APD achieves substantially higher throughput with minimal quality degradation across downstream benchmark tasks.
Takeaways & Limitations
APD offers a tunable approach for making dLLM inference a more viable and efficient alternative for fast text generation.
Takeaways & Limitations
APD assumes access to a small autoregressive model that can compute sequence likelihoods in parallel.
Abstract
from arXiv · showhide
The generation speed of LLMs are bottlenecked by autoregressive decoding, where tokens are predicted sequentially one by one. Alternatively, diffusion large language models (dLLMs) theoretically allow for parallel token generation, but in practice struggle to achieve the speed of autoregressive models without significantly sacrificing quality. We therefore introduce adaptive parallel decoding (APD), a novel method that dynamically adjusts the number of tokens sampled in parallel. We achieve this by defining a multiplicative mixture between the dLLM marginal probabilities and the joint probability of sequences under a small auxiliary autoregressive model. This inverts the standard setup of speculative decoding, where the goal is to sample from a large autoregressive verifier by drafting from a smaller model. We further optimize APD by enabling KV caching and limiting the size of the masked input. Altogether, our method puts forward three tunable parameters to flexibly tradeoff throughput and quality. We show that APD provides markedly higher throughput with minimal quality degradations on downstream benchmarks.
1 Introduction
Autoregressive decoding limits LLM throughput because it generates tokens sequentially, while dLLMs’ practical parallelism can reduce quality. APD dynamically adjusts parallel sampling and, with efficiency optimizations, targets higher throughput with minimal quality degradation.
- Motivation: Autoregressive generation produces tokens sequentially, limiting throughput and hindering real-time applications as models scale.Generation speed is also important for test-time scaling and reasoning models.
- Motivation: Diffusion LLMs theoretically generate multiple tokens simultaneously, but exploiting this parallelism reduces quality in current open-source models.Dream and Llada achieve their best quality when generating one token at a time.
- Adaptive Parallel Decoding: APD dynamically modulates the number of tokens sampled in parallel by combining dLLM marginals with joint-dependence checks from a smaller autoregressive model.The dLLM generation order is fixed left to right, enabling the auxiliary model to assess candidate subsets.
- Efficiency Optimizations: APD adds KV caching and limits masked-input size to increase diffusion inference efficiency.These optimizations complement the adaptive decoding mechanism.
- Results: Three tunable parameters let practitioners trade off throughput and output quality, with empirical evidence of higher throughput and minimal quality degradation across downstream benchmarks.The paper presents this as a step toward faster and more viable dLLM inference.
2 Background
Diffusion language models offer non-autoregressive generation, but current decoding methods face a quality-throughput tradeoff. Naive parallel decoding increases speed by assuming within-group independence, whereas left-to-right denoising and adaptive methods address quality concerns differently.
- Discrete Diffusion Language Models: Discrete diffusion language models are masked language models trained to reverse a token-corruption process through iterative denoising.The corruption process gradually converts clean tokens to [MASK].
- Discrete Diffusion Language Models: At inference, decoding heuristics determine which tokens to unmask, including lowest-entropy decoding for Dream and highest-probability confidence decoding for Llada.These heuristics are used although the models are theoretically trained for random unmasking.
- Practical Performance: Current open-source dLLMs can approach autoregressive quality, but their throughput remains a fraction of Qwen2.5 7B and competitive GSM8K performance requires sequential decoding.Reducing timesteps increases throughput but causes quality loss.
- From Sequential to Parallel Sampling: Left-to-right denoising is equivalent to autoregressive sampling and can compute an exact autoregressive likelihood.This establishes the sequential baseline for semi-autoregressive extensions.
- From Sequential to Parallel Sampling: Semi-autoregressive decoding samples groups of k tokens in parallel, increasing throughput while introducing an intra-group independence assumption.For k = 1, the semi-autoregressive distribution equals the autoregressive distribution.
3 Method
APD balances decoding speed and output quality by choosing contiguous token groups adaptively, using a multiplicative mixture of diffusion marginals and auxiliary autoregressive joint probabilities. Universal coupling accepts matching tokens, while KV caching and bounded masked lookahead provide additional speed controls.
- Problem statement: APD selects contiguous token groups G by balancing fewer sequential iterations against closeness to the autoregressive distribution.The method targets speed by minimizing |G| and quality by minimizing the distance between pAPD and pAR.
- Adaptive parallel decoding: The target distribution combines diffusion-model marginals with a small autoregressive model’s joint probabilities through a multiplicative mixture weighted by R.Higher R gives the diffusion model more weight, while the auxiliary model captures inter-token dependencies that marginals omit.
- Adaptive parallel decoding: APD samples diffusion and target sequences with shared Gumbel randomness, then accepts matching tokens until the first disagreement while always accepting the first token.Universal coupling preserves adaptivity without fixing a lookahead window.
- Adaptive parallel decoding: The parameter R controls the speed-quality tradeoff: R = 1 accepts the diffusion sample in one shot, whereas R = 0 relies on autoregressive acceptance.The algorithm accepts every diffusion token at R = 1 and does not trust the diffusion model at R = 0.
- Architectural optimizations: Architectural optimizations add KV caching outside a sliding window and cap the masked suffix length M to reduce redundant attention computation.The masked-input cap can alter the EOS probability, so M is tunable; KV caching may induce out-of-distribution behavior but is motivated by low attention weights for distant tokens.
- Empirical tradeoff: APD can generate over 5 tokens per iteration while maintaining approximately 80% GSM8K accuracy, and can exceed 100 tokens per second with some quality loss.Smaller R values use fewer parallel tokens per iteration while maintaining high quality.
4 Experiments
The experiments evaluate how APD’s tunable parameters affect throughput and quality, finding distinct trade-offs and strong speed-quality configurations relative to baselines.
- Experimental Configuration: APD uses three tunable parameters: multiplicative mixture weight R, recompute KV window W, and maximum masked lookahead M.The experiments vary each parameter individually and jointly to measure speed-quality trade-offs.
- Implementation: Dream 7B Instruct serves as the diffusion model, while Qwen2.5 0.5B serves as the approximate autoregressive model.Both models share a tokenizer; Dream 7B was distilled from Qwen2.5 7B.
- Tradeoffs: Over 5 tokens per iteration can be generated on average with nearly the accuracy of generating 1 token per iteration when varying R.Increasing R lowers accuracy, but the decline is not drastic; APD also exhibits a high parallel acceptance rate.
- Tradeoffs: Decreasing W provides a nontrivial speedup at almost no expense to quality.The speed-quality trade-off for W is weaker than that observed for the multiplicative mixture weight.
- Tradeoffs: Decreasing M increases throughput but can significantly reduce quality by shortening generation length.The effect may be especially damaging for complex reasoning tasks.
- Pareto Frontier: Dream 7B with APD achieves substantially higher speed with minimal performance degradation versus K = 1 and exceeds the speed of Qwen 7B and Qwen 0.5B.The reported frontier includes two APD hyperparameter configurations and identifies APD as Pareto-optimal.
5 Related Work
The related work covers architectural and inference-time approaches to multi-token prediction and LLM acceleration, while distinguishing APD from methods with different requirements or trade-offs.
- Multi-token Prediction: Prior work enables multi-token prediction through architectural modifications such as Mask-Predict, Medusa, and DynaMo.
- Diffusion LLM Acceleration: Block diffusion enables KV caching but requires training, whereas discrete copula diffusion reduces denoising steps without offering a tunable speed-quality trade-off.
- Inference Acceleration: Cascades, lookahead decoding, and speculative decoding generally target autoregressive models, while drafting in parallel with a large dLLM presents distinct challenges.
- Orthogonal Approaches: Quantization is presented as a promising orthogonal approach for achieving a strong speed-quality trade-off in LLMs.
6 Conclusion
APD accelerates dLLM sampling by restructuring generation autoregressively, using a smaller autoregressive model, and adding KV caching and limited masked inputs. Its tunable parameters trade generation speed against output quality.
- APD restructures dLLM generation into a left-to-right autoregressive process and uses a smaller autoregressive model to assess parallel token candidates.
- KV caching and limited masked inputs complement APD by improving diffusion-model inference efficiency.
- Three tunable parameters let practitioners trade generation speed against output quality.
A Limitations
APD presents a throughput–quality tradeoff rather than a universally improving decoding method. Its performance also remains bounded by the quality of the underlying diffusion model.
- Higher throughput leads to lower quality, so APD does not provide a free improvement in both objectives.
- APD does not claim to improve over the base diffusion model from which it samples.
- If Dream 7B is weak in a particular domain, APD will also perform poorly there.
B Recompute KV Tradeoff
The recompute KV window governs a throughput–accuracy tradeoff in left-to-right diffusion-model generation. KV caching can modestly improve throughput with very little accuracy loss.
- Autoregressive models keep the attention matrix over a fixed token set invariant throughout generation, enabling KV caching.
- KV caching in a diffusion model provides a modest throughput gain with very minimal accuracy loss.
C Experimental Details
Experiments evaluate instruction-tuned models zero-shot with task-specific chat prompts. Diffusion and autoregressive models use different maximum generation lengths because diffusion models materialize the maximum context length.
- The instruction-tuned models are evaluated zero-shot using a chat format and system prompt.
- GSM8K and HumanEval use “You are a helpful assistant,” while GPQA and MATH use prompts specifying the reasoning task and answer format.
- Diffusion models use maximum generation lengths of 256 or 512 tokens, whereas autoregressive Qwen models use 16384 tokens.
- This length difference is necessary because diffusion models must materialize the maximum context length.
- Dream 7B uses temperature 0.2 and top-p 0.95 as default hyperparameters.
D Qualitative Examples
APD supports rapid generation for open-ended persuasive writing, while parallelism varies across prompts and remains lower than on more regular reasoning tasks.
- APD generates persuasive arguments for open-ended prompts, including gas-car bans, identity verification, and unrestricted emotional-companion AI.
- Open-ended generation performs well with APD, but its parallel-generation rate is lower than for grade-school math questions.
- Average Parallel Tokens per Iteration reaches 3.38 for another qualitative example.
- Average Parallel Tokens per Iteration reaches 2.92 for a third qualitative example.
- Average Parallel Tokens per Iteration reaches 2.88 for one qualitative example.