Source-linked AI summary
Dependency-Aware Revocable Decoding for Efficient Diffusion Large Language Model Inference
Wooje Park, Insu Lee, Minyoung Noh, Jaeyun Jang, Sungmin Lee, Kyuhong Shim, Byonghyo Shim
TL;DR
dLLM parallel decoding can reduce inference cost, but unreliable tokens may contaminate later contexts and compromise verification. DARD is a training-free, three-state revocable decoding framework that verifies candidates selectively and regulates their contextual influence. Across 12 textual and multimodal benchmarks on 3 open-source dLLMs, DARD consistently improves the speed-quality Pareto frontier over recent revocable methods.
Problem
Increasing dLLM decoding parallelism can produce inconsistent tokens, while existing revocable methods may let unreliable tokens corrupt the verification context.
Method
DARD uses masked, candidate, and unmasked token states to verify candidates with selective reliable context and adaptively regulate candidate influence.
Results
DARD consistently improves the speed-quality Pareto frontier across 12 benchmarks and 3 open-source dLLMs, achieving a 2.71× speedup and 4.35-point CIDEr gain over Saber on Flickr30K.
Takeaways & Limitations
Controlling verification context is a key ingredient for reliable and efficient parallel decoding in dLLMs.
Takeaways & Limitations
DARD preserves the base model’s generation behavior, so absolute gains remain moderate, and its augmented shadow sequence and state-specific masks add per-step computation.
Abstract
from arXiv · showhide
Diffusion large language models (dLLMs) offer a promising alternative to autoregressive generation by decoding multiple tokens in parallel through iterative denoising. However, increasing decoding parallelism often degrades generation quality, as early errors can contaminate later contexts. Revocable decoding mitigates this issue by re-evaluating decoded tokens and remasking unreliable ones, but existing methods overlook that unreliable tokens may also corrupt the verification context itself. We identify this failure mode and propose Dependency-Aware Revocable Decoding (DARD), a training-free framework that separates tokens into masked, candidate, and unmasked states. DARD verifies candidate tokens using a selective context that excludes less reliable tokens and adaptively regulates their influence on subsequent decoding. Experiments across 12 textual and multimodal benchmarks on 3 open-source dLLMs show that DARD consistently improves the speed-quality Pareto frontier over recent revocable decoding methods, achieving a 2.71$\times$ speedup and a 4.35-point CIDEr score gain over Saber on Flickr30K.
1 Introduction
dLLMs accelerate generation by decoding multiple tokens in parallel, but unreliable tokens can corrupt later contexts and even undermine revocable verification. DARD addresses this with reliability-aware token states and selective context, improving the speed-quality frontier across broad evaluations.
- Motivation: Parallel dLLM decoding speeds inference by predicting multiple masked positions simultaneously, but limited same-step context can produce inconsistent tokens.These errors may remain visible as misleading context for future predictions.
- Motivation: Existing revocable methods re-evaluate decoded tokens and remask unreliable ones, but may still use erroneous tokens when verifying other tokens.Saber tracks confidence drops, while WINO uses an auxiliary verification path.
- Method: DARD uses masked, candidate, and unmasked states to selectively control contextual dependencies during verification.Candidate tokens attend to all unmasked tokens and only higher-confidence candidates, while masked-token reliance on candidates is adaptively regulated.
- Results: DARD consistently improves the speed-quality Pareto frontier across 12 benchmarks and 3 open-source dLLMs.On Flickr30K, it achieves a 2.71× speedup and a 4.35-point CIDEr gain over Saber.
- Problem: DARD identifies context contamination as a failure mode that can cause persistent errors or unnecessary remasking.The “Los Diego” example illustrates how mutually inconsistent tokens can generate conflicting verification signals.
2 Related Work
Related work accelerates dLLM inference through efficient attention-state reuse or by decoding more tokens per denoising step. These approaches address computation or step count using caching, pruning, and model-internal confidence signals.
- dLLM generation: dLLMs generate text by iteratively denoising masked sequences, enabling flexible-order and parallel token generation.This differs from autoregressive models, which generate tokens sequentially from left to right.
- Efficient inference: One line of work reduces per-step computation by adapting Key-Value caching to dLLMs.Because bidirectional attention changes KV states across denoising steps, methods use block-wise generation, approximate reuse, delayed caching, or pruning.
- Efficient inference: Another line reduces denoising steps by decoding more tokens at each step.Some methods use confidence, entropy, or probability margins to determine which tokens can be safely committed.
3 Dependency-Aware Revocable Decoding
DARD performs diffusion decoding with state-specific contexts, separating masked, candidate, and unmasked tokens for selective verification. It orders candidate-token dependencies by confidence and adaptively combines verification-informed predictions for masked positions.
- Parallelized inference over distinct decoding contexts: DARD augments the partially decoded sequence with a fully masked shadow sequence to compute predictions under distinct contexts in parallel.The original and shadow sequences share positional embeddings, while attention masks determine their accessible keys.
- State transition: DARD assigns each position to masked, candidate, or unmasked states according to confidence thresholds, then updates states and token values at each decoding step.Masked positions remain [Mask], whereas candidate and unmasked positions receive their corresponding predictions.
- Verification of U tokens with reliable context: Unmasked tokens are verified using reliable-token context while uncertain candidate tokens are excluded, preventing candidate contamination of verification results.Unmasked queries attend to unmasked keys in the original sequence and to masked or candidate keys in the shadow sequence.
- Verification of C tokens with confidence-ordered context: Candidate tokens are verified with confidence-ordered context, allowing higher-confidence candidates to provide context for lower-confidence candidates.The attention mask blocks information flow from lower-confidence candidates to higher-confidence candidates, approximating confidence-ordered multi-step decoding.
- Adaptive prediction of M tokens based on verification results: For masked positions, DARD combines original- and shadow-sequence predictions using distance-weighted promotion and demotion outcomes from candidate tokens.The geometric kernel makes nearby verification outcomes exert greater influence on each masked position, while the guidance weight favors predictions conditioned on predominantly promoted candidates.
- Evaluation: Figure 5 evaluates task performance against average decoding steps, with method-specific Pareto frontiers, best configurations, TPS annotations, and a 64-step standard-decoding reference.The figure covers six language benchmarks using LLaDA-8B-Instruct.
4 Experiments
DARD is evaluated on language and vision-language benchmarks using speed-quality Pareto curves, with results showing favorable trade-offs across both task families. It generally matches or exceeds recent revocable decoding methods with fewer decoding steps.
- Language Benchmarks: DARD generally achieves a more favorable Pareto frontier than WINO and Saber across six language benchmarks.It attains comparable or higher performance with fewer decoding steps.
- Language Benchmarks: DARD performs well on Countdown and Sudoku, which involve structured reasoning and strict-format requirements.
- Language Benchmarks: On MBPP and GSM8K, DARD shows a slight performance drop but achieves similar performance with substantially fewer decoding steps.
- Vision-Language Benchmarks: DARD achieves a better Pareto frontier than recent revocable decoding methods across most of six vision-language benchmarks.The comparison plots task performance against average decoding steps, with dashed curves marking each method’s Pareto frontier.
- Vision-Language Benchmarks: Compared with Saber, DARD reduces step count by more than 2× while improving Flickr30K by over 4 CIDEr points and AI2D by over 2 accuracy points.
5 Analysis
The analysis examines DARD’s attention-mask and logit-mixing choices, robustness across decoding configurations, and qualitative behavior on GSM8K. Results favor confidence-based verification and adaptive context control over unrestricted candidate interaction.
- Attention Mask Design: A bidirectional mask reduces decoding steps by only 0.8 but causes a 3.2-percentage-point accuracy drop.Allowing uncertain candidate tokens to freely interact makes verification unstable despite slightly improving efficiency.
- Attention Mask Design: Confidence-based ordering maintains substantially higher accuracy while achieving comparable decoding efficiency to alternative ordering strategies.The alternatives include entropy, margin, and left-to-right ordering.
- Adaptive Logit Mixing: Fully incorporating candidate-token context with w = 1.0 achieves the fewest decoding steps, while adaptive logit mixing delivers higher accuracy with only a slight increase in steps.The mixing strategy controls how much candidate-token context contributes to masked-token prediction.
- Configuration Robustness: DARD remains stable across different generation lengths and block lengths.This indicates robustness to these generation configurations.
- Qualitative Comparison: On GSM8K, WINO can revise a correct answer into an incorrect one, while Saber fails to properly correct early errors and DARD refines the reasoning path more reliably.DARD performs verification in confidence order.
6 Conclusion
The paper identifies verification-context contamination as a failure mode in dLLM revocable decoding and introduces DARD to address it. Across diverse benchmarks and models, DARD improves the speed-quality Pareto frontier while emphasizing controlled verification context.
- Incorrectly decoded tokens can distort verification context and lead to incorrect remasking decisions.
- DARD is a training-free three-state framework that separates tokens by confidence and decouples decoding and verification across states.Its states are masked, candidate, and unmasked.
- Across six textual and six multimodal benchmarks on three open-source dLLMs, DARD consistently improves the speed-quality Pareto frontier over recent revocable decoding baselines.The results highlight controlling verification context as important for reliable and efficient parallel decoding.
Limitations
DARD’s evaluation covers diverse language and vision-language benchmarks, but its gains remain moderate and its state-specific processing adds computation. The method preserves the base model’s generation behavior while incurring a small practical runtime overhead.
- DARD’s absolute performance gain remains moderate because it preserves the original dLLM prediction behavior rather than altering the model distribution.
- DARD introduces additional per-step computation through the augmented shadow sequence and state-specific attention masks.The overhead is reported as small in practice, especially with block decoding.
- Evaluation Scope: The evaluation spans six language benchmarks and six vision-language benchmarks, with task-specific metrics including accuracy and CIDEr.The reported benchmark suite covers mathematical, code, arithmetic, logical, commonsense/science, and vision-language tasks.
A.2 Implementation Details
The experiments use block decoding with open-source dLLMs and evaluate speed–quality trade-offs under varied generation budgets.
- All experiments adopt block decoding.
- Language benchmarks use LLaDA-8B-Instruct and LLaDA-1.5, while vision-language benchmarks use MMaDA-8B-MixCoT.
- Unless otherwise specified, experiments use generation length 256 and block length 128.
- Generation budgets are varied to analyze the speed–quality trade-off.
B.1 Evaluation on LLaDA 1.5
DARD improves speed–quality trade-offs across models, generation budgets, and multimodal benchmarks, while adaptive verification regulates unreliable candidate context. Its confidence ordering also tracks subsequent decoding trajectories.
- B.1 Evaluation on LLaDA 1.5: DARD consistently improves the speed–quality trade-off on LLaDA 1.5, showing effectiveness beyond the main model.
- B.1 Evaluation on LLaDA 1.5: DARD achieves a better speed–quality trade-off than competing methods on MATH-500 with 512- and 1024-token generation budgets.
- B.1 Evaluation on LLaDA 1.5: DARD achieves a better speed–quality Pareto frontier across MMVP, BLINK, and HRBench and remains robust to threshold selection.
- B.1 Evaluation on LLaDA 1.5: Adaptive mixing assigns high weights to candidate context in most decoding steps but lowers them below 0.5 when that context is unreliable.
- B.1 Evaluation on LLaDA 1.5: Confidence-based ordering aligns better with the one-token-per-step reference trajectory than left-to-right ordering across both evaluated models.
- B.1 Evaluation on LLaDA 1.5: DARD achieves higher peak accuracy than the best default one-token-per-step configuration across all six evaluated datasets.
- B.1 Evaluation on LLaDA 1.5: DARD’s fixed threshold settings remain above prior Pareto frontiers across all five datasets evaluated for each architecture, without task-specific tuning.
- B.1 Evaluation on LLaDA 1.5: Increasing τu generally increases decoding steps and performance, whereas decreasing τc reduces steps but often lowers accuracy.
C.5 Analysis of Computational Efficiency and Overhead
DARD generally delivers higher throughput and task performance with limited computational overhead, while its dependency-aware verification models joint token probabilities under confidence-ordered attention.
- Tokens-per-second equals generated output tokens divided by total wall-clock decoding time, so higher TPS means shorter end-to-end time when output length is fixed.
- DARD usually achieves higher TPS and better task performance simultaneously; when Saber is faster, DARD exceeds it by 7.4 points on ARC-C and 5.6 points on Countdown.
- DARD adds negligible single-pass inference overhead compared with WINO and compensates for slightly higher overhead than Saber with fewer decoding steps.
- Peak GPU memory increases by less than 6.3% in the worst case, including block length 256.
- DARD duplicates only the currently decoded block in its shadow sequence, producing a small relative attention overhead.
- DARD’s mask lets each candidate attend only to higher-confidence vocabulary tokens and unmasked tokens, allowing verification to reflect joint committed-token probabilities.
F Handling Contamination Propagated to Low-Confidence Tokens
DARD limits contamination from erroneous high-confidence context by enforcing confidence-ordered information flow and removing demoted tokens from later verification.
- A high-confidence token can still affect lower-confidence tokens, but iterative verification mitigates this propagation after the token is demoted.
- After demotion, an erroneous token is excluded from subsequent conditioning, lowering the confidence of tokens it affected and making their demotion more likely.
- In the “football” and “field” example, higher-confidence “field” demotes because it cannot use lower-confidence “football” as context, while “football” promotes using “field.”
- Qualitative comparisons report that DARD resolves decoding errors and reduces their propagation during verification.
I Ethics Statement
The work uses open-source datasets and publicly available model weights under their respective licenses. Its implementation is intended for reproducible research on efficient dLLM decoding.
- Open-source datasets and publicly available model weights are used in accordance with their respective licenses.
- The materials are intended for research or evaluation purposes.
- The DARD implementation is intended to support reproducible research on efficient dLLM decoding.