Source-linked AI summary
Orchestrating Dual-Boundaries: An Arithmetic Intensity Inspired Acceleration Framework for Diffusion Language Models
Linye Wei, Wenjue Chen, Pingzhi Tang, Xiaotian Guo, Le Ye, Runsheng Wang, Meng Li
TL;DR
dLLM inference is constrained by cache refreshes that alternate prefill and decoding, with fixed response lengths adding redundant computation. ODB-dLLM combines adaptive length prediction with jump-share speculative decoding. It reports 46–162× speedups over baseline dLLMs and 2.63–6.30× over Fast-dLLM while mitigating accuracy degradation.
Problem
Bidirectional attention prevents dLLMs from using standard KV caching, and cache refreshes create alternating compute- and memory-bound phases that constrain inference speed.
Source: §1 · pp. 1–2Method
ODB-dLLM coordinates the two phases with adaptive length prediction to reduce redundant prefill and jump-share speculative decoding to reduce decoding steps.
Source: §1 · p. 2, §4.1 · p. 5Results
46–162× speedups over baseline dLLMs and 2.63–6.30× over Fast-dLLM, while mitigating accuracy degradation in existing acceleration frameworks.
Source: §6 · p. 10Takeaways & Limitations
Across multiple datasets, ODB-dLLM improves inference speed over baseline dLLMs and Fast-dLLM while mitigating accuracy degradation.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Diffusion-based large language models (dLLMs) have recently gained significant attention for their exceptional performance and inherent potential for parallel decoding. Existing frameworks further enhance its inference efficiency by enabling KV caching. However, its bidirectional attention mechanism necessitates periodic cache refreshes that interleave prefill and decoding phases, both contributing substantial inference cost and constraining achievable speedup. Inspired by the heterogeneous arithmetic intensity of the prefill and decoding phases, we propose ODB-dLLM, a framework that orchestrates dual-boundaries to accelerate dLLM inference. In the prefill phase, we find that the predefined fixed response length introduces heavy yet redundant computational overhead, which affects efficiency. To alleviate this, ODB-dLLM incorporates an adaptive length prediction mechanism that progressively reduces prefill overhead and unnecessary computation. In the decoding phase, we analyze the computational characteristics of dLLMs and propose a dLLM-specific jump-share speculative decoding method to enhance efficiency by reducing the number of decoding iterations. Experimental results demonstrate that ODB-dLLM achieves 46-162x and 2.63-6.30x speedups over the baseline dLLM and Fast-dLLM, respectively, while simultaneously mitigating the accuracy degradation in existing acceleration frameworks.
1 Introduction
Existing dLLM acceleration frameworks interleave compute-bound prefill and memory-bound decoding, while fixed response lengths and cache refreshes retain substantial overhead. ODB-dLLM targets both boundaries with adaptive length prediction and jump-share speculative decoding.
1 Introduction
- Bidirectional attention prevents dLLMs from using standard KV caching, forcing repeated prefill and creating a compute-bound bottleneck.Block-wise cache refresh reduces per-step prefill but still incurs inference cost; fixed response lengths also add redundant computation and can degrade performance.Source: §1 · pp. 1–2
- ODB-dLLM is a training-free framework that coordinates compute- and memory-bound phases using adaptive length prediction and jump-share speculative decoding.Adaptive length prediction reduces redundant prefill, while jump-share reduces decoding steps.Source: §1 · p. 2
- Periodic cache refreshes create alternating compute- and memory-bound phases; decoding has limited arithmetic intensity, while response-length prediction offers a way to reduce prefill work.The proposed framework is motivated by these distinct opportunities at the two boundaries.Source: §1 · p. 2
- 46–162× and 2.63–6.30× speedups over baseline dLLMs and Fast-dLLM, respectively, while mitigating accuracy degradation in Fast-dLLM.The reported speedups are from the introduction’s summary of experiments.Source: §1 · p. 2
2 Background
The background describes masked diffusion generation and block-wise caching, then reviews speculative decoding approaches for autoregressive and diffusion language models. It highlights fixed-length inefficiency and the challenge of reusing rejected dLLM drafts.
2.1 Diffusion Language Models and Acceleration Techniques
- Masked diffusion models progressively replace tokens with [MASK] and generate by unmasking multiple tokens while moving from noise level t to an earlier level s.Fast-dLLM uses blocks and DualCache to cache tokens outside the active block, refreshing the global cache once that block is filled.Source: §2.1 · pp. 2–3
- Fixed response lengths trade performance against redundant computation; dLLM-Var requires training, while DAEDAL uses a multi-stage training-free heuristic.The paper motivates a more direct training-free strategy for adjusting generation length during cache refreshes.Source: §2.1 · p. 3
2.2 Speculative Decoding in LLM/dLLM
- Autoregressive speculative decoding drafts tokens and verifies them in parallel; dLLM methods instead use the dLLM itself for block-wise proposals and verification.SSD uses hierarchical verification trees, while Spiffy uses a directed draft graph; rejected blocks can leave computation and information unused.Source: §2.2 · p. 3
3 Motivation
dLLM inference alternates prefill and decoding, with prefill accounting for 30–40% of latency. Fixed response lengths waste prefill computation, while hard acceptance thresholds defer some correct tokens.
Observation 1
- 30–40% of total inference latency comes from prefill across GSM8K, Minerva Math, and BBH, and finer-grained decoding may make it increasingly dominant.The profiling motivates an arithmetic-intensity-aware approach to both compute- and memory-bound phases.Source: Figure 4 · p. 4, §3 · p. 4
Observation 2
- Most queries need only 1/4 to 1/2 of their preallocated tokens for correct answers, leaving redundant response space that can dominate latency.The fixed default is typically 1024 tokens; the remainder may become meaningless or repetitive content.Source: §3 · pp. 4–5, Figure 5 · p. 5
Observation 3
- Many correct tokens below the fixed confidence threshold are accepted in later rounds, adding decoding iterations; lowering the threshold naively compromises accuracy.The authors identify speculative acceptance of these below-threshold tokens as an optimization opportunity, but note that dLLM speculation can become compute-bound as candidates grow.Source: §3 · p. 5
4 ODB-dLLM Framework
ODB-dLLM addresses compute- and memory-bound costs in dLLM inference with adaptive response-length prediction and jump-share speculative decoding. Its strategies progressively reduce redundant prefill work and decoding steps while keeping speculative computation within the memory-bound region.
4 ODB-dLLM Framework
- ODB-dLLM explicitly orchestrates the interleaved compute- and memory-bound phases with adaptive length prediction and jump-share speculative decoding.Length prediction targets redundant prefill computation; jump-share targets decoding steps under limited arithmetic intensity.Source: §4 · p. 5
4.1 Adaptive Length Prediction
- During prefill, ODB-dLLM reads [EOS] positions as length signals and truncates only when an identified token’s confidence exceeds a predefined threshold.If no [EOS] is found, or all detected tokens fall below the threshold, response length remains unchanged; the prediction progressively aligns with required output length.Source: §4.1 · pp. 5–6
4.2 Jump-share Speculative Decoding
- Naive speculation can exceed hardware’s memory-bound limit as arithmetic intensity grows with speculative blocks; jump-share strategies improve acceptance while maintaining that limit.The framework uses accept-jump and decoded-share in two stages.Source: §4.2 · p. 7
4.2.1 Accept-jump Speculative Strategy
- Accept-jump uses three speculative blocks for the top-confidence token, the second-highest-confidence token, or both, then jumps to a matching decoding result when verified.This lets more tokens be unmasked within one decoding iteration under a limited candidate budget.Source: §4.2.1 · p. 7
4.2.2 Decoded-share Speculative Strategy
- Decoded-share first uses accept-jump, then recomputes already decoded tokens once in block0 and shares their KV pairs across speculative blocks as decoding advances.The strategy adds four extra candidates to simplify control logic and improve end-to-end speedup.Source: §4.2.2 · p. 7
5 Experimental Results
Across two diffusion language models and five benchmarks, ODB-dLLM reports higher speedups than vanilla baselines and Fast-dLLM while mitigating accuracy degradation. Ablations and speculative-variant analyses examine the contributions to efficiency and accuracy.
5.1 Experimental Setup
- Evaluation uses LLaDA-Instruct and LLaDA-1.5 across GSM8K, MATH, BBH, HUMANEVAL, and MBPP on one NVIDIA A100 80GB GPU.The implementation builds on Fast-dLLM and disables padding and batch inference for consistency with vanilla LLaDA.Source: §5.1 · pp. 7–8
5.2 Main Results & Ablation Study
- 46×–162× speedup on LLaDA-Instruct and 50–182× on LLaDA-1.5 over vanilla, plus 2.63–6.30× and 2.60–7.22× over Fast-dLLM, respectively; Fig. 9 shows the highest speedup across baselines.The authors also report that ODB-dLLM substantially mitigates Fast-dLLM’s accuracy degradation, matching or sometimes surpassing vanilla models.Source: §5.2 · p. 8, Table 2 · p. 8, Figure 9 · p. 9
- 2.84× average speedup over Fast-dLLM with ALP; response length averages 272 tokens versus 1024 fixed and DAEDAL’s 625, with a 1.16-point overall improvement.The authors attribute the improvement to reduced prefill redundancy and report higher accuracy than truncating at the first [EOS].Source: Figure 10 · p. 9, §5.2 · p. 9
- 4.19× speedup over Fast-dLLM and a 1.48-point accuracy improvement with Jump-Share Speculative Decoding.In dLLMs, speculative decoding can alter token acceptance order and response content because of bidirectional attention and parallel generation.Source: Figure 10 · p. 9, §5.2 · p. 9
5.3 Speculative Variants
- Accept-jump raises accepted tokens per step, while decoded-share further reduces decoding rounds; vanilla speculation’s speedup is limited despite fewer function evaluations.The authors associate vanilla speculation’s limited speedup with its small candidate set and an attention mask incompatible with Flash Attention.Source: §5.3 · pp. 9–10, Figure 11 · p. 10
- Vanilla speculation degrades accuracy, whereas accept-jump improves it and can surpass the non-speculative baseline; decoded-share yields the highest overall accuracy among variants.The paper introduces effective NFE to measure effective state changes; it reports that lower effective NFE indicates greater likelihood of conflicts among independently sampled tokens.Source: Figure 11 · p. 10, §5.3 · p. 10
6 Conclusion
ODB-dLLM combines adaptive length prediction and jump-share speculative decoding to accelerate diffusion language model inference. Experiments report speedups over baseline dLLMs and Fast-dLLM while mitigating accuracy degradation.
6 Conclusion
- ODB-dLLM achieves 46–162× speedup over baseline dLLMs and 2.63–6.30× over Fast-dLLM while mitigating accuracy degradation.The framework targets interleaved compute- and memory-bound phases with adaptive length prediction and jump-share speculative decoding.Source: §6 · p. 10