PaperlayerView exact revision on arXiv

Source-linked AI summary

dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Caching

Zhiyuan Liu, Yicun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyan Wei, Shaobo Wang, Yichen Zhu, Linfeng Zhang

arXiv:2506.06295v3cs.LGcs.AIcs.CL

TL;DR

dLLMs suffer high inference latency because repeated bidirectional-attention recomputation makes traditional ARM KV caching incompatible. dLLM-Cache uses long-interval prompt caching and similarity-guided partial response updates, with reported acceleration up to 9.1× while maintaining competitive quality. The approach remains approximate and can add memory overhead for long sequences.

  • Problem

    dLLM inference has O(N^3) complexity from repeated bidirectional-attention recomputation, while traditional ARM KV caching is incompatible with dLLMs.

  • Method

    dLLM-Cache is a training-free adaptive framework combining long-interval prompt caching, short-interval response caching, and Value-similarity-guided partial updates.

  • Results

    Up to 9.1× FLOPs reduction is reported across LLaDA and Dream while maintaining competitive generation quality.

  • Takeaways & Limitations

    The method brings dLLM inference latency close to ARM latency under many settings while preserving a competitive speed-quality trade-off.

  • Takeaways & Limitations

    Caching intermediate features increases memory overhead for long-context scenarios, and the method approximates rather than exactly reproduces KV caching.

Abstract

from arXiv · show

Autoregressive Models (ARMs) have long dominated the landscape of Large Language Models. Recently, a new paradigm has emerged in the form of diffusion-based Large Language Models (dLLMs), which generate text by iteratively denoising masked segments. This approach has shown significant advantages and potential. However, dLLMs suffer from high inference latency. Traditional ARM acceleration techniques, such as Key-Value caching, are incompatible with dLLMs due to their bidirectional attention mechanism. To address this specific challenge, our work begins with a key observation that dLLM inference involves a static prompt and a partially dynamic response, where most tokens remain stable across adjacent denoising steps. Based on this, we propose dLLM-Cache, a training-free adaptive caching framework that combines long-interval prompt caching with partial response updates guided by feature similarity. This design enables efficient reuse of intermediate computations without compromising model performance. Extensive experiments on representative dLLMs, including LLaDA 8B and Dream 7B, show that dLLM-Cache achieves up to 9.1x FLOPs reduction on LongBench-HotpotQA while maintaining competitive output quality. Notably, our method brings dLLM inference latency close to that of ARMs under many settings. The code for this work is publicly available at: https://github.com/maomaocun/dLLM-cache.

1. Introduction

dLLM inference is expensive because bidirectional attention repeatedly recomputes all tokens, while prompt and response features exhibit distinct temporal redundancies. dLLM-Cache exploits these patterns with differentiated, training-free caching and similarity-guided response updates.

  • Motivation: dLLM inference has O(N^3) computational complexity because each of N denoising iterations recomputes bidirectional attention across the sequence.By contrast, ARMs reduce overall effort to O(N^2) through Key-Value caching.
  • Motivation: Prompt features are quasi-static, whereas response features evolve non-uniformly, making uniform caching strategies ineffective.The prompt remains constant, but only some response tokens change substantially between adjacent denoising steps.
  • Method: dLLM-Cache combines long-interval prompt caching with short-interval, similarity-guided partial updates for response tokens.The framework is training-free and differentiates caching behavior according to prompt and response dynamics.
  • Method: V-verify uses Value-vector cosine similarity to identify the most changed response tokens for selective feature recomputation.Value similarity closely tracks changes in attention and feedforward outputs, supporting it as an update indicator.
  • Results: Up to 9.1× acceleration is reported on LLaDA while maintaining competitive output quality.The method achieves a superior speed-quality trade-off compared with the baseline and simpler caching methods.

2. Related Work

Related work establishes dLLMs as bidirectional diffusion-based alternatives to ARMs and reviews caching approaches developed for each paradigm. Because dLLM bidirectional attention is incompatible with traditional ARM KV caching, dLLM-Cache targets a distinct caching problem.

  • Diffusion Language Models: Masked diffusion language models have emerged as discrete-text adaptations of diffusion models, with recent scaling work showing performance comparable to similarly sized ARMs.Their bidirectional design also addresses limitations such as the reversal curse and supports reasoning and multimodal extensions.
  • ARM Caching: ARM Key-Value caching exploits causal attention to reuse past tokens’ states, reducing computation at the cost of cache growth with input length.Prior work addresses long-context bottlenecks by sparsifying caches retrospectively.
  • dLLM Caching: Traditional KV caching is incompatible with dLLMs’ bidirectional attention, while prior dLLM caching methods may require cache-aware training or restrictive conditions.This creates a distinct acceleration challenge for diffusion language models.
  • dLLM Caching: dLLM-Cache exploits asymmetric prompt-response dynamics and caches intermediate features beyond KV states, including Attention Output and FFN Output.Concurrent methods instead emphasize delayed KV reuse or block-wise approximate KV caching with confidence-aware decoding.

3. Methodology

dLLMs generate responses through repeated bidirectional denoising, motivating a cache that reuses stable prompt and response features. dLLM-Cache periodically refreshes caches and selectively recomputes response tokens using Value-vector similarity.

  • Inference Process: dLLMs iteratively denoise a fully masked response through K discrete steps, using a mask predictor, greedy clean-sequence estimate, and transition function.The transition selectively updates tokens based on the estimated clean sequence, but repeated recomputation causes high latency as K grows.
  • Cache Design: dLLM-Cache stores prompt and response features separately, with prompt refresh interval Kp, response refresh interval Kr, and adaptive update ratio ρ controlling reuse.The caches include layer-wise Key, Value, Attention Output, and FFN Output features.
  • Prompt Cache: Prompt features are recomputed every Kp steps and reused otherwise because the static prompt remains largely stable over time.This reduces the cost of processing prompt tokens, particularly when Kp is large.
  • Response Cache: Response features receive periodic full refreshes every Kr steps and adaptive partial updates between refreshes.At full refreshes all response features are recomputed; otherwise the cache updates only selected response tokens.
  • Adaptive Update: V-verify selects the ⌊ρ|y(k)|⌋ response tokens with the lowest Value-vector similarity for recomputation and reuses cached values for the rest.Value changes correlate with downstream Attention Output and FFN Output changes, enabling lightweight identification of dynamic tokens.
  • Adaptive Update: The adaptive strategy leverages temporal stability to reduce computation while preserving accuracy.The implementation overwrites the cached response Value vectors while selectively recomputing and scattering back Key, Attention Output, and FFN Output for selected tokens.

4. Experiments

Experiments evaluate dLLM-Cache on LLaDA 8B and Dream 7B across eight benchmarks using throughput, FLOPs per token, and task performance. Results show broad efficiency gains with largely preserved quality, including strong comparisons against other acceleration approaches and favorable update-ratio behavior.

  • Main Results: Across eight benchmarks, dLLM-Cache consistently improves inference efficiency while largely preserving task performance for LLaDA 8B and Dream 7B.The evaluation includes Base and Instruct variants of both models, with throughput, FLOPs, and benchmark scores reported.
  • Main Results: 8.08× FLOPs speedup reduces LLaDA Instruct’s GPQA cost from 22.07T to 2.73T without accuracy degradation.
  • Comparisons: 2.8× speedup preserves 70.66% accuracy for accelerated LLaDA, whereas reducing denoising steps to 32 lowers accuracy to 22.25%.Combining dLLM-Cache with SlowFast Sampling raises throughput to 49.86 TPS, comparable to Llama 3 8B under this efficiency reference.
  • Comparisons: 5.33× speedup on GPQA with Dream Base outperforms dKV-Cache’s 1.74× and Fast-dLLM’s 3.83× while maintaining 34.15% accuracy.The comparison attributes the efficiency advantage to V-verify-guided bypassing of entire transformer layers, including Attention and FFN projections.
  • Ablations: At ρ = 0.25, value-based selection achieves the highest GSM8K accuracy while requiring significantly fewer FLOPs than full recomputation.Both similarity-based strategies outperform random selection across a wide range of update ratios.

5. Discussion

dLLM-Cache reduces inference cost through selective caching while preserving output quality, but its speedup is limited by fixed overheads from activating selective updates.

  • Effect of Denoising Steps: 5× lossless speedup at 256 denoising steps matches the computational cost of an approximately 48-step baseline while more than doubling its accuracy on GSM8K.Simple step reduction accelerates inference but causes severe performance degradation.
  • Storage Overhead of Caching: dLLM-Cache stores K, V, AttnOut, and FFNOut features per layer, with cache cost scaling as T × d × 4 × L.Only one version per layer is cached, keeping the overall footprint stable.
  • Storage Overhead of Caching: Less than 1 GB of additional memory, a 5% increase over baseline, accompanies up to 9× generation acceleration on GSM8K with LLaDA 8B Base.Peak GPU usage is 16.94 GB without caching and 17.93 GB with dLLM-Cache.
  • Cost of V-verify and the Fixed Update Overheads: Fixed operational overheads from GPU kernel management and data movement limit runtime savings at very low response-update fractions ρ.An effective ρ must balance these activation costs against saved dynamic computation and model quality.
  • Cost of V-verify and the Fixed Update Overheads: ρ ≈0.25 offers an effective trade-off between selective-update activation costs, reduced computation, and output fidelity.The reported setting optimizes overall efficiency and fidelity.

6. Conclusion

The paper presents dLLM-Cache as a training-free, model-agnostic caching method for diffusion-based language models. Evaluations on LLaDA and Dream report up to 9.1× FLOPs reduction while maintaining competitive generation quality.

  • dLLM-Cache is a training-free, model-agnostic method for accelerating inference in diffusion-based large language models.
  • Evaluations on LLaDA and Dream show up to 9.1× FLOPs reduction with competitive generation quality.

A.1. Performance on Long and Semantically Diverse Scenarios

dLLM-Cache is especially effective for long static prompts and retains performance across diverse LongBench tasks. On LongBench-HotpotQA, it reduces FLOPs while improving F1 for LLaDA 8B Base.

  • Performance on Long and Semantically Diverse Scenarios: 9.1× FLOPs reduction over the baseline on LongBench-HotpotQA accompanies an F1 increase from 34.56 to 36.10 for LLaDA 8B Base.
  • Performance on Long and Semantically Diverse Scenarios: Table 6 compares LongBench performance for LLaDA Instruct and Dream Instruct with and without dLLM-Cache.
  • Performance on Long and Semantically Diverse Scenarios: LLaDA Instruct retains a 38.61 average LongBench score compared with the 39.14 baseline across six major task categories.

A.2. Detailed Comparison with Concurrent Caching Methods

The paper provides a comprehensive comparison of dLLM-Cache with concurrent caching methods on both LLaDA and Dream. These comparisons are presented in Table 7.

  • Detailed Comparison with Concurrent Caching Methods: dLLM-Cache is compared with the concurrent methods dKV-Cache and Fast-dLLM in a comprehensive evaluation.
  • Detailed Comparison with Concurrent Caching Methods: Table 7 compares dLLM-Cache with other concurrent methods on LLaDA and Dream.

A.3. Detailed Sensitivity Analysis on Dream 7B

dLLM-Cache shows consistent behavior across LLaDA and Dream, with Dream 7B sensitivity analyses examining adaptive update ratio, prompt refresh interval, and response refresh interval.

  • dLLM-Cache is reported as effective across LLaDA and Dream architectures, supporting generalizability beyond model-specific artifacts.
  • Dream 7B sensitivity trends are highly consistent with those observed for LLaDA.The analysis presents results for Dream 7B and describes the cross-model trends as stable and predictable.
  • The Dream 7B analysis evaluates sensitivity to the adaptive update ratio ρ on GPQA.
  • The Dream 7B analysis evaluates sensitivity to the prompt refresh interval Kp on GPQA.
  • The Dream 7B analysis evaluates sensitivity to the response refresh interval Kr on GPQA.

A.4. Cross-Layer Behavior of Token Selection

dLLM-Cache selects response tokens independently at each Transformer layer because cached features and V-verify scores are layer-specific. Under a stated margin condition, selected tokens remain stable across adjacent layers, while experiments favor a fixed global update ratio.

  • Cross-Layer Behavior of Token Selection: Cached features and V-verify scores are maintained separately for each Transformer layer, making token selection layer-wise.The caches store K(l), V(l), AttnOut(l), and FFNOut(l) for prompt and response features.
  • Cross-Layer Behavior of Token Selection: If score variation ε is smaller than the selection-boundary margin ∆, bottom-⌊ρ|y|⌋ membership is preserved between adjacent layers.This follows under smoothness assumptions for the layer map and Value projection.
  • Cross-Layer Behavior of Token Selection: Unless many tokens lie near the cutoff, the selected response-token set should remain largely stable across depth.
  • Cross-Layer Behavior of Token Selection: A fixed global ρ = 0.25 achieves the highest GSM8K accuracy among update-ratio allocations with the same average budget.The comparison uses LLaDA Instruct with Kp = 50 and Kr = 7, against front-heavy and back-heavy allocations.

A.5. Design Choice Between Full V-Cache Overwrite and Selective Update

dLLM-Cache reduces dLLM computation by combining periodic full refreshes, response-only refreshes, and adaptive partial updates. Its full V-cache overwrite and cosine-similarity design choices preserve accuracy while enabling substantial FLOPs savings, although caching introduces memory and approximation limitations.

  • V-cache design choice: 79.83% accuracy with full V-cache overwrite exceeds 78.85% from updating only selected Value vectors on GSM8K with LLaDA 8B Instruct.The comparison supports overwriting the full cached Value vectors before the next similarity check.
  • V-verify metric: 78.54% accuracy with cosine similarity exceeds 55.95% with L2 distance on GSM8K with LLaDA 8B Instruct.Cosine similarity is therefore adopted as the default metric for V-verify.
  • Complexity and caching strategy: dLLM-Cache partitions inference into full refreshes, response-only refreshes, and adaptive partial updates.Prompt and response computations are handled at different refresh frequencies, while adaptive updates selectively recompute response tokens.
  • Complexity and caching strategy: The adaptive strategy reduces the quadratic attention term from O(n^2) to O(rn) during response-only refreshes.For partial updates, attention and FFN costs are reduced from 4n^2d to 4r̂nd and from 6ndm to 6r̂dm, respectively.
  • Efficiency: 9.1× FLOPs reduction is achieved in practical scenarios through sparse updates to attention and FFN layers.The adaptive partial-update term uses ρ typically set to 25%, while V-verify adds a lightweight fixed Value-vector computation.
  • Error control and limitations: Theoretical error growth is bounded by response refreshes and tightened by V-verify, but caching increases long-sequence memory overhead and is not fundamentally lossless.The refresh interval prevents indefinite accumulation, while the approximation trades a small amount of accuracy for speed.
Loading 2506.06295v3…