DPad: Efficient Diffusion Language Models with Suffix Dropout

Xinhua Chen; Sitao Huang; Cong Guo; Chiyue Wei; Yintao He; Jianyi Zhang; Hai "Helen" Li; Yiran Chen

Source-linked AI summary

DPad: Efficient Diffusion Language Models with Suffix Dropout

Xinhua Chen, Sitao Huang, Cong Guo, Chiyue Wei, Yintao He, Jianyi Zhang, Hai "Helen" Li, Yiran Chen

arXiv:2508.14148v2Aug 19, 2025cs.CLcs.LG

TL;DR

dLLMs parallelize denoising but repeatedly compute redundant future suffix tokens, creating a major efficiency bottleneck. DPad is a training-free method that retains nearby suffix context through a sliding window and distance-decay dropout. It achieves up to 61.4× speedup over vanilla dLLMs while maintaining comparable accuracy, though performance degrades at very long sequence lengths.

Problem
dLLMs compute many redundant suffix tokens during parallel denoising, increasing computational cost and potentially reducing generated-content fidelity.
Method
DPad combines a fixed-length sliding window with distance-decay dropout to prune distant suffix tokens before attention computation.
Results
Up to 61.4× speedup over vanilla dLLMs is achieved while maintaining comparable model accuracy.
Takeaways & Limitations
DPad provides a simple, training-free route to efficient long-sequence dLLM inference and works with existing optimizations such as prefix caching.
Takeaways & Limitations
Performance degrades in very long-sequence generation, particularly at context length 2048, which the authors attribute to distributional shift from suffix dropout.

Abstract

from arXiv · show

Diffusion-based Large Language Models (dLLMs) parallelize text generation by framing decoding as a denoising process, but suffer from high computational overhead since they predict all future suffix tokens at each step while retaining only a small fraction. We propose Diffusion Scratchpad (DPad), a training-free method that restricts attention to a small set of nearby suffix tokens, preserving fidelity while eliminating redundancy. DPad integrates two strategies: (i) a sliding window, which maintains a fixed-length suffix window, and (ii) distance-decay dropout, which deterministically removes distant suffix tokens before attention computation. This simple design is compatible with existing optimizations such as prefix caching and can be implemented with only a few lines of code. Comprehensive evaluations across multiple benchmarks on LLaDA-1.5 and Dream models demonstrate that DPad delivers up to $\mathbf{61.4\times}$ speedup over vanilla dLLMs while maintaining comparable accuracy, highlighting its potential for efficient and scalable long-sequence inference. Our code is available at https://github.com/Crys-Chen/DPad.

1. Introduction

dLLMs enable parallel denoising but waste computation on redundant future suffix tokens that can also reduce fidelity. DPad addresses this with localized, training-free suffix attention and reports large speedups while maintaining comparable accuracy.

Motivation: dLLMs compute predictions for all future suffix tokens at each denoising step even though only a small fraction are retained, undermining their throughput gains.The resulting computation is a key bottleneck to widespread dLLM adoption.
Motivation: Most suffix tokens act as a low-entropy information reservoir rather than providing direct semantic information, and their redundancy increases with distance from the current block.Distant suffix attention scores drop sharply, while redundant tokens can impose overhead and degrade generated-content fidelity.
DPad: DPad restricts attention to nearby suffix tokens using a fixed-length sliding window and distance-decay dropout before attention computation.The sliding window moves with the current block, while farther tokens receive higher dropout ratios and tokens beyond the window are omitted.
DPad: The method is training-free, requires only a few lines of code, and constructs a sparse suffix-token subset that can preserve accuracy comparable to vanilla dLLMs.The paper frames this subset as an inference-time “winning ticket” for suffix tokens.
Results: 61.39× speedup on LLaDA-1.5/GSM8K and 30.58× on Dream/HumanEval at sequence length 1024, while maintaining comparable model accuracy.These gains combine DPad with parallel generation and prefix caching and are measured against vanilla dLLMs.
DPad: DPad avoids fixed sequence-length enforcement and remains compatible with prefix caching, dual caching, and dLLM-Cache while accelerating shorter generations.The paper reports stable acceleration at generation lengths 256 and 512 and compatibility with existing optimizations.

2. Preliminary

dLLMs learn to denoise masked sequences with bidirectional attention, then iteratively unmask high-confidence positions. Related acceleration methods dynamically unmask tokens or reuse and evict cache entries based on persistent attention patterns.

2.1. Foundational Principles of Diffusion Large Language Models (dLLM): dLLMs formulate generation as non-autoregressive denoising, training a model to recover original tokens from partially masked sequences.Forward masking replaces clean tokens with [MASK], while reverse unmasking predicts the original tokens from the corrupted sequence.
2.1. Foundational Principles of Diffusion Large Language Models (dLLM): Bidirectional multi-head attention processes masked and unmasked tokens together to learn the distribution for each masked position.The Transformer is trained to model pθ(x0|xt) over the sequence.
2.2. Inference and Sampling in dLLM: At inference, the model initializes a prompt followed by mask tokens and repeatedly predicts masked positions, selecting high-confidence positions for unmasking until none remain.A scheduling function determines how many positions to update at each step, and the masked-index set shrinks after each update.
2.3. Related work on dLLM acceleration: Acceleration methods use dynamic confidence-aware unmasking and cache management because bidirectional attention prevents direct application of conventional KV caching.Prefix and distant suffix attention values can be reused across steps, while finer-grained methods evict cache entries dynamically using attention scores.

3. Method

DPad interprets suffix tokens as a cross-layer scratchpad and reduces their redundancy with bounded, distance-aware dropout strategies. The method preserves nearby suffix memory, dynamically resamples retained tokens, and frames sparse inference as a training-free lottery-ticket search.

3.1. Scratchpad Mechanism: DPad models suffix tokens as an information reservoir that aggregates prefix and current signals, then returns them to the current block across consecutive layers.The scratchpad follows a write–store–read cycle, while suffix-to-prefix influence is considered negligible in practice.
3.2. Suffix Dropout Strategies: DPad combines a fixed-length sliding window with distance-decay dropout to retain nearby suffix tokens while progressively pruning more distant ones.Both strategies are implemented through Gaussian sampling that enforces bounded retention and distance-dependent decay.
3.2. Suffix Dropout Strategies: Suffix dropout keeps suffix-related computation bounded independently of sequence length, reducing vanilla dLLM operations from 𝑂(𝐿) per step to a constant number of retained tokens.In vanilla block-wise dLLMs, suffix computation requires 𝐿/𝐵 steps with 𝑂(𝐿) suffix-token operations per step.
3.2. Suffix Dropout Strategies: Gaussian retention probability decreases with distance from the suffix boundary, and dynamic block-wise resampling restores dropped tokens before each subsequent decoding block.Resampling prevents sampling bias and avoids systematically ignoring any token.
3.2. Suffix Dropout Strategies: DPad also halts fixed-length generation after an <eos> token by filling the remaining sequence with <eos>.The conditional check runs after each decoded block.
3.2. Suffix Dropout Strategies: The implementation preserves original RoPE positions after sparse suffix selection through a lightweight positional remapping inside attention.The functional form of RoPE remains unchanged, and the adjustment is described as nearly cost-free.
3.2. Suffix Dropout Strategies: Far suffix attention decays sharply, while pruning distant high-attention spikes causes nearby tokens to absorb the displaced information with little accuracy impact.The reported examples include spikes at distances 199, 298, and 362, with token 359 absorbing attention after token 362 is pruned.
3.3. Diffusion Lottery Tickets Hypothesis: The Diffusion Lottery Tickets hypothesis interprets suffix dropout as training-free inference-time search for a sparse subset that preserves semantic consistency and generation quality.This view treats retained suffix tokens as dynamically reorganized winning tickets within the forward pass.

4. Experiments

Across four benchmarks and three dLLM models, DPad reduces latency while preserving flexible accuracy and often improving strict-match accuracy. Its largest efficiency gains appear in long-sequence settings, while ablations support concentrating preserved tokens near the current block and using Gaussian dropout under tight budgets.

4.2. Main Results: DPad consistently lowers latency, keeps Flexible Match comparable, and substantially improves Strict Match, although throughput varies across models and benchmarks.Across the three-model suite, latency speedups over vanilla range from 1.18× to 4.17×.
4.2. Main Results: DPad’s latency gains mainly come from reducing quadratic suffix computation to linear complexity, with early termination and shorter outputs contributing less.On LLaDA-1.5, generation length falls about 10% on MATH and 27% on GSM8K, while MBPP is nearly unchanged.
4.2. Main Results: The advantage is bounded on short sequences because suffix attention is a small computation share and Gaussian dropout retains about 62.5% sparsity within the window.The paper cites prompt-dominated GSM8K and MATH settings and Amdahl-law limits near 1.1×.
4.2. Main Results: LLaDA-Instruct strict-match accuracy increases by 26.46% on GSM8K and 19.62% on MATH, whereas Dream-Base accuracy remains broadly comparable to baseline.The paper attributes Dream-Base’s stability to training-protocol differences, especially the absence of supervised fine-tuning.
4.3.1. Maximum Generation Length: 61.39× overall speedup over vanilla LLaDA is achieved by combining DPad with parallel decoding and prefix caching at a 1024-token limit.Standalone DPad reaches 20.3×, remains 4.8× faster with early termination, and the combined system improves 8.7× over Fast-dLLM.
4.3.1. Maximum Generation Length: 97.32× speedup is reached on Dream-Base HumanEval at 2048 tokens when DPad is combined with Fast-dLLM, versus 30.58× at 1024 tokens.DPad alone achieves 9.13× and 17.1× at 1024 and 2048 tokens, respectively.
4.3.2. Sliding Window Size and Dropout Function: A critical 64–128-token window follows the current block: preserving more tokens helps inside it, while spreading a limited budget farther can reduce accuracy.The recommended principle is to maintain high token density near the current block rather than expand the window under a fixed budget.
4.3.2. Sliding Window Size and Dropout Function: Gaussian dropout is especially advantageous under tight token budgets, and the selected settings are k=4.0 with 25.0% density for GSM8K and k=3.0 with 37.5% density for HumanEval.At large budgets, Gaussian and uniform dropout perform comparably; the exact decay form is less important than emphasizing nearby tokens.

5. Discussion

The discussion identifies a long-sequence performance boundary for DPad and outlines training-based extensions, while positioning distance-decay dropout between suffix-free block diffusion and bidirectional semi-autoregressive diffusion.

5.1. Beyond a Training-free Method: Suffix Dropout with SFT: At 2048-token contexts, DPad can degrade performance because distance-decay dropout creates a training–inference distribution gap between continuous masked suffixes and discontinuous retained suffixes.The authors suggest supervised finetuning as a possible mitigation.
5.1. Beyond a Training-free Method: Suffix Dropout with SFT: SFT would sample distance-decay dropout masks during training and compute loss on current-block tokens, encouraging robustness without overfitting to one deterministic pattern.The revised objective is intended to reduce redundant information written into distant suffix tokens likely to be pruned at inference.
5.1. Beyond a Training-free Method: Suffix Dropout with SFT: Future pretraining with sparse suffix attention could align training and inference conditions and potentially improve the efficiency–accuracy trade-off.
5.2. Comparison to Semi-Autoregressive Diffusion and Block Diffusion: Block Diffusion excludes subsequent blocks and therefore cannot use long-range bidirectional dependencies or suffix tokens as a scratchpad.
5.2. Comparison to Semi-Autoregressive Diffusion and Block Diffusion: DPad interpolates between these paradigms by retaining suffix tokens as a scratchpad while preserving bidirectional writing and retrieval.

6. Conclusion

The conclusion presents DPad as a training-free solution to redundant full suffix attention in dLLMs. It combines fixed-window attention and distance-decay pruning to improve efficiency while preserving accuracy, reaching up to 61.4× speedup with existing optimizations.

Up to 61.4× speedups are achieved when DPad is combined with existing optimizations, while experiments report improved efficiency without sacrificing accuracy.
DPad reduces suffix-attention complexity from quadratic to linear with a fixed-length sliding window and prunes low-entropy suffix tokens before computation.
The method constructs an inference-time “winning ticket” from the sparse subset of suffix tokens needed for generation.

A.1. Experiment Details

The experiment details report benchmark-specific Gaussian-sampler hyperparameters because attention-score distributions vary across datasets and models. Table 4 summarizes the selected settings for the main experiments.

DPad does not use one universal Gaussian-sampler setting because attention-score distributions vary across datasets, even for the same model.
For each benchmark, the authors sample a small data subset to select hyperparameters that perform best for the corresponding model.
Table 4 summarizes the Gaussian-sampler hyperparameters used in the main experiments.

A.2. Case Studies

The case study shows DPad changing Dream’s generation behavior from exhausting the token budget to producing shorter, logically sound solutions. A HumanEval example contrasts a 1021-token baseline output with a 69-token DPad output.

DPad mitigates Dream’s tendency to produce verbose outputs when shorter responses would suffice.
69 tokens versus 1021 tokens: DPad generates a concise, logically sound HumanEval solution while Dream’s baseline exhausts its budget by unrolling a loop.