PaperlayerView exact revision on arXiv

Source-linked AI summary

DAWN: Dependency-Aware Fast Inference for Diffusion LLMs

Lizhuo Luo, Zhuoran Shi, Jiajun Luo, Zhi Wang, Shen Ren, Wenya Wang, Tianwei Zhang

arXiv:2602.06953v1cs.CL

TL;DR

dLLM parallel decoding is constrained because position predictions are often coupled, forcing conservative strategies that limit speed or degrade quality. DAWN uses attention-derived dependency graphs, anchor-guided decoding, and conflict-based scheduling to select safer parallel updates. Across models and datasets, it reports 1.80–8.06× baseline speedups while preserving comparable generation quality.

  • Problem

    Nonindependent position predictions make aggressive parallel decoding unreliable, while conservative selection criteria leave dLLM inference efficiency underused.

  • Method

    DAWN is a training-free dependency-aware decoder that constructs attention-based dependency graphs, relaxes thresholds for anchor-linked positions, and avoids coupled low-confidence updates.

  • Results

    1.80–8.06× speedup over the baseline is reported across multiple models and datasets while generation quality is preserved.

  • Takeaways & Limitations

    DAWN improves the quality–speed trade-off by enabling highly parallel updates without sacrificing comparable generation quality.

Abstract

from arXiv · show

Diffusion large language models (dLLMs) have shown advantages in text generation, particularly due to their inherent ability for parallel decoding. However, constrained by the quality--speed trade-off, existing inference solutions adopt conservative parallel strategies, leaving substantial efficiency potential underexplored. A core challenge is that parallel decoding assumes each position can be filled independently, but tokens are often semantically coupled. Thus, the correct choice at one position constrains valid choices at others. Without modeling these inter-token dependencies, parallel strategies produce deteriorated outputs. Motivated by this insight, we propose DAWN, a training-free, dependency-aware decoding method for fast dLLM inference. DAWN extracts token dependencies and leverages two key motivations: (1) positions dependent on unmasked certain positions become more reliable, (2) simultaneously unmasking strongly coupled uncertain positions induces errors. Given those findings, DAWN leverages a dependency graph to select more reliable unmasking positions at each iteration, achieving high parallelism with negligible loss in generation quality. Extensive experiments across multiple models and datasets demonstrate that DAWN speedups the inference by 1.80-8.06x over baselines while preserving the generation quality. Code is released at https://github.com/lizhuo-luo/DAWN.

1. Introduction

dLLM inference can exploit parallel decoding but is limited by nonindependent position predictions and conservative selection criteria. DAWN addresses this with dependency-aware scheduling that expands parallelism while preserving generation quality.

  • Motivation: dLLMs refine masked sequences through parallel-capable denoising, but practical inference still trails autoregressive models because of KV-Cache management and nonindependent position predictions.The marginal distributions at different positions often violate the independence assumption required for parallel decoding.
  • Motivation: Conservative confidence- and entropy-based selection limits achievable parallelism, leaving efficiency potential underused.The paper motivates modeling positional relationships directly rather than relying only on coarse-grained heuristics.
  • Method: DAWN constructs a dependency graph and uses anchor-guided decoding plus conflict-based scheduling to select disjoint sets of positions for parallel updates.Anchors relax thresholds for strongly coupled masked positions, while conflict scheduling avoids simultaneous updates of strongly coupled low-confidence positions.
  • Results: DAWN improves the quality–speed trade-off by enabling additional unmasking while preserving generation quality.The method is positioned as a complementary approach to failures caused by nonindependent parallel unmasking.
  • Key observations: Attention sinks can distort dependency estimates, whereas positions dependent on committed high-confidence tokens can remain reliably predictable at lower confidence.DAWN filters attention sinks and exploits reliable anchor-induced predictions.
  • Results: 1.80–8.06× speedup over the baseline is reported across multiple models and datasets, with ablations supporting the contributions of DAWN’s components.The experiments are presented as evidence of effectiveness across representative baselines.

2. Preliminaries

dLLMs generate responses by iteratively unmasking a fully masked sequence. Although each step predicts all masked positions, decoding multiple coupled positions together can cause inconsistencies and quality degradation.

  • Inference process: dLLMs start from a heavily masked sequence and progressively recover positions over denoising steps until no [MASK] remains.At each step, token distributions are predicted for all masked positions conditioned on the current sequence state.
  • Inference process: A response is initialized as a fully masked sequence of predefined length L.The response state is represented over the vocabulary and the special [MASK] token.
  • Inference process: In naive decoding, one highest-confidence masked position is committed at each denoising step, producing a fully unmasked response after L steps.The model concatenates the prompt with the current response state before each commitment.
  • Parallel decoding: Single-token unmasking is slow, while directly unmasking multiple tokens can substantially degrade quality.This creates the central quality–speed trade-off for practical dLLM inference.
  • Parallel decoding: Strongly coupled positions can yield inconsistent outputs when committed together, whereas decoding one position first can increase the probability of recovering a valid pair.The paper illustrates this with poker-hand terms such as “high card” and “full house.”

3. Observations

Attention in dLLMs can concentrate on shifting, semantically irrelevant sink tokens, making attention-based dependency estimates unreliable. The paper also examines whether dependency-linked masked positions remain consistent with final outputs.

  • Induced consistency: Figure 2 measures induced-token consistency by comparing each prediction with the final decoded output for anchor–induced position pairs.The experiments use LLaDA-8B-Instruct on sampled GSM8K and HumanEval instances; gray bins contain negligible sample fractions.
  • Attention sinks: Attention sinks concentrate disproportionate attention on a small subset of positions, and their locations can shift across denoising steps.The observed sink tokens are often punctuation or special tokens with little lexical meaning.
  • Attention sinks: Because sink tokens are largely unrelated to semantics, their concentration can mislead attention-based proxies for token dependencies.This motivates identifying and filtering sink positions before constructing dependency relations.

dLLMs

DAWN addresses dependency-related errors in parallel dLLM decoding by filtering misleading attention patterns and exploiting reliable context from high-confidence anchors.

  • DAWN’s overview combines dependency-graph construction with Anchor-Guided Decoding and Conflict-Based Scheduling to select positions for simultaneous unmasking.
  • Attention sinks can absorb disproportionate attention and mislead dependency estimates despite being semantically irrelevant.
  • Positions strongly dependent on high-confidence anchors can remain consistent with the final output despite relatively low instantaneous confidence.When anchors exceed 0.9 confidence, induced tokens are reported as consistently correct at lower confidence.

4. Methodology

DAWN constructs iteration-specific dependency graphs from attention maps, then combines anchor-guided selection with conflict-aware scheduling to expand parallel updates while limiting coupled low-confidence errors.

  • Dependency Graph Construction: Attention maps provide a lightweight proxy for token dependencies, which DAWN converts into a sparse directed graph at each denoising iteration.Attention is averaged across recent layers and heads, while attention sinks are filtered before graph construction.
  • Dependency Graph Construction: The graph retains thresholded salient edges, where an edge j →i indicates that position i is significantly conditioned by position j.
  • Anchor-Guided Decoding: Anchor-Guided Decoding selects approximately independent high-confidence positions and relaxes the threshold for masked positions reachable from committed anchors.Anchors are previously unmasked positions meeting τhigh; induced positions can be selected using τinduced.
  • Conflict-Based Scheduling: Conflict-Based Scheduling prevents strongly coupled lower-confidence positions from being unmasked simultaneously by treating connected positions as conflicts.
  • Conflict-Based Scheduling: A greedy descending-confidence independent-set procedure selects additional nonconflicting positions for parallel decoding.Selected positions and their conflicting neighbors are removed from further consideration until no candidates remain.
  • Conflict-Based Scheduling: Explicitly avoiding simultaneous unmasking of strongly coupled positions supports a lower confidence threshold while helping maintain decoding quality and speed inference.

5. Experiments

Across multiple models, datasets, and baselines, DAWN improves dLLM inference speed while preserving comparable accuracy. Ablations and sensitivity analyses show how its components and decoding settings shape the quality–speed trade-off.

  • 5.2. Main Results: DAWN achieves up to 8.06× speedup on MBPP with LLaDA-1.5 while maintaining comparable accuracy overall.On LLaDA-8B-Instruct, it matches the original method’s 77.94 GSM8K accuracy and slightly improves MBPP accuracy from 29.60 to 30.80.
  • 5.2. Main Results: DAWN improves throughput over confidence-aware parallel and KLASS with nearly identical accuracy, and outperforms LocalLeap in both quality and speed.Against LocalLeap, throughput improves by approximately 0.05–5.17 tokens per second and accuracy by up to 3.04% across most benchmarks.
  • 5.3. Ablation Study: Removing Anchor-Guided Decoding reduces TPS from 44.72 to 22.31 on GSM8K with LLaDA-8B-Instruct, identifying it as a primary speed contributor.Anchor-guided decoding expands safely unmaskable positions under reliable anchor context.
  • 5.3. Ablation Study: Removing Conflict-Based Scheduling increases accuracy but lowers efficiency on Dream-v0-Instruct-7B, indicating a speed–accuracy trade-off from avoiding inconsistent joint updates.CBS unlocks parallelism by preventing simultaneous updates at strongly coupled positions.
  • 5.3. Ablation Study: Across generation lengths, DAWN consistently improves throughput over the original sampler while maintaining comparable accuracy on both LLaDA and Dream models.Throughput decreases for both methods as generation length increases, but DAWN retains a more favorable quality–speed trade-off.
  • 5.3. Ablation Study: As block length increases, throughput rises while accuracy first increases and then decreases, with DAWN retaining a robust quality–speed trade-off.The block-length study evaluates L ∈ {8, 16, 32, 64} on HumanEval for LLaDA-8B-Instruct and Dream-v0-Instruct-7B.
  • 5.3. Ablation Study: Reducing τlow increases TPS by admitting more parallel updates but can hurt accuracy; τlow = 0.80 preserves high quality and improved efficiency.This trade-off is observed on both models in the HumanEval sensitivity study.

6. Related Work

Related work positions dLLMs as parallel alternatives to autoregressive generation, while highlighting ongoing inference-efficiency challenges. Existing solutions address caching, stopping, distillation, and sampling optimization.

  • Diffusion Large Language Models: dLLMs refine entire sequences through multiple denoising iterations with full attention, unlike autoregressive models that generate tokens sequentially.This parallel formulation supports text generation while addressing limitations associated with sequential decoding.
  • Diffusion Large Language Models: Recent diffusion-based language models include systems pretrained from scratch, built on autoregressive models, and extended to multimodal settings.Commercial systems and multimodal dLLMs further demonstrate the practical scope of diffusion-based generation.
  • Efficient Inference of dLLMs: Despite parallel updates, dLLM inference faces practical challenges that motivate faster and more reliable decoding methods.Related efficiency work studies KV-cache optimization, early stopping, distillation-based acceleration, and optimized sampling.

7. Conclusion

DAWN is a training-free, dependency-aware method that uses a sparse directed dependency graph to select parallel unmasking positions. Experiments show consistent speedups while maintaining comparable quality, improving the quality–speed trade-off for dLLM inference.

  • 7. Conclusion: DAWN extracts positional dependencies and selects reliable unmasking positions through a sparse directed dependency graph.The method targets nonindependent position predictions and enables highly parallel updates without training.
  • 7. Conclusion: DAWN consistently speeds dLLM inference across multiple models and datasets while maintaining comparable generation quality.The reported outcome is a better quality–speed trade-off that narrows the practical gap to state-of-the-art language models.

A. Experiment Details

The experiments cover four model variants and four reasoning or code-generation benchmarks, with model-specific dependency thresholds selected through accuracy–throughput studies. Final choices favor configurations near the Pareto frontier while retaining high accuracy.

  • A. Experiment Details: DAWN’s τedge, τinduced, and τsink values are tuned with HumanEval accuracy–TPS curves across candidate settings.Figures 7–10 visualize the resulting model-specific threshold studies.
  • A. Experiment Details: Most hyperparameter settings exhibit a throughput–accuracy trade-off, so configurations are selected near the Pareto frontier.The criterion favors high accuracy with meaningful efficiency gains rather than marginal speed improvements that sacrifice accuracy.
  • A. Experiment Details: For LLaDA-8B-Instruct, the selected threshold is τedge = 0.07 because it gives the highest tested accuracy while exceeding the throughput of τedge = 0.08.The τinduced and τsink settings are also model-specific, while τlow uses benchmark-specific configurations.
  • A. Experiment Details: The lower confidence threshold τlow is tuned across two models and benchmarks in 16 experimental settings, with method-specific hyperparameters optimized within each setting.Final selected results are reported in Table 3.

B. Discussion

The discussion frames nonindependent position predictions as a central challenge for dLLM decoding and argues that decoding order should account for positional dependencies. It presents dependency-aware analysis as a perspective for addressing parallel-decoding difficulty and motivating more efficient inference methods.

  • Nonindependent position predictions motivate decoding orders that account for dependencies among dLLM positions.
  • Treating high confidence as sufficient evidence of consistency or approximate independence can limit achievable parallelism.
  • Dependency structures offer a complementary perspective on the difficulty of parallel decoding under non-independence.
  • The study encourages further investigation of dLLM dependency structures and inference methods that leverage them.
Loading 2602.06953v1…