PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast-dLLM v2: Efficient Block-Diffusion LLM

Chengyue Wu, Hao Zhang, Shuchen Xue, Shizhe Diao, Yonggan Fu, Zhijian Liu, Pavlo Molchanov, Ping Luo, Song Han, Enze Xie

arXiv:2509.26328v1cs.CL

TL;DR

AR language models provide strong generation quality but decode sequentially, limiting inference efficiency. Fast-dLLM v2 adapts pretrained AR models into block-diffusion decoders with complementary masking and hierarchical caching. It uses about 1B fine-tuning tokens and achieves up to 2.5× speedup over standard AR decoding without compromising generation quality.

  • Problem

    AR models decode tokens sequentially, while diffusion models face practical efficiency and caching limitations despite offering greater decoding parallelism.

  • Method

    Fast-dLLM v2 adapts pretrained AR models into block-diffusion decoders using complementary masking, blockwise attention, hierarchical caching, and parallel decoding.

  • Results

    Fast-dLLM v2 achieves up to 2.5× speedup over standard AR decoding while maintaining comparable generation quality across diverse tasks.

  • Takeaways & Limitations

    Fast-dLLM v2 provides a data-efficient route to practical, low-latency diffusion-style decoding for large language models.

Abstract

from arXiv · show

Autoregressive (AR) large language models (LLMs) have achieved remarkable performance across a wide range of natural language tasks, yet their inherent sequential decoding limits inference efficiency. In this work, we propose Fast-dLLM v2, a carefully designed block diffusion language model (dLLM) that efficiently adapts pretrained AR models into dLLMs for parallel text generation, requiring only approximately 1B tokens of fine-tuning. This represents a 500x reduction in training data compared to full-attention diffusion LLMs such as Dream (580B tokens), while preserving the original model's performance. Our approach introduces a novel training recipe that combines a block diffusion mechanism with a complementary attention mask, enabling blockwise bidirectional context modeling without sacrificing AR training objectives. To further accelerate decoding, we design a hierarchical caching mechanism: a block-level cache that stores historical context representations across blocks, and a sub-block cache that enables efficient parallel generation within partially decoded blocks. Coupled with our parallel decoding pipeline, Fast-dLLM v2 achieves up to 2.5x speedup over standard AR decoding without compromising generation quality. Extensive experiments across diverse benchmarks demonstrate that Fast-dLLM v2 matches or surpasses AR baselines in accuracy, while delivering state-of-the-art efficiency among dLLMs - marking a significant step toward the practical deployment of fast and accurate LLMs. Code and model will be publicly released.

1. Introduction

Fast-dLLM v2 adapts pretrained AR models into scalable block-diffusion decoders, addressing diffusion LLM limitations in efficiency, caching, and large-scale validation. It uses data-efficient fine-tuning and achieves faster generation while preserving quality.

  • AR decoding generates tokens sequentially, limiting parallelism and inference efficiency.
  • Diffusion LLMs enable joint token or block prediction but often struggle with KV caching, latency, and flexible generation lengths.
  • Block diffusion combines intra-block diffusion with autoregressive conditioning across blocks, supporting flexible lengths and KV caching, but had limited large-scale validation.
  • Fast-dLLM v2 adapts pretrained AR models using blockwise attention and complementary masking, requiring ∼1B fine-tuning tokens versus ∼500B for Dream.
  • 2.5× speedup over standard AR decoding is achieved while maintaining comparable generation quality across large-scale, diverse-task experiments.

2. Related Work

Related work develops discrete diffusion foundations, blockwise diffusion models, and inference acceleration through caching and advanced decoding. These approaches motivate Fast-dLLM v2’s combination of block structure, cache reuse, and parallel refinement.

  • Discrete diffusion evolved from Markov-chain formulations to masked diffusion models that replace tokens with [MASK] according to a mask ratio.
  • Blockwise diffusion methods interpolate between denoising diffusion and autoregressive generation by applying diffusion within blocks under a left-to-right structure.
  • Diffusion LLM acceleration primarily uses caching mechanisms and advanced decoding strategies to reduce bidirectional-attention costs.
  • DualCache reuses prefix and suffix KV activations, while other methods delay, adapt, sparsify, or window cache reuse.

3. Methodology

Fast-dLLM v2 organizes training and inference around blocks: complementary masking preserves token supervision, structured attention supports causal inter-block and bidirectional intra-block processing, and hierarchical caching accelerates refinement.

  • 3.1. Preliminary: Autoregressive models predict each token from its preceding clean context, whereas diffusion models predict original tokens from independently masked sequences.
  • 3.1. Preliminary: Training decodes sequences block by block with partial masking and applies loss only to predictions of masked tokens.
  • 3.2. Adaptation to Block Diffusion LLM: Sequences are padded to block-size multiples, packed into fixed-length contexts, and split into aligned non-overlapping blocks.
  • 3.2. Adaptation to Block Diffusion LLM: Complementary mask views duplicate each sample with masks m and 1 − m so tokens receive supervision in masked and unmasked contexts.
  • 3.2. Adaptation to Block Diffusion LLM: Shifted labels use the preceding position’s logit to predict masked tokens, preserving AR-like temporal representations during intra-block diffusion.
  • 3.2. Adaptation to Block Diffusion LLM: Inference decodes one block at a time, reuses decoded blocks as clean prefix context, and refines the current block in parallel.
  • 3.2. Adaptation to Block Diffusion LLM: A hybrid attention mask processes corrupted and clean sequences together while supporting block parallelism and causal dependencies between blocks.
  • 3.2. Adaptation to Block Diffusion LLM: Confidence-aware decoding finalizes high-confidence tokens in parallel while retaining uncertain positions for later refinement.

4. Experiments

Experiments evaluate Fast-dLLM v2 across diverse language, reasoning, and knowledge benchmarks, showing competitive accuracy alongside substantial decoding-throughput gains. Ablations identify complementary masking, sub-block decoding, and caching as important design choices for the quality–efficiency trade-off.

  • Main Results: Fast-dLLM v2 reaches average scores of 45.0 at 1.5B and 60.3 at 7B, surpassing the listed baselines at both scales.At 7B, it exceeds Qwen2.5-7B-Nemo-FT (59.6) and Dream (57.6).
  • Main Results: Both model sizes perform on par with or better than counterparts trained with standard next-token prediction on identical data and training steps.The comparison uses original Qwen-2.5 models tuned with next-token prediction under matched training conditions.
  • Performance and Speed: At threshold 0.9, GSM8K throughput rises from 39.1 to 101.7 tokens/s, yielding a 2.6× speedup with only a marginal accuracy drop.Threshold 1.0 recovers the standard non-parallel decoding process.
  • Performance and Speed: On GSM8K, diffusion generation reaches up to 1.5× higher throughput than autoregressive generation on A100 and 1.8× on H100 at batch size 64.The comparison uses threshold 0.9 and sub-block cache across varying batch sizes.
  • Ablation Study: The full training recipe with padding and complementary masking improves average accuracy by +3.7 points over naive token shifting.Padding prevents cross-sample leakage, while complementary masking provides supervision for all input tokens.
  • Ablation Study: Sub-block size 8 gives the highest average accuracy across tasks, whereas mismatched inference block sizes substantially degrade performance.For example, GSM8K drops from 62.0 to 58.5 under a mismatched block size; larger sub-blocks increase throughput but slightly reduce accuracy.
  • Ablation Study: Sub-block caching provides substantial speedups in compute-bound settings without observable accuracy changes.Its gains are negligible at small batch sizes but substantial at practical compute-bound batch sizes such as 32.

5. Conclusion

Fast-dLLM v2 adapts pretrained AR models into scalable block-diffusion decoders, combining blockwise attention with hierarchical caching and parallel decoding. Experiments show efficient generation with preserved quality, while decoding performance depends on sub-block configuration.

  • 5. Conclusion: Fast-dLLM v2 integrates blockwise diffusion, complementary masking, hierarchical caching, and parallel decoding for efficient diffusion-style generation.The framework includes block-level context reuse and sub-block refinement caching.
  • 5. Conclusion: Sub-block caching preserves accuracy and provides substantial speedup in compute-bound settings such as batch size 32.Its gains are negligible at small batch sizes, where memory bandwidth is underutilized.
  • 5. Conclusion: Larger sub-block sizes increase decoding throughput by reducing sequential forward passes and increasing intra-step parallelism, but may slightly reduce accuracy.The reported optimal sub-block size is 8, while mismatched training and inference sizes reduce performance.
  • 5. Conclusion: 2.5× speedup over standard AR decoding is achieved without loss of generation quality on large-scale Qwen2.5-Instruct models.Experiments cover the 1.5B and 7B model sizes.

A.1. Training Setup

Fast-dLLM v2 fine-tunes pretrained Qwen2.5-Instruct models using block-aligned instruction data and fixed experimental settings. The setup uses different training durations and learning rates for the 1.5B and 7B models.

  • A.1. Training Setup: Pretrained Qwen2.5-Instruct models with 1.5B and 7B parameters are fine-tuned under the block-wise diffusion framework.Experiments generally use context length 2048 and batch size 256 on 64 NVIDIA A100 GPUs.
  • A.1. Training Setup: The fine-tuning data comes from a high-quality LLaMA-Nemotron instruction-following subset spanning broad domains.Sequences are block-wise packed, and padding tokens are excluded from loss and gradient updates.
  • A.1. Training Setup: The 1.5B model uses 6,000 steps at 2 × 10^-5, while the 7B model uses 2,500 steps at 1 × 10^-5.Both settings use AdamW and linear warmup for the first 500 steps.
  • A.1. Training Setup: Training processes approximately 3.15 billion tokens for the 1.5B model and 1.31 billion tokens for the 7B model.These totals follow from the stated step counts and 524,288 tokens per step.
  • A.1. Training Setup: All experiments use block size 32 with right-padded, block-aligned sequences to support consistent batch construction.The alignment is designed to utilize model context efficiently under hardware constraints.

A.2. Attention Mask Design

The attention design jointly processes noised and clean sequences using complementary block-aware masks. During inference, decoded blocks are cached while the current block is refined bidirectionally against a causal prefix.

  • A.2. Attention Mask Design: Training concatenates the noised sequence x_t and clean sequence x_0 into a single 2L-token input processed with a hybrid attention mask.The mask is defined over the combined sequence and supports simultaneous noised and clean representations.
  • A.2. Attention Mask Design: The block-diagonal mask provides bidirectional self-attention among tokens within each noised block for within-block refinement.This enables tokens in the same block to attend to one another during diffusion training.
  • A.2. Attention Mask Design: The offset block-causal mask lets noised tokens attend to clean tokens in previous blocks, preserving inter-block causal conditioning.It connects the current noised block with its clean causal context.
  • A.2. Attention Mask Design: The block-causal mask lets clean tokens attend to all previous and current block positions, supporting autoregressive-like progression.This retains left-to-right structure for the clean sequence.
  • A.2. Attention Mask Design: During inference, previously generated blocks are cached, while only the current noised block is refined with bidirectional within-block attention.The current block attends causally to the unmasked tokens in prior blocks, enabling KV-cache reuse and reduced memory footprint.

A.3. Details on Training Objective

Fast-dLLM v2 trains with a masked-token-only cross-entropy objective paired with complementary masks. The complementary construction ensures that every original sequence position contributes to the loss.

  • A.3. Details on Training Objective: The training objective minimizes cross-entropy only over masked tokens.The loss predicts masked positions from their available contextual representations.
  • A.3. Details on Training Objective: The objective omits the usual normalization by the number of masked tokens intentionally because complementary masks are used.Each sample pairs masks m and 1 − m at times t and 1 − t.
  • A.3. Details on Training Objective: Complementary masking guarantees that the total number of tokens contributing to each sample’s loss equals the full sequence length L.The paired masks cover the sequence across the two training views.

A.4. Evaluation Protocol

The evaluation uses standardized decoding and benchmark protocols across reasoning, knowledge, and code-generation tasks. Inference settings are fixed to align training and evaluation for consistent assessment of block-wise diffusion.

  • Evaluation setup: Evaluations cover reasoning, knowledge, and code-generation benchmarks using greedy decoding, with GPQA as the exception under 5-shot prompting.All other tasks use zero-shot settings.
  • Evaluation setup: LM-Eval evaluates non-code tasks, while EvalPlus calculates reliable pass rates for HumanEval and MBPP.
  • Inference configuration: Block size = 32, sub-block size = 8, and parallel decoding is disabled with threshold = 1.
  • Inference configuration: The fixed configuration matches training and inference setups to facilitate evaluation of Fast-dLLM v2’s block-wise diffusion capability.

B. Case Study

The case studies examine Fast-dLLM v2’s behavior in single-turn and multi-turn dialogue. Examples cover mathematical reasoning, recursive code generation, temporal understanding, and contextual coherence.

  • Case-study design: Fast-dLLM v2 is examined in representative single-turn and multi-turn dialogue scenarios presented in Tables 5 and 6.
  • Single-turn Dialogue Scenarios: In the single-turn math example, the model factors a rational function and solves for undefined x values to determine vertical asymptotes.
  • Single-turn Dialogue Scenarios: In the single-turn code example, the model generates a recursive Python implementation of Tower of Hanoi with an appropriate explanation.
  • Multi-turn Dialogue Scenarios: In multi-turn dialogue, Fast-dLLM v2 retains context and builds on previous turns for temporal reasoning and step-by-step arithmetic problem solving.The arithmetic example includes intermediate calculations and reports the final result in hours and minutes.
  • Overall findings: Together, the cases demonstrate mathematical reasoning, code generation, temporal understanding, and contextual coherence across both dialogue settings.
Loading 2509.26328v1…