PaperlayerView exact revision on arXiv

Source-linked AI summary

Introspective Diffusion Language Models

Yifan Yu, Yuqing Jian, Junxiong Wang, Zhongzhu Zhou, Donglin Zhuang, Xinyu Fang, Sri Yanamandra, Xiaoxia Wu, Qingyang Wu, Shuaiwen Leon Song, Tri Dao, Ben Athiwaratkun, James Zou, Fan Lai, Chenfeng Xu

arXiv:2604.11035v1cs.AI

TL;DR

DLMs offer parallel generation but lag AR models because they are not trained to agree with their own outputs, while practical serving incurs substantial overhead. I-DLM combines introspective-consistency training with introspective strided decoding and AR-compatible serving. It matches same-scale AR quality and improves prior DLM quality and serving efficiency across the reported benchmarks.

  • Problem

    DLMs remain behind AR models in quality, while their theoretical parallelism is not reliably converted into practical speedups because of compute and serving-system limitations.

  • Method

    I-DLM trains introspective consistency and uses introspective strided decoding to verify prior tokens while generating new ones, with an AR-compatible inference stack.

  • Results

    I-DLM matches its same-scale AR counterpart while outperforming prior DLMs in model quality and practical serving efficiency across 15 benchmarks.

  • Takeaways & Limitations

    Introspective consistency provides a path to parallel DLM generation that preserves AR-level quality while retaining practical serving efficiency.

Abstract

from arXiv · show

Diffusion language models promise parallel generation, yet still lag behind autoregressive (AR) models in quality. We stem this gap to a failure of introspective consistency: AR models agree with their own generations, while DLMs often do not. We define the introspective acceptance rate, which measures whether a model accepts its previously generated tokens. This reveals why AR training has a structural advantage: causal masking and logit shifting implicitly enforce introspective consistency. Motivated by this observation, we introduce Introspective Diffusion Language Model (I-DLM), a paradigm that retains diffusion-style parallel decoding while inheriting the introspective consistency of AR training. I-DLM uses a novel introspective strided decoding (ISD) algorithm, which enables the model to verify previously generated tokens while advancing new ones in the same forward pass. From a systems standpoint, we build I-DLM inference engine on AR-inherited optimizations and further customize it with a stationary-batch scheduler. To the best of our knowledge, I-DLM is the first DLM to match the quality of its same-scale AR counterpart while outperforming prior DLMs in both model quality and practical serving efficiency across 15 benchmarks. It reaches 69.6 on AIME-24 and 45.7 on LiveCodeBench-v6, exceeding LLaDA-2.1-mini (16B) by more than 26 and 15 points, respectively. Beyond quality, I-DLM is designed for the growing demand of large-concurrency serving, delivering about 3x higher throughput than prior state-of-the-art DLMs.

1 Introduction

Diffusion language models offer parallel generation but remain behind autoregressive models in quality and practical speed. I-DLM addresses these gaps by preserving AR introspective consistency while retaining parallelism and adapting AR-compatible serving.

  • DLMs promise parallel generation by iteratively refining token blocks, but quality gaps and limited inference-system support hinder practical adoption.
  • AR causal masking and logit shifting train models to revisit and validate previously generated tokens, whereas DLMs generally lack this self-agreement.
  • I-DLM combines introspective-consistency training with introspective strided decoding, generating new tokens while verifying prior tokens in the same forward pass.
  • I-DLM retains parallel generation while inheriting AR introspective consistency and uses an inference stack compatible with AR serving systems.
  • Introspective consistency is identified as the missing principle that limits prior DLMs’ ability to realize the underlying model’s full capability.

2 Background and Motivation

The paper attributes DLM shortcomings to weak self-consistency, excessive compute overhead, and incompatibility with AR serving engines. It formalizes these bottlenecks and motivates ISD as a more efficient alternative.

  • Current DLMs suffer from low introspective consistency, higher FLOPs per token, and inference engines poorly aligned with multi-token, multi-step denoising.
  • The introspective acceptance rate measures whether generated tokens sampled from q are accepted by the model’s corresponding causal distributions p.
  • AR models have p = q by construction and therefore achieve α = 1, while SDAR and LLaDA 2.0-flash score 0.699 and 0.568 on IFEval, respectively.
  • 4.5B training tokens let I-DLM match its AR base model, whereas SDAR uses 54B tokens and reaches 10.0 versus 69.6 on AIME-24.
  • At N=4, ISD exceeds break-even compute efficiency above acceptance rates of approximately 0.83 or 0.86, reaching 1.08–2.29× at observed rates.
  • ISD’s adaptive stride and causal attention produce a TPS growth rate of 549, compared with SDAR’s nearly flat slope of 84 with respect to TPF.

3 Introspective Diffusion Language Model

I-DLM combines causal, introspective-consistency training with introspective strided decoding to preserve AR-style verification while enabling parallel generation. Its decoding and serving design adapts stride to self-consistency, unifies proposal and verification, and approaches AR behavior when acceptance is low.

  • 3.3 Systems: I-DLM preserves causal structure for direct AR-stack integration, while ISD reduces introspection overhead compared with prior multi-step diffusion and unified draft methods.The design is presented as a drop-in replacement within AR serving infrastructure; its logit shift reduces TiDAR’s token-query cost from N(N+1) to N2.
  • 3.1 Introspective-Consistency Training: I-DLM converts pretrained AR models into introspective diffusion models using causal attention, logit shifting, and an all-masked objective.Clean positions learn causal anchor distributions while masked positions learn the decode distribution, using a shared cross-entropy objective.
  • 3.1 Introspective-Consistency Training: Auto-balanced loss rescales clean-position supervision to match masked-position loss, preventing the decode pathway from overwhelming verification training.The balancing coefficient is recomputed from loss magnitudes at each training step without differentiation through it.
  • 3.2 Introspective Strided Decoding: ISD produces a quality-guaranteed token alongside strided proposals, then verifies proposals against causal anchors while advancing fresh tokens in the same forward pass.The procedure bootstraps with an AR-equivalent token and fuses subsequent introspection with the next proposal step.
  • 3.2 Introspective Strided Decoding: ISD dynamically adapts its stride using the model’s p/q acceptance criterion, accepting easy tokens in parallel and reverting toward AR-quality generation for difficult tokens.It begins with AR prefill and uses the model’s own self-consistency rather than a fixed block size or separate draft model.
  • 3.2 Introspective Strided Decoding: With typical acceptance rates of p ≥0.85, stride N=3 achieves TPF ≈2.3–2.4× with only ∼2× compute overhead; at p=0, ISD degenerates to AR.In the memory-bound regime, approximately constant forward-pass latency makes TPF translate into wall-clock speedup.

4 Experiments

Experiments evaluate I-DLM across 15 benchmarks, quality baselines, serving workloads, and component ablations. I-DLM improves over prior DLMs, matches same-scale AR quality, and gains efficiency from its training and systems design.

  • End-to-end quality: I-DLM-8B exceeds LLaDA-2.1-mini (16B) across all benchmarks, including +26.3 on AIME-24 and +15.3 on LiveCodeBench-v6.
  • End-to-end quality: I-DLM matches same-scale AR quality, matching Qwen3-8B on ARC-C and IFEval, remaining within approximately 1 point on MMLU, and surpassing it on Math-500.
  • Serving efficiency: At concurrency C=64, I-DLM sustains approximately 125 tok/s and delivers 2.9–4.1× higher throughput than prior DLMs.
  • Serving efficiency: I-DLM outperforms EAGLE3 in per-request throughput from C=1 through C=32 across all evaluated benchmarks.
  • Training ablations: Removing causal attention and logit shift sharply reduces reasoning performance, including HumanEval from 92.7 to 60.3 and MathBench from 89.1 to 71.6.
  • Systems ablations: CUDA graph capture contributes +42–76% efficiency, while stationary-batch scheduling and argmax proposals contribute +11–21% and +11–15%, respectively.

N TPF TPS MATH-500 MBPP

ISD’s stride controls the trade-off between parallelism and quality, while relaxed acceptance increases throughput with limited quality degradation. Across tested strides, quality remains stable as TPF rises.

  • Stride impact: TPF scales nearly linearly from 1.80 to 4.01 as stride N increases from 2 to 8.
  • Stride impact: Accuracy remains stable across increased strides, with MATH-500 within 94.6–96.8% and MBPP within 88.3–93.4%.
  • Relaxed acceptance: At relaxation τ=1.0, HumanEval decreases from 93.3 to 91.2 while TPF increases from 2.63 to 2.73.
  • Relaxed acceptance: At τ=0, ISD provably matches the AR distribution; increasing τ relaxes this guarantee to obtain higher TPF without retraining.

5 Conclusion

The paper identifies introspective consistency as the missing principle in DLMs and introduces I-DLM to combine AR-like consistency with parallel generation. Results report AR-level quality and higher throughput at large concurrency.

  • I-DLM combines logit-shifted causal training with introspective strided decoding to unify parallel generation and self-verification.
  • I-DLM matches AR-level quality, substantially outperforms existing DLMs, and achieves 2.9–4.1× better throughput at large concurrency.

A Detailed Related work

Related work has pursued diffusion models, AR-to-diffusion conversion, speculative decoding, and DLLM-specific decoding. I-DLM differs by combining strict causal attention with logit-shifted prediction to preserve AR patterns during conversion.

  • Diffusion language models: Diffusion language models progressively evolved from continuous and uniform-state formulations to discrete, blockwise, and causal-mask decoding.
  • AR-to-diffusion conversion: AR-to-diffusion conversion reduces training cost, with prior approaches using masked objectives, distillation, full-model training, causal prefixes, or autoregressive verification.
  • I-DLM’s distinction: I-DLM combines strict causal attention with logit-shifted prediction from MASK positions to preserve pretrained AR attention and prediction patterns.
  • Speculative decoding: Speculative decoding accelerates AR models by drafting and parallel verification, but standard DLLMs lack the well-trained verify distribution that supports its guarantee.
  • DLLM-specific decoding: Prior DLLM-specific methods use confidence-based acceptance or iterative denoising, while fundamental gaps remain in training scalability, inference alignment, compute efficiency, and infrastructure compatibility.

B TPF and Compute Overhead Analysis

ISD combines proposal generation with introspective verification, while its expected throughput and overhead depend on token acceptance. Compared with SDAR and TiDAR, ISD avoids mandatory commits and wasteful branching.

  • ISD: ISD alternates propose-only NP steps with P steps that verify previous proposals while generating new ones.A rejection finalizes the free token, accepted proposals, and one resampled token.
  • ISD: 2 + p + p^2 + · · · + p^(N−2) is ISD’s expected tokens per forward pass.The expression applies under uniform per-token acceptance probability p and stride size N.
  • ISD: At p = 1, ISD reaches TPF = N; at p = 0, it reduces to AR with TPF = 1.The limiting cases show how acceptance controls parallel decoding benefit.
  • Comparison: As p →1, the ISD efficiency gap vanishes; in memory-bound regimes, wall-clock speedup is approximately TPF.Fixed-query implementations use 2N−1 queries for both NP and P steps.
  • SDAR: SDAR processes N tokens across E[S | N] denoising steps plus a mandatory KV commit, with TPF = N E[S | N] + 1 and overhead = E[S | N] + 1.The separate commit pass processes all N tokens but produces no new tokens.
  • TiDAR: TiDAR achieves high TPF by verifying N drafts and pre-drafting N branches, but discards (N−1) · N mask queries every forward.Even at p = 1, its overhead is N+1 and its efficiency remains below 1.

C Why Block Diffusion Requires a Separate KV Commit Pass

Block diffusion requires a separate forward pass to commit denoised tokens to the KV cache. Fusing that commit with the next denoising step would require mixed attention masks unsupported by current kernels.

  • Commit requirement: After denoising produces N tokens, block diffusion performs a mandatory KV commit forward that generates zero new tokens.This remains necessary even at p=1 and caps TPF at N/2.
  • Fusion barrier: Fusing commit with the next block would require bidirectional attention for committed tokens and block-causal attention for MASK tokens.The two position types need different attention behavior within one forward pass.
  • Fusion barrier: FlashAttention and FlashInfer expose only a single global causal flag, so fusion would require custom kernels, doubled query size, and complex concurrent batching logic.The cited implementation notes that the relevant SGLang commit flags are defined but inactive.
  • I-DLM contrast: I-DLM avoids the commit pass because strict causal attention lets every ISD step use a standard AR-style extend operation that incrementally commits KV entries.This avoids mixed masks and custom attention kernels.

D Attention Kernel Overhead

Block diffusion’s block-causal execution uses a three-kernel attention cascade that is inefficient for short decode steps. I-DLM’s strict causal attention enables one paged kernel per layer instead.

  • Block diffusion: Block diffusion uses a three-kernel cascade: ragged attention, paged attention over the prefix, and a merge kernel.The cascade is optimized for large prefills rather than short DLLM decode steps.
  • Block diffusion: For N=4–5 token decode steps, the ragged kernel’s advantage vanishes and three kernel launches dominate overhead.This makes the standard extend path poorly matched to block diffusion decoding.
  • I-DLM: I-DLM reduces attention launches from 3L to L by bypassing the cascade with one paged attention kernel per layer.The optimization follows from strict causal attention and small extended sizes.
  • Concurrency: +4% at C=1 and +20% at C=64 are the reported cascade overheads as concurrency increases.The larger-concurrency penalty reflects compounded kernel-launch overhead during batching.
  • Attention masks: I-DLM training uses strict causal attention within noisy blocks and in the clean region, unlike SDAR’s bidirectional noisy blocks and block-causal clean region.The mask is composed of noisy self-attention, noisy-to-clean cross-attention, and clean self-attention.
  • ISD illustration: Figure 9 illustrates ISD at stride N=3 in both all-accept and rejection cases.The procedure introspects previous proposals while advancing new ones in a single forward pass.

G Lossless ISD with Gated LoRA

Lossless ISD uses gated LoRA to improve proposal quality while preserving an AR-identical causal anchor. Its single-pass introspection-and-proposal design supports AR-compatible training and serving configurations.

  • Gated LoRA: Gated LoRA activates on [MASK] proposal positions, while clean introspection positions use base-model-only weights.The per-token mask applies Wx + BAx only where the input is [MASK].
  • R-ISD pipeline: A single forward pass unifies introspection and proposal generation.The illustrated R-ISD pipeline alternates accepted, resampled, and discarded positions across steps.
  • Gated LoRA: The causal anchor p remains the exact base AR distribution because strict causality prevents introspection positions from attending to [MASK] positions.Their KV entries are computed entirely from base weights over clean tokens.
  • Training: I-DLM-8B and I-DLM-32B are converted from Qwen3-8B and Qwen3-32B using introspective-consistency training.The reported training data contains 4.5B response tokens generated across reasoning datasets.
  • Training: R-ISD additionally trains rank-128 LoRA adapters on the same data with a learning rate of 2e-4.Inference follows the gated residual design.
  • Implementation: The experiments use H100 80GB SXM GPUs with NVLink, CUDA 12.9, FlashInfer, and CUDA graphs.Serving configurations use SGLang, while ISD configurations vary stride and block size.

H.4 Evaluation Configuration

The evaluation section specifies benchmark settings, answer extraction procedures, baseline reproduction sources, and throughput and ablation protocols. It also describes the configurations underlying the peak per-request TPS comparison.

  • Quality evaluation: Benchmark evaluations use temperature t=1.0, topk=50, and top-p=0.95, with results averaged over three runs.The same sampling parameters are used for the Qwen3-8B/32B AR baseline.
  • Answer extraction: Math, multiple-choice, and code benchmarks use separate answer-extraction procedures tailored to their output formats.Math answers come from final boxed answers; multiple-choice answers use ANSWER: [A-D]; code is executed against test cases with a 10-second timeout.
  • Baseline reproduction: LLaDA-2.1-mini, SDAR, and EAGLE-3 results are reproduced with SGLang configurations, while other baselines come from their original publications.The reproduced settings include block size=4, threshold=0.95, and edit threshold=0.9 for LLaDA-2.1-mini.
  • Serving ablation: The serving ablation is cumulative: optimizations are added one at a time after a naive configuration and throughput is measured at C=1, C=8, and C=32 on one H100.The study disables optimizations individually through environment variables, with toggles listed in Table 12.
  • Throughput evaluation: Peak per-request TPS is reported across stride, hardware, and model configurations under low-concurrency, long-generation conditions.The base I-DLM-8B uses N=3 training, while N=4 and N=8 are produced through successive stride-extension training; the LoRA variant uses a rank-128 R-ISD adapter.
Loading 2604.11035v1…