Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed

Yonggan Fu; Lexington Whalen; Zhifan Ye; Xin Dong; Shizhe Diao; Jingyu Liu; Chengyue Wu; Hao Zhang; Enze Xie; Song Han; Maksim Khadkevich; Jan Kautz; Yingyan Celine Lin; Pavlo Molchanov

Source-linked AI summary

Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed

Yonggan Fu, Lexington Whalen, Zhifan Ye, Xin Dong, Shizhe Diao, Jingyu Liu, Chengyue Wu, Hao Zhang, Enze Xie, Song Han, Maksim Khadkevich, Jan Kautz, Yingyan Celine Lin, Pavlo Molchanov

arXiv:2512.14067v2Dec 16, 2025cs.CLcs.AIcs.LG

TL;DR

dLMs promise faster parallel generation but remain less learning-efficient than AR models and often lack practical speed advantages. This paper continuously pretrains pretrained AR models with block-wise attention and position-dependent masking, producing Efficient-DLM models with improved accuracy–throughput trade-offs. Efficient-DLM 8B reports higher accuracy and throughput than Dream 7B and Qwen3 4B while maintaining comparable or slightly better accuracy than Qwen3 8B.

Problem
dLMs enable parallel generation, but their learning efficiency and practical speed have lagged behind AR models, motivating effective AR-to-dLM conversion.
Method
The paper continuously pretrains pretrained AR models using block-wise attention and position-dependent token masking to preserve AR abilities and narrow the training–test gap.
Results
+5.4%/+2.7% higher accuracy with 4.5×/2.7× higher throughput than Dream 7B and Qwen3 4B, respectively, while Efficient-DLM 8B remains comparable or slightly better in accuracy than Qwen3 8B.
Takeaways & Limitations
The Efficient-DLM family provides a practical framework for converting pretrained AR models into faster dLMs while retaining strong task accuracy.
Takeaways & Limitations
Fully bidirectional AR-to-dLM conversion can hinder KV caching, over-corrupt context, and cause larger weight drift from pretrained AR models.

Abstract

from arXiv · show

Diffusion language models (dLMs) have emerged as a promising paradigm that enables parallel, non-autoregressive generation, but their learning efficiency lags behind that of autoregressive (AR) language models when trained from scratch. To this end, we study AR-to-dLM conversion to transform pretrained AR models into efficient dLMs that excel in speed while preserving AR models' task accuracy. We achieve this by identifying limitations in the attention patterns and objectives of existing AR-to-dLM methods and then proposing principles and methodologies for more effective AR-to-dLM conversion. Specifically, we first systematically compare different attention patterns and find that maintaining pretrained AR weight distributions is critical for effective AR-to-dLM conversion. As such, we introduce a continuous pretraining scheme with a block-wise attention pattern, which remains causal across blocks while enabling bidirectional modeling within each block. We find that this approach can better preserve pretrained AR models' weight distributions than fully bidirectional modeling, in addition to its known benefit of enabling KV caching, and leads to a win-win in accuracy and efficiency. Second, to mitigate the training-test gap in mask token distributions (uniform vs. highly left-to-right), we propose a position-dependent token masking strategy that assigns higher masking probabilities to later tokens during training to better mimic test-time behavior. Leveraging this framework, we conduct extensive studies of dLMs' attention patterns, training dynamics, and other design choices, providing actionable insights into scalable AR-to-dLM conversion. These studies lead to the Efficient-DLM family, which outperforms state-of-the-art AR models and dLMs, e.g., our Efficient-DLM 8B achieves +5.4%/+2.7% higher accuracy with 4.5x/2.7x higher throughput compared to Dream 7B and Qwen3 4B, respectively.

1. Introduction

Efficient-DLM converts pretrained AR models into faster diffusion language models by preserving AR capabilities through block-wise attention and narrowing the training–test masking gap. The resulting family improves the accuracy–throughput trade-off over AR and dLM baselines.

1. Introduction: dLMs offer parallel non-autoregressive generation, but existing systems often fail to exceed AR speed because of limited KV-cache compatibility and decoding parallelism.Training dLMs from scratch is also more difficult because they learn all token permutations rather than only left-to-right modeling.
1. Introduction: Block-wise attention preserves pretrained AR weight distributions better than fully bidirectional modeling while enabling KV caching and a win-win in accuracy and efficiency.It remains causal across blocks and bidirectional within each block.
1. Introduction: Position-dependent masking assigns higher masking probabilities to later tokens to better match confidence-based test-time sampling.The strategy reflects dLMs’ retained left-to-right generation tendency.
1. Introduction: +5.4%/+2.7% higher accuracy with 4.5×/2.7× higher throughput than Dream 7B and Qwen3 4B, respectively.Efficient-DLM 8B maintains comparable, slightly better accuracy than Qwen3 8B.
1. Introduction: The study finds that attention patterns, block size, masking behavior, and training dynamics are central design choices for scalable AR-to-dLM conversion.The reported takeaways include the importance of clean context, suitable block sizes, omitting token shift, and exploiting improving likelihood estimates for more aggressive decoding.

2. Efficient-DLM: Attention Pattern Analysis

Block-wise attention preserves pretrained AR behavior better than fully bidirectional modeling while retaining KV-cache efficiency, especially when each noisy block uses clean context. Accuracy depends on balanced training and evaluation block sizes, with larger evaluation blocks enabling more parallel decoding.

Analyzing Different Attention Patterns: Block-wise attention remains causal across blocks and bidirectional within blocks, enabling KV caching while better preserving pretrained AR abilities than fully bidirectional attention.Fully bidirectional attention diverges from AR causality and causes larger weight drifts in attention and FFN layers.
Comparison of Attention Patterns: 19.12% higher average accuracy comes from block-wise attention with clean context and no token shift versus bidirectional attention.Block-wise attention also improves average accuracy by 8.94% even without clean context; Table 1 reports the full comparison.
Comparison of Attention Patterns: Clean context improves accuracy by 9.46% over noisy context, and doubling corrupted-context training tokens does not recover the gap.Clean-context training also produces smaller FFN weight drifts, indicating better preservation of pretrained behavior.
Analysis of the Optimal Block Sizes: A sweet-spot training block size balances insufficient context from small blocks against excessive corruption and weight changes from large blocks.Larger models tolerate larger training blocks, while smaller models exhibit a more pronounced optimum.
Analysis of the Optimal Block Sizes: Proper training block sizes generalize across evaluation block sizes, while larger evaluation blocks generally support more aggressive parallel token generation.Evaluation block size affects the accuracy–parallelism trade-off under varying numbers of function evaluations.

3. Efficient-DLM: Position-dependent Token Masking

Inference shows a left-to-right denoising tendency, so the paper introduces position-dependent masking that assigns stronger late-position emphasis as denoising completes. This narrows the training–test gap and improves accuracy, especially during aggressive parallel decoding.

3.1. The Training-Test Gap in Token Masking: Inference exhibits a left-to-right denoising tendency, with later token positions requiring more denoising steps and becoming masked nearer the end.Tokens gain confidence after neighboring tokens are decoded, and later mask tokens have larger losses from more corrupted context.
3.2. Our Token Masking Strategy: The method makes masking probabilities depend on noise level and relative position, increasing late-token masking as denoising approaches completion.At high noise, masking becomes more uniform; at low noise, later tokens receive larger weights.
3.2. Our Token Masking Strategy: The positional bias is controlled by β, with β=0 yielding uniform sampling and larger β producing stronger positional bias.The mask count is k=⌊tL′⌉ and positions are sampled after normalizing the weights with Gumbel-top-k sampling.
3.3. Comparison of Token Masking Schemes: The experiments use a half-life ratio λ to parameterize positional bias, comparing finite λ values with uniform and fully right-to-left masking.Lower λ means a stronger positional prior, while λ→∞ represents uniform masking and λ→0 represents right-to-left masking.
3.3. Comparison of Token Masking Schemes: Up to 4.38% higher average accuracy results from positional priors under aggressive parallel decoding, while fully right-to-left masking performs poorly.The comparison covers six generation tasks and varying tokens per forward (TPF).
3.3. Comparison of Token Masking Schemes: The paper concludes that mimicking dLMs’ left-to-right generation tendency during training can boost generation quality.The authors caution that positional priors help only when introduced properly.

4. Training Dynamics Analysis

Longer continuous training improves likelihood estimation and enables more aggressive parallel generation, although generation accuracy can fluctuate on some tasks. The analysis therefore treats accuracy under parallel decoding as a distinct dimension of dLM performance.

Setting: The study trains diffusion Qwen2.5 1.5B for 200B tokens and evaluates likelihood and generation tasks across training token budgets.Figure 8 tracks likelihood-task accuracy and accuracy–NFE trade-offs.
Observations and analysis: With roughly 10B training tokens, converted dLMs can largely recover task accuracy, while additional training consistently improves likelihood-based accuracy.Generation-task accuracy also improves on average, though it fluctuates on certain tasks.
Observations and analysis: Longer training improves the accuracy–NFE trade-off, enabling more aggressive parallel token generation through stronger likelihood estimation.The authors attribute this to more accurate and reliable confidence scores under confidence-based sampling.
Observations and analysis: Accuracy can diverge under aggressive parallel decoding even when models have comparable accuracy with one token generated per step.This motivates evaluating dLMs by both task accuracy and parallel token generation ability.

5. Efficient-DLM: A New Family of Efficient dLMs

The Efficient-DLM family combines block-wise clean-context attention, no token shift, position-dependent masking, and continuous pretraining. Across broad benchmarks, it improves accuracy–throughput trade-offs over leading dLM and AR baselines and offers configurable parallel decoding.

5. Efficient-DLM: A New Family: The family consists of 1.5B, 4B, and 8B models continuously pretrained from Qwen2.5-1.5B, Qwen3 4B, and Qwen3 8B, respectively.The models use block sizes 16, 64, and 64, and integrate clean-context block-wise attention without token shift with λ=0.1 masking.
5.1. Benchmark with SOTA AR and dLMs: Efficient-DLM 8B achieves 5.35% higher average accuracy and 4.50× throughput than Dream 7B, while also reaching 2.68% higher accuracy and 2.77× throughput than Qwen3 4B.The benchmark covers 12 tasks spanning coding, math, factual knowledge, and commonsense reasoning.
5.2. One-for-all Flexibility: Efficient-DLM 8B achieves better accuracy–throughput frontiers than AR Qwen3 models from 1.7B to 8B across four math and coding tasks.Confidence thresholds control the trade-off between accuracy and throughput for one model.
5.3. Text Embedding: Efficient-DLM outperforms same-size AR Qwen models on text embedding tasks by 7.71% at 1.5B and 9.91% at 4B.The evaluation spans 15 MTEB datasets across six categories.
5.4. Ablation Study: Ablations show that proper attention patterns, removing token shift, position-dependent masking, and longer training each contribute to AR-to-dLM conversion.The study progressively adds these components to Dream’s bidirectional 25B-token baseline on Qwen3 4B.

6. Related Work

Related work develops diffusion language models for parallel generation and explores acceleration through caching, confidence-based sampling, and block-wise designs. These efforts address the speed limits created by bidirectional attention and one-token-per-step denoising.

Diffusion language models: Diffusion language models replace token-by-token AR decoding with non-autoregressive parallel generation, including scalable masked dLMs such as LLaDA and Dream.The literature includes both continuous and discrete diffusion language models.
Diffusion language model acceleration: Bidirectional attention and one-token-per-step denoising limit dLM speed-ups, motivating dedicated caching strategies and confidence-based sampling.Caching methods reuse computations or approximate bidirectional attention, while confidence-based sampling supports parallel token generation.
Diffusion language model acceleration: Block-wise dLMs support native KV caching and have been developed by converting pretrained AR models or existing dLMs.The related work includes in-block diffusion and concurrent block-wise conversion approaches.

7. Conclusion

The work presents Efficient-DLM as a framework for converting pretrained autoregressive models into faster diffusion language models while retaining strong accuracy.

7. Conclusion: Efficient-DLM combines continuous pretraining, block-wise attention, and position-dependent token masking to deliver diffusion language models with strong accuracy and speed.The framework also includes comprehensive analyses of attention patterns, training dynamics, and other design choices for scalable AR-to-dLM conversion.

A. Detailed Experimental Settings

The experiments evaluate Efficient-DLM and baselines across diverse generation and embedding tasks using specified decoding, masking, and representation settings.

A. Detailed Experimental Settings: The generation evaluation covers 12 tasks spanning math, coding, factual knowledge, and commonsense reasoning.The settings use task-specific few-shot configurations and a maximum of 512 generated tokens, except GSM8K with 256.
A. Detailed Experimental Settings: Throughput and token efficiency are reported as tokens per forward and tokens per second, averaged across six generation tasks.Parallel decoding uses a confidence threshold and decodes tokens exceeding it at each denoising step.
A. Detailed Experimental Settings: Text-embedding evaluation covers 15 datasets across retrieval, reranking, clustering, pair classification, and classification categories.Embeddings are obtained by mean pooling the final-layer hidden states in zero-shot evaluation without fine-tuning.
A. Detailed Experimental Settings: Qwen models use causal attention for embedding evaluation, whereas Efficient-DLM models use bidirectional attention.Switching Qwen to bidirectional attention consistently degraded performance.

B. Per-task Accuracy Achieved by Efficient-DLM and Baselines

This section provides per-task accuracy results for Efficient-DLM and state-of-the-art autoregressive and diffusion baselines, complementing the main benchmark table.

B. Per-task Accuracy Achieved by Efficient-DLM and Baselines: Per-task accuracy for Efficient-DLM and state-of-the-art autoregressive and diffusion language models is reported in Table 6.The table complements Table 3 of the main paper.
B. Per-task Accuracy Achieved by Efficient-DLM and Baselines: Figure 10 compares the accuracy-throughput trade-off for Dream and LLaDA with Fast-dLLM.The figure provides a visual comparison of these accelerated diffusion baselines.

C. More Benchmarks with SOTA AR LMs and dLMs

Additional benchmarks compare Efficient-DLM with autoregressive and diffusion baselines across accelerated decoding and varying batch sizes, revealing strong small-batch trade-offs but limitations at larger batches.

C. More Benchmarks with SOTA AR LMs and dLMs: Figure 10 benchmarks Efficient-DLM 8B against Dream and LLaDA accelerated with Fast-dLLM using confidence thresholds to vary the accuracy-throughput trade-off.The comparison uses Fast-dLLM dual-cache and parallel-decoding acceleration.
C. More Benchmarks with SOTA AR LMs and dLMs: Efficient-DLM 8B consistently improves the accuracy-efficiency trade-off over Qwen3 1.7B–8B and diffusion models up to batch size 16.At batch size 32, Efficient-DLM 8B falls behind Qwen3 1.7B in throughput.
C. More Benchmarks with SOTA AR LMs and dLMs: The efficiency benefits of diffusion models over autoregressive models are more pronounced at small batch sizes and diminish as batch size increases.The experiments identify large-batch serving as a current limitation for diffusion language models.
C. More Benchmarks with SOTA AR LMs and dLMs: Figure 11 visualizes accuracy-throughput trade-offs for multiple models on GSM8K under different inference batch sizes.The figure complements the corresponding benchmark discussion for larger-batch serving.

D. The Impact of Initial LR

Initial learning rate balances preserving pretrained abilities against adapting to dLM attention patterns, with a sweet spot observed at 1e-5 in the reported Qwen3 4B experiment.

D. The Impact of Initial LR: 1e-5 balances preserving pretrained abilities with adapting to the new attention pattern in Qwen3 4B trained for 25B tokens.The experiment uses a cosine learning-rate schedule and compares different initial learning rates.
D. The Impact of Initial LR: Overly large learning rates increase weight drift and degrade original abilities, whereas overly small rates hinder adaptation to the new attention pattern.
Loss distributions differ by token position: AR models have higher loss initially, while dLM losses show periodic block-aligned patterns.In dLMs, later tokens within each block have higher loss because they have less clean context.
LoRA can reasonably convert pretrained AR models into dLMs, but rank 64 remains 7.63% behind the best full-model training scheme.The best scheme uses block-wise attention conditioned on clean context without token shift.