PaperlayerView exact revision on arXiv

Source-linked AI summary

SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation

Shuang Cheng, Yihan Bian, Dawei Liu, Linfeng Zhang, Qian Yao, Zhongbo Tian, Wenhai Wang, Qipeng Guo, Kai Chen, Biqing Qi, Bowen Zhou

arXiv:2510.06303v3cs.LGcs.AI

TL;DR

AR models provide more compute-efficient training than masked diffusion models, motivating SDAR’s conversion of pretrained AR models into blockwise diffusion models. SDAR generates blocks autoregressively while decoding tokens within each block in parallel, preserving AR-level performance while improving reasoning on selected scientific benchmarks. The paper positions this combination as a practical route to scalable, high-throughput reasoning, while deployment efficiency remains dependent on inference-engine optimizations and workload configuration.

  • Problem

    AR models train efficiently but decode sequentially, whereas diffusion models support parallel decoding yet face training and inference inefficiencies.

  • Method

    SDAR adapts a pretrained AR model into a blockwise diffusion model that generates autoregressively across blocks and denoises tokens in parallel within each block.

  • Results

    SDAR is competitive with AR models overall and achieves gains of +5.5% on GPQA-diamond and +12.3% on ChemBench.

  • Takeaways & Limitations

    SDAR provides a practical paradigm combining AR training efficiency with diffusion-style parallel generation for scalable reasoning.

  • Takeaways & Limitations

    Deployment throughput still depends on separate KV-cache computation, specialized kernels, batch size, block size, and workload-specific configuration.

Abstract

from arXiv · show

We propose SDAR, a Synergistic Diffusion-Autoregression paradigm that unifies the training efficiency of autoregressive models with the parallel inference capability of diffusion. Instead of costly end-to-end diffusion training, SDAR performs a lightweight paradigm conversion that transforms a well-trained autoregressive (AR) model into a blockwise diffusion model through brief, data-efficient adaptation. During inference, SDAR generates sequences autoregressively across blocks for global coherence while decoding all tokens within each block in parallel via a discrete diffusion process. Extensive experiments show that AR models remain substantially more compute-efficient than masked diffusion models, providing a strong foundation for adaptation. Building on this insight, SDAR achieves efficient AR-to-diffusion conversion with minimal cost, preserving AR-level performance while enabling parallel generation. Scaling studies across dense and Mixture-of-Experts architectures confirm that SDAR scales without compromise: larger models exhibit stronger robustness to block size and decoding thresholds, yielding greater speedups without accuracy loss. Beyond efficiency, SDAR demonstrates enhanced reasoning and domain adaptability. Our 30B MoE model surpasses its AR counterpart on challenging scientific reasoning benchmarks such as GPQA and ChemBench, and gains further improvements under test-time scaling methods like majority voting and pass@k. Together, these results establish SDAR as a practical paradigm that combines the strengths of autoregression and diffusion for scalable, high-throughput reasoning.

1 Introduction

SDAR addresses the tension between AR models’ efficient training and diffusion models’ parallel inference by converting pretrained AR models into block-wise diffusion models through lightweight adaptation. Experiments compare training efficiency, adaptation bases, scaling behavior, and released model configurations.

  • Motivation: AR decoding is slow because it generates tokens sequentially, while diffusion models enable more flexible generation through parallel decoding.The paper motivates a hybrid design for tasks requiring non-local or holistic reasoning.
  • Motivation: MDLMs face training and inference inefficiencies from NELBO optimization and implementations that lack KV caching.Hybrid block-wise approaches also introduce substantial training overhead.
  • SDAR: SDAR uses full-scale AR pretraining followed by lightweight adaptation to enable block-wise diffusion inference while retaining AR advantages.The design combines inter-block causal structure with parallel intra-block generation.
  • Evidence: Controlled experiments find that AR models have substantially higher training efficiency than MDLMs under identical computational settings.This comparison supports using AR models as the foundation for SDAR adaptation.
  • Evidence: AR-based SDAR adaptation consistently outperforms adaptation from MDLM bases on downstream benchmarks using a 100B-token subset.The results identify AR backbones as the preferred base models.
  • Scaling and release: Scaling studies cover 1.7B–30B dense and MoE models and block sizes of 4–64, while showing negligible-cost block-size increases with minimal supervised fine-tuning.The authors also provide an adaptation recipe using an arbitrary 50B-token open-source dataset and release models and inference engines.

2 Preliminary: Language Modeling Paradigm

The preliminary section contrasts sequential autoregressive modeling with masked diffusion and introduces blockwise diffusion as a hybrid that is autoregressive across blocks but parallel within each block.

  • Autoregressive Models: AR models factorize sequence probability into next-token conditionals and train with maximum likelihood on next-token prediction.The preceding-token context is denoted x<ℓ.
  • Autoregressive Models: Generating a sequence of length L with AR decoding requires L sequential forward passes, creating a low-latency bottleneck.This limitation follows from the iterative nature of token-level generation.
  • Masked Diffusion Models: Discrete masked diffusion models learn to reverse independently applied token corruption, allowing non-autoregressive generation.Their noise process interpolates original token one-hot vectors with the [MASK] token under a predefined schedule.
  • Masked Diffusion Models: Masked diffusion models are trained by minimizing NELBO, which in the absorbing-state formulation becomes a reweighted cross-entropy objective over masked tokens.The weighting depends on diffusion time under the linear noise schedule.
  • Blockwise Diffusion: Blockwise diffusion partitions a sequence into B contiguous blocks, modeling dependencies autoregressively across blocks and non-autoregressively within each block.Each block conditional is represented by a self-contained diffusion process conditioned on preceding clean blocks.

3 Method

SDAR converts a pretrained AR model into a block-wise diffusion model with efficient adaptation, then generates blocks autoregressively while denoising tokens within each block in parallel. Its decoding process iteratively fills masked positions using confidence-based remasking.

  • Training Paradigm: SDAR starts from a well-trained AR model and applies efficient adaptation instead of costly end-to-end block-diffusion training.A subsequent SFT stage aligns the adapted model with conversational and reasoning abilities.
  • Training Paradigm: The conversion changes the attention mask and replaces the NLL objective with NELBO to obtain block-wise diffusion behavior.The stated conversion does not use logits shift or attention mask annealing.
  • Training Paradigm: SDAR partitions sequences into non-overlapping blocks and trains each corrupted block to denoise conditioned on preceding clean blocks.This yields inter-block autoregression and intra-block parallel diffusion.
  • Training Paradigm: The adaptation uses a significantly smaller dataset of about 30B–50B tokens than pretraining and relies on pretrained AR initialization for rapid convergence.The conversion is intended to unlock holistic reasoning within local blocks for parallel inference.
  • Hierarchical Inference: Inference generates blocks sequentially from the full preceding context while applying diffusion-based parallel decoding within each block.Previously generated blocks can provide reusable KV cache, whereas the current block is decoded without storing KV cache.
  • Hierarchical Inference: Each block begins fully masked and is iteratively refined across denoising steps until all positions are filled.At each step, selected masked positions receive sampled or greedy tokens while the remaining positions stay masked.
  • Decoding Strategies: Static low-confidence remasking decodes a fixed number of highest-confidence tokens per step, ensuring a constant number of forward passes.Dynamic low-confidence remasking instead decodes positions whose confidence exceeds threshold τ, with a minimum-selection fallback to guarantee progress.

4 From AR and Diffusion to SDAR

SDAR converts an efficiently trained AR model into a blockwise diffusion model, combining AR training and global sequence modeling with parallel intra-block decoding. Controlled experiments show that AR-based conversion preserves performance better than MDLM-based alternatives.

  • Motivation and approach: AR models provide a stronger foundation than MDLMs because they achieve lower training loss and better downstream performance under matched training conditions.AR training directly optimizes negative log-likelihood, whereas MDLMs typically optimize a less efficient ELBO-based objective.
  • Motivation and approach: SDAR converts an existing AR or MDLM backbone into Block Diffusion through short continued training instead of full Block Diffusion pretraining.The conversion retains AR training efficiency while adding parallel decoding within blocks.
  • Evaluation results: AR-2B-Chat outperforms MDLM-2B-Chat across nearly all benchmarks, including MATH at 29.9 versus 12.6 and HumanEval at 42.1 versus 21.3.These results were obtained under the controlled 2B-model training setup using the same 1T-token corpus and matched architecture components.
  • Evaluation results: AR-BD preserves AR performance while enabling parallel decoding, with MMLU improving by +2.2 and GPQA by +1.9 relative to AR-2B-Chat.The conversion incurs small decreases on MATH (-3.1) and HumanEval (-2.1).
  • Evaluation results: MDLM-to-Block-Diffusion conversion improves MDLM performance, but MDLM-BD remains behind AR-BD in most domains.MATH nearly doubles from 12.6 to 23.3 after MDLM conversion, yet the AR-derived model remains the stronger starting point.

5 Scaling SDAR: Principles and Practice

Scaling studies examine how SDAR’s efficiency and performance vary with model size and block size across dense and MoE architectures. Results show that larger SDAR models retain or improve benchmark performance while supporting efficient adaptation and stronger comparisons with AR and diffusion baselines.

  • Scaling principles and setup: SDAR’s central scaling objective is to measure intrinsic inference speedup and model how efficiency depends on model size and block size.The stated goal is a predictive scaling law for configuring larger models and block sizes.
  • Scaling results: Larger SDAR models improve consistently across benchmark categories, with MMLU increasing from 62.9% to 82.8% as scale grows from 1.7B to 30B.At 30B, SDAR-Chat matches or surpasses AR-Chat on 11 of 18 benchmarks.
  • Scaling results: SDAR-Chat exceeds AR-Chat on HumanEval by +2.4% and HumanEval-X by +2.8%, while maintaining strong mathematics performance.These results indicate that scaling preserves, and sometimes enhances, the base model’s reasoning capabilities.
  • Comparisons and adaptation cost: SDAR-8B reaches 78.6% on MMLU, compared with 65.9% for LLaDA-8B and 69.5% for Dream-7B.The comparison positions SDAR above the cited diffusion-based baselines on this benchmark.
  • Comparisons and adaptation cost: SDAR requires 50B tokens of continued pretraining for conversion, versus 65B for DiffuLLaMA and 580B for Dream.Smaller 20B, 30B, and 40B adaptation budgets produce functional models but slightly lower downstream performance and reduced block-size flexibility.

5.2 Scaling Dynamics: A Trade-off Analysis of Performance, Efficiency, and Model Dimensions

SDAR’s scaling behavior links model size, block size, confidence threshold, and deployment regime to a joint performance–efficiency trade-off. Larger models and carefully chosen decoding settings support more parallelism, while hardware and batch size determine realized gains.

  • Analysis Framework: The analysis varies performance, model scale, block size, decoding strategy, and efficiency measured by Effective Tokens Per Forward Pass (TPF).Models span 1.7B, 4B, 8B, and 30B-A3B; block sizes span 4–64, with static and confidence-thresholded dynamic decoding.
  • Decoding Strategy: Dynamic decoding selects tokens whose confidence exceeds τ, while guaranteeing progress by selecting high-confidence positions when too few tokens meet the threshold.The number of denoising steps matches the selected block size for each step, with τ evaluated from 0.80 to 0.95.
  • Performance Scaling: Larger models remain stable across wider block-size ranges, whereas 1.7B and 4B models degrade noticeably when B exceeds 4.The larger 8B and 30B-A3B models therefore tolerate more aggressive parallelization.
  • Performance Scaling: Intermediate block sizes B ∈{8, 16} preserve or improve performance on some reasoning and coding tasks, while B = 32 introduces task-dependent trade-offs.At B = 4, performance is nearly indistinguishable from autoregressive counterparts.
  • Confidence Thresholds: Performance improves as τ approaches 1.0, while the 30B-A3B model remains nearly saturated for τ ≥0.90 across most tasks and block sizes.MBPP is more sensitive to τ because the SFT models are less suited to base-model evaluation.
  • Efficiency Dynamics: TPF increases monotonically with block size and is consistently higher for larger models, whose higher confidence enables more tokens to be generated in parallel.Lower thresholds do not always increase TPF because low-quality blocks can propagate uncertainty to subsequent decoding steps.
  • Efficiency Dynamics: Higher quality and efficiency are intertwined in SDAR: conservative thresholds or more capable models can preserve low-entropy, high-confidence decoding states that improve acceleration.Thus, confidence and low entropy function as drivers of speed rather than merely outcomes traded against it.
  • Deployment Efficiency: SDAR reaches peak TGS values of 6600, 5510, 4236, 2520, and 1509 for block sizes 4, 8, 16, 32, and 64, respectively.Its advantage is strongest at smaller batch sizes and diminishes as batch size increases, especially for larger blocks.

6 Reasoning potential of SDAR

SDAR preserves AR-derived reasoning while adding local bidirectional context and parallel decoding. Experiments show competitive general reasoning, stronger scientific results, and substantial gains from test-time scaling.

  • Reasoning potential: SDAR preserves reasoning inherited from strong AR models and supports domain transfer during lightweight adaptation.The paper highlights reasoning preservation and domain adaptability as central properties.
  • General reasoning and mathematics: SDAR matches AR performance on MMLU-Pro, AIME-2024, and AIME-2025, while improving LiveMathBench-Hard by +5.3%.Code-generation results are mixed, with improvement on LCB-v6 but a deficit on LCB-v5.
  • Scientific domains: +5.5% on GPQA-diamond and +12.3% on ChemBench distinguish SDAR in scientific reasoning, while matching AR on PHYSICS and ProteinLMBench.The paper attributes these gains to local bidirectional attention for structured and associative knowledge.
  • Test-time scaling: +11.8% on AIME-2024, +19.3% on AIME-2025, and +15.7% on LiveMathBench-hard result from majority voting versus the AR baseline.The evaluation samples multiple reasoning paths before selecting the most frequent answer.
  • Test-time scaling: Pass@k reaches 84.3% on GPQA-diamond, 93.3% on AIME-2024, 86.7% on AIME-2025, and 87.5% on LiveMathBench-hard.These correspond to gains of +23.1%, +18.4%, +26.0%, and +32.1% over baseline, respectively.

7 Related Work

Related work contrasts AR’s efficient causal training with diffusion’s parallel, bidirectional generation. Hybrid and conversion approaches address this tension but traditionally incur substantial training or adaptation costs.

  • Autoregressive models: AR models use left-to-right factorization with efficient cross-entropy training, but their causal decoding limits parallelism and holistic reasoning.The limitation is especially relevant for global constraints, future dependencies, and revision.
  • Diffusion models: Diffusion models generate sequences through holistic denoising, enabling parallel decoding and a more flexible bidirectional inductive bias.These properties target latency and non-local reasoning limitations of AR models.
  • Diffusion models: Diffusion language models remain costly because ELBO-like objectives are harder to optimize and converge more slowly than AR cross-entropy.Reconstructing structured sequences from varying noise levels adds to the training burden.
  • Hybrid models: Block-wise hybrids combine global inter-block AR structure with parallel intra-block diffusion, preserving causal flow, variable-length generation, and KV caching.Their joint-from-scratch training regime compounds diffusion inefficiency with hybrid-loss complexity.
  • Conversion models: AR-to-diffusion conversion leverages AR pretraining but can require extensive adaptation and degrade performance relative to the original AR model.Dream 7B is cited as requiring 580B adaptation tokens.

8 Conclusion

SDAR decouples efficient AR training from parallel diffusion inference through lightweight adaptation. Controlled and scaling experiments report preserved accuracy, increasing decoding efficiency, and strong reasoning gains under test-time scaling.

  • Conclusion: SDAR converts a well-trained AR model into a blockwise diffusion model, retaining variable-length generation and KV caching while enabling parallel intra-block decoding.The design separates full-scale AR pretraining from lightweight adaptation.
  • Conclusion: Controlled experiments show SDAR maintains the performance of AR counterparts under identical computational budgets.This supports the feasibility and robustness of the adaptation process.
  • Conclusion: Larger models tolerate greater block sizes and more aggressive decoding thresholds while preserving accuracy and achieving higher parallel decoding efficiency.The paper describes this as a scaling synergy between model capability and throughput.
  • Conclusion: SDAR preserves AR reasoning strength and gains substantially from majority voting and pass@k test-time scaling.The paper connects these results to compatibility with reinforcement learning and other post-training optimization methods.
  • Conclusion: SDAR is presented as a scalable paradigm combining AR optimization efficiency with diffusion’s holistic representation and parallelizable generation without quality loss.The conclusion frames SDAR as an extensible alternative beyond purely autoregressive modeling.

A.1 Training Infrastructure

SDAR training requires specialized attention and loss infrastructure rather than standard AR kernels. The implementation improves memory efficiency, but effective training speed is approximately half that of standard AR models.

  • Training Infrastructure: SDAR’s specialized masks and objectives prevent direct use of FlashAttention and fused cross-entropy kernels.FlexAttention is adopted for attention computation instead.
  • Training Infrastructure: A custom fused Triton loss kernel avoids fully materializing logits, reducing GPU memory usage and improving efficiency.The kernel performs the loss computation as a fused operation.
  • Training Infrastructure: Effective SDAR training speed is approximately halved relative to standard AR models because each sample requires both clean and noised versions.Raw throughput per GPU per second is comparable to conventional AR training.
  • Training Infrastructure: The full-scale industry training framework is not entirely open source, though a research-oriented implementation is planned for release.The planned release is based on existing widely used frameworks.

A.2 Inference Infrastructure

SDAR extends autoregressive inference infrastructure with block-oriented optimizations while preserving compatibility with standard KV-cache management. Its modular implementation supports efficient interactive generation and future extensions.

  • Cache compatibility: SDAR maintains a mathematically identical KV cache to standard autoregressive models, enabling established management techniques such as PagedAttention.This compatibility supports integration with vLLM.
  • Inference optimization: The implementation uses a custom attention kernel for prefill and PagedAttention during decoding to support efficient block-oriented inference.Memory allocation is aligned to blocks for decoding.
  • Modularity: SDAR’s JetEngine2 implementation is lightweight and modular, supporting future integration of new sampling strategies and reinforcement learning algorithms.
  • Interactive serving: lmdeploy integration enables prefix-based KV-cache sharing across requests, lowering computational and memory overheads for interactive generation.

B Example Outputs

The example outputs illustrate SDAR’s reasoning behavior on chemistry and combinatorics tasks. The chemistry example identifies an oligosaccharide by its linked sugar units, while the combinatorics example counts valid assignments and obtains a remainder of 16.

  • Chemistry example: An oligosaccharide is identified as a molecule containing multiple monosaccharide units connected by glycosidic bonds.Option D is selected because it contains multiple sugar rings and oxygen-bridge linkages.
  • Chemistry example: Option D is favored over the alternatives because its structure shows multiple sugar rings and glycosidic linkages, unlike the non-oligosaccharide structures described for A, B, and C.
  • Combinatorics example: The baseball assignment problem requires positive counts C, V, and S satisfying C > V > S and C + V + S = 9.The valid triples are (6,2,1), (5,3,1), and (4,3,2).
  • Combinatorics example: The total is 2016 assignments, so the requested value modulo 1000 is 16.
  • Combinatorics example: The number of assignments is computed with multinomial coefficients for each valid count triple, giving 252, 504, and 1260 assignments.
Loading 2510.06303v3…