Source-linked AI summary
Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Model
Chenghao Fan, Wen Heng, Bo Li, Sichen Liu, Yuxuan Song, Jing Su, Xiaoye Qu, Kai Shen, Wei Wei
TL;DR
Existing code DLLMs lag strong AR baselines under comparable budgets, motivating a controlled study of whether diffusion training can improve code modeling. The paper introduces Stable-DiffCoder, which reuses Seed-Coder’s architecture and data while adding block diffusion continual pretraining and stabilization techniques. Under matched architecture and data, Stable-DiffCoder outperforms its AR counterpart broadly and achieves strong results against 8B-scale AR and diffusion models, while its broader-domain benefits remain uncertain.
Problem
Existing code DLLMs lag strong AR baselines under comparable budgets, leaving diffusion training’s value for code modeling insufficiently established.
Method
Stable-DiffCoder reuses Seed-Coder’s architecture, data, and pipeline, adding block diffusion continual pretraining with tailored warmup and block-wise clipped noise scheduling.
Results
Under the same architecture and data, Stable-DiffCoder broadly outperforms its AR Seed-Coder counterpart and achieves stronger performance than a wide range of approximately 8B AR and diffusion models.
Takeaways & Limitations
Diffusion-based training can improve code modeling quality beyond AR training alone, while diffusion sampling provides effective data augmentation within the evaluated code setting.
Takeaways & Limitations
Because Stable-DiffCoder focuses on code and lacks large-scale data from other domains, its mathematical reasoning and general-purpose text performance may be limited.
Abstract
from arXiv · showhide
Diffusion-based language models (DLLMs) offer non-sequential, block-wise generation and richer data reuse compared to autoregressive (AR) models, but existing code DLLMs still lag behind strong AR baselines under comparable budgets. We revisit this setting in a controlled study and introduce Stable-DiffCoder, a block diffusion code model that reuses the Seed-Coder architecture, data, and training pipeline. To enable efficient knowledge learning and stable training, we incorporate a block diffusion continual pretraining (CPT) stage enhanced by a tailored warmup and block-wise clipped noise schedule. Under the same data and architecture, Stable-DiffCoder overall outperforms its AR counterpart on a broad suite of code benchmarks. Moreover, relying only on the CPT and supervised fine-tuning stages, Stable-DiffCoder achieves stronger performance than a wide range of \~8B ARs and DLLMs, demonstrating that diffusion-based training can improve code modeling quality beyond AR training alone. Moreover, diffusion-based any-order modeling improves structured code modeling for editing and reasoning, and through data augmentation, benefits low-resource coding languages.
1 Introduction
Code development often requires non-sequential operations that strictly left-to-right decoding does not support well. Stable-DiffCoder studies whether block diffusion can improve code modeling under controlled conditions while retaining the Seed-Coder architecture and data.
- Code workflows include infilling, revision using later context, scratch-to-final iteration, and parallel generation, exposing limits of strictly sequential AR decoding.
- Stable-DiffCoder reuses the Seed-Coder architecture, data, and training pipeline while adding a block diffusion continual pretraining stage.The stage uses small-block diffusion with block size 4 and 1.3T tokens.
- The training pipeline adds tailored warmup and block-wise clipped noise scheduling to improve DLLM knowledge learning and training stability.
- Under matched architecture and data, Stable-DiffCoder almost uniformly surpasses the same-size AR Seed-Coder across base and instruction-tuned settings.
- Stable-DiffCoder establishes new state-of-the-art results among 8B-scale diffusion code models on many metrics.
2 Preliminary
AR models predict tokens left to right, whereas masked DLLMs corrupt sequences and train denoisers to reconstruct clean text. Block diffusion extends this framework by corrupting contiguous spans while preserving surrounding context, supporting partially parallel generation and data reuse.
- AR language models factorize sequence probabilities left to right and train Transformer parameterizations with token-level cross-entropy.
- Token-by-token AR generation can be slow and lacks mechanisms for global planning or simultaneous reasoning across multiple sequence parts.
- Masked DLLMs replace tokens with [MASK] during forward corruption and iteratively reconstruct clean text through reverse denoising.
- DLLM training can reduce to a weighted cross-entropy objective over masked tokens, with weights depending on corruption level.
- Block diffusion corrupts only a contiguous token block while keeping surrounding context clean, enabling span generation conditioned on context.
- Diffusion training supports parallel decoding and repeatedly reuses examples under diverse corruption trajectories, potentially extracting more information from rare or long-tail samples.
3 Approach
The analysis identifies efficient diffusion knowledge learning as requiring reliable reasoning evidence, controlled contexts, and alignment between training and inference. These principles motivate AR-to-diffusion curricula, small-block training, and warmup for stable continual pretraining.
- Token Reasoning Knowledge: Training efficiency depends on small candidate sets and repeated context–label exposure, while fully bidirectional or large-block diffusion often enters correlation or noise regimes.Clean, contiguous left-side evidence generally yields smaller candidate sets and more efficient knowledge compression.
- Token Reasoning Knowledge: Random masking improves data reuse but can create weakly constrained contexts whose gradients capture noisy co-occurrence rather than clear reasoning rules.Heavy masking can prevent the model from seeing clean evidence needed to learn a stable mapping from inputs to outputs.
- Training–Inference Alignment: Good diffusion training should expose clean reasoning evidence, limit context proliferation, and align training contexts with inference trajectories.The analysis explicitly formulates these as two conditions for effective knowledge acquisition and data augmentation.
- Curriculum Design and Empirical Results: Under equal compute after CPT, the three training schemes rank (1) > (2) > (3), with scheme (1) strongest for small-block decoding before bidirectional conversion.Pure AR training aligns closely with small-block inference, but its causal structure does not cover bidirectional patterns required by large blocks.
- Curriculum Design and Empirical Results: After conversion to a bidirectional DLLM, scheme (1) performs well across block sizes because AR training compresses knowledge efficiently before CPT adapts to larger-block contexts.The subsequent CPT primarily adapts the model to the larger-block context distribution.
- Warmup for Stable DLLM Continued Pretraining: Warmup produces a smoother CPT trajectory by reducing the AR-to-DLLM gradient spike and lowering peak loss toward AR continual-pretraining levels.Stable behavior is also observed for block-diffusion training without logit shifting.
4 Experiments
The experiments compare both base and instruction-tuned models across diverse code-centric benchmarks.
- Both the base model and its instruction-tuned variant are evaluated.
- The evaluation spans a diverse set of code-centric benchmarks.
4.1 Experiments Setting
The study reuses the Seed-Coder pipeline and compares Stable-DiffCoder with broad autoregressive and diffusion-language-model baselines.
- Experimental setup: Stable-DiffCoder reuses the Seed-Coder training pipeline and supervised fine-tuning dataset while continuing pretraining from a pre-annealing checkpoint.Continuous pretraining uses 1.3T subsampled tokens and packed sequences with context length 8192.
- Implementation: Training efficiency is supported by packed-sample attention reuse and the same packing strategy during supervised fine-tuning.Randomly appending 1–4 <eos> tokens preserves variable-length generation within packed blocks.
- Baselines: The comparison includes strong autoregressive baselines such as StarCoder2, DeepSeek-Coder, CodeQwen1.5, OpenCoder, Qwen2.5-Coder, Seed-Coder, CodeLlama, and Llama3.1.
- Baselines: The diffusion-language-model baselines include LLaDA, Dream, DiffuCoder, Dream-Coder, LLaDA-MoE, Fast-dLLMv2, SDAR, and other recent models.
4.2 Benchmarks
The benchmark suite covers code generation, execution-centric reasoning, multilingual generalization, and instruction-following code editing across standard and challenging evaluations.
- Benchmark coverage: The evaluation spans function-level code generation, execution-centric reasoning, multilingual generalization, and instruction-following code editing.
- Code generation: HumanEval(+), MBPP(+), MHPP, and BigCodeBench assess increasingly strict, difficult, or realistic code-generation performance.HumanEval+ and MBPP+ expand test suites, while BigCodeBench includes tool-like calls across 139 libraries and 1,140 Python tasks.
- Code reasoning: CRUXEval evaluates Python reasoning by asking models to predict inputs from outputs and outputs from inputs.
- Multilingual evaluation: MultiPL-E and MBXP measure multilingual code generation across multiple programming languages, including 18 and 13 languages respectively.
- Contamination-resistant evaluation: LiveCodeBench uses time-stamped competitive-programming problems to reduce contamination and overfitting to static benchmarks.
- Editing and practical coding: NaturalCodeBench, Aider, and CanItEdit test realistic user-query coding and instructional editing, including underspecified edit requests.
4.3 Evaluation of Base Models
Stable-DiffCoder performs strongly across function generation, multilingual generation, and code reasoning, often improving over Seed-Coder and comparable diffusion models.
- HumanEval and MBPP: Stable-DiffCoder-8B-Base achieves the best overall performance among comparable-scale diffusion models on HumanEval(+) and MBPP(+).
- HumanEval and MBPP: Against similarly sized autoregressive baselines, it wins on HumanEval, HumanEval+, and MBPP, while being slightly inferior on MBPP+.
- Multilingual generation: Stable-DiffCoder shows particularly large gains in C# and PHP on MultiPL-E, where those languages are sparsely represented in the training corpus.The paper hypothesizes that stochastic corruption and denoising provide multiple views of low-resource examples.
- Code reasoning: Stable-DiffCoder outperforms Seed-Coder-Base on both CRUXEval input and output reasoning tasks.The authors relate this result to moderate masking and any-order modeling of structured components.
4.4 Evaluation of Instruction Models
Stable-DiffCoder-8B-Instruct performs strongly across code generation, reasoning, and editing benchmarks, with especially strong results on MHPP, BigCodeBench, and CanItEdit. Its performance is mixed on LiveCodeBench, multilingual coding, NaturalCodeBench, CRUXEval, and long-context Aider.
- 4.4.1 Code Generation: Stable-DiffCoder-8B-Instruct significantly improves upon Seed-Coder-8B-Instruct on HumanEval(+) and MBPP(+), while surpassing other ∼8B diffusion models.
- 4.4.1 Code Generation: Stable-DiffCoder-8B-Instruct achieves the best performance among compared models on MHPP, matching Qwen2.5-Coder-32B-Instruct.
- 4.4.1 Code Generation: On BigCodeBench, Stable-DiffCoder-8B-Instruct substantially improves over Seed-Coder-8B-Instruct and is surpassed only by the much larger DeepSeek-Coder-V2-Instruct.
- 4.4.1 Code Generation: Stable-DiffCoder-8B-Instruct scores 23.5% on LiveCodeBench, below Seed-Coder-8B-Instruct at 24.7%, but matches Qwen3-8B and exceeds other ∼8B-scale models.
- 4.4.1 Code Generation: Under the 10+ language MBXP setting, Stable-DiffCoder-8B-Instruct has a comparable overall average to Seed-Coder-8B-Instruct and the highest scores in most languages among ∼8B instruct models.
- 4.4.1 Code Generation: On NaturalCodeBench, Stable-DiffCoder-8B-Instruct is slightly weaker on Chinese Python queries but otherwise performs on par with Seed-Coder-8B-Instruct overall.
- 4.4.2 Code Reasoning: Stable-DiffCoder-8B-Instruct performs better than Seed-Coder-8B-Instruct on CRUXEval Output-CoT and slightly better on average, although Qwen3-8B remains ahead.
- 4.4.3 Code editing: Stable-DiffCoder-8B-Instruct substantially outperforms all other models on CanItEdit, while remaining slightly weaker than Seed-Coder-8B-Instruct on Aider under tries=2.The Aider result is associated with long contexts exceeding the 8192-token training window; the CanItEdit gain is hypothesized to reflect denoising-based edit and infill patterns.
5 Conclusion, Limitation, and Future Work
The paper presents Stable-DiffCoder as evidence that diffusion training can improve code-model performance under matched architecture and data, while broader-domain performance remains an open limitation.
- With architecture and data identical to Seed-Coder, Stable-DiffCoder’s diffusion-based training yields consistently better results across comprehensive code benchmarks.
- Stable-DiffCoder achieves state-of-the-art results among ∼8B autoregressive and diffusion-based code models.
- Because Stable-DiffCoder focuses on code and lacks large-scale data from other areas, its mathematical-reasoning and general-purpose-text performance may be limited.Whether text-diffusion sampling provides greater benefits in broader domains remains open.
6 Contributions
The listed contribution information identifies the project lead, core contributor, contributors, and two affiliations associated with the work.
- Chenghao Fan is listed as the project lead.
- Chenghao Fan is also listed as a core contributor.
- The contributor list includes Wen Heng, Bo Li, Sichen Liu, Yuxuan Song, Jing Su, Xiaoye Qu, Kai Shen, and Wei Wei.