Source-linked AI summary
Dream-Coder 7B: An Open Diffusion Language Model for Code
Zhihui Xie, Jiacheng Ye, Lin Zheng, Jiahui Gao, Jingwei Dong, Zirui Wu, Xueliang Zhao, Shansan Gong, Xin Jiang, Zhenguo Li, Lingpeng Kong
TL;DR
Autoregressive code models generate left-to-right, limiting holistic reasoning, flexible infilling, and refinement for complex programming tasks. Dream-Coder 7B adapts a pretrained autoregressive checkpoint into an open-source discrete diffusion model with bidirectional context and parallel generation, achieving competitive coding performance and adaptive generation patterns. Dream-Coder 7B Instruct reaches 21.4% pass@1 on LiveCodeBench and performs competitively across coding benchmarks.
Problem
Autoregressive models’ sequential left-to-right generation struggles with global program understanding, flexible infilling, and multi-step refinement in complex programming scenarios.
Method
Dream-Coder 7B adapts the Qwen2.5-Coder 7B autoregressive architecture into a discrete diffusion language model using shift operations to retain pretrained knowledge while enabling bidirectional context and parallel generation.
Results
Dream-Coder 7B Instruct achieves competitive performance with autoregressive baselines across coding benchmarks, including 21.4% pass@1 on LiveCodeBench.
Takeaways & Limitations
Diffusion language models support adaptive generation patterns and offer distinct advantages for complex coding scenarios through non-autoregressive generation and iterative refinement.
Takeaways & Limitations
Future work is needed on context extension, data curation, and specialized coding domains requiring advanced planning and reasoning.
Abstract
from arXiv · showhide
We present Dream-Coder 7B, an open-source discrete diffusion language model for code generation that exhibits emergent any-order generation capabilities. Unlike traditional autoregressive (AR) models that decode strictly left-to-right, Dream-Coder 7B adaptively determines its decoding strategy based on the coding task: sketch-first generation for complex algorithms, left-to-right generation for straightforward completions, and interleaved reasoning generation for code understanding tasks. We adapt a pretrained AR checkpoint to a discrete diffusion frameworks with a continuous-time weighted cross-entropy objective. Our post-training recipe comprises (i) supervised fine-tuning, where we mitigate padding pathologies via random truncation and a padding penalty to improve sample efficiency and stabilize generation; and (ii) reinforcement learning with verifiable rewards over a curated high-quality prompt set drawn from open-source datasets, using a tailored reinforcement learning recipe for diffusion language models. The resulting Dream-Coder 7B Instruct attains 21.4\% pass@1 on LiveCodeBench (2410--2505) and demonstrates competitive performance on HumanEval, MBPP, BigCodeBench, and CRUXEval. We release Dream-Coder-7B and Dream-Coder-7B-Instruct checkpoints, training recipes, preprocessing pipelines, and inference code to facilitate reproducibility and further research.
1 Introduction
Dream-Coder 7B addresses limitations of left-to-right code generation with an open-source diffusion model that adaptively uses multiple generation patterns. The paper introduces the model and releases its artifacts for reproducibility.
- Autoregressive code generation remains dominant, but complex refactoring and debugging require global understanding, flexible infilling, and multi-step refinement.
- Diffusion modeling offers bidirectional context, iterative refinement, controllable generation, and adjustable quality-speed trade-offs.
- Dream-Coder 7B combines AR-based initialization, context-adaptive noise scheduling, and verifiable reward-driven reinforcement learning for code generation.
- Dream-Coder 7B is an open-source diffusion code model designed to achieve competitive performance with autoregressive baselines while offering unique generation capabilities.
- The model exhibits sketch-first scaffolding, left-to-right completion, and interleaved reasoning that adapt to different coding task complexities.
- The authors release checkpoints, training recipes, preprocessing pipelines, and inference code to facilitate reproducibility and further research.
2 Dream-Coder-7B
Dream-Coder 7B uses discrete diffusion to corrupt code with masks and learn iterative denoising, while continuous-time training permits flexible noise levels. Its shift operation adapts a pretrained autoregressive architecture for bidirectional and parallel generation.
- 2.1 Background: Discrete Diffusion Modeling: Discrete diffusion combines a forward noising process with a learned reverse denoising process over masked sequences.
- 2.1 Background: Discrete Diffusion Modeling: The forward process progressively corrupts clean sequences, while the backward process predicts masked tokens as the timestep decreases.
- 2.1 Background: Discrete Diffusion Modeling: Denoising steps use full-sequence context rather than autoregressive left-context prediction.
- 2.1 Background: Discrete Diffusion Modeling: Continuous-time parameterization with t ∈ [0, 1] enables flexible noise schedules and sampling at arbitrary noise levels.
- 2.1 Background: Discrete Diffusion Modeling: The weighted cross-entropy objective provides a tractable variational upper bound on negative log-likelihood for large-scale diffusion language-model training.
- 2.2 Architecture: Dream-Coder 7B builds on Qwen2.5-Coder 7B and uses a shift operation to retain pretrained knowledge while enabling bidirectional context and parallel generation.
3 Pre-training
Dream-Coder 7B is trained to reconstruct masked code tokens from full noisy-sequence context using continuous-time diffusion. Its pretraining uses a large public-data mixture and context-adaptive token weighting.
- Training samples clean sequences, draws t ∼ U(0, 1), masks tokens according to α_t, and predicts the original masked tokens from full context.
- Context-Adaptive Token-Level Noise Rescheduling is applied during pretraining.
- The token-level weighting function can depend on the noised sequence structure and assigns higher noise rates to harder-to-predict tokens.
- Pretraining uses a mixture of publicly available code, math, and general-language datasets totaling 322B tokens.
- The corpus includes OpenCoder1, Stack-Edu, Dolmino, and DCLM-Baseline data sources.
4 Post-Training
Dream-Coder 7B’s post-training combines supervised fine-tuning with padding-focused stabilization and reinforcement learning using verifiable rewards. The resulting strategy improves sample utilization and stabilizes generation length while targeting code reasoning.
- Supervised Fine-Tuning: 5 million high-quality instruction-based code examples from Ling-Coder-SFT are used for supervised fine-tuning with rule-based filtering.
- Supervised Fine-Tuning: Random truncation reduces wasted computation on padding tokens by focusing learning on meaningful response-token predictions.
- Supervised Fine-Tuning: A gradually decaying inference penalty on [PAD] logits discourages premature sequence termination and improves control over generation length.
- Supervised Fine-Tuning: The combined truncation and padding-penalty strategy enhances sample utilization and stabilizes generation length without sacrificing accuracy.
- Reinforcement Learning with Verifiable Rewards: Reinforcement learning with verifiable rewards is applied to boost Dream-Coder’s reasoning capabilities for code using the GRPO algorithm.
- Reinforcement Learning with Verifiable Rewards: The reinforcement-learning data comprise 17k prompts with unit-test-based rewards, while quality filtering, deduplication, and difficulty calibration support data curation.
- Reinforcement Learning with Verifiable Rewards: For diffusion-language-model training, GRPO uses omitted entropy/KL penalties, asymmetric Clip-Higher bounds of εlow = 0.2 and εhigh = 0.28, and coupled sampling with informative substitution.
5 Experiments
Dream-Coder 7B is evaluated across coding, general, mathematical, and scientific reasoning benchmarks, with post-training results for instruction-following models. The experiments also examine task-adaptive generation patterns that distinguish diffusion decoding from fixed left-to-right generation.
- Base-model results: Dream-Coder 7B closely matches Qwen2.5-Coder-7B on coding benchmarks and significantly outperforms other diffusion-based models.The model also substantially outperforms baseline diffusion models on general benchmarks while approaching state-of-the-art autoregressive models.
- Instruction-tuned results: 21.4% pass@1 on LiveCodeBench (2410-2505) places Dream-Coder 7B Instruct near Mercury Coder Small and above OpenCoder 8B Instruct.The instruction-tuned model also performs on par with top-tier models across HumanEval, MBPP, EvalPlus, and BigCodeBench.
- Generation patterns: Dream-Coder 7B uses sketch-first scaffolding for complex algorithmic tasks, generating structure before progressively filling implementation details.This pattern is described for template-heavy tasks such as LiveCodeBench, where global program flow is important.
- Generation patterns: For standalone function completion, Dream-Coder 7B follows a mostly left-to-right order while retaining diffusion-based local refinement and correction.This hybrid pattern is associated with HumanEval and MBPP and is reported to improve success rates on standard completion benchmarks.
- Generation patterns: For logic-intensive tasks, the model generates logical components non-linearly and selects among patterns based on prompt structure, task complexity, and learned representations.Interleaved reasoning is associated with CRUXEval, while the broader pattern-selection behavior emerges from diffusion training rather than explicit programming.
6 Related Work
Diffusion models offer flexible generation orders and iterative refinement as an alternative to autoregressive code generation. Recent work has applied these ideas to programming languages, with DiffuCoder pioneering masked diffusion strategies for code.
- Autoregressive Code Models: Autoregressive code models remain the dominant paradigm, generating tokens sequentially from left to right and establishing strong pretrained baselines.Recent work has also enhanced coding capabilities through advanced reasoning and agentic tool-use frameworks.
- Diffusion Language Models: Diffusion language models provide flexible generation orders and iterative refinement capabilities as alternatives to autoregressive generation.Discrete diffusion established the foundation for text diffusion modeling and subsequent work improved these approaches from multiple perspectives.
- Diffusion Models for Code Generation: Code generation’s iterative refinement workflow makes discrete diffusion a natural fit for programming tasks.The approach can leverage the inherently iterative nature of programming workflows.
- Diffusion Models for Code Generation: DiffuCoder pioneered masked diffusion strategies for programming languages and examined how diffusion objectives capture code structure and dependencies.
7 Conclusion
Dream-Coder-7B extends open-source diffusion language models through post-training improvements for code generation and reports competitive performance with state-of-the-art autoregressive models. The authors identify complex coding scenarios as a particular strength and plan future work on context extension, data curation, and specialized domains.
- 7 Conclusion: Dream-Coder-7B focuses on post-training improvements and code-generation capabilities within open-source diffusion language models.
- 7 Conclusion: Diffusion language models show particular strength in complex coding scenarios through non-autoregressive generation and iterative refinement.
- 7 Conclusion: Future work will explore context extension, improved data curation, and specialized coding domains requiring advanced planning and reasoning.