Source-linked AI summary
Dream 7B: Diffusion Large Language Models
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, Lingpeng Kong
TL;DR
Autoregressive language models generate sequentially but have limitations in complex reasoning, long-term planning, and extended coherence. Dream 7B applies discrete diffusion with iterative denoising, AR-based initialization, and context-adaptive noise rescheduling, achieving competitive autoregressive-level performance while adding planning and flexible inference capabilities.
Problem
Autoregressive generation is limited in complex reasoning, long-term planning, and maintaining coherence across extended contexts, motivating alternative architectural paradigms.
Method
Dream 7B uses discrete diffusion to iteratively denoise sequences, combining AR-based LLM initialization with context-adaptive token-level noise rescheduling.
Results
Dream 7B consistently outperforms existing diffusion language models and achieves competitive performance with Qwen 2.5 across general, mathematical, and coding tasks.
Takeaways & Limitations
Dream 7B particularly excels in planning-intensive and constraint-satisfaction tasks while supporting arbitrary-order generation, infilling, and adjustable quality-speed trade-offs.
Abstract
from arXiv · showhide
We introduce Dream 7B, the most powerful open diffusion large language model to date. Unlike autoregressive (AR) models that generate tokens sequentially, Dream 7B employs discrete diffusion modeling to refine sequences in parallel through iterative denoising. Our model consistently outperforms existing diffusion language models on general, mathematical, and coding tasks. Dream 7B demonstrates superior planning abilities and inference flexibility, including arbitrary-order generation, infilling capabilities, and tunable quality-speed trade-offs. These results are achieved through simple yet effective training techniques, including AR-based LLM initialization and context-adaptive token-level noise rescheduling. We release both Dream-Base and Dream-Instruct to facilitate further research in diffusion-based language modeling.
1 Introduction
Dream 7B addresses limitations of sequential autoregressive generation with a scaled discrete diffusion model that iteratively denoises sequences. It matches strong autoregressive performance while adding planning and flexible inference capabilities.
- Motivation and contribution: Discrete diffusion starts from corrupted sequences and progressively denoises them, enabling bidirectional context integration and flexible generation patterns.This contrasts with left-to-right token-by-token generation in autoregressive models.
- Motivation and contribution: Dream 7B is a 7-billion-parameter diffusion language model designed to bridge the performance gap with autoregressive models.Its training framework uses AR-based initialization and context-adaptive noise scheduling.
- Results: Dream 7B consistently outperforms existing diffusion language models across diverse benchmarks.The comparison is reported across general, mathematical, and coding evaluations.
- Results: Dream 7B achieves competitive performance with Qwen 2.5 on general language understanding, mathematical reasoning, and code generation.The model also demonstrates superior planning abilities and diffusion-specific inference flexibility.
- Capabilities: Dream 7B provides superior planning abilities and arbitrary-order generation with tunable quality-speed trade-offs.These capabilities support flexible completion and infilling use cases.
2 Related Work
Research on diffusion language models has progressed from continuous and discrete formulations to billion-parameter systems. Recent work includes both adapting autoregressive models and training diffusion models from scratch, alongside commercial evidence for efficient code generation.
- Diffusion language modeling: Continuous diffusion language models model text in embedding space, while discrete diffusion operates directly over the vocabulary space.Discrete diffusion was introduced to fit the discrete nature of text.
- Diffusion language modeling: Discrete diffusion progressively corrupts text with [MASK] or random tokens and learns a reverse process to reconstruct the original sequence.These choices produce absorbing-state or uniform-state formulations.
- Scaling: Scaling efforts reached billion-parameter regimes through both adaptation of pretrained autoregressive models and training diffusion models from scratch.Examples include DiffuLLaMA and DiffuGPT from AR models, and LLaDA trained from scratch.
- Scaling: Continuous diffusion models require substantially longer training than autoregressive counterparts for optimal computational efficiency.In contrast, masked discrete diffusion models have achieved GPT-2-level-comparable or better perplexities.
- Applications: Mercury Coder demonstrates commercial applicability and inference efficiency for diffusion-based code generation.This provides evidence beyond academic language-model scaling efforts.
3 Preliminary
Autoregressive modeling factorizes sequence probability into left-to-right conditional predictions, whereas discrete diffusion corrupts sequences and learns iterative full-context denoising. Dream uses continuous-time noise levels and a weighted masked-token cross-entropy objective.
- 3.1 Auto-regressive Modeling: Autoregressive sequence modeling decomposes joint probability into conditional probabilities over preceding tokens.The hidden representation at each position is associated with predicting the next token.
- Generation paradigms: Autoregressive generation uses progressive left-context prediction, while diffusion uses progressive full-context prediction during denoising.The two labels characterize the different sequence-generation directions.
- 3.2 Discrete Diffusion Modeling: Discrete diffusion progressively corrupts clean sequences with [MASK] tokens and reverses this process by iteratively predicting masked tokens from t = T to 0.The forward process uses a noise schedule, and the backward process reconstructs the original data distribution.
- 3.2 Discrete Diffusion Modeling: The model parameters are optimized by minimizing the negative log-likelihood of the clean data.This objective is implemented through a tractable weighted cross-entropy formulation for masked positions.
- 3.2 Discrete Diffusion Modeling: Continuous-time parameterization lets t vary over [0, 1], enabling sampling across arbitrary noise levels rather than predetermined discrete levels.The forward process can therefore be expressed as q(x_t|x_s) for any 0 ≤ s < t ≤ 1.
- 3.2 Discrete Diffusion Modeling: The weighted cross-entropy loss is computed only on masked token positions, with w(t) determined by the noise schedule.For α_t = 1 − t, the corresponding time-dependent weight is specified in the formulation.
4 Approach
Dream combines Transformer compatibility and AR-based initialization with diffusion training, then adapts noise levels per token according to contextual informativeness. This design targets token-specific dependencies while preserving alignment with existing autoregressive models.
- AR-based LLM Initialization: Dream uses shifted prediction of masked tokens to maximize architectural alignment and weight initialization compatibility with autoregressive models.
- AR-based LLM Initialization: AR initialization builds on established sequence-modeling capabilities instead of learning representations from scratch, enabling faster iteration from stronger pretrained AR models.
- Context-Adaptive Token-Level Noise Rescheduling: Standard sentence-level timesteps assign one noise level across tokens despite differing contextual dependencies, producing suboptimal learning for tokens with unequal information.
- Context-Adaptive Token-Level Noise Rescheduling: CART measures each token’s contextual informativeness and dynamically reassigns token-level noise levels to provide fine-grained guidance.
- Context-Adaptive Token-Level Noise Rescheduling: A geometric-distribution mixture quantifies each clean token’s information contribution relative to noised tokens; smaller p spreads influence, while larger p emphasizes nearby masked tokens.
- Training: Dream uses the Transformer architecture, adopts the Qwen2.5-7B configuration, and trains on text, mathematics, and code with supervised fine-tuning for instruction following.
5.1 Setup
Dream is evaluated across general language understanding, mathematical and scientific reasoning, and coding-related benchmarks. The setup is designed to test knowledge, reasoning, commonsense, and programming abilities.
- Evaluation Tasks: General language understanding is evaluated with MMLU, BBH, ARC-E, ARC-C, HellaSwag, WinoGrande, PIQA, and RACE.
- Evaluation Tasks: Mathematical and scientific reasoning is assessed with GSM8K and MATH.
- Evaluation Tasks: The benchmark suite spans multiple domains to test knowledge, reasoning, commonsense abilities, and code generation.
5.2 Results of Dream-Base
Dream 7B remains competitive with Qwen2.5 7B on general language tasks while showing its clearest advantages on reasoning and planning benchmarks. Against LLaDA 8B, it improves across evaluation domains using one quarter of the training data.
- Planning and reasoning: 16.0 vs. 6.2 on Countdown, 81.0 vs. 21.0 on Sudoku, and 17.8 vs. 3.6 on Trip planning show Dream 7B’s largest advantages over Qwen2.5 7B.
- General and reasoning tasks: Dream 7B achieves comparable general-task performance with modest MMLU and HellaSwag gaps while outperforming Qwen2.5 7B on ARC-E and ARC-C.
- Comparison with LLaDA: Dream 7B improves ARC-C from 47.5 to 59.8 and GSM8K from 70.9 to 77.2 relative to LLaDA 8B.
- Comparison with LLaDA: Dream 7B improves Countdown from 13.2 to 16.0 and Sudoku from 46.0 to 81.0 relative to LLaDA 8B.
5.3 Results of Dream-Instruct
Dream-Instruct uses lightweight supervised fine-tuning to align the diffusion model with user instructions. Its results indicate potential to match autoregressive models on instruction-following tasks.
- Supervised fine-tuning: Dream-Instruct is fine-tuned for three epochs on 1.8M instruction-response pairs curated from Tulu 3 and SmolLM 2.
- Results: Dream-Instruct demonstrates potential to match autoregressive language models on instruction-following tasks.
5.4 Effect of AR Initialization
Dream uses autoregressive initialization to accelerate diffusion-model training, while learning-rate calibration helps preserve inherited autoregressive knowledge and support diffusion learning.
- AR initialization gives Dream a strong early-training advantage over training from scratch.The inherited left-to-right knowledge provides a foundation for developing any-order generation capabilities more quickly.
- Learning-rate choice must balance retaining autoregressive knowledge against learning the diffusion process effectively.Excessively high rates degrade inherited left-to-right knowledge, whereas overly conservative rates impede diffusion learning.
- Figure 4 compares loss during 200B-token Dream 1B training initialized from LLaMA3.2 1B or trained from scratch.
- AR initialization begins with high loss because training transitions from causal attention to full attention, but remains lower than training from scratch throughout.
5.5 Discussion
Dream 7B shows strong planning performance and flexible inference behavior, including controllable quality-speed trade-offs, infilling, and configurable decoding order.
- 5.5.1 Analysis of Planning Ability: Dream’s planning advantage is especially pronounced for problems involving multiple constraints or specific objective optimization.
- 5.5.2 Quality-Speed Trade-Offs: Reducing diffusion timesteps accelerates inference but may lower quality, whereas increasing them improves quality at additional computational cost.
- 5.5.2 Quality-Speed Trade-Offs: With 5-20 diffusion steps, Dream achieves superior speed and quality to Qwen2.5 7B on Countdown.
- 5.5.3 Inference Flexibility: Users can configure decoding order from structured left-to-right generation toward partially or fully random-order synthesis.This flexibility allows output construction to be adapted to different preferences and task requirements.
- 5.5.3 Inference Flexibility: Dream supports completion and infilling without specialized training.Infilling can be unconstrained or guided by requirements such as producing an exact ending sentence.
6 Conclusion
Dream 7B combines competitive autoregressive-level performance with advantages in planning and flexible text generation. Its arbitrary-order generation, infilling, adjustable timesteps, and autoregressive initialization support more adaptable diffusion language modeling.
- Dream 7B achieves competitive performance with state-of-the-art autoregressive models while providing flexible text generation capabilities.
- Diffusion language models excel particularly in constraint-satisfaction and planning tasks through bidirectional processing and iterative refinement.
- Arbitrary-order generation supports completion and infilling, while adjustable timesteps provide fine-grained control over inference quality and speed.
- Autoregressive initialization provides a practical pathway for developing more flexible and capable language models.