Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference

1 Introduction

• Diffusion models reverse incremental noise corruption through coarse-to-fine representations and have succeeded in image and video synthesis and natural-science challenges.The introduction cites diffusion models [3] [4] [5] as learning to reverse the corruption process.Source: §1 · p. 2
• Discrete language diffusion adapts continuous diffusion through token latents [8] [9] [10] or explicit discrete-state transition matrices [11–14], with recent approaches scaling via advanced architectures and training recipes [15].Standard diffusion is naturally defined over continuous state spaces and is not directly applicable to discrete natural language.Source: §1 · p. 2
• Random-order learning can be inefficient or detrimental because natural language is overwhelmingly processed sequentially, despite discrete diffusion's ability to generate tokens in arbitrary orders [13] [14].This mismatch can dampen model performance.Source: §1 · p. 2
• Although diffusion is non-autoregressive, its iterative, step-sensitive denoising introduces severe latency that undermines its practical advantage over autoregressive models.The introduction identifies this latency as a practical bottleneck.Source: §1 · p. 2
• 2146 tokens/second on H20 GPUs accompanies competitive performance against similarly-sized standard language models across code benchmarks, placing Seed Diffusion Preview on the speed-quality Pareto frontier.The paper describes this as a new state of the art on the speed-quality Pareto frontier.Source: §1 · p. 2

2 Related Work

• Early non-autoregressive methods enabled parallel inference and performed well on tasks such as machine translation, but lacked rigorous density-estimation foundations for general-purpose probabilistic language modeling [16] [17] [18].This limited their viability as general-purpose probabilistic language models.Source: §2 · p. 2
• Discrete diffusion models [11] [12] [13] [14] use ELBO optimization to provide a principled probabilistic framework for language modeling.Source: §2 · p. 2
• Mercury Coder and Gemini Diffusion show that discrete diffusion can narrow the quality gap with autoregressive systems while offering substantial speedups.Their results challenge the conventional view of a quality-speed trade-off and renew interest in non-autoregressive models for modern LLMs.Source: §2 · p. 2

3.1 TSC: A Two-Stage Curriculum for Robust Diffusion Training

• The two-stage curriculum uses mask corruption for the first 80% of diffusion training steps, then adds edit-based corruption for the last 20% to improve calibration and reduce sampling repetitions.The edit-based augmentation also forces the model to re-evaluate unmasked tokens, addressing the overconfidence and inability to self-correct that can result from mask-only training.Source: §3.1 · pp. 3–4

3.2 Tailoring the Trajectory Space of Diffusion

• To address diffusion training across redundant or misaligned generation orders, the method generates candidate trajectories with a pretrained diffusion model, selects them by ELBO, and fine-tunes on the resulting set.The selected high-quality trajectories constrain the generation orders used during subsequent training.Source: §3.2 · p. 4

3.3 On-policy Diffusion Learning

• On-policy learning optimizes reverse-process trajectories from all-mask sequences using a model-based verifier for reasonable or correct outputs; because direct trajectory-length minimization was unstable, it uses a progressive surrogate loss.The reported speed-up dynamics have an effect analogous to mode filtering in non-autoregressive text generation.Source: §3.3 · pp. 4–5

3.4 Inference and Infrastructure

• Inference uses causal block-level parallel sampling with KV-caching, avoids block-specific training to allow arbitrary partitions, and reports no significant quality degradation from caching.Specialized diffusion-sampling infrastructure supports the scheme, and block size is selected to trade off single-pass latency against token generation rate.Source: §3.4 · pp. 5–6

4 Experiments

• Evaluation protocols and primary baselines follow [20], with Mercury and Gemini-Diffusion added for comparison.Source: §4 · p. 6

4.1 Benchmarks

• Benchmarks cover basic coding ability with HumanEval and MBPP, plus BigCodeBench [28] for real-world, multi-tool programming tasks.BigCodeBench includes 1,140 Python tasks across 7 domains and 139 libraries, with an average of 5.6 test cases and 99% branch coverage per task.Source: §4.1 · p. 6
• LiveCodeBench [29] evaluates competitive-programming problems with release-date filtering to assess contamination-free slices, including stages v1–v6 and v6 (250201–250501).Source: §4.1 · p. 6
• MBXP [30] extends Python-centric MBPP problems and unit tests to more than ten programming languages, while NaturalCodeBench [31] tests realistic Python and Java user queries.NaturalCodeBench contains 402 problems across six domains with complex test inputs, including varied file types and data structures.Source: §4.1 · pp. 6–7

4.2 Performance

• Seed-Diffusion-Preview matches advanced autoregressive models at significantly higher speeds and boosts editing performance.The paper presents these results across Tables 1–3 and Figure 1.Source: Table 2 · p. 7, Table 3 · p. 7, §4.2 · p. 8

4.1 Benchmarks

• Aider and CanItEdit assess code editing, including automated application of edits and responses to explicit or ambiguous instructions.Aider has 133 Exercism exercises requiring edits to existing code; CanItEdit [32] has 105 hand-crafted problems with descriptive and lazy instructions.Source: §4.1 · pp. 7–8

• The experimental Seed Diffusion model’s technical components demonstrate its potential to significantly accelerate inference in large-scale language models.Source: §5 · p. 8
• Discrete diffusion may offer benefits beyond speed, including exploring alternatives to conventional left-to-right modeling order.The paper frames moving beyond this pervasive, human-centric assumption as a valuable research direction.Source: §5 · p. 8
• Unlocking discrete diffusion’s full capabilities will require community efforts to study its scaling properties and applications to complex reasoning tasks.Source: §5 · p. 8