Source-linked AI summary
Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference
Yuxuan Song, Zheng Zhang, Cheng Luo, Pengyang Gao, Fan Xia, Hao Luo, Zheng Li, Yuehang Yang, Hongli Yu, Xingwei Qu, Yuwei Fu, Jing Su, Ge Zhang, Wenhao Huang, Mingxuan Wang, Lin Yan, Xiaoying Jia, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Yonghui Wu, Hao Zhou
TL;DR
Discrete diffusion language models target the latency of token-by-token decoding while retaining competitive language-modeling quality. Seed Diffusion Preview applies discrete diffusion to code generation with training and sampling strategies designed for efficient inference, achieving 2,146 tokens/s on H20 GPUs and a new speed-quality Pareto frontier for code models.
Problem
Iterative denoising latency and token-order modeling challenges limit the practical deployment of discrete diffusion models for language.
Method
Seed Diffusion Preview is a code-focused discrete diffusion language model using code-only data, edit-based augmentation, on-policy learning, and block-level parallel sampling.
Results
2,146 tokens/second on H20 GPUs with competitive performance across diverse code evaluation benchmarks establishes a new speed-quality Pareto frontier.
Takeaways & Limitations
Discrete diffusion shows potential for significant inference acceleration and for exploring alternatives to conventional left-to-right language modeling.
Abstract
from arXiv · showhide
We present Seed Diffusion Preview, a large-scale language model based on discrete-state diffusion, offering remarkably fast inference speed. Thanks to non-sequential, parallel generation, discrete diffusion models provide a notable speedup to mitigate the inherent latency of token-by-token decoding, as demonstrated recently (e.g., Mercury Coder, Gemini Diffusion). Seed Diffusion Preview achieves an inference speed of 2,146 token/s over H20 GPUs while maintaining competitive performance across a sweep of standard code evaluation benchmarks, significantly faster than contemporary Mercury and Gemini Diffusion, establishing new state of the art on the speed-quality Pareto frontier for code models.
1 Introduction
Discrete diffusion extends diffusion modeling to language but faces challenges from discrete state spaces, token-order inductive bias, and iterative denoising latency. Seed Diffusion Preview addresses these constraints for code generation, reaching 2,146 tokens/s while maintaining competitive benchmark performance.
- Diffusion models reverse incremental data corruption through a hierarchy of simplified representations, succeeding across image, video, and natural-science applications.
- Natural-language diffusion requires adaptations because standard diffusion is defined over continuous rather than discrete state spaces.Approaches include continuous latent projections and explicit discrete-state transition matrices.
- Random-order token modeling can be inefficient or detrimental because natural language is predominantly processed sequentially.
- Iterative step-sensitive denoising creates severe latency that undermines diffusion’s non-autoregressive advantage over autoregressive models.
- 2,146 tokens/second on H20 GPUs accompanies competitive performance across diverse code evaluation benchmarks, establishing a new speed-quality Pareto frontier.Seed Diffusion Preview is a code-focused language model designed to balance speed and quality.
2 Related Work
Non-autoregressive language models offer parallel inference but early approaches lacked a rigorous probabilistic foundation. Discrete diffusion addresses this gap through ELBO-based modeling and has narrowed the quality gap with autoregressive systems while improving speed.
- Early non-autoregressive methods enabled parallel inference and performed strongly on selected tasks such as machine translation.
- Early non-autoregressive methods often lacked a rigorous theoretical foundation for density estimation, limiting their viability as general-purpose probabilistic language models.
- Discrete diffusion models close this gap by optimizing the Evidence Lower Bound as a principled probabilistic framework for language modeling.
- Mercury Coder and Gemini Diffusion narrow the quality gap with autoregressive systems while offering substantial speedup.
3 Seed Diffusion
Seed Diffusion Preview combines discrete diffusion training with trajectory selection, on-policy learning, and block-level parallel sampling to balance code-generation quality and inference efficiency. Its training curriculum addresses diffusion-specific token-order and self-correction issues, while system design targets practical latency.
- Model scope: Seed Diffusion Preview is a code-focused dense Transformer using code-only data pipelines and processing methodology from Seed Coder.The initial version omits LongCoT reasoning to establish a strong and efficient performance baseline.
- TSC: A Two-Stage Curriculum for Robust Diffusion Training: The two-stage curriculum uses mask-based corruption for the first 80% of training steps and edit-based augmentation for the final 20%.The edit-based process uses deletions, insertions, and substitutions to control corruption through an approximate signal-to-noise schedule.
- TSC: A Two-Stage Curriculum for Robust Diffusion Training: Edit-based augmentation forces the model to re-evaluate unmasked tokens, mitigating the overconfidence and self-correction problems associated with carrying unmasked tokens directly to the output.The paper links purely mask-based training to a spurious correlation that unmasked tokens are always correct.
- Tailoring the Trajectory Space of Diffusion: Constrained-order training distills candidate generation trajectories and fine-tunes the model on trajectories selected by maximizing Evidence Lower Bound.The procedure is introduced after two-stage diffusion learning to reduce the burden of learning from redundant or misaligned generation orders.
- On-policy Diffusion Learning: On-policy learning optimizes sampled reverse-process trajectories with a model-based verifier, starting from all-mask sequences and ending with generated samples.The verifier ensures that sampling converges to a reasonable or correct sample, and the paper reports speed-up dynamics during training.
- Inference and Infrastructure: Block-level parallel diffusion maintains causal ordering between blocks, uses previously generated blocks as context, and preserves arbitrary block partitioning at inference.KV-caching and infrastructure optimizations support the latency–token-generation-rate trade-off across block sizes.
4 Experiments
The experiments evaluate Seed Diffusion across diverse code-generation, multilingual, competitive-coding, and code-editing benchmarks, comparing it with standard and diffusion language models. Results indicate performance comparable to advanced autoregressive models at substantially higher speeds, with a notable boost on editing tasks.
- Baselines: The evaluation includes Mercury and Gemini-Diffusion as state-of-the-art diffusion-language-model baselines.
- Evaluation scope: Seed Diffusion is evaluated across a diverse suite of code-generation and code-editing benchmarks.The suite includes HumanEval, MBPP, BigCodeBench, LiveCodeBench, MBXP, NaturalCodeBench, Aider, and CanItEdit.
- Benchmark design: BigCodeBench tests real-world programming tasks involving multi-tool use across 1,140 Python tasks, 139 libraries, and seven domains.Its evaluation averages 5.6 test cases and 99% branch coverage per task.
- Benchmark design: LiveCodeBench provides contamination-free slices by evaluating problems released after the model’s training-data cutoff.The evaluation covers stages v1-v6 and the most recent v6 stage.
- Code editing: Aider and CanItEdit assess code-editing capabilities, including editing existing code and following descriptive or ambiguous instructions.Aider contains 133 Exercism coding exercises, while CanItEdit contains 105 hand-crafted problems.
- Results: Seed Diffusion achieves performance comparable to advanced autoregressive models at significantly higher speeds and provides a notable boost on editing tasks.
5 Discussion
The discussion presents Seed Diffusion Preview as an experimental model demonstrating significant inference acceleration, while identifying broader research directions beyond speed. These include alternative modeling orders, scaling properties, and complex reasoning tasks.
- Discussion: Seed Diffusion Preview demonstrates potential for significant inference acceleration in large-scale language models.
- Discussion: Discrete diffusion also motivates alternatives to conventional left-to-right modeling order.The paper frames this as moving away from a pervasive human-centric assumption in machine learning.
- Open directions: Further progress requires community investigation of scaling properties and applications to complex reasoning tasks.
Contributions
The paper lists a large author team affiliated with Tsinghua University and the SIA-Lab of Tsinghua AIR and ByteDance Seed.
- The paper credits Yuxuan Song, Zheng Zhang, Cheng Luo, Pengyang Gao, Fan Xia, and Hao Luo among its authors.
- Additional authors include Zheng Li, Yuehang Yang, Hongli Yu, Xingwei Qu, Yuwei Fu, Jing Su, Ge Zhang, and Wenhao Huang.
- The author list also includes Mingxuan Wang, Lin Yan, Xiaoying Jia, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Yonghui Wu, and Hao Zhou.
- One listed affiliation is the Institute for AI Industry Research at Tsinghua University.
- Another listed affiliation is the SIA-Lab of Tsinghua AIR and ByteDance Seed.