Source-linked AI summary
Sumi: Open Uniform Diffusion Language Model from Scratch
Mengyu Ye, Keito Kudo, Wataru Ikeda, Ryosuke Matsuda, Keisuke Sakaguchi, Jun Suzuki
TL;DR
Large-scale, scratch-pretrained uniform diffusion language models remain unexplored. Sumi addresses this gap with a fully open 7B model trained on 1.5T tokens, performing competitively with comparable-token autoregressive models on knowledge, reasoning, and coding while underperforming on commonsense benchmarks.
Problem
Uniform diffusion language models lack a scratch-pretrained model at both large parameter and token scales, leaving their data-rich behavior unexplored.
Method
Sumi is a fully open 7B-parameter uniform diffusion language model pretrained from scratch on 1.5T tokens using the GIDD objective.
Results
Sumi performs competitively with comparable-token autoregressive models on knowledge, reasoning, and coding benchmarks but underperforms on commonsense benchmarks.
Takeaways & Limitations
The release provides an open reference for studying uniform diffusion at scale, while inference probes identify canvas-length sensitivity and modest coding parallelism.
Takeaways & Limitations
Inference-time analyses are directional rather than conclusive because they use small samples, a rough fluency proxy, and lack matched model comparisons.
Abstract
from arXiv · showhide
Diffusion models have become a promising alternative to autoregressive models. Among these, uniform diffusion language models (UDLMs) permit any token to be updated at any step, in principle enabling more flexible generation. However, no UDLM has yet been pretrained from scratch at both large parameter scale and large token budget. Both autoregressive modeling and masked diffusion modeling already have capable models at scale that the community can study and build on; uniform diffusion has none. A scratch-pretrained UDLM at scale would provide a clean reference point for studying scaling behavior, generation dynamics, controllability, and trade-offs against established autoregressive and masked diffusion models. To this end, we introduce Sumi ("ink" in Japanese), a fully open 7B uniform diffusion language model pretrained from scratch on 1.5T tokens. Sumi performs competitively with autoregressive models trained at comparable token budgets on knowledge, reasoning, and coding benchmarks, while under-performing on commonsense benchmarks, where our education-heavy data mixture is a likely contributor. We release our model weights, checkpoints, and full training recipe, including a complete specification of the data mixture over publicly available corpora. We hope this release enables the community to study native uniform diffusion at scale and catalyzes work on its as-yet poorly understood aspects.
1. Introduction
The introduction identifies a gap in large-scale from-scratch uniform diffusion pretraining and presents Sumi, a fully open 7B UDLM trained on 1.5T tokens. It also frames the release as a reference for studying native uniform diffusion and reports exploratory evidence that generation is strongest within the trained canvas range.
- Motivation: Uniform diffusion permits any token to be updated, unlike masked diffusion, where filled tokens cannot be revised.Masked diffusion models have nevertheless reached 8B parameters and over 2T training tokens with performance competitive with strong autoregressive baselines.
- Research gap: No UDLM had been pretrained from scratch at both large parameter and token scales.Existing large models were compute-optimal checkpoints trained on comparatively small token budgets, while the only data-rich model had 1.7B parameters.
- Contribution: 7B-parameter Sumi is a fully open UDLM pretrained from scratch on 1.5T tokens.Sumi builds on the GIDD framework and its improved SNR-based formulation.
- Inference probes: Exploratory inference probes indicate that Sumi generates fluently within its trained canvas range and degrades outside it.The probes used 30 questions per task and are directional rather than conclusive; degradation is sharpest at short canvases across all four tasks.
- Release motivation: The release is intended to enable study of native uniform diffusion at scale and catalyze work on its poorly understood aspects.The paper positions Sumi as an open resource for investigating uniform diffusion models trained natively from scratch.
2. Training
Sumi is a 7B-parameter time-agnostic bidirectional Transformer trained with the GIDD objective under pure uniform noise. Training comprises approximately 1.3T-token pre-training followed by 250B-token domain-specific mid-training on 288 NVIDIA H100 GPUs.
- Architecture: Sumi uses a 7B-parameter time-agnostic bidirectional Transformer with the SNR-reparameterized GIDD objective and log-SNR restricted to λ∈[−9, 9].
- Architecture: The model has 36 LLaMA-style layers, hidden size 4096, SwiGLU FFN size 12288, grouped-query attention, RMSNorm, untied embeddings, and no biases or dropout.Attention uses 32 heads, 8 KV groups, and head dimension 128; RMSNorm uses 𝜖=10−5.
- Hardware and compute: 43,308 GPU-hours were used in total: 35,776 for pre-training and 7,531 for two mid-training stages on 288 NVIDIA H100 GPUs.
- Training data: The approximately 250B-token mid-training mixture comprised coding 81.4B (32.5%), math 74.3B (29.7%), general 52.4B (21.0%), and reasoning 42.0B (16.8%).Pre-training used English llm-jp-corpus-v4 subsets, StarCoder code, Python code, and en_fineweb-rescored data; all data came from publicly available corpora.
3. Evaluation
Sumi is evaluated across 13 benchmarks using diffusion-based scoring, with a fixed 2048-token canvas and task-specific generation lengths. Under a shared protocol, it leads on general knowledge and coding, remains competitive on reasoning and mathematics, but is among the weakest on commonsense benchmarks.
- Evaluation protocol: The evaluation uses a 2048-token canvas, with generation lengths of 512 for BBH, 64 for GSM8K, and 256 for HumanEval and MBPP.For likelihood tasks, the remaining canvas positions are filled with random tokens; diffusion-based scoring is implemented in the lm-evaluation-harness.
- Evaluation coverage: Sumi is evaluated on 13 benchmarks spanning general knowledge, reasoning and mathematics, coding, and commonsense.The benchmarks include MMLU, RACE, TruthfulQA, ARC-Easy, ARC-Challenge, GPQA, BIG-Bench Hard, GSM8K, HumanEval, MBPP, WinoGrande, PIQA, and HellaSwag.
- Baselines: Sumi is compared under the same protocol with Falcon-7B, Llama 2-7B, and OLMo-7B, while LLaDA-8B and Llama 3-8B provide contextual reference scores only.Sumi is trained on 1.5T tokens, and the primary comparison models have similar parameter counts and comparable token budgets.
- Results: Sumi achieves the best scores among protocol-evaluated models on general knowledge and coding, and is competitive with Llama 2-7B while mostly ahead of Falcon-7B on reasoning and mathematics.The paper attributes the knowledge and coding results partly to the educational and code-heavy data mixture, while noting comparatively limited relevant reasoning data in mid-training.
- Results: Sumi is among the weakest evaluated models on commonsense benchmarks, with its education-heavy data mixture identified as a likely contributor to the gap.The authors caution that data composition alone cannot fully explain the gap.
4. Discussion
Exploratory probes suggest that Sumi is fluent only within a task-dependent canvas-length band, while confidence sampling structures its order-free generation and enables limited coding parallelism. Additional denoising does not produce meaningful self-correction, and these findings are directional rather than controlled claims.
- Scope and limitations: The analyses use 30 randomly sampled questions from each of GSM8K, HumanEval, MBPP, and BBH-Logic3 and are reported as exploratory probes rather than controlled claims.BBH-Logic3 was selected because its answer format extracts cleanly and Sumi’s score there is 74.8.
- Canvas length: Sumi’s fluency depends on both canvas length and task, with GSM8K degrading sharply outside the training range while the other tasks degrade more gently.At long canvases, the other three tasks remain stable or improve out to roughly 2.5 times the longest trained sequence length.
- Adaptive sampling: Confidence sampling induces self-organized commit order, unlike ancestral sampling, which remains essentially unstructured.The sampler selects positions with large potential improvement, producing task-shaped structure in the otherwise order-agnostic generation process.
- Parallel decoding: Outside GSM8K, accuracy is largely preserved through k = 4 tokens per step, whereas GSM8K loses accuracy already at k = 2.HumanEval and MBPP stay within one to two samples of the single-token baseline through k = 4; MBPP drops sharply only at k = 8, while BBH-Logic3’s apparent k = 4 peak is likely sampling noise.
- Revision and self-correction: 58% to 100% of revision steps overwrite committed tokens, yet at most 1% of final tokens change, extracted answers almost never flip, and accuracy is unchanged.The overwrites are predominantly A→B→A round trips rather than directed edits.
5. Conclusion and Future Work
Sumi is presented as the first natively trained UDLM at both large parameter and token scale, performing competitively on knowledge, reasoning, and coding benchmarks but underperforming on commonsense benchmarks. Inference probes reveal task-dependent generation behavior, modest coding parallelism, order-sensitive arithmetic, and no effective self-correction, motivating instruction tuning and controlled comparisons.
- Contributions: Sumi is a fully open 7B UDLM pretrained from scratch on 1.5T tokens, described as the first UDLM trained natively at both large parameter and token scale.The model is released as an open research reference for studying uniform diffusion at scale.
- Contributions: Sumi performs competitively with autoregressive models at comparable token budgets on general knowledge, reasoning, and coding benchmarks, but underperforms on commonsense benchmarks.The education- and code-heavy data mixture is identified as a likely contributor to the commonsense gap.
- Inference observations: Generation fluency is sensitive to canvas length within a task-dependent band, while confidence sampling induces a self-organized commitment order within that band.A canvas length of 2048 lies within the identified band for all evaluated tasks and is used throughout evaluation.
- Inference observations: Confidence sampling enables modest parallel decoding on coding tasks, whereas multi-step arithmetic remains order-sensitive.These findings expose task-dependent differences in how uniform diffusion supports parallel generation.
- Limitations and future work: An explicit revision budget does not produce self-correction: extra denoising overwrites committed tokens, with edits predominantly following A→B→A round trips.The result leaves open whether a revision setup designed to target likely errors could recover self-correction.
- Limitations and future work: Future work includes releasing an instruction-tuned Sumi variant and conducting controlled comparisons under a matched evaluation protocol.The authors state that such comparisons are needed to determine which observed generation behaviors are intrinsic to uniform diffusion.
Limitations & Risks
The inference-time analyses are directional rather than conclusive, and they do not establish whether observed behaviors arise from uniform diffusion or Sumi specifically. As an unaligned pretrained base model, Sumi also retains risks including harmful or biased text and confident false information.
- Analysis limitations: Inference analyses use only 30 sampled questions per task and Falcon-7B perplexity as a rough fluency proxy, limiting their conclusiveness.The study does not run matched comparisons against masked diffusion and autoregressive models.
- Analysis limitations: The analyses cannot attribute observed behaviors to uniform diffusion itself rather than to Sumi specifically.Matched comparisons against masked diffusion and autoregressive models would be needed for that attribution.
- Analysis limitations: Sumi’s commonsense gap is larger than its education- and code-heavy data mixture can explain on its own.
- Safety risks: As a pretrained base model without instruction tuning, alignment, or safety filtering, Sumi can produce harmful, offensive, sensitive, or biased text.Such outputs may be elicited by adversarial or careless prompts or arise unprompted from biases in the pretraining corpus.
- Safety risks: Sumi has no mechanism for ensuring factual accuracy and may state false information with apparent confidence.