Diffusion Language Models are Super Data Learners

Jinjie Ni; Qian Liu; Longxu Dou; Chao Du; Zili Wang; Hang Yan; Tianyu Pang; Michael Qizhe Shieh

Source-linked AI summary

Diffusion Language Models are Super Data Learners

Jinjie Ni, Qian Liu, Longxu Dou, Chao Du, Zili Wang, Hang Yan, Tianyu Pang, Michael Qizhe Shieh

arXiv:2511.03276v1Nov 5, 2025cs.LG

TL;DR

The paper studies which language-modeling paradigm extracts more signal when unique data is scarce. Under matched pre-training conditions, it compares diffusion and autoregressive models and finds a crossover: DLMs win in data-bound regimes, with the trade-off of greater compute and inference complexity.

Problem
The central question is which modeling paradigm extracts more signal per unique token as high-quality data becomes the primary bottleneck while compute continues to scale.
Method
The paper compares diffusion and autoregressive language models under a common pre-training regime while varying unique data, data quality, model scale, architecture sparsity, and training repetition.
Results
DLMs consistently surpass equally sized AR models when unique data is limited, with the crossover shifting later for more or better data, earlier for larger models, and persisting across dense and sparse architectures.
Takeaways & Limitations
DLMs are compelling for data-bound scaling because they trade greater compute for higher data potential per unit of unique data.
Takeaways & Limitations
DLMs require more training and inference FLOPs, and practical comparisons remain sensitive to under-explored inference choices and non-comparable perplexity objectives.

Abstract

from arXiv · show

Under strictly controlled pre-training settings, we observe a Crossover: when unique data is limited, diffusion language models (DLMs) consistently surpass autoregressive (AR) models by training for more epochs. The crossover shifts later with more or higher-quality data, earlier with larger models, and persists across dense and sparse architectures. We attribute the gains to three compounding factors: (1) any-order modeling, (2) super-dense compute from iterative bidirectional denoising, and (3) built-in Monte Carlo augmentation; input or parameter noise improves AR under data constraint but cannot close the gap. At scale, a 1.7B DLM trained with a ~1.5T-token compute budget on 10B unique Python tokens overtakes an AR coder trained with strictly matched settings. In addition, a 1B-parameter DLM achieves > 56% accuracy on HellaSwag and > 33% on MMLU using only 1B tokens, without any special tricks, just by repeating standard pre-training data. We also show that rising validation cross-entropy does not imply degraded downstream performance in this regime.

1. Introduction

The paper asks which modeling paradigm extracts more signal per unique token when data, rather than compute, is scarce. It finds that diffusion language models outperform matched autoregressive models in data-bound regimes, with gains attributed to several modeling and compute factors.

1. Introduction: DLMs consistently surpass equally sized AR models when total training tokens are fixed but unique data is limited.The crossover shifts later with more unique or higher-quality data, earlier with larger models, and persists across dense and sparse architectures.
1. Introduction: Three factors jointly drive the gains: any-order modeling, super-dense iterative computation, and built-in noisy augmentation.Input or parameter noise helps AR models under data scarcity but does not close the gap.
1. Introduction: A 1B DLM trained for 480 epochs on 1B tokens reaches approximately 56% HellaSwag and 33% MMLU without clear saturation.The paper reports that DLMs overfit significantly later than AR models in this small-data regime.
1. Introduction: The paper argues that rising validation cross-entropy does not necessarily indicate degraded downstream performance in this regime.The introduction presents this as a diagnostics caveat alongside the data-efficiency findings.
1. Introduction: A 1.7B diffusion coder trained on 10B unique Python tokens with a 1.5T-token compute budget reaches parity with state-of-the-art AR code models trained on trillions of unique tokens.The comparison uses matched 1.7B-parameter AR and diffusion models trained for approximately 150 epochs.

2. Preliminaries

Autoregressive models factorize sequences left to right and train with next-token prediction, whereas masked diffusion models corrupt and iteratively denoise sequences using bidirectional, any-order conditioning.

2.1. Autoregressive Language Models: AR models parameterize sequence likelihood through a causal chain, conditioning each token on its preceding prefix.Causal self-attention restricts each position to visible prefix context while preserving left-to-right generation.
2.1. Autoregressive Language Models: Teacher forcing predicts every next token in parallel during training, while inference generates sequentially with KV-caching.This yields exact normalized likelihoods and efficient streaming generation.
2.2. Masked Diffusion Language Models: Bidirectional attention lets diffusion models perform any-order modeling, modify context during generation, and support multi-token generation with more parallelizable compute.These properties are presented as advantages for tasks involving non-causal dependencies and iterative reasoning.
2.2. Masked Diffusion Language Models: Masked diffusion independently masks tokens according to a noise schedule, then reverses the process by progressively revealing masked positions.The expected unmasked fraction at noise level t is α_t; reverse steps preserve revealed tokens and sample predictions for masked ones.
2.2. Masked Diffusion Language Models: A time-agnostic masked-diffusion property makes the clean-token conditional depend only on visible unmasked context, not directly on the noise level.This permits parameterizing the denoiser without an explicit time embedding.
2.2. Masked Diffusion Language Models: Masked diffusion trains by minimizing a variational objective whose schedule-dependent weight compensates for differing numbers of masked positions across noise levels.For the linear schedule α_t = 1 − t, the weight reduces to 1/t.

3. The Intelligence Crossover

Under fixed compute and controlled settings, diffusion language models outperform autoregressive models when unique data is scarce, with the crossover shaped by data quality, model size, sparsity, and augmentation. The results identify any-order modeling, super-dense computation, and noisy augmentation as contributors to this data-efficiency advantage.

3.2. Data Budget Decides the Crossover Timing: More than 3× effective data efficiency lets a DLM trained on 0.5B unique tokens match an AR model trained on 1.5B unique tokens.Both models use the fixed 96B-token training budget in the data-budget comparison.
3.2. Data Budget Decides the Crossover Timing: Diffusion models consistently surpass AR models at low unique-data budgets, while increasing unique data shifts the crossover later or beyond the observed range.AR models retain stronger end-of-training performance when data is abundant and compute is limiting.
3.3. Varying Data Quality: Higher-quality data benefits both paradigms but shifts the crossover slightly later, while validation loss can disagree with benchmark performance.The medium-quality run has the lowest validation loss, showing that cross-entropy is not always a reliable comparator here.
3.4. How Much Does Model Size Impact the Data-Constrained Training?: Larger models make the crossover arrive earlier, and even the smallest diffusion model outperforms AR models across the tested model sizes.AR models saturate or overfit scarce data, whereas diffusion models continue improving within the tested token window.
3.5. Sparse, Dense, Super-Density: Diffusion models surpass AR models across dense and sparse architectures, with crossover timing ordered as 8B dense < 8B1A MoE < 1B dense.The sparse comparisons separately expose parameter-matching and FLOPs-matching trade-offs.
3.6. Is Noise Deciding the Game?: Input masking and parameter noise improve AR models under data constraints but do not eliminate diffusion’s advantage.Input masking helps at modest noise levels and collapses when inputs are overly corrupted; dropout produces a similar incomplete recovery.

4. Scaling Crossovers to Trillion-Level Total Tokens

At trillion-token compute scale, DLMs still show downstream crossovers over AR models on code-generation benchmarks under limited unique data. The crossover timing varies by evaluation protocol, while the DLM remains unconverged at the end of training.

Scaling Crossovers: Early downstream crossovers appear for 1.7B DLMs trained on 10B unique code tokens for approximately 150 epochs, with DLMs surpassing AR models.The result extends crossover evidence to larger-scale training and coding benchmarks.
Training Dynamics: The DLM had not converged by the end of the 1.5T-token training cycle, indicating additional training potential.
Evaluation Protocols: Crossover timing differs across generative benchmarks, occurring near the end of annealing for HumanEval and HumanEval+ but earlier on MBPP and MBPP+.The paper links this discrepancy to evaluation protocols and calls for studies separating training dynamics from evaluation artifacts.
Generative Tasks: The crossover extends robustly to generative tasks with larger unique-token budgets, supporting its broader relevance beyond standard evaluations.The paper frames code generation as a constrained setting where the 10B-token budget is practically meaningful for some programming languages.

5. High Validation Loss ≠Degraded Intelligence

Rising validation loss under repeated training does not necessarily signal worse downstream intelligence. The paper explains this through the distinction between absolute NLL and relative discrimination among answer options.

Downstream Performance: Increasing validation loss can coexist with continued downstream improvement, so apparent pre-training overfitting does not necessarily reduce evaluation performance.This pattern is shown for models whose downstream performance keeps improving through the end of training.
Mechanism: Multiple-choice accuracy depends on relative NLL differences between correct and alternative options, whereas validation loss measures absolute cross-entropy.Thus, higher absolute NLL can accompany stronger separation between ground-truth and competing options.
Mechanism: Repeated exposure may increase confidence on certain text segments and inflate NLL for incorrect predictions, while relative NLL separation continues to grow.The paper proposes that this growing separation reflects improved discriminative power.
Generative Evaluations: The same reasoning is hypothesized to extend to generative evaluations because token-level decisions may be less affected by overconfidence on non-essential tokens.

6. Diffusion Language Models also Overfit the Data

DLMs can eventually overfit repeated data, but they extract substantial downstream signal from extremely small corpora before saturation. More unique data delays overfitting, whereas larger models reach it earlier.

Fixed-Data Scaling: 56% HellaSwag accuracy and 33% MMLU accuracy are achieved by a 1B-parameter DLM trained on the same 1B-token dataset for 480 epochs.These results exceed the reported AR accuracies of 41% and 29%, respectively, and performance had not saturated under this repetition.
Overfitting: DLMs eventually overfit when unique data is sufficiently small and training continues for enough epochs.The paper examines training runs extending to 1000 epochs.
Overfitting: The onset of overfitting is delayed by larger unique-data budgets and advanced by larger model sizes.

7. Discussions

The discussion attributes DLMs’ data potential to any-order modeling, greater training and inference computation, and richer corruption-based augmentation. These benefits trade compute efficiency for data reuse and introduce deployment, evaluation, and safety constraints.

Reduced Inductive Bias via Any-Order Modeling: Any-order modeling removes AR’s strict causal inductive bias and lets DLMs use bidirectional attention to fit diverse textual patterns.The paper argues this can squeeze more value from each data point, including non-causal data types such as code and biological sequences.
Super-Density: DLMs use substantially more training and inference FLOPs through temporal refinement and iterative prediction, enabling greater compute per data point.Training experiments indicate more than 100× the FLOPs of AR models may be required to reach optimal performance.
Super-Density: At inference, parallelizable diffusion computation can keep the speed gap acceptable before GPU compute becomes the bottleneck, despite higher total computation.AR generation of N tokens is described as roughly equivalent in cost to one diffusion sampling step.
Monte Carlo Sampling: The diffusion objective averages over masking configurations, turning each unique example into many noisy variants and making data repetition intrinsic to training.The paper contrasts this with implementations that corrupt each example only once because of computational limitations.
Monte Carlo Sampling: For an input of length L, masked DLMs are described as expanding the learning space to 2^L corrupted sequences, compared with L causal sequences for AR.The paper notes that not every corruption is substantially different and that larger models may be needed to fit the expanded space.
Limitations: DLMs trade data potential for higher energy, time, memory, inference-tuning, contamination, and deployment costs than mature AR systems.The discussion also notes that diffusion perplexity is not directly comparable to normalized AR likelihood and that benchmark gains may not transfer uniformly to streaming or long-horizon use.

8. Related Work

Prior work establishes diffusion language models as competitive with autoregressive models, including emerging hybrid and commercial systems. A parallel literature addresses scarce unique data through repeated training, data mixing, and higher-utility token construction.

Large-scale diffusion language models have achieved performance competitive with leading open-source autoregressive models.
Commercial diffusion language models report strong coding and math capabilities alongside significantly lower generation latency.
Hybrid approaches bridge autoregressive and diffusion modeling, including block-wise diffusion that reduces to autoregressive modeling at block size 1.
Data-constrained scaling studies find that repeating data for up to approximately 4 epochs incurs little loss penalty at fixed compute, followed by sharply diminishing returns.
Other strategies improve token utility through web paraphrasing, multilingual rephrasing, targeted math and code rewriting, and large-scale agentic data synthesis.