PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion Beats Autoregressive in Data-Constrained Settings

Mihir Prabhudesai, Mengning Wu, Amir Zadeh, Katerina Fragkiadaki, Deepak Pathak

arXiv:2507.15857v7cs.LGcs.AIcs.CVcs.RO

TL;DR

The paper asks whether autoregressive models remain preferable when high-quality data is limited and training repeatedly reuses a small dataset. It systematically studies masked diffusion models in this regime and develops scaling laws for their behavior. Diffusion models outperform autoregressive models when compute is sufficient but data is scarce, motivating their use in data-constrained settings.

  • Problem

    Comparisons based on single-epoch training leave it unclear whether diffusion models require more compute or simply more data, especially as high-quality data becomes scarce.

  • Method

    The paper trains hundreds of masked diffusion and autoregressive models across model sizes, data quantities, and training epochs to fit data-constrained scaling laws.

  • Results

    Diffusion models consistently outperform autoregressive models in data-constrained regimes, benefiting from repeated data for up to 100 epochs and yielding a closed-form critical-compute threshold.

  • Takeaways & Limitations

    When data is constrained but compute is available, diffusion models offer a compelling alternative to autoregressive models.

  • Takeaways & Limitations

    The scaling laws are fit over a limited range of unique data sizes, and hybrid models interpolating between the paradigms require further study.

Abstract

from arXiv · show

Autoregressive (AR) models have long dominated the landscape of large language models, driving progress across a wide range of tasks. Recently, diffusion-based language models have emerged as a promising alternative, though their advantages over AR models remain underexplored. In this paper, we systematically study masked diffusion models in data-constrained settings where training involves repeated passes over limited data and find that they significantly outperform AR models when compute is abundant but data is scarce. Diffusion models make better use of repeated data, achieving lower validation loss and superior downstream performance. We find new scaling laws for diffusion models and derive a closed-form expression for the critical compute threshold at which diffusion begins to outperform AR. Finally, we explain why diffusion models excel in this regime: their randomized masking objective implicitly trains over a rich distribution of token orderings, acting as an implicit data augmentation that AR's fixed left-to-right factorization lacks. Our results suggest that when data, not compute, is the bottleneck, diffusion models offer a compelling alternative to the standard AR paradigm. Our code is available at: https://diffusion-scaling.github.io.

1 Introduction

The paper studies masked diffusion models under repeated training on limited data, finding that they can surpass autoregressive models when compute is sufficient and data is scarce. It develops scaling laws and attributes this advantage to exposure to diverse token orderings.

  • Study setting: Masked diffusion models substantially outperform autoregressive models when training repeatedly reuses limited data.The study spans multiple data scales and compute budgets.
  • Compute regime: At low compute, autoregressive models initially lead, but diffusion models improve beyond a critical compute threshold and ultimately achieve better performance.This pattern holds across a wide range of unique token budgets.
  • Repeated data: Diffusion models can train on repeated data for up to 100 epochs, whereas prior work found autoregressive models benefited from repetition for only up to 4 epochs.For diffusion models, repeated data was almost as effective as fresh data.
  • Repeated data: Diffusion models have an effective epoch count of approximately 500, indicating limited degradation from repeated data across many epochs.This effective epoch count is reported as D ≈500.
  • Scaling laws: The critical compute required for diffusion to outperform autoregressive models follows a power law with the number of unique tokens.The paper derives a closed-form expression predicting when diffusion becomes favorable for a given dataset size.
  • Downstream performance: The best diffusion model consistently outperforms the best autoregressive model on a range of downstream language tasks in data-constrained settings.The advantage extends beyond validation loss to downstream performance.
  • Mechanism: Randomized masking exposes diffusion models to diverse token orderings, functioning as implicit data augmentation beyond autoregressive left-to-right factorization.Explicit token-ordering augmentations to autoregressive training support this explanation.
  • Implication: These findings indicate that diffusion models make better use of repeated data, achieving lower validation loss and better downstream generalization.The paper presents this as a compelling alternative when data, rather than compute, is the primary bottleneck.

2 Method

The method compares masked diffusion and autoregressive models under repeated-data training while holding architecture and most training variables constant. It combines controlled experiments with an effective-data scaling framework to study data scarcity and diminishing returns from reuse.

  • Controlled comparison: The study compares masked diffusion and autoregressive models in data-constrained settings while fixing the core architecture and data pipeline.Both families share a GPT-2-style Transformer backbone with rotary positional embeddings, and other training variables are held constant.
  • Autoregressive objective: Autoregressive models predict each token from a growing prefix using causal attention and next-token prediction on clean sequences.The causal mask prevents attention to future positions, enforcing a left-to-right factorization.
  • Masked diffusion objective: Masked diffusion models randomly mask tokens, then predict the masked originals using full bidirectional attention over the corrupted sequence.The masking ratio is sampled from U(0, 1), and the masked positions are denoised from both past and future unmasked tokens.
  • Masked diffusion objective: Resampling mask patterns implicitly trains diffusion models on a broad collection of token-ordering tasks rather than one fixed factorization.This randomized objective provides varied conditioning contexts, while the absence of a causal mask permits bidirectional attention.
  • Data-constrained scaling: The repeated-data scaling framework replaces raw training tokens with effective quantities that capture diminishing utility across repeated epochs.It defines D = U · E and models the kth exposure's utility as (1 − δ)^(k−1), producing an empirically fitted effective data size D′.
  • Data-constrained scaling: As repeated-data exposure grows without bound, effective data approaches a finite limit, so additional repetition eventually provides no additional effective data.The framework describes this as a natural saturation point in returns from repeated training data.
  • Data-constrained scaling: The modified Chinchilla-style loss uses effective parameter and data quantities to model validation loss and guide apples-to-apples comparisons across compute, data, and model scale.The formulation is intended for regimes where data is reused multiple times and captures loss behavior under data scarcity.
  • Experimental setup: Experiments use English C4 with GPT-2 BPE sequences of 2048 tokens, unique-token budgets of 25, 50, and 100M, up to 800 epochs, and models from 7M to 2.5B parameters.Model width and depth are increased proportionally following a Chinchilla-style scaling strategy.

3 Experiments

Experiments show that diffusion models become advantageous when compute is abundant and data is reused, overcoming AR models’ early compute efficiency through greater robustness to repetition. Scaling-law analyses identify a compute crossover, while downstream evaluations and ordering experiments support diffusion’s broader sample efficiency.

  • 3.2 Fitting Data-Constrained Scaling Laws: Diffusion’s repeated-data value decays more slowly than AR’s across compute budgets, with nearly unchanged validation curves as repetition increases.AR validation loss rises and diverges under repetition, while diffusion curves overlap and show little overfitting.
  • 3.2 Fitting Data-Constrained Scaling Laws: Repeated data remains nearly as effective as fresh data for approximately 100 diffusion epochs, compared with only about 4 AR epochs.This difference quantifies diffusion’s greater robustness to data reuse under compute-optimal extrapolation.
  • 3.1 Does Diffusion Beat AR in Data-Constrained Settings?: Diffusion models initially trail AR at low compute but surpass them beyond a critical compute threshold, especially with limited, repeated data.The heatmap shows low-compute regions favoring AR and higher-compute, data-limited regions favoring diffusion.
  • 3.3 When to Use Diffusion over AR?: The critical compute frontier follows Ccrit(U) ∝ U^2.174, with empirical crossover points aligning closely with the fitted prediction.The fitted closed form is Ccrit(U) = 2.12 × 10^1.956 · U^2.174.
  • 3.4 Downstream Results: Across diverse downstream tasks and data scales, diffusion models consistently outperform AR counterparts, including comparisons with flop-matched overfitted AR models.The evaluation includes best-validation AR models and AR models trained for the same epoch counts as diffusion models.
  • 3.5 Why do Diffusion models outperform AR models in data-constrained settings?: AR models augmented with fixed random token-order permutations test whether task diversity accounts for diffusion’s low-data advantage.The results support diffusion’s advantage arising from exposure to a richer distribution of conditional prediction tasks; controlling task diversity may enable hybrid designs.

4 Conclusion

The paper argues that masked diffusion models consistently outperform autoregressive models when training repeatedly reuses limited data. It presents diffusion as a compelling alternative when data, rather than compute, is the primary bottleneck.

  • Diffusion models consistently outperform autoregressive models in data-constrained regimes involving repeated passes over limited data.
  • Diffusion models extract value from repeated data far beyond what autoregressive models can achieve.
  • Practitioners should use autoregressive models when compute is constrained and diffusion models when data is constrained.

6 Related Work

Related work situates this paper within the scaling of language models and the emergence of diffusion-based alternatives to autoregressive modeling. It distinguishes continuous and discrete diffusion approaches for text.

  • Deep Learning in Data-Constrainted Settings: Deep learning progress has been driven by scaling compute and data, while analyses suggest a future data bottleneck could inhibit continued advancement.
  • Deep Learning in Data-Constrainted Settings: Autoregressive models dominate language modeling and are typically trained for a single epoch to maximize exposure to unique tokens.
  • Diffusion-Based Language Models: Diffusion language models offer a different paradigm and divide broadly into continuous and discrete approaches.
  • Diffusion-Based Language Models: Continuous approaches inject Gaussian noise, whereas discrete methods corrupt tokens using noise sampled from distributions such as Bernoulli.

7 Additional Results

Additional results examine augmentation, compute-loss tradeoffs, and downstream evaluation across data-constrained settings. They also document a large-scale diffusion run that was stopped because of compute limitations.

  • Attention dropout and token masking fail to improve autoregressive validation loss, while autoregressive models continue to overfit and lag behind longer-trained diffusion models.Autoregressive baselines use 140M parameters and 50 epochs; the best diffusion model uses 500 epochs.
  • Table 4 reports downstream negative log-likelihood, where lower is better, across four diverse corpora.The corpora are OSCAR, TinyStories, WikiText, and IndustryCorpus2 EN Sub.
  • Downstream results include best-performing autoregressive and diffusion models trained with 500M unique tokens, alongside a random baseline.
  • Figure 9 compares validation loss against training FLOPs for autoregressive and masked diffusion models across 25M, 50M, and 100M unique-token settings.

8 Hyperparameter details

The experiments use AdamW with specified optimization, scheduling, regularization, batching, and gradient-clipping settings.

  • Training uses batches of 256 sequences with AdamW optimization.
  • The learning rate peaks at 2e-4, has a minimum of 2e-5, uses 1% warm-up, and follows cosine decay.
  • Optimization uses β1=0.9, β2=0.95, ϵ=10−8, weight decay 0.1, and gradient clipping at 1.0.

9 Model Architecture

The models use a transformer-based implementation with RoPE, SwiGLU, RMSNorm, and no bias terms. The revised parameter-count formula accounts for these architectural changes and the SwiGLU hidden dimension.

  • The implementation builds on Megatron-DeepSpeed and modifies a general transformer architecture for masked diffusion training.The supplied passage does not specify the individual architectural modifications beyond describing the framework and transformer basis.
  • RoPE replaces absolute positional embeddings, while SwiGLU replaces the standard MLP activation and RMSNorm replaces LayerNorm.These changes are described as improving extrapolation, convergence, downstream performance, or training stability, respectively.
  • The feed-forward hidden size hf is rounded so that it remains divisible by 64 while closely matching the effective SwiGLU dimensionality.The rounding preserves the original MLP capacity while aligning parameter sizes with hardware constraints.
  • The revised parameter formula is P = 4lh2 + 3lh · hf + 6lh + V h.It reflects attention, SwiGLU MLP, normalization terms, and token embeddings under the revised architecture.
  • Table 5 lists the model configurations used in the experiments together with their parameter counts.

10 Discussion

The discussion frames masked diffusion and AR models as complementary extremes: diffusion is more data-efficient, whereas AR is more compute-efficient. It motivates hybrids and broader scaling-law evaluation as open directions.

  • Discussion: AR models specialize in one left-to-right prediction task, whereas diffusion models generalize across many random token orderings.The discussion identifies order specialization as one hypothesized contributor to AR's compute efficiency.
  • Discussion: The experimental setup includes predefined token orderings, beginning with raster order and generating further permutations from Gaussian perturbations.

11 Limitations

The paper characterizes masked diffusion and AR models as complementary extremes in the data–compute trade-off and notes limits to the current scaling-law analysis.

  • 11 Limitations: Masked diffusion models are presented as more data-efficient, while AR models are presented as more compute-efficient.
  • 11 Limitations: Hybrid models could provide a better balance between the two modeling extremes, but require more detailed evaluation through the data–compute-efficiency lens.
  • 11 Limitations: The scaling laws are fit over a limited range of unique data sizes, and extending them could improve predictive accuracy.

12 Order Permutation Details

The order-permutation procedure trains AR models with sampled alternatives to left-to-right order while preserving a first-token positional anchor. The broader evaluation materials also describe reproducibility and reporting considerations.

  • 12 Order Permutation Details: The experiment trains AR models with different token orderings and evaluates them using left-to-right ordering.
  • 12 Order Permutation Details: The ordering list starts with standard left-to-right order, then adds Gaussian noise to position IDs with increasing standard deviation to create permutations.
  • 12 Order Permutation Details: Each sequence randomly samples one predefined permutation during training.
  • 12 Order Permutation Details: The first token remains fixed at position 0, providing a soft absolute positional anchor under RoPE.
  • 12 Order Permutation Details: For a sequence with T predicted tokens, only indices 1:T are shuffled while the first token is retained.
  • 12 Order Permutation Details: Algorithm 2 returns shuffled tokens and position IDs after selecting an order independently for each batch sequence.
  • Evaluation: The paper reports that its abstract and introduction accurately reflect the comparison, sample-efficiency claims, and scaling-law contributions.
  • Evaluation: The paper reports limitations involving restricted model-size and dataset ranges for scaling-law fitting.
Loading 2507.15857v7…