PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting the 16x Pretraining Compute Gap between Diffusion and Autoregressive Language Models

Guanghan Wang, Bing He, Zhan Shi, Benoit Dumoulin, Hanqing Lu, Volodymyr Kuleshov

arXiv:2610.17590v1cs.LGcs.AI

TL;DR

A widely cited estimate put the training compute needed for diffusion language models at roughly 16× that of autoregressive models, based on evaluation and model choices that differ from current practice. This paper revises the scaling analysis using exact likelihood, block diffusion, and encoder–decoder architectures, finding much smaller reported compute multiples as block size decreases.

  • Problem

    A widely cited study estimated that masked diffusion models need about 16× the training FLOPs of autoregressive models to reach the same validation perplexity, using evaluation and architecture choices that differ from current practice.

    Source: §1 · p. 2

  • Method

    The study revisits scaling laws with exact likelihood under top-1 confidence decoding, block diffusion, encoder–decoder models, and a sweep of practical block sizes.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    12.8× with exact likelihood, 7.8× with encoder–decoder block diffusion at b=32, 4.6× at b=8, and 3.0× at b=4 are the reported compute multiples.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The revised analysis finds that pretraining diffusion language models is far cheaper than the widely cited 16× estimate suggests.

    Source: §1 · p. 2

    All experiments use a single fixed random seed.

    Source: §6 · p. 9

Abstract

from arXiv · show

Diffusion language models (DLMs) have attracted wide attention for their fast, parallel decoding, but they are believed to be far less efficient to train than autoregressive language models (ARMs): a widely cited study reports that a DLM needs roughly 16x the training compute of an ARM to match its likelihood, a figure that has made pretraining DLMs from scratch look prohibitive at scale. We find that the methodology behind this figure has fallen out of step with current DLM practice: it scores the model with its evidence lower bound (ELBO) and trains a full-sequence, decoder-only model. We therefore revisit the scaling laws of DLMs, adopting the exact likelihood under the model's own decoding order, block diffusion training, an encoder-decoder architecture, and a sweep over the block sizes used in practice. Under this new protocol, we study across ten model variants, about 1,700 models in total, and find that pretraining DLMs is far cheaper than previously thought. Concretely, scoring with the exact likelihood reduces the compute multiple to 12.8x; block diffusion training and an encoder-decoder architecture further reduce it to 7.8x at block size 32, and smaller block sizes narrow it to 4.6x at block size 8 and 3.0x at block size 4. We complement these scaling analyses with an empirical rule for the optimal decoder depth of encoder-decoder block diffusion, a held-out compute budget beyond the fitting range at which the gap between variants persists, and downstream benchmark results where likelihood improvements carry over to accuracy. Overall, the 16x figure substantially overstates the pretraining cost gap between DLMs and ARMs, and pretraining DLMs from scratch is far more viable than previously assumed.

1 INTRODUCTION

The widely cited 16× compute gap made pretraining masked diffusion models from scratch appear prohibitive, but it was measured with a protocol that differs from current practice. A revised scaling analysis finds a substantially smaller gap using exact likelihood, block diffusion, and encoder–decoder models.

Motivation

  • The 16× figure is presented as a widely cited estimate that made pretraining masked diffusion models from scratch look prohibitive at scale.The paper frames its analysis as a reassessment of that estimate under current diffusion-language-model practice.Source: §1 · p. 2
  • The prior 16× estimate used a full-sequence, decoder-only model scored by ELBO, unlike current diffusion-model practices described here.The ELBO upper-bounds NLL and evaluates uniformly random token orders that differ from the model's inference decoding order.Source: §1 · p. 2

Revised protocol

  • The revised protocol evaluates exact likelihood under the top-1 confidence decoding rule, trains block-aware encoder–decoder models, and sweeps commonly used block sizes.It also counts training FLOPs exactly and reads compute multiples within the measured scaling-law range.Source: §1 · p. 2

Results

  • Exact-likelihood evaluation reduces the compute multiple to 12.8×; encoder–decoder block diffusion lowers it to 7.8× at b=32, 4.6× at b=8, and 3.0× at b=4.Source: §1 · p. 2

Additional findings

  • The paper reports that the E2D2–ARM gap persists at a held-out budget three times larger than the largest fitted FLOPs.It also reports that likelihood improvements carry over to downstream accuracy.Source: §1 · p. 2

2 BACKGROUND

Masked diffusion models progressively mask tokens and train predictions for masked positions, but their conventional ELBO evaluation does not match the decoding order used at inference. Block-wise and encoder–decoder designs change how sequences are modeled and the training compute required.

Masked diffusion models

  • A masked diffusion model independently masks tokens with probability 1−α_t as α_t decreases from 1 to 0, and predicts masked positions from partially masked sequences.With the linear schedule α_t=1−t, training uses a negative ELBO; the paper follows a low-variance version that drops the 1/t weight.Source: §2 · p. 3

Likelihood evaluation

  • The ELBO is an upper bound on NLL and averages autoregressive factorizations over random token orders, unlike the adaptive orders used in practice.DUEL scores likelihood under a deterministic unmasking rule; this paper uses the top-1 confidence rule and reports its negative log-likelihood.Source: §2 · p. 3

Block diffusion

  • Block diffusion partitions a sequence into contiguous b-token blocks, models each block with masked diffusion conditioned on the clean prefix, and factorizes across blocks.The decoder-only BD3LM concatenates clean and noised sequences, so a training step runs over 2n tokens and costs roughly twice the FLOPs of an ARM or MDM step at the same size.Source: §2 · p. 3

Encoder–decoder block diffusion

  • E2D2 divides transformer layers between an encoder for the clean prefix and a decoder that denoises the block using self- and cross-attention.Because clean and noised tokens do not pass through the same large network, each training step is cheaper than a BD3LM step.Source: §2 · p. 3

IsoFLOP scaling laws

  • IsoFLOP analysis fits compute-optimal loss across model sizes and budgets, then compares the compute required by two model families at a fixed loss.The compute multiple is the ratio of the two families' budgets for reaching that loss.Source: §2 · p. 3

3 SCALING LAW PROTOCOL

The protocol updates the model evaluation and architecture choices, corrects FLOP accounting and compute-multiple readout, and retains a consistent training recipe for comparison with prior work. It fits compute-optimal loss across budgets and reads the resulting comparison within the ARM's measured range.

3 SCALING LAW PROTOCOL

  • The protocol replaces prior choices with exact likelihood, block diffusion at practical sizes, and encoder–decoder models, while correcting FLOP counts and compute-multiple readout.Diffusion models are scored with DUEL under top-1 confidence unmasking; ELBO scores are also computed for comparison on a frozen validation set of 49,152 sequences.Source: §3 · p. 4, §3.1 · p. 4

3.1 INCORPORATING RECENT ADVANCES

  • Block sizes b∈{4, 8, 16, 32} cover serving configurations; b=4 serves as a reference for the compute savings of a smaller block.For mid-size serving batches of 8–32, sizes 8–32 raise arithmetic intensity to about 256 or more, near the H100's roughly 300 FLOPs-per-byte ratio.Source: §3.1 · p. 4

3.2 TRAINING RECIPE AND ISOFLOP SWEEP

  • The study compares ARM, MDM, four BD3LM block sizes, and four E2D2 block sizes, sweeping E2D2 decoder depth at each grid point.It largely retains SMDM's model, data, tokenizer, and hyperparameter choices; the grids contain about 1,700 trained models across ten variants.Source: §3.1 · p. 4, §3.2 · pp. 4–5
  • Training FLOPs include the output head and attention score computation, rather than relying on the coarse 6N estimate.The omitted attention computation is non-negligible at sequence length 2048 and differs across variants.Source: §3.2 · p. 5

3.3 FITTING SCALING LAWS

  • For each budget, a parabola over model sizes estimates compute-optimal NLL; power laws across budgets are then used to read the compute multiple within the ARM's measured range.The multiple compares diffusion compute at 10^20 FLOPs with ARM compute at the same NLL, avoiding extrapolation beyond the measured ARM range.Source: §3.3 · p. 5

4 RESULTS

The results show that exact likelihood scoring, block diffusion, and encoder–decoder architecture each reduce the estimated compute gap between diffusion models and ARMs. The paper also finds a practical decoder-depth rule, persistence of the variant gap at a held-out budget, and downstream accuracy patterns that broadly track likelihood.

4.1 THE 16× OVERSTATES THE COST OF PRETRAINING DLMS

  • The ELBO-scored MDM requires 17.8× ARM compute; BD3LM lowers the multiple to 9.6× at b=32, 7.8× at b=16, and 5.9× at b=8.Source: §4.1 · pp. 5–6
  • Block-wise training’s likelihood gain outweighs BD3LM’s more expensive steps, which roughly halve its training steps relative to MDM at matched compute.Source: §4.1 · p. 7
  • E2D2 reaches 3.0× at b=4 and 4.6× at b=8, while exact likelihood alone lowers the MDM compute multiple to 12.8×.Source: §4.1 · pp. 5, 7
  • At matched compute, E2D2 consistently improves on BD3LM; the authors suggest separate encoder and decoder roles use pretraining compute more efficiently.Source: §4.1 · p. 7

4.2 WHERE THE LAYERS GO: OPTIMAL DECODER DEPTH IN E2D2

  • The optimal E2D2 decoder fraction depends mainly on block size and is decoder-heavy; the simple rule is within 5 m-nats of the sweep minimum at 90 of 104 grid points.Source: §4.2 · p. 7

4.3 THE GAP BETWEEN VARIANTS PERSISTS BEYOND THE FITTING RANGE

  • At the held-out 3×10^20-FLOP budget, ARM–E2D2 gaps are 261.6 m-nats measured versus 257 predicted at b=16, and 296.8 versus 294 at b=32, despite shared upward deviations.Source: §4.3 · p. 8

4.4 DOWNSTREAM PERFORMANCE BROADLY TRACKS LIKELIHOOD

  • DUEL yields statistically significantly higher mean accuracy than the ELBO for every diffusion variant.Source: Table 3 · p. 8
  • Across variants, accuracy order tracks NLL order; the orders coincide at the three largest budgets and differ only within noise at the two smallest.Source: §4.4 · pp. 8–9, Figure 6 · p. 8

5 RELATED WORK

Prior scaling studies generally found large compute gaps between diffusion language models and autoregressive models, while this work reports substantially lowering the widely cited 16× figure.

5 RELATED WORK

  • 16× compute was needed for masked diffusion models to match autoregressive likelihood in the first scaling study, by Nie et al. (2025).Uniform-state and continuous diffusion studies reported 23× and 20×, respectively.Source: §5 · p. 9
  • Under multi-epoch training in the data-constrained regime, masked diffusion models eventually overtake autoregressive models, according to Prabhudesai et al. (2026) and Ni et al. (2025a).Source: §5 · p. 9
  • The authors describe their study as the first to substantially lower the 16× figure, finding diffusion pretraining far cheaper than previously assumed.Source: §5 · p. 9

6 CONCLUSION

The revised analysis finds that the 16× compute gap substantially overstates diffusion pretraining costs, using exact likelihood, block-aware training, encoder–decoder models, and practical block sizes. It also documents limitations and the accounting and model-design details behind the analysis.

6 CONCLUSION

  • Exact likelihood reduces the compute multiple to 12.8×; block diffusion with an encoder–decoder reduces it to 7.8× at block size 32, 4.6× at 8, and 3.0× at 4.The study trained about 1,700 models across ten variants and reports downstream benchmarks where likelihood improvements carry over to accuracy.Source: §6 · p. 9
  • Experiments use one random seed and one learning-rate and batch-size configuration; DUEL uses only the top-1 sampler, and the held-out budget excludes block size 8.The authors fit pure power laws without an irreducible term and treat fitted exponents as local to their compute range.Source: §6 · p. 9

A.1 EXISTING LARGE-SCALE DIFFUSION LANGUAGE MODELS

  • Table 4 catalogs block sizes adopted by recent large-scale masked diffusion models, including inference block lengths for models trained with full-sequence masking.Source: §A.1 · p. 16, Table 4 · p. 16

A.2 ATTENTION PATTERNS OF THE MODEL VARIANTS

  • BD3LM concatenates clean and noised sequences in one tower, while E2D2 separates them into an encoder and decoder with block-restricted attention.In E2D2, the decoder attends within each noised block and cross-attends to encoder outputs from strictly preceding blocks.Source: §A.2 · p. 16, Figure 7 · p. 17

B.1 FLOP ACCOUNTING

  • Attention-score FLOPs are substantial, reach about 30% for smaller MDMs, and differ across variants, so attention-agnostic 6ND counts distort scaling and compute multiples.At matched size, MDM full attention costs about twice ARM causal attention, while E2D2 decoder attention is cheapest.Source: §B.1 · pp. 16–17

B.2 SCALING-LAW RUNS

  • The scaling-law grids list trained ladder sizes by variant and budget, and for E2D2 also sweep decoder depths; the held-out 3×10^20-FLOP budget is listed separately.Source: §B.2 · pp. 17–18, Table 6 · p. 18

B.3 COMPUTE COST

The scaling-law runs consumed approximately 136,000 GPU hours, with evaluation costs and decoder-depth sweeps contributing substantially to the total.

B.3 COMPUTE COST

  • Approximately 136,000 B200 GPU hours covered 1,741 training runs, excluding downstream benchmark evaluation.E2D2 rows include decoder-depth sweeps, while DUEL evaluation is especially costly for full-sequence MDMs.Source: §B.3 · p. 18
  • Table 7 organizes E2D2 decoder-depth sweeps by compute budget, ladder size, and block size.Source: Table 7 · p. 19

B.4 BENCHMARK EVALUATION PROTOCOL

The benchmark protocol specifies task metrics and diffusion readouts, while the b=4 IsoFLOP curves complement the main scaling-law comparisons.

B.4 BENCHMARK EVALUATION PROTOCOL

  • The protocol evaluates 11 benchmarks, using acc norm for HellaSwag, ARC-easy, PIQA, SciQ, RACE, and OpenBookQA, and acc for the other five.Diffusion variants are scored with top-1-confidence DUEL and 128-sample ELBO; the readouts coincide on single-word LAMBADA completion.Source: §B.4 · p. 19
  • Table 8 reports training and DUEL-evaluation GPU hours by variant, with E2D2 totals including decoder-depth sweeps.Source: Table 8 · p. 20

C.1 ISOFLOP CURVES

  • Figure 8 plots IsoFLOP curves for BD3LM and E2D2 at block size 4; darker shades indicate larger budgets and stars mark fitted vertices.For E2D2, each plotted point uses the best decoder depth at that model size.Source: Figure 8 · p. 20, §C.1 · p. 20

C.2 COMPUTE MULTIPLE

The compute multiple matches diffusion and ARM compute at equal NLL, but its reported value depends on the readout and whether the ARM law is extrapolated.

  • Below 6×10^19 FLOPs, most diffusion variants exceed the ARM’s measured NLL range of 2.39–2.84 nats, requiring ARM-law extrapolation below 6×10^18 FLOPs.Those entries are shown for reference only.Source: §C.2 · p. 20
  • At 10^20 FLOPs, the extrapolating readout differs from the measured-range readout by at most 22% for MDM under DUEL, while preserving variant ordering.Source: §C.2 · p. 20
  • Because diffusion exponents are slightly steeper than the ARM’s, the extrapolating readout’s compute multiple decreases as compute increases.Source: §C.2 · p. 20

C.3 DECODER DEPTH

The decoder-depth analysis finds a block-size-dependent optimal split and tests its scaling-law predictions at a held-out budget. A shared irreducible term improves that held-out fit, but the available budgets do not determine the term’s value.

C.3 DECODER DEPTH

  • Three readouts of the optimal decoder fraction agree closely at every block size, making the estimated split insensitive to the readout method.Source: §C.3 · p. 20, Table 11 · p. 21
  • The decoder-depth rule reproduces mean decoder fractions at b=4, 8, 16, and 32 within 0.005, with individual-column residual RMS of 0.045.Block size dominates the regression; model size has no detectable effect, and budget has a smaller effect.Source: §C.3 · p. 21
  • For b=4, the optimal decoder fraction is 0.63–0.65 versus 0.64 predicted; rounding Eq. 2 is within 10 m-nats in 87% of columns and all five compute-optimal columns.Source: §C.3 · p. 21, Table 12 · p. 22

C.4 HELD-OUT BUDGET

  • At 3×10^20 FLOPs, E2D2–ARM NLL gaps are 261.6 m-nats measured versus 257 predicted at b=16, and 296.8 versus 294 at b=32.Source: Table 13 · p. 22, §C.4 · p. 22
  • With a shared E≈1.5, the held-out misses become −7, +0.4, and 0.0 m-nats, but five budgets cannot pin down E.The held-out cells use three model sizes rather than a full IsoFLOP search, so the authors retain the pure power law in the paper’s main analysis.Source: Table 14 · p. 22, §C.4 · p. 22

C.5 DOWNSTREAM BENCHMARKS

The downstream analysis retains eight benchmarks whose accuracy rises with compute beyond standard error and excludes three that are uninformative or unusually noisy at these scales.

C.5 DOWNSTREAM BENCHMARKS

  • Eight benchmarks show accuracy increases with compute well beyond their standard errors; WinoGrande, BoolQ, and COPA are excluded for limited discrimination or high variance.WinoGrande stays at 50.8% against 50% chance at 10^20 FLOPs, BoolQ declines with compute, and COPA has only 100 questions.Source: Figure 11 · p. 23, §C.5 · p. 23
  • Table 15 reports zero-shot accuracy on the eight retained benchmarks and their mean for each variant’s compute-optimal cell at each budget.These are the values plotted in Figure 6.Source: §C.5 · p. 23, Table 15 · p. 24

D ASSETS

  • Table 16 lists the study’s third-party code, data, and benchmarks together with their licenses.Source: D ASSETS · p. 24

E PER-CELL RESULTS OF THE ISOFLOP SWEEPS

The appendix records per-cell IsoFLOP results for ARM and diffusion variants, plus trained E2D2 decoder-depth sweeps across block sizes.

E PER-CELL RESULTS OF THE ISOFLOP SWEEPS

N Ldec/L DUEL ELBO

  • E2D2 decoder-depth tables report trained models at b=32, 16, 8, and 4, with nominal size, DUEL and ELBO validation NLLs, and decoder-depth fraction where specified.The best decoder depth in each budget-and-size group is marked in the tables.Source: Table 18 · p. 27, Table 20 · p. 29, Table 22 · p. 31
  • The b=32 and b=16 decoder-depth tables include rows at the held-out budget of 3×10^20 FLOPs.Source: Table 18 · p. 27, Table 20 · p. 29

Related papers

Loading 2610.17590v1…