PaperlayerView exact revision on arXiv

Source-linked AI summary

On-Policy Residual Learning for Diffusion Language Models

Yifan Yu, Yuqing Jian, Jisen Li, Yuezhou Hu, Tianyi Zhang, Xiaoxia Wu, Rui-Jie Zhu, Junxiong Wang, Zhongzhu Zhou, Fan Lai, James Zou, Chenfeng Xu

arXiv:2610.20687v1cs.LG

TL;DR

Autoregressive decoding is sequential, while diffusion language models expose parallelism that can be learned; however, there is no clear recipe for learning residual context. ORL trains on the current model’s residual trajectories and uses entropy contraction to sharpen predictions, improving tokens per forward and throughput at comparable quality.

  • Problem

    Autoregressive decoding remains sequential, and there is no clear recipe for learning residual context in diffusion language models.

    Source: §1 · pp. 1–2

  • Method

    ORL learns from fresh, model-induced residual trajectories and combines verified-output supervision with adjacent entropy contraction to sharpen predictions across denoising steps.

    Source: §1 · p. 3, §3 · p. 5

  • Results

    On QwenDLM, ORL improves TPF by 2.5–6.2× and throughput by approximately 5× (approximately 6.9× with DLMRT) at comparable quality, and raises TPF on both transferred DLMs across four benchmarks.

    Source: §4.1 · p. 9, §5 · p. 13

  • Takeaways & Limitations

    ORL transfers to LLaDA-2.1-mini and DiffusionGemma, improving TPF while maintaining or improving quality.

    Source: §4.1 · p. 9, §5 · p. 13

    Throughput comparisons between Base and ORL use the same engine, so they reflect training rather than engine differences.

    Source: §3.3 · p. 8

Abstract

from arXiv · show

Residual context is key to unlocking the speed-quality Pareto frontier of diffusion language models (DLMs) by carrying forward predictive distributions across denoising steps. Yet turning this seemingly simple mechanism into an effective learning paradigm is far from simple: there is no clear recipe for how residual context should be learned. We introduce ORL, a framework for on-policy residual learning in DLMs. Algorithmically, we identify that residual learning is inherently on-policy. ORL learns from model-induced residual trajectories on the fly and introduces an entropy-contraction objective that sharpens predictions as the model revisits its own predictions. System-wise, exact residual supervision requires capturing, retaining, and reprocessing many trajectory-dependent states. ORL co-designs the algorithm with asynchronous residual capture and transport, specialized block-diffusion attention, and packed compute-balanced training, making it practical at scale. On a 4B DLM converted from Qwen3-4B, ORL improves tokens per forward (TPF) by up to 6.2x and, with our dlmrt engine, end-to-end throughput by approximately 6.9x at comparable or better quality. In single-request serving on one H100, this mode achieves a stronger speed-quality Pareto frontier than autoregressive models with EAGLE3 and DFlash speculative decoding under both strict and relaxed verification, running 1.74x faster than DFlash at matched accuracy. ORL also transfers to existing DLMs: on LLaDA-2.1-mini, 36 H100-hours and 4.2M self-generated tokens, about 10x less data than prior LLaDA recipes, yield up to 1.5x higher TPF and up to 3x higher throughput at comparable quality, and on DiffusionGemma, ORL raises TPF by 1.1-1.2x. With dlmrt and NVFP4 quantization, ORL models decode a single request at up to 5,046 tokens/s on one B200, 7,708 on eight B200s, and 2,788 on a consumer RTX 5090, with comparable quality.

1 INTRODUCTION

Diffusion language models expose a learned parallelism dimension, but effective residual learning lacks a clear recipe. ORL addresses the learning and systems challenges, improving speed-quality results across DLMs.

1 INTRODUCTION

  • Autoregressive decoding remains sequential and underuses modern hardware, while DLMs make parallelism a joint learning problem with prediction quality.Speculative decoding accelerates a fixed target, whereas DLMs can learn how much reliable progress to make per forward pass.Source: §1 · pp. 1–2
  • Existing residual methods improve the speed-quality frontier, but leave no clear recipe for learning residual information effectively and efficiently.Residual predictions carry forward distributions that would otherwise be discarded and recomputed.Source: §1 · p. 2
  • ORL learns from fresh residual trajectories generated by the current model and adds adjacent entropy contraction to sharpen predictions across denoising steps.Supervision is gated by rollout entropy, and residual updates are stabilized to keep the residual pathway aligned with backbone updates.Source: §1 · p. 3
  • Asynchronous residual capture, memory-adaptive transport, packed training, compute-aware scheduling, and staircase attention make exact on-policy residual learning practical at scale.DLMRT further converts fewer forward passes into higher inference throughput.Source: §1 · p. 3
  • ORL improves QwenDLM TPF by 2.5–6.2× and throughput by approximately 5× at comparable quality, or approximately 6.9× with DLMRT.In single-request serving on one H100, ORL is 1.74× faster than DFlash at matched accuracy with relaxed verification.Source: §1 · p. 3
  • On LLaDA-2.1-mini, ORL uses 4.2M self-generated tokens and 36 H100-hours, about 10× less data than prior recipes, and raises TPF by up to 1.5×.The same contribution summary reports up to 3× higher throughput on LLaDA-2.1-mini and 1.1–1.2× higher TPF on DiffusionGemma.Source: §1 · p. 3

2 RELATED WORK AND MOTIVATION

Residual diffusion decodes blocks through repeated denoising and can carry prior predictions forward as residual context. Because those residuals shape subsequent states, effective learning needs fresh trajectories, an explicit refinement objective, and systems support for exact replay.

2 RELATED WORK AND MOTIVATION

  • Block DLM decoding denoises B-token blocks over several steps; TPF counts completion tokens per denoising forward, and residual decoding carries prior predictions as context.The decoding rule accepts confident unresolved positions, re-noises the rest, and commits a block once no positions remain unresolved or the step cap is reached.Source: §2 · p. 4
  • Residuals are injected as expected token embeddings through a lightweight adapter; alignment differs between shifted QwenDLM outputs and same-position LLaDA outputs.The adapter is model-specific: QwenDLM uses a warmed-up gated MLP, LLaDA adds a zero-initialized gated MLP, and DiffusionGemma uses its native residual module.Source: §2 · p. 4
  • Fresh trajectories jointly improve accuracy and TPF, while increasing trajectory staleness sacrifices accuracy; offline training barely improves TPF.Staleness N counts optimizer steps between rollout-policy synchronizations; N=1 is on-policy, while offline uses fixed cached trajectories.Source: §2 · p. 5
  • Standard supervision can produce correct final answers without faster refinement: unresolved-position entropy stays nearly flat, so TPF barely improves.The paper argues parallel decoding needs an explicit objective for refinement speed, not accuracy alone.Source: §2 · p. 5
  • Exact on-policy supervision generates states dynamically, with irregular trajectory lengths and compute costs that create memory waste and GPU load imbalance.A naïve implementation repeatedly materializes states, and its time and memory costs grow rapidly.Source: §2 · p. 5

3 METHOD

ORL trains residual predictions on fresh model-generated trajectories, replaying captured states with objectives for both prediction quality and refinement speed. Its training and inference systems preserve exact residual inputs while making state capture, replay, and single-request decoding practical.

3.1 ON-POLICY RESIDUAL LEARNING

  • Each update generates fresh residual-conditioned rollouts, keeps verified responses, replays captured states, and combines verified-output supervision with adjacent entropy contraction.The objectives teach what each state should predict and how quickly predictions should sharpen.Source: §3 · p. 5, §3.1 · p. 6
  • Exact replay preserves each selected state's canvas and consumed pre-adapter residual, then evaluates the captured input without backpropagating through earlier steps.Each captured state is replayed once by the parameters that generated it; exactness means preserving the input, not bitwise-equal outputs.Source: §3.1 · p. 6
  • AEC penalizes only steps whose next-state unresolved entropy fails to contract by the target fraction, avoiding further sharpening where refinement is already progressing.The hinge loss targets positions still unresolved after acceptance.Source: §3.1 · p. 6
  • To stabilize the feedback loop, ORL slows residual-pathway updates relative to the backbone and gates cross-entropy supervision to uncertain rollout predictions.For confident positions, it uses a capped reverse KL to keep predictions near the rollout distribution.Source: §3.1 · p. 7

3.2 EFFICIENT EXACT RESIDUAL TRAINING

  • Asynchronous capture transfers rollout states off the decode critical path; packed cost-balanced rows reuse clean context, while staircase attention skips invisible tiles under block-diffusion visibility.Packing and redistribution preserve the response-normalized objective up to floating-point rounding.Source: §3.2 · p. 7

3.3 EFFICIENT RESIDUAL INFERENCE

  • Inference folds the clean-block check into the next block's first denoising step and uses a GPU-graph engine specialized for single-request diffusion decoding.Commit folding saves one forward pass per block; the passage reports 53% higher throughput from this change.Source: §3.3 · p. 8

4 EXPERIMENTS

Across converted and existing DLMs, ORL improves tokens per forward while maintaining or improving quality, and its gains extend to single-request throughput. Ablations support the importance of fresh rollouts, residual scaling, and system co-design; deployment results show further speedups with DLMRT and NVFP4.

4.1 MAIN RESULTS

  • ORL improves QwenDLM by 2.5–6.2× in TPF while preserving or improving quality, and raises TPF on all four benchmarks for both LLaDA-2.1-mini and DiffusionGemma.On HumanEval, QwenDLM reaches 89.6 pass@1 at 11.1 TPF versus 73.5/2.4 for SDAR-8B; LLaDA-2.1-mini reaches 89.0/13.1 versus 86.6/11.3 in pass@1/TPF.Source: §4.1 · p. 9
  • Across ten additional model–benchmark pairs, ORL raises TPF by 1.26× on average and up to 1.46×, with little quality change.The evaluation spans additional mathematics, knowledge-intensive reasoning, code-generation, and instruction-following benchmarks for two independently developed DLMs.Source: §4.1 · p. 9, Table 2 · p. 9
  • At comparable accuracy, QwenDLM with ORL reaches 1,298 tok/s and 91.0% mean GSM8K/HumanEval accuracy on one H100, about 6.9× faster than its starting DLM on SGLang.Against measured speculative settings at comparable accuracy, ORL is faster than EAGLE3 and DFlash under both relaxed and strict verification; Figure 4 also reports clean-TPS gains of 5.60× for QwenDLM and 2.95× for LLaDA-2.1-mini.Source: §4.1 · p. 9, Figure 4 · p. 10
  • With every-step rollout refresh, QwenDLM reaches 89.8% mean GSM8K accuracy and 13.7 TPF, versus 77.1%/11.7 at N = 25 and 78.1%/9.9 at N = 75.N controls rollout-weight refresh; the reported results motivate frequent refresh for the evaluated recipe.Source: §4.1 · pp. 9–10

4.2 ABLATIONS AND ANALYSIS

  • System co-design reduces optimizer-step time by 3.9/5.2/3.5× and raises token throughput by 2.9/3.4/2.7× across n = 8/16/24 rollouts.The cumulative optimizations add staircase attention, packed state training, compute-aware balancing, and asynchronous GPU-resident handoff.Source: §4.2 · p. 10, Figure 5 · p. 10
  • ORL uses 4.2M self-generated tokens—about 10× less than the roughly 40M used by dParallel and d3LLM-LLaDA—yet reaches higher average TPF at higher average accuracy than the listed baselines.The LLaDA-2.1-mini run uses 36 H100 GPU-hours.Source: §4.2 · p. 10, Figure 7 · p. 11
  • For QwenDLM trained with ORL on GSM8K prompts, residual scaling yields nearly identical verifier reward over the first 250 steps, then remains at 0.677 over longer training, while unscaled training collapses to 0.225.Removing scaling changes backbone update magnitude by only 0.96× but increases residual-adapter updates by 31.3×.Source: §4.2 · p. 10, Figure 8 · p. 11
  • AEC removal cuts TPF by 28% on GSM8K and 22% on MATH-500, while removing entropy-gated reverse KL or the residual pathway degrades both accuracy and TPF.ORL’s HumanEval TPF also increases from 11.4 to 15.0 at block size 32 and from 9.2 to 19.2 at block size 128; DLMRT raises throughput by 1.22–1.40× for ORL models, and ORL plus NVFP4 reaches 5,046 tokens/s on one B200 for LLaDA-2.1-mini and 7,708 tokens/s on eight B200s for DiffusionGemma.Source: §4.2 · pp. 11–12, Table 5 · p. 12, §4.3 · p. 13

5 CONCLUSION

ORL combines fresh on-policy residual trajectories, entropy-based refinement, and system co-design; it improves speed-quality outcomes across converted and existing DLMs. The conclusion also identifies a memory cost for retaining intermediate states.

5 CONCLUSION

  • On QwenDLM, ORL raises TPF 2.5–6.2× and throughput approximately 5× (approximately 6.9× with DLMRT) at comparable quality, above measured EAGLE3 and DFlash operating points.The conclusion also reports TPF gains on LLaDA-2.1-mini and DiffusionGemma, and single-request throughput measurements with DLMRT and NVFP4.Source: §5 · p. 13

A RELATED WORKS

  • ORL uses fresh trajectories from the current model and replays intermediate states with the exact residual inputs consumed during generation.This addresses changing residual predictions and trajectories as the model is updated.Source: A RELATED WORKS · p. 17
  • ORL supervises unresolved-position predictions toward verified final responses, beyond sampled actions, and uses adjacent entropy contraction to encourage uncertainty reduction.Both objectives operate in a fully on-policy framework that regenerates trajectories after every update.Source: A RELATED WORKS · p. 18
  • Residual DLMs implement recurrent computation through continuous predictive feedback between denoising steps without adding layer repetitions within a forward pass.ORL complements this architecture by training feedback on fresh trajectories with exact residual context and an explicit refinement objective.Source: A RELATED WORKS · p. 18

B.1 PREDICTIVE INFORMATION AND CAPTURE COST

  • Naive capture adds about 28 GB of host memory per 100k training tokens, versus 36 GB for the combined host/GPU footprint reported in Figure 2(c).Intermediate predictions can contain the final decoded token before it is committed; this agreement is with the trajectory output, not reference-answer correctness.Source: §B.1 · p. 18

C OPTIMIZATION DETAILS AND STABILIZATION

The optimization details describe response weighting, entropy-based refinement, teacher construction, and model-specific residual initialization.

C OPTIMIZATION DETAILS AND STABILIZATION

  • Each verified response receives equal token-loss weight, and batches with no valid supervised positions are skipped.The valid-position definition excludes padding and positions after terminal EOS.Source: C OPTIMIZATION DETAILS AND STABILIZATION · p. 18
  • AEC weights valid adjacent state pairs equally before gating, excludes responses with no valid pair, and uses a canvas-pair mean for DiffusionGemma.A valid pair has a selected target state and nonempty unresolved sets at both states; its source need not be selected.Source: C OPTIMIZATION DETAILS AND STABILIZATION · p. 19
  • Only QwenDLM’s final training stage uses the entropy gate, while other reported stages use CE; its teacher uses the current output head under stop-gradient.Both head outputs are cast to float32, clipped, and normalized over the vocabulary; replay gradients accumulate before one optimizer step.Source: C OPTIMIZATION DETAILS AND STABILIZATION · p. 19
  • Exact captured inputs do not guarantee rollout/replay numerical parity, and this recipe does not isolate RKL’s contribution from removing low-entropy CE.Identical transformed distributions yield zero RKL loss and gradient before capping.Source: C OPTIMIZATION DETAILS AND STABILIZATION · p. 19
  • QwenDLM initialization scales adapter matrices with compensating input and output factors to preserve function; LLaDA instead uses small gate/up weights, zero down weights, and no post-normalization.For QwenDLM, the compensation constants are 1/8 and the residual sum receives non-affine RMS normalization scaled by 0.02; DiffusionGemma retains its native self-conditioning module.Source: C OPTIMIZATION DETAILS AND STABILIZATION · pp. 19–20

D SYSTEM IMPLEMENTATION DETAILS

The system implementation combines asynchronous state capture, specialized block-diffusion attention, and compute-balanced packing.

D.1 ASYNCHRONOUS EXACT-STATE CAPTURE

  • Asynchronous capture snapshots tokens, masks, residuals, and required statistics, then materializes records in a background worker after a CUDA event.The decode critical path performs no host copy or host synchronization; bounded queue exhaustion blocks briefly or fails explicitly rather than dropping states.Source: §D.1 · p. 20

D.2 STAIRCASE ATTENTION

  • Staircase attention restricts noisy queries to the prompt, preceding clean blocks, and their own block, while clean queries never attend to noisy keys.The Triton implementation skips invisible tiles and avoids allocating a dense N × N mask.Source: §D.2 · p. 20

D.3 PACKING AND COST BALANCING

  • Replay rows are packed as mutually invisible documents and assigned to data-parallel groups using a weighted cost proxy for tokens, supervised positions, and resident pages.Chunked materialization bounds live input/page footprint; label ownership and global reduction preserve the unpacked objective up to rounding.Source: §D.3 · p. 20

E.1 EVALUATION PROTOCOLS

The evaluation protocols specify serving conditions, task and scoring procedures, aggregation, and runtime configurations. They distinguish within-family comparisons from literature comparisons and identify the scope of reported throughput estimates.

E.1 EVALUATION PROTOCOLS

  • Measured throughput uses one H100, TP=1, and a single request in flight; length-terminated responses count as incorrect and are excluded only from clean efficiency statistics.Request time includes prefill, decoding, scheduling, and host overhead, but excludes model loading and server startup.Source: §E.1 · p. 20
  • Reproducing specified throughput results requires DLMRT, and absolute TPS depends on hardware and runtime; engine and decoding configurations are recorded in the repository.Separate B200 NVFP4 measurements include the quantization procedure and FP8 KV-cache configuration.Source: §E.1 · p. 21
  • The main suite covers GSM8K, MATH-500, HumanEval, and MBPP427; Base/ORL share task and scorer protocols within each family, while cross-family and literature comparisons are not scorer-matched.QwenDLM uses its CodeTACO code-scoring path, whereas LLaDA and DiffusionGemma use EvalPlus base tests for HumanEval.Source: §E.1 · p. 21
  • Metrics are averaged within each checkpoint and then equally across independent ORL runs; reported means alone do not establish statistical significance for small score differences.Each evaluation generates one completion per task, with three generation-seed settings per DiffusionGemma checkpoint.Source: §E.1 · p. 21
  • Transfer-evaluation table metrics average independent training runs and repeated evaluations; its AIME-24 score uses 30 tasks from a shared 60-task, two-year generation input.Cross-benchmark summaries use equal-weight means of benchmark-level metrics.Source: Table 8 · p. 22, §E.1 · p. 22
  • Published baseline comparisons retain their original backbones and evaluation settings rather than using scorer-matched reruns; derived TPS comparison points are estimates, not measured wall TPS.The estimates convert TPF using measured reference ratios or disaggregate a paper’s aggregate TPS under its task, hardware, and decoding protocol.Source: §E.1 · p. 22

E.2 ADDITIONAL BENCHMARKS

The section reports transfer quality and implementation details for initialization, on-policy training, inference optimization, and DLMRT. Transfer results show TPF gains across all evaluated model–benchmark pairs, while accuracy changes vary by task.

E.2 ADDITIONAL BENCHMARKS

  • TPF increases on all ten transfer model–benchmark pairs, while accuracy varies; declines include 3.4 points for LLaDA on GPQA-Diamond and 2.1 points for DiffusionGemma on LiveCodeBench-v6.Quality and clean per-response TPF are averaged over repeated checkpoint evaluations, then equally across independent ORL training runs; the authors note incomplete historical source/RNG binding for some reproduction scorers and a context override.Source: §E.2 · p. 22

E.3 OFFLINE SDAR INITIALIZATION

  • QwenDLM starts from a non-residual Qwen3-4B-Thinking-2507 SDAR checkpoint, receives a zero-output residual adapter, and is initialized with CE training on ReasonMix/SFTMix.The recipe uses block size 256 and an 8,192-token cutoff; an auxiliary sweep found later checkpoints did not monotonically improve accuracy.Source: §E.3 · p. 23

E.4 ON-POLICY TRAINING RECIPES

  • On-policy recipes train until TPF stabilizes, using one prompt and eight rollouts per iteration and updating only on verified responses.Training data and reward verification vary by model; LLaDA uses block size 32 and CE plus AEC without entropy-gated RKL, while DiffusionGemma retains native self-conditioning and keeps intermediate states.Source: §E.4 · p. 23

F INFERENCE OPTIMIZATION

  • Inference optimization combines cache reuse, fused execution, and CUDA graphs, with a final output-parity gate on HE164, EvalPlus MBPP378, and GSM8K100.In Figure 11’s LLaDA Quality-mode sequence, the 864-to-1,324 transition had 153/164 identical outputs and accuracy moved from 0.896 to 0.890; later changes pass the final gate.Source: F INFERENCE OPTIMIZATION · p. 23, Figure 11 · p. 24

F.1 THE DLMRT ENGINE

  • DLMRT runs each block’s denoising loop as one CUDA graph, with the host launching the block and reading back its tokens.The graph’s device-side conditional loop avoids host waits between rounds; parity checks report 4,816/4,820 Qwen answers reproduced and all 164 LLaDA Speed-mode HumanEval answers reproduced.Source: Figure 12 · p. 24, §F.1 · pp. 24–25

F.2 NVFP4 QUANTIZATION

The section describes NVFP4 quantization and a scale-search method that accounts for input-feature importance and within-block correlations. Reported B200 throughput rises substantially versus H100 BF16, with some task-specific LLaDA accuracy losses.

F.2 NVFP4 QUANTIZATION

  • 2.31–2.39× for LLaDA-2.1-mini-ORL and 2.69–2.89× for DiffusionGemma-ORL are the B200 NVFP4 throughput gains over H100 BF16.Accuracy remains broadly comparable, although LLaDA loses 3.0 points on MATH-500 and 1.4 points on MBPP; TPF decreases modestly across all four tasks for both models.Source: Table 11 · p. 26, §F.2 · p. 26
  • Local-Hessian scale search chooses the NVFP4 scale for each 16-weight block by minimizing local output reconstruction error across 126 positive, nonzero FP8 E4M3 candidates.It approximates the input second-moment matrix as block diagonal, retaining within-block correlations while ignoring interactions across blocks; scale-only changes add no runtime overhead.Source: §F.2 · pp. 25–26
  • The evaluated Diagonal-Hessian variant ignores off-diagonal input-feature correlations, and the selected quantized checkpoints use Local-Hessian for LLaDA and Diagonal-Hessian for DiffusionGemma.Both variants are evaluated, and the checkpoint with the best generation quality is selected.Source: §F.2 · p. 26

G.1 FULL-TASK REQUEST ACCOUNTING

The section accounts for full-task throughput, speculative-decoding comparisons, additional model comparisons, quality–throughput sweeps, canvas-TPF results, and system costs. It reports large QwenDLM throughput gains while distinguishing throughput from refinement-TPF diagnostics and documenting evaluation and systems boundaries.

G.1 FULL-TASK REQUEST ACCOUNTING

  • 1,298 tok/s is QwenDLM ORL’s DLMRT benchmark-macro clean throughput, about 6.9× the SGLang Base’s 186.88 tok/s; the DLMRT Base is 289 tok/s.All-output/completed-output ratios are 4.59/4.70; the full-task accounting includes length failures as incorrect for quality.Source: §G.1 · p. 26, Table 12 · p. 27

G.2 SPECULATIVE-DECODING COMPARISON

  • At ORL’s 91.03% mean accuracy, the matched-quality profiles are EAGLE3 QE02 at 94.23% and 404.7 TPS and DFlash QD03 at 90.76% and 745.5 TPS.The matching rule selects the fastest measured setting per drafter within 0.5 percentage points below ORL; the DFlash drafter targets a different Qwen release.Source: §G.2 · p. 27, Table 13 · p. 28

G.4 QUALITY–THROUGHPUT SWEEPS

  • Figure 14 shows 48 operating points across Qwen Base, Qwen ORL, and LLaDA checkpoints, sweeping acceptance thresholds, entropy budgets, or LLaDA profiles.Its accuracy–throughput axes average GSM8K and HumanEval, and the highlighted matched-accuracy pairs are specified in the figure caption.Source: §G.4 · p. 27

G.5 ACCURACY–CANVAS-TPF TRADEOFFS

  • At the highlighted HumanEval pair, both checkpoints pass 146/164 tasks while canvas TPF rises from 4.8395 to 13.9086, a 2.874× ratio.Canvas TPF includes final-block padding and does not subtract edits or charge refinement/commit forwards, so it is a refinement diagnostic rather than wall TPS.Source: §G.5 · p. 28

H.1 SYSTEM COST

  • At n = 16 rollouts, packing reduces actor-update time from 368 to 131 seconds, and cost balancing further reduces it to 96 seconds.GPU-resident handoff at n = 8, 16, and 24 also reduces rollout and replay-materialization time substantially, with little actor-update change.Source: §H.1 · pp. 28–29

G.3 ADDITIONAL MODEL COMPARISONS

  • Figure 13 adds model families and methods, but throughput for CForce, Nemotron-Diff, and LightningRL is estimated from published results or converted TPF rather than directly measured completion throughput.The remaining points use measured single-request, one-H100 wall-clock completion throughput.Source: §G.3 · p. 27, Figure 13 · p. 29

H.2 RESIDUAL SCALING AND BLOCK SIZE

  • With residual rescaling γ = 8, the residual MLP’s median normalized update ratio is 0.00142%, versus 0.04431% without scaling, while backbone values are 0.00122% and 0.00117%.The MBPP block-size study reports ORL TPF rising from 8.6 to 13.0 at block size 32 and from 7.2 to 16.6 at block size 128.Source: §H.2 · p. 29

H.3 TRAINING DATA AND COMPUTE

The section compares post-training data budgets and presents quality–throughput operating points alongside compute-accounting conventions. It reports ORL’s LLaDA data usage and highlights selected Qwen and LLaDA sweep pairs.

H.3 TRAINING DATA AND COMPUTE

  • LLaDA training uses 4.2M total tokens from 5,000 responses, including 2.37M verified response tokens entering the loss.The stated total includes prompts and responses, including failed rollouts, and is about one-tenth of the roughly 40M-token dParallel and d3LLM-LLaDA budgets.Source: §H.3 · p. 29
  • The highlighted Qwen pair gives 86.09/86.92% mean accuracy and 5.60/5.09× clean/completed TPS gains, while the LLaDA pair gives 90.49% accuracy on both sides and 2.95/3.39× gains.The sweep axes average GSM8K and HumanEval; settings are measured with SGLang.Source: Figure 14 · p. 30
  • Table 14 counts allocated H100 GPU-hours along the training chain to each reported checkpoint, while excluding initializer production, checkpoint evaluation, other runs, idle gaps, and some failed attempts.Including attempts that fail before their first checkpoint adds about 4.5 GPU-hours for QwenDLM and at most 8.5 for DiffusionGemma.Source: §H.3 · p. 30, Table 14 · p. 30

I.1 THE UPDATE AND STALE DATA

The section bounds frozen-data gradient mismatch under a gradient-norm assumption, and describes limits on entropy contraction and optimizer rescaling. It also treats quality-versus-throughput curves as descriptive approximations with measured-range constraints.

I.1 THE UPDATE AND STALE DATA

  • Under bounded gradient norms, stale-gradient discrepancy is at most 2G times total-variation distance between record distributions; this bounds mismatch, not harm.The gradient bound is an assumption, not automatic for cross-entropy.Source: §I.1 · p. 33

I.2 ENTROPY AND TOKEN ACCEPTANCE

  • Entropy sharpening can lower uncertainty without correcting a wrong argmax, and removing confident positions can raise the remaining-set mean without worsening predictions.The contraction argument also depends on conditions such as eligible positions staying unresolved only until accepted.Source: §I.2 · pp. 33–34

I.3 OPTIMIZER AND RESCALING

  • Compensated projection scaling reduces effective SGD updates by γ^2 and Adam updates approximately by γ, while stability remains empirical.Clipping, moment handling, and epsilon can change the optimizer relationship.Source: §I.3 · p. 34

J.1 A DESCRIPTIVE MODEL

  • A quality-independent speed factor requires matching curve plateaus and shapes; half-plateau estimates beyond canvas bound 32 are unattainable.The log-logistic curve is a compact empirical approximation, not an identified latent-difficulty law or proof of an AEC mechanism.Source: §J.1 · pp. 34–35

J.2 LOCKED LLADA SWEEP

In the locked LLaDA sweep, ORL's HumanEval operating point delivered a 2.874× canvas-TPF ratio while both checkpoints passed 146/164 tasks. Shared-profile comparisons and conditional uncertainty estimates qualify that result, while exploratory curve fits were not uniformly better than isotonic regression.

J.2 LOCKED LLADA SWEEP

  • 2.874× canvas-TPF ratio was observed on HumanEval with both checkpoints passing 146/164 tasks.The paired task-bootstrap interval for the canvas ratio was [2.737, 3.012]; it does not establish population equivalence.Source: §J.2 · p. 35
  • At L11, Base/ORL HE accuracy was 143/164 versus 146/164 and completion TPF was 8.720 versus 13.077; GSM accuracy was 1219/1319 versus 1213/1319, with TPF 4.045 versus 8.535.Table 15 compares all fourteen shared profiles; its R metric is clean completion-TPF ratio, not padded canvas or wall TPS.Source: §J.2 · p. 35, Table 15 · p. 35
  • Bootstrap intervals are conditional on evaluated checkpoints and archived records; they do not estimate variation across independent training runs or repeated evaluations.The joint task-vector bootstrap intervals cover the displayed profiles and benchmarks.Source: §J.2 · pp. 35–36
  • ORL's estimated half-plateau values were 44.03/80.39, above canvas bound 32; log-logistic fits were not uniformly better than isotonic regression.Held-out folds excluded one operating point or one adjacent pair; the fit is a compact empirical approximation, not an identified latent-difficulty law.Source: §J.2 · p. 36
Loading 2610.20687v1…