PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Continuous Diffusion Models

Zhihan Yang, Wei Guo, Jean-Marie Lemercier, Simon Welker, Yonggan Fu, Mohammad Mahdi Kamani, Sajad Norouzi, Julius Berner, Tomas Geffner, Karsten Kreis, Yongxin Chen, Molei Tao, John Thickstun, Pavlo Molchanov, Ante Jukić, Arash Vahdat, Morteza Mardani

arXiv:2610.02665v1cs.CLcs.AIcs.LG

TL;DR

Continuous dLMs had not been shown to scale to complex conditional reasoning, despite their potential for smoother trajectory control. Sigma is a large-scale blockwise continuous dLM trained with AR initialization, and it achieves competitive benchmark performance while exhibiting steering and low-NFE properties that support its potential for fast inference.

  • Problem

    Whether continuous dLMs scale to complex conditional reasoning remains open, as existing methods were below 2B or focused on unconditional generation while discrete models’ high-dimensional categorical spaces impede trajectory control.

    Source: §1 · p. 1

  • Method

    Sigma is a large-scale continuous dLM trained blockwise with AR initialization and regularization, paired with reverse-diffusion sampling using CFG, score temperature, and self-conditioning.

    Source: §1 · p. 2, §2 · p. 3, §7 · p. 10

  • Results

    Sigma achieves competitive performance with state-of-the-art discrete dLMs and ARs across base and instruct benchmarks, including standard math and coding tasks and harder reasoning after supervised fine-tuning.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Embedding-space CFG and score temperature govern the quality–diversity trade-off, while low-NFE robustness and trajectory distillation position continuous diffusion as a promising foundation for fast inference.

    Source: §1 · p. 2

    At high step counts, continuous dLM decoding remains slower per generated token than fully optimized AR baselines, while DDIM-4/8 drops substantially relative to DDPM-256.

    Source: B. Limitations and Future Work · p. 22

Abstract

from arXiv · show

Despite the success of discrete diffusion language models (dLMs) for fast parallel decoding, their non-smooth, high-dimensional space hinders trajectory steering for reasoning and inference acceleration. To overcome this, we present Sigma, the first large-scale (3B/8B) continuous dLM built on steerable, low-dimensional ODE/SDE latent trajectories. Trained blockwise via likelihood optimization, Sigma jointly denoises Gaussian-corrupted token embeddings while learning an optimal embedding geometry. To accelerate training, Sigma leverages pre-trained weights from autoregressive (AR) models for warm-starting. During inference, we identify classifier-free guidance and score temperature as essential for high-fidelity reasoning and coding. Across comprehensive math reasoning and coding evaluations against state-of-the-art discrete counterparts (masked dLMs and AR baselines), Sigma achieves competitive performance with discrete models on standard benchmarks (e.g., GSM8K, Minerva, HumanEval, MBPP) after pre-training and on challenging reasoning tasks (e.g., MATH-500, AIME) after supervised fine-tuning. Beyond performance parity, we uncover key structural properties unique to continuous dLMs: (i) embedding-space steering effectively governs the quality-diversity trade-off, yielding strong pass@k performance and (ii) continuous trajectories enable graceful degradation for low NFEs and efficient distillation. These establish continuous dLMs as a promising paradigm for efficient language generation.

1. Introduction

Sigma addresses the open question of whether continuous diffusion language models can scale to complex conditional reasoning by introducing large-scale models trained and sampled in a blockwise fashion. It reports competitive math and coding performance and identifies steering and low-NFE behavior as distinctive properties of continuous dLMs.

1. Introduction

  • Prior continuous text-generation methods were below 2B or unconditional, leaving their ability to scale to complex conditional reasoning unresolved.The introduction contrasts global updates to continuous block embeddings with semi-autoregressive token commitment in discrete dLMs.Source: §1 · p. 1
  • Sigma scales continuous diffusion language modeling to 3B and 8B parameters, using blockwise likelihood optimization over 16-dimensional embeddings and warm-starting from pretrained AR weights.Its inference recipe identifies embedding-space CFG and score temperature as important for reasoning and coding.Source: §1 · p. 1
  • Sigma is competitive with state-of-the-art discrete dLMs and AR models across base and instruct benchmarks, including standard math and coding tasks and harder reasoning after supervised fine-tuning.The reported evaluations include GSM8K, Minerva, HumanEval, MBPP, MATH-500, and AIME.Source: §1 · pp. 1–2
  • Embedding-space classifier-free guidance and score temperature tune the quality–diversity trade-off, while regularization and trajectory distillation support generation at low NFEs.The paper reports strong quality down to 16 NFEs per block and improved performance at 4–8 NFEs after distillation.Source: §1 · p. 2

2. Background

The background presents continuous dLMs as Gaussian diffusion over learned, low-dimensional token embeddings, with a denoiser that predicts vocabulary distributions. It also describes the training objective, noise schedule, learned-embedding distinction, and self-conditioning.

2. Background

  • Plaid-style continuous dLMs map tokens to learned low-dimensional embeddings, add variance-preserving Gaussian noise, and use categorical predictions to parameterize the reverse Gaussian process.The denoiser predicts a vocabulary distribution at each position, whose mean embedding prediction is used in the reverse process.Source: §2 · pp. 2–3
  • The negative evidence lower bound combines prior KL, reconstruction, and diffusion terms, with the predicted mean embedding corresponding to the denoiser’s categorical output.The background identifies the prior as a standard Gaussian.Source: §2 · p. 3
  • The learned noise schedule uses endpoints to optimize the objective and an interior shape to control Monte Carlo variance.The diffusion term is invariant to the interior shape, while the endpoints determine it.Source: §2 · p. 3
  • Unlike fixed-encoding continuous dLMs, Plaid likelihood-trains its embeddings; the cited work reports worse performance without learned embeddings.Source: §2 · p. 3
  • Self-conditioning feeds the denoiser its prior clean-embedding estimate; training uses it for 25% of samples, adding about 8% non-embedding FLOPs, while inference adds no overhead.The training pass without gradients produces the estimate used as input to the gradient-tracked pass.Source: §2 · p. 3

3. Large Language Continuous Diffusion Models

Sigma models the likelihood blockwise, training continuous diffusion alongside an autoregressive stream. The paper aligns its pretraining setup with discrete baselines and finds that AR initialization and regularization improve the training path and final model quality.

3.1. Blockwise Probabilistic Model

  • The blockwise model factorizes sequence likelihood across contiguous blocks, denoising each block conditioned on its clean prefix.The training figure shows separate clean-token AR and noisy-embedding diffusion streams.Source: §3.1 · p. 3, Figure 2 · p. 4
  • Training samples reconstruction or diffusion objectives per block using an unbiased estimator; RePlaid sets the reconstruction probability to 0.1 to minimize objective variance.The schedule endpoints optimize the likelihood objective, while its interior shape targets Monte Carlo variance.Source: §3.1 · pp. 3–4

3.2.1. Unified Pre-training of Continuous and Discrete dLMs

  • For controlled comparison, Sigma’s data, architecture, and optimization are aligned with discrete dLMs; its transformer additions account for less than 1% of parameters.The models train for 150K steps, corresponding to over 300B tokens at batch size 512.Source: §3.2 · p. 4, §3.2.1 · pp. 4–5

3.2. Pre-Training

  • Pretraining investigates AR initialization and regularization to address whether discrete-model weights transfer effectively to continuous representations.The section first establishes an aligned baseline before testing initialization and regularization.Source: §3.2 · p. 4

3.2.2. AR Initialization and Regularization

  • For self-conditioned Sigma-3B (L′=64), AR initialization from a 1T-token-pretrained baseline significantly accelerates convergence and lowers final training loss; tuned AR regularization improves stability and can outperform pure perplexity optimization for the diffusion stream.Source: §3.2.2 · p. 5

4. Language Generation via Continuous Diffusion Sampling

Sigma combines blockwise continuous diffusion sampling with inference-time trajectory steering, then evaluates its reasoning and coding performance against discrete diffusion models. Its results show competitive benchmark performance, with CFG and score temperature jointly strengthening steering and pass@k performance improving as rollout budgets grow.

4. Language Generation via Continuous Diffusion Sampling

  • Sigma evaluates continuous diffusion samplers on standard math-reasoning and coding benchmarks, using DDPM sampling with CFG, score temperature, and self-conditioning for its 256-step diffusion evaluation.That Sigma-3B evaluation uses 256 steps per block, an EDM rho=7 schedule, CFG strength 2, and score temperature 0.1.Source: §4 · p. 5, §4.3 · p. 7

4.1. Block diffusion sampling

  • Blockwise sampling denoises one noisy block at a time while its positions attend bidirectionally within the block and causally to the full clean prefix.The clean prefix uses causal attention and cached key-values; after a block is decoded, its committed tokens are appended to that cache for the next block.Source: §4.1 · p. 6

4.2. Trajectory Steering

  • Combining CFG and score temperature yields larger gains than either alone, and embedding-space steering performs better than logit-space sharpening.CFG and score temperature commute and can be combined into one operation; score temperature adds no model query, while packed CFG has less than 5% per-forward-pass overhead at block size 32 and 64.Source: §4.2 · pp. 6–7, Figure 5 · p. 7

4.3. Mathematical Reasoning and Coding Performance

  • Sigma-3B's diffusion run is comparable to Nemotron-Labs-Diffusion-3B on average, sets the best base-model results on GSM8K, HumanEval, and HumanEval+, and exceeds LLaDA-8B and Dream-7B by 22.6 and 5.6 average points, respectively.The cited tables cover Sigma-3B and Sigma-8B base-model math and coding performance; the passage gives no specific cell values for Sigma-8B.Source: §4.3 · p. 7, Table 1 · p. 8, Table 2 · p. 8
  • On MBPP, Sigma overtakes NLD at k=32 and reaches 91.8% versus 87.6% at k=256.The study sweeps CFG and score temperature for Sigma and confidence threshold and sampling temperature for NLD, reporting each model's best tested configuration at each budget.Source: §4.3 · p. 8

5. Few-Step Generation

The few-step study compares samplers and time discretizations, tests embedding regularization, and distills a many-step trajectory into a student. DDIM improves low-NFE performance, while SUSReg and parallel decoding distillation improve few-step and distillation results.

5.1. ODE Sampler and Time Discretization

  • DDIM substantially outperforms DDPM at low NFEs while matching it at high NFEs, and uniform-γ discretization performs best across a wide range of budgets.The comparison spans budgets from 4 to 256 steps and reports the best CFG-score-temperature pair for each sampler, discretization, and budget.Source: §5.1 · pp. 8–9

5.2. Embedding Regularization

  • Sigma-SUSReg-3B improves few-step and distillation results over Sigma-3B with nearly no validation-PPL penalty.During training, embeddings collapse to about four effective principal-component dimensions; SUSReg penalizes deviation of the embedding distribution from uniformity on the unit sphere.Source: §5.2 · p. 9

5.3. Parallel Decoding Distillation for Few-step Generation

  • Parallel decoding distillation is most helpful at 4 diffusion NFEs, and pre-training with SUSReg improves PDD performance.Because parallel student heads cannot use recurrent computation, the teacher is distilled without self-conditioning; the student is trained to match the two-call guided teacher in one conditional call.Source: §5.3 · p. 9

6. Supervised Fine-Tuning

Sigma-Instruct is produced by supervised fine-tuning the pre-trained base model on assistant-response data using the pre-training objective. Its average performance is on par with NLD at both model scales, and it surpasses Dream and LLaDA at 8B.

6. Supervised Fine-Tuning

  • Supervised fine-tuning uses the pre-training objective and restricts training data to assistant responses.The data processing and optimization recipe, along with continuous-diffusion-specific controls, are detailed in Appendix H.Source: §6 · p. 10
  • Sigma-Instruct's average performance is on par with NLD at both 3B and 8B, and exceeds Dream and LLaDA at 8B.For each sampler-step combination, the evaluation reuses the best CFG-score-temperature configuration found in earlier sections.Source: §6 · p. 10, Table 3 · p. 10, Table 4 · p. 10
  • The sampling configurations tuned before fine-tuning are reused across sampler-step combinations, and the reported results indicate they are reasonably robust to fine-tuning and scale.Source: §6 · p. 10

7. Conclusions

The paper concludes that Sigma provides a blockwise continuous diffusion language model and a reverse-diffusion sampling recipe. It reports competitive same-size masked-dLM performance and non-trivial 4/8-step results when embedding regularization and distillation are adapted to Sigma.

7. Conclusions

  • Sigma combines blockwise continuous diffusion, continually pre-trained from an AR checkpoint, with a reverse-diffusion recipe using DDPM or DDIM, CFG, score temperature, and self-conditioning.Source: §7 · p. 10
  • In diffusion mode, Sigma performs comparably to same-size Nemotron-Labs-Diffusion under an aligned pipeline, with generally similar conclusions after supervised fine-tuning.Source: §7 · p. 10
  • Adapting sphere-normalized embedding regularization and few-step distillation to Sigma yields non-trivial 4/8-step results.Source: §7 · p. 10

A. Related Work

Continuous dLMs diffuse over token embeddings, autoencoded latents, or statistical manifolds, while related approaches also reach discrete data through dequantization or Gaussian–discrete duality. Sigma belongs to the token-embedding family and distinguishes itself through large-scale likelihood training and inference-time embedding-space steering.

A. Related Work

  • Continuous dLMs lift tokens into Euclidean space for Gaussian or Riemannian diffusion, and can reuse image-diffusion tools such as ODE solvers, guidance, and distillation.Methods differ in whether diffusion acts on token embeddings, autoencoded latents, or simplex and other statistical-manifold representations.Source: A. Related Work · p. 20
  • Sigma follows the learned token-embedding and variational-diffusion approach of Plaid and RePlaid, rather than latent, simplex, or fixed one-hot representations.Related large models scale through different routes, including autoencoded latents, flow maps, bitwise diffusion heads, and adaptation of autoregressive backbones.Source: A. Related Work · p. 20
  • Sigma is, to the authors’ knowledge, the first likelihood-trained continuous dLM at 3B/8B and the first evaluated head-to-head with a matched masked dLM on conditional reasoning and code.Earlier continuous methods remained below 2B or focused on unconditional generation, leaving complex conditional reasoning open.Source: A. Related Work · pp. 20–21
  • For conditional guidance, prior continuous dLMs extrapolate prompt-conditioned and unconditional predictions, while Sigma extrapolates predicted clean embeddings or scores using the clean prefix as its condition.Sigma’s unconditional branch drops the prefix, and its steering is an inference-time operation requiring no training-time prompt dropout.Source: A. Related Work · p. 21
  • Sigma’s block-causal design couples autoregression across blocks with bidirectional Gaussian denoising within each block, paralleling causal-frame video diffusion approaches.The paper identifies per-block noise schedules, rolling KV caches, and distribution-matching distillation as techniques that transfer naturally from that work.Source: A. Related Work · p. 21

B. Limitations and Future Work

The paper identifies speed, decoder robustness, memory use, and reward-driven methods as limitations or open directions for continuous dLMs. It also reviews masked dLMs, which iteratively unmask tokens and can factor generation across blocks.

B. Limitations and Future Work

  • At high step counts, continuous dLM decoding remains slower per generated token than optimized autoregressive baselines, while DDIM-4/8 drops substantially relative to DDPM-256.Higher-order solvers and commit-stride sampling do not fully close the gap; preliminary parallel decoding distillation gains are encouraging, but self-conditioning distillation remains open.Source: B. Limitations and Future Work · p. 22
  • Diffusion ELBO training can collapse embedding geometry, leaving the readout decoder hypersensitive to continuous perturbations, especially for out-of-distribution latents.The paper suggests auxiliary cross-entropy across diffusion timesteps or VAE-style decoder pre-training as possible mitigations.Source: B. Limitations and Future Work · p. 22
  • Blockwise continuous diffusion repeatedly updates and queries KV states across noise steps, increasing peak memory-bandwidth demand relative to standard single-pass autoregressive caching.Persistent cache management, state reuse, and extension to contexts beyond 32k tokens remain systems challenges.Source: B. Limitations and Future Work · p. 22
  • Reward-driven inference steering and post-training remain largely unexplored for continuous dLMs, including whether diffusion reinforcement-learning methods transfer to language-verifier rewards.Proposed directions include reward-model guidance, sequential Monte Carlo over the denoising SDE, and verifier-directed post-training.Source: B. Limitations and Future Work · p. 22
  • The paper presents joint multimodal continuous trajectories as a future direction for connecting text with image, video, or audio generation.It notes that the probability-flow ODE is well-defined in its continuous formulation and points to trajectory-matching techniques as a possible route.Source: B. Limitations and Future Work · p. 22

C. Additional Background: Masked Diffusion Language Models

  • Masked diffusion corrupts tokens by independently replacing them with a mask token, then reverses the process by unmasking positions that remain fixed once revealed.Confidence-based samplers choose which positions to reveal, while ancestral sampling makes independent unmasking decisions at each step.Source: C. Additional Background: Masked… · pp. 23–24
  • Block masked dLMs denoise one block conditioned on its clean prefix, factorizing the likelihood autoregressively across blocks and bounding each factor.NLD instead shares one sampled noise time across blocks and normalizes its training loss by the number of masked tokens; the paper notes this is not a per-token likelihood bound and overstates per-token NELBO by roughly a factor of two.Source: C. Additional Background: Masked… · pp. 24–25
  • Masked dLMs have scaled both from scratch and by adapting pretrained autoregressive checkpoints into blockwise models that preserve KV caching across blocks.The review describes LLaDA-8B in the from-scratch line and systems including Nemotron-Labs-Diffusion, FLARE, and SDAR in the adaptation line.Source: C. Additional Background: Masked… · p. 25

D. Training and Inference Algorithms

Sigma’s algorithms train a blockwise diffusion model with a likelihood objective and sample blocks using guidance, score temperature, and optional self-conditioning. The accompanying details cover optimization, inference cost, and regularization of the learned embedding geometry.

D. Training and Inference Algorithms

  • Training uses one forward pass to denoise all blocks, with Gaussian-corrupted embeddings, per-block noise levels, and a block-causal mask that exposes each block’s clean prefix.The objective combines block-diffusion NELBO with optional autoregressive loss and SUSReg; the diffusion schedule’s endpoints tighten the bound while its interior shape is trained to reduce estimator variance.Source: D. Training and Inference Algorithms · p. 26
  • Conditional and unconditional guidance branches are packed into one forward pass, adding slightly more wall-clock time per step at batch size one.Turning off CFG, score temperature, or self-conditioning recovers a strictly weaker sampler in the reported comparison.Source: D. Training and Inference Algorithms · pp. 26–27
  • Inference samples each block from Gaussian noise through CFG- and score-temperature-adjusted predictions, with optional self-conditioning and commit stride.For generation length T_gen, stride S, and N denoising steps, the algorithm uses approximately (N + 1)/S effective forward evaluations per committed token.Source: D. Training and Inference Algorithms · p. 27

E. Implementation Details and Additional Results for Pre-training

  • The implementation uses Ministral-style 3B and 8B transformer configurations and separate optimizer settings for the diffusion embeddings and schedule versus other parameters.The 3B and 8B configurations have 3.8B and 8.5B total parameters, respectively; optimization uses a 150K-step warmup-stable-decay learning-rate schedule.Source: E. Implementation Details and… · p. 28
  • The flagship 3B and 8B models use diffusion loss weight 0.3 after 200B tokens, a configuration reported to give the best convergence for both diffusion and autoregressive losses.The paper reports that with diffusion weight 1.0, adding autoregressive regularization improved diffusion perplexity.Source: E. Implementation Details and… · p. 28

E.1. Sketched Uniform Spherical Regularization (SUSReg)

  • SUSReg regularizes the distribution of learned unit-sphere embeddings by comparing random one-dimensional projections with the uniform-sphere target.For 3B pre-training, it uses 512 directions and 17 quadrature points; its computation is described as negligible relative to a transformer forward pass.Source: §E.1 · pp. 28–29

E.2. Ablation Study: MDLM Initialization for Pre-training

The initialization ablation compares Sigma-3B trained from autoregressive versus masked-diffusion checkpoints under the same 300B-token recipe. Masked-diffusion initialization wins slightly on average in the reported configurations, but the direction varies by task.

E.2. Ablation Study: MDLM Initialization for Pre-training

  • The ablation holds the 300B-token training recipe fixed and changes the warm start from the AR checkpoint to Nemotron-Labs-Diffusion-3B.Both trained models are evaluated with self-conditioning at the DDPM and DDIM CFG-ST configurations; reported values average ten sampling lanes with two standard errors.Source: §E.2 · p. 29
  • MDLM initialization has a small, consistent average win at both CFG-ST configurations, but the change is task-dependent rather than uniform.MBPP and MBPP+ favor MDLM initialization, while HumanEval and HumanEval+ show the opposite trend.Source: §E.2 · p. 29, Table 5 · p. 30

F. Implementation Details and Additional Results for Likelihood-based Benchmarks

The appendix details likelihood-based scoring and inference evaluation: a count-based estimator is presented as a lower-variance alternative, alongside benchmark protocols and decoding settings.

F. Implementation Details and Additional Results for Likelihood-based Benchmarks

  • Estimator (19) is a Rao–Blackwellized, lower-variance alternative to (18), avoiding zero-mask cases and stabilizing estimates when few tokens are masked.Its finite mask count also permits stratified sampling so each batch covers all masking rates.Source: F. Implementation Details and… · p. 32
  • In early experiments, estimator (19) worked better than (18); for MMLU, single-token candidates make (19) exact in one forward pass.For Nemotron-Labs-Diffusion, the method uses 128 low-discrepancy draws and random mask sets.Source: F. Implementation Details and… · p. 32
  • Length generalization assumes bidirectional attention over responses generalizes beyond the training block length, while prompts use causal attention.The attention map follows the conditional branch in Fig. 4a.Source: F. Implementation Details and… · p. 32

G. Implementation Details and Additional Results for Inference

  • Table 8 reports low-NFE Pass@1 for Sigma-3B with DDIM and SUSReg; DDIM uses SC, whereas PDD does not.Guidance settings are tuned on six-task average for DDIM-256 and on three tasks for other runs except PDD.Source: Table 8 · p. 33

G.1. Reasoning and Coding Benchmark Setup

  • GSM8K and Minerva use 8- and 4-shot budgets; HumanEval variants are 0-shot, and MBPP variants are 3-shot, with task-specific exact-match, equivalence, or pass@1 scoring.Generation budgets are 256 tokens for GSM8K and 512 for the other five tasks.Source: §G.1 · p. 34
  • Baseline decoding differs by model: NLD and Sigma AR modes use greedy argmax, while masked diffusion baselines use confidence-based or highest-confidence unmasking.NLD diffusion uses block size 32 and the better of thresholds 0.8 and 0.9 per task; scoring is consistent across base models and generation modes.Source: §G.1 · pp. 33–34

G.2. Sweeping Classifier-Free Guidance and Score Temperature

  • The appendix provides full CFG and score-temperature sweeps for Sigma-3B models trained with and without AR regularization.The supplied passage identifies the sweep tables but does not report their outcomes.Source: §G.2 · p. 34

G.3. Ablation Study: Steering in Logit Space Instead of Embedding Space

This section examines Sigma’s steering and inference choices through ablations, alongside analyses of reasoning performance, inference cost, and related properties. It also documents the weight-decay settings used for supervised fine-tuning.

G.3. Ablation Study: Steering in Logit Space Instead of Embedding Space

  • Embedding-space CFG remains the default: logit-space guidance does not outperform it generally, and logit CFG adds negligible benefit at w=2 before harming accuracy at larger weights.Logit temperature can compose with score temperature, but adding it when CFG and score temperature are already active harms accuracy.Source: Table 9 · p. 35, §G.3 · p. 36, Table 11 · p. 36

G.4. Ablation Study: Choosing the Bad Prediction for CFG with Self-conditioning

  • The fully unconditional bad prediction almost wins every matched CFG-weight comparison and achieves a higher tuned ceiling on both GSM8K and HumanEval across both samplers.The comparison tests removing both clean context and self-conditioning against retaining context while removing self-conditioning.Source: §G.4 · pp. 37–38, Table 12 · p. 37, Table 13 · p. 37

G.5. Ablation Study: Commit-Stride Sampling

  • At nominal NFE 256, DDIM with stride 8 beats tuned DDPM-256 by +2.66 average points on Sigma-3B and +0.76 on Sigma-8B; smaller strides help down to about nominal NFE 64.Below roughly 16 steps per window, the coarse time grid increases discretization error and can reverse the smaller-stride benefit.Source: §G.5 · p. 38, Table 14 · p. 39, Table 15 · p. 40

G.6. Ablation Study: Time Discretization

  • Uniform-γ and EDM ρ=7 perform similarly at high NFE, while uniform-γ generally performs best for low-NFE sampling.Source: §G.6 · pp. 38–39

G.7. Higher-order ODE Solvers

  • None of the tested higher-order solvers improves low-NFE accuracy over DDIM, and ρAB2-DEIS and ρRK-DEIS become especially unstable at 16 steps.At 8 steps, DDIM’s mean off-sphere distance is 0.014, versus 1.29 for ρAB1-DEIS and 26.2 for DPM-Solver++(2M); higher-order extrapolation can strand endpoints off the embedding manifold.Source: Table 16 · p. 40, §G.7 · pp. 41–42

G.8.2. Base Model: Minerva Geometry

  • At matched pass@1 on Minerva Geometry, Sigma tracks best-tuned NLD within 0.007 across pass@k, while gaining 1.9 to 6.6 points at pass@256 against lower-temperature NLD.In the unconstrained comparison, Sigma’s best-of-grid lead grows from 0.1 point at k=8 to 1.3 points at k=512; the fixed comparison’s confidence interval spans zero.Source: §G.8.2 · p. 43, Table 17 · p. 43, Table 18 · p. 43

G.10. Wall-clock Cost of the Packed CFG Forward

  • Packed CFG adds +1.8% transformer-forward cost at query length 32 and is within measurement noise at length 64; per-NFE overhead comes mainly from logits and embedding operations.The measurements use batch size 1 on an H100; the extra attention work is hidden under the memory-bandwidth floor.Source: Table 20 · p. 45, §G.10 · p. 45

G.11. Proofs

  • CFG and score temperature commute in the mean prediction, and the EDM-ρ discretization converges pointwise to the uniform-γ discretization as ρ→∞.The proof uses affine correspondence between mean predictions and scores.Source: §G.11 · pp. 45–46

G.8.1. Base Model: MBPP

  • At pass@256 on MBPP, one fixed Sigma configuration reaches 0.918 versus 0.876 for NLD; the best-of-grid curves show Sigma overtaking NLD at k=32.At selected fixed settings, Sigma repeats 10.4% of output strings versus 53.1% for NLD, while producing 71.4 distinct correct strings versus 46.9.Source: §G.8.1 · p. 42, Figure 16 · p. 51

G.8.3. Base Model: GSM8K

  • On GSM8K, Sigma’s fixed configuration leads NLD by +0.7 points from k=32 through k=256, solving 1314 of 1319 problems at least once versus NLD’s 1305.NLD leads at small k, the curves cross near k=16, and Sigma’s paired advantage is significant at every k≥32.Source: §G.8.3 · p. 43, Figure 18 · p. 52

G.8.4. Instruct Models: AIME 2025

  • +1.7 points at pass@1 rising to +10.0 at pass@256, Sigma’s best instruct setting leads NLD’s best on AIME 2025, though only 1 of 27 paired comparisons excludes zero.At pass@256, four of five Sigma settings exceed every NLD setting and the fifth ties NLD’s best at 0.533.Source: §G.8.4 · p. 44, Figure 19 · p. 52

G.9. Reasoning Trajectory Analysis

  • Sigma tends to commit early positions of a block later than MDLM and its final positions earlier.The analysis uses 256-step DDPM and defines commitment by a stable final-token argmax assigned probability of at least 0.9.Source: §G.9 · p. 44, Figure 20 · p. 53

H.1. Supervised Fine-Tuning Setup

  • SFT initializes from pretrained Sigma checkpoints and trains the transformer and heads while freezing the diffusion schedule, endpoints, and embedding; an ablation unfreezes them with a 1000× learning-rate multiplier.Prompts are left-padded so answers begin on the 32-token block grid; the joint objective combines continuous block NELBO and next-token cross-entropy.Source: §H.1 · pp. 54–55
  • Weight decay is zero for the schedule network and diffusion embedding matrix, while the two scalar endpoints retain the base weight decay of 0.1.These settings are used under the same scheduler.Source: §H.1 · p. 55

H.2. Evaluation Setup and Frozen-Parameter Ablation

The section reports the instruct evaluation protocol and a frozen-versus-unfrozen ablation, then describes parallel decoding distillation (PDD), its training objective, and few-step inference. PDD reduces sequential denoiser evaluations while retaining fine-grid DDIM updates.

H.2. Evaluation Setup and Frozen-Parameter Ablation

  • The instruct evaluation covers ten tasks, uses two sampler operating points, and scores math, coding, multiple-choice, and instruction-following outputs with task-specific procedures.Baselines are mainly copied from Fu et al. (2026); NLD-8B diffusion results are reproduced under an aligned protocol.Source: §H.2 · p. 55
  • Under DDPM-256, frozen and unfrozen configurations are within the seed interval on seven of ten tasks; under DDIM-32, the average ordering reverses.The configurations are generally close, so the frozen configuration remains the simpler 8B default.Source: §H.2 · p. 55, Table 22 · p. 56

I. Implementation Details and Additional Results for Parallel Decoding Distillation

  • PDD partitions a fine DDIM grid into fewer segments, emitting multiple clean-embedding predictions per transformer evaluation to reduce expensive denoiser evaluations.Each segment spans Δ = G/M fine intervals for a target budget M below G.Source: §I.1 · p. 57

I.1. Parallel Predictions and Intra-Segment Rollout

  • Within each PDD segment, predictions are composed by ordinary DDIM updates; only the endpoint is passed to the next transformer evaluation.Parallelism applies to denoiser predictions, not the inexpensive state updates within a segment.Source: §I.1 · p. 57

I.2. Residual Parallel Heads

  • PDD residual heads transform the pretrained diffusion readout, and zero-initialized final layers make each head initially reproduce that anchor prediction.The additive residual can also leave the convex hull of token embeddings to match extrapolated CFG-guided teacher predictions.Source: §I.2 · pp. 57–58

I.3. Distillation Objective and Optimization

  • The student receives only the conditional prefix, allowing one student call to amortize the teacher’s conditional and unconditional evaluations.The teacher remains frozen during distillation.Source: §I.3 · p. 58
  • The distillation objective matches raw student predictions to CFG-guided teacher predictions at states generated by the current student within each segment.The segment start comes from the forward marginal, while later supervised states are student-generated; ST affects rollout states but not the regression target.Source: §I.3 · p. 58

I.4. Few-Step Inference

  • At target budget M, the student evaluates at M segment starts and retains fine-grid DDIM integration between those evaluations.Only each segment endpoint is carried forward to the next transformer evaluation.Source: §I.4 · p. 59

I.5. Training and Parameterization Details

The section gives PDD training and parameterization details, reports its few-NFE results and a SUSReg ablation, and presents a Sigma-Base-3B worked example.

I.5. Training and Parameterization Details

  • The student uses a shared bank of 32 residual heads and is trained flexibly at NFE budgets 4 and 8.Training uses a 512-token clean context and a 32-token signal block.Source: §I.5 · p. 59
  • PDD delivers 2× acceleration from NFE-8 to NFE-4 relative to the self-conditioning-enabled Sigma-3B-SUSReg model, with 68% GSM8K flexible-extract accuracy at NFE=4.Coding remains more NFE-sensitive: NFE-4 pass@1 falls short of performance at NFE>16.Source: §I.5 · pp. 59–60
  • A SUSReg-regularized checkpoint improves distillation by 4.67pp at NFE-8 and 3.71pp at NFE-4.The authors identify embedding-geometry regularization as important in this few-NFE setup.Source: §I.5 · p. 60

J.1. Sigma-Base-3B

  • A Sigma-Base-3B example solves a fish-count word problem by adding 22 red-striped fish to 20 blue-striped fish, giving 42.The blue-striped count is computed from the 44 fish remaining after the red-striped fish are counted.Source: §J.1 · p. 61

J.2. Sigma-3B-Instruct

The section presents Sigma-3B-Instruct example outputs across math and science questions, including worked solutions and final answers.

J.2. Sigma-3B-Instruct

  • In the AIME 2024 lottery example, the model gives 116 for the sum of the numerator and denominator of the conditional probability.It counts 115 prize-winning outcomes and one grand-prize outcome, yielding probability 1/115.Source: §J.2 · p. 62
  • In the AIME 2025 geometry example, the model concludes that triangle BGE has area 468.The solution uses base 39 and height 24.Source: §J.2 · pp. 62–63
  • For the GPQA impulse-approximation question, the model selects assumptions 1, 2, and 4, answering D.Its explanation says the non-relativistic-quarks assumption is not relevant to the approximation.Source: §J.2 · p. 63
Loading 2610.02665v1…