PaperlayerView exact revision on arXiv

Source-linked AI summary

Loopholing All the Way: Language Modeling with pure Self-Conditioning

Adrien Deschenaux, Nadezhda Ilieva, Justin Deschenaux, Manuel Madeira, Dorina Thanou, Aurélien Lucchi, Antonio Orvieto, Caglar Gulcehre, Pascal Frossard

arXiv:2610.30068v1cs.LG

TL;DR

Discrete diffusion can lose sample quality in few-step generation and discard uncertainty from previous predictions, while Self-Conditioning has remained auxiliary. LAW uses the Expected Embedding as its propagated state and decodes tokens only at the final step, with reported gains across reasoning tasks and competitive OpenWebText results.

  • Problem

    Discrete diffusion's independent token sampling can hurt few-step sample quality and discarding previous predictions loses associated uncertainty, motivating the question of whether a continuous generative model can be based purely on Self-Conditioning.

    Source: §1 · p. 2

  • Method

    LAW propagates the Expected Embedding under the denoiser's prediction as its only state and decodes tokens once, at the final sampling step.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    LAW is most accurate on Sudoku and Maze with few steps, outperforms same-FLOPs diffusion and flow baselines on GSM8K, and raises fine-tuned Duo's GSM8K accuracy from 17.2% to 48.1%.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The results suggest that pure Self-Conditioning can match or outperform Self-Conditioning used as an auxiliary input.

    Source: §6 · p. 10

    Whether the results hold at larger scale remains open because experiments use the standard academic setting.

    Source: §6 · p. 10

Abstract

from arXiv · show

Recent Diffusion Language Models (DLMs) use Self-Conditioning (SC), or Loopholing, to improve their performance. While SC is traditionally seen as an auxiliary heuristic, its empirical success raises the question: can a continuous generative model be based purely on Self-Conditioning? We answer affirmatively with LAW (``Loopholing All the Way''), which propagates the Expected Embedding under the denoiser's prediction across steps, decoding the tokens once, at the end. On Sudoku, LAW outperforms previous Discrete and Continuous Diffusion Models, including Loopholing Discrete Diffusion. On Maze, LAW is the most accurate from 2 to 64 NFE. After training on TinyGSM and sampling at temperature 1 with 1024 NFE, LAW solves 42.2% of GSM8K problems against 34.5% for uniform Loopholing Discrete Diffusion. Additionally, fine-tuning a pre-trained Uniform Diffusion Model (Duo) with LAW increases its accuracy from 17.2% to 48.1%. Finally, on OpenWebText, LAW pushes the Pareto frontier and approaches Autoregressive Models (ARMs) with 64 NFE. Find our code at https://github.com/ilievanadezhda/law.

1 INTRODUCTION

Discrete diffusion can lose sample quality in few-step generation and discard uncertainty from previous predictions, while continuous models have struggled to surpass discrete baselines on reasoning. LAW makes Self-Conditioning the basis of a continuous generative model and performs strongly across reasoning and language-modeling benchmarks.

1 INTRODUCTION

  • Discrete diffusion models sample positions independently and discard denoiser predictions, hurting few-step sample quality and losing associated uncertainty.The paper describes the latter as information collapse.Source: §1 · p. 2
  • Self-Conditioning feeds previous denoiser predictions or hidden representations back as input, but had remained auxiliary despite empirical gains from LDDMs and DiffusionGemma.On hard Sudoku, LDDM-U solves 91.0% against Duo's 58.4%.Source: §1 · p. 2
  • LAW propagates the Expected Embedding under the denoiser's prediction and decodes tokens only at the final sampling step.Sampling starts from uniformly random tokens.Source: §1 · p. 2
  • The Expected Embedding as SC state yields 35.8% on GSM8K versus below 1.5% with Jo et al.'s (2026) final-layer hidden representation; temperature regularization raises accuracy at T = 0.1 from 24.8% to 48.8%.Temperature regularization randomly scales the denoiser's logits during training.Source: §1 · p. 2
  • 96.5% vs. 91.0% on hard Sudoku, and the most accurate model on Maze through 64 NFE; LAW also raises fine-tuned Duo's GSM8K accuracy from 17.2% to 48.1%.On OpenWebText, LAW comes close to autoregressive models at matched entropy with 64 NFE.Source: §1 · p. 2

2 BACKGROUND

Figure 2 contrasts LDDM's joint propagation of discrete tokens and an SC state with LAW's propagation of only the SC state and final-step decoding. The background introduces uniform discrete diffusion and LDDM's use of Self-Conditioning.

2 BACKGROUND

  • Figure 2 contrasts LDDM's token-plus-SC-state updates with LAW's SC-only propagation and decoding back to tokens at the end.Both schemes repeatedly apply the denoising backbone, starting from tokens drawn from the uniform prior.Source: Figure 2 · p. 3

2.1 UNIFORM DISCRETE DIFFUSION MODELS (UDMS)

  • UDMs independently corrupt positions, retaining each clean token with probability α_t or drawing uniformly; they train a denoiser with NELBO and sample backward from uniform-prior tokens.The denoiser predicts clean tokens to construct the reverse process.Source: §2.1 · pp. 3–4

2.2 LOOPHOLING DISCRETE DIFFUSION MODELS (LDDMS)

  • LDDMs (Jo et al., 2026) propagate a layer-normalized final-layer hidden state alongside sampled tokens despite a train–test mismatch; on hard Sudoku, LDDM-U solves 91.0% versus Duo's 58.4%.Their state is added to token embeddings at the next step.Source: §2.2 · p. 4

3 LOOPHOLING ALL THE WAY (LAW)

LAW makes Self-Conditioning the denoiser’s sole state, propagating Expected Embeddings instead of discrete tokens and decoding only at the final step. Its training uses multi-pass updates and randomized temperature regularization to prepare the denoiser for its own evolving states.

3 LOOPHOLING ALL THE WAY (LAW)

  • LAW replaces the discrete sequence with the Expected Embedding as the denoiser’s sole propagated state, decoding tokens only at the final step.The state is a convex combination of token embeddings under the denoiser’s prediction.Source: §3 · p. 4, §3.1 · p. 5

3.1 GENERATIVE PROCESS

  • Sampling initializes from embeddings of uniformly drawn tokens, repeatedly maps the Self-Conditioning state through the denoiser, and fixes prompt positions to clean embeddings for conditional generation.The sampling temperature is used at each step; with the argmax decoder, sampling is deterministic given the initial state.Source: §3.1 · pp. 4–5

3.2 TRAINING THE CONTINUOUS TRAJECTORY

  • Multi-pass training unrolls M passes with probability p, feeds later passes the Self-Conditioning state, and applies the loss only to the final prediction.Otherwise training uses a single pass as in standard Uniform Diffusion Model training; the objective is per-token NELBO or Cross Entropy.Source: §3.2 · p. 5
  • Randomized temperature regularization perturbs Self-Conditioning states during training while computing loss from the untempered prediction.The temperature is shared across passes and sequence tokens, and its range allows both sharpening and flattening logits.Source: §3.2 · p. 5
  • LAW uses the standard Uniform Diffusion Model denoiser architecture and can be trained from scratch or fine-tuned from a pretrained checkpoint.A learned mode embedding is used only on OpenWebText, where it prevents representation collapse when ρ = 3.Source: §3.2 · p. 5

4 EXPERIMENTS

The experiments evaluate LAW on Sudoku, Maze, GSM8K, and OpenWebText against autoregressive, discrete-diffusion, and continuous-model baselines. LAW and LDDMs use fewer training steps to keep training FLOPs similar to the baselines.

4.1 REASONING ON SUDOKU

  • At 180 NFE, LAW is most accurate on all Sudoku difficulty levels, solving 96.5% with temperature regularization (ρ = 2, p = 0.9).Training with CE performs better than with the NELBO.Source: Table 1 · p. 6, §4.1 · p. 6

4.2 PATH FINDING ON MAZE

  • At 4 NFE, LAW solves 90.9% of mazes and leads every baseline through 64 NFE; from 512 NFE, LDDM-M is slightly more accurate, with both near 92%.LDDM-U plateaus below 89%, and continuous baselines below 82%.Source: §4.2 · p. 6, Figure 3 · p. 7

4.3 REASONING ON GSM8K

  • At T = 1 and 1024 NFE, LAW solves 42.2% of GSM8K problems versus 34.5% for LDDM-U, and beats all diffusion and flow baselines from 32 NFE onward.At T = 0.1, it is most accurate from 4 to 512 NFE; at 2 to 64 NFE it also exceeds the listed few-step approaches.Source: §4.3 · p. 7
  • After 50k fine-tuning steps, LAW reaches 48.1% GSM8K accuracy at T = 1 and 1024 NFE, versus 17.2% for Duo and 42.2% for LAW trained from scratch.This fine-tuning adds 26% to Duo's training compute.Source: §4.3 · p. 7, Figure 4 · p. 8
  • Fine-tuned LAW exceeds AlphaDLM from 16 NFE onward at T = 1 for about 8% more training compute; AlphaDLM's k trades few-step accuracy against many-step performance.The authors describe AlphaDLM as complementary to LAW and leave studying their combination for future work.Source: §4.3 · pp. 7–8
  • GSM8K ablations show that the Expected Embedding state yields 35.8% versus below 1.5%, temperature regularization raises accuracy from 24.8% to 48.8%, and stop-gradient yields 42.2% versus 20.2%.The temperature-regularization comparison is at T = 0.1 with 1024 NFE; the stop-gradient comparison is reported at matched training FLOPs and equal training steps.Source: §4.3 · p. 8

4.4 LANGUAGE MODELING ON OPENWEBTEXT

  • At 64 NFE, LAW beats diffusion and flow baselines and nears the ARM; at 1024 NFE it beats the ARM at matched entropy but reaches only 5.1–5.2 entropy, versus 5.8 at 64 NFE.The comparison sweeps sampling temperatures and assesses Gen. PPL at matched entropy.Source: §4.4 · pp. 8–9, Figure 5 · p. 9

5 RELATED WORK

Prior Self-Conditioning methods use continuous states alongside discrete-token sampling, while continuous diffusion language models use ODE or SDE trajectories. LAW propagates only its Self-Conditioning state and decodes tokens once at the end.

Self-Conditioning

  • Self-Conditioning feeds prior denoiser predictions or hidden representations back as input; existing methods still sample discrete tokens between steps, whereas LAW propagates only the state and decodes once at the end.LDDMs (Jo et al., 2026) and DiffusionGemma (Google DeepMind, 2026) propagate a continuous state; Relay (Rozonoyer et al., 2026) and PUMBA (Madeira et al., 2026) pass hidden states between teacher-forced steps.Source: §5 · p. 9

Continuous Language Models

  • Unlike continuous DLMs that corrupt embeddings or one-hot vectors with Gaussian noise or use the hypersphere, LAW uses uniform discrete corruption and no Gaussian noise or ODE solver.Like those models, LAW has a continuous sampling trajectory.Source: §5 · p. 9

Looped Models

  • Looping changes computation within a denoising step, while LAW changes what passes between steps; the authors describe the approaches as complementary.Looped DLMs reuse denoiser layers to vary inference-time depth.Source: §5 · p. 10

6 CONCLUSION AND LIMITATIONS

The conclusion presents LAW as a model based purely on Self-Conditioning and reports results across Sudoku, Maze, GSM8K, and OpenWebText. It also identifies limitations in the evaluation setting and in the entropy range covered on OpenWebText.

6 CONCLUSION AND LIMITATIONS

  • LAW propagates only the Expected Embedding under the denoiser’s prediction and decodes tokens after the final step, with results suggesting pure Self-Conditioning can match or outperform its auxiliary use.The reported comparisons span Sudoku, Maze, GSM8K, and OpenWebText; fine-tuning Duo with LAW raises GSM8K accuracy from 17.2% to 48.1%.Source: §6 · p. 10
  • LAW’s evaluation uses the standard academic setting, so whether the results hold at larger scale remains future work.On OpenWebText at 1024 NFE, LAW samples cover a narrower entropy range than the baselines.Source: §6 · p. 10

AI USE STATEMENT

  • The authors report that LLMs assisted manuscript writing and code, and that the TinyGSM dataset was fully generated by GPT-3.5.They state that the main ideas were proposed without LLM use and that they take responsibility for the paper’s content, claims, citations, and conclusions.Source: AI USE STATEMENT · p. 11

A.2 TRAINING AND SAMPLING ALGORITHMS

The appendices describe LAW’s sampling and training procedures, compare them with LDDMs, and explain gradient handling and compute matching.

A.3 COMPARISON TO LOOPHOLING DISCRETE DIFFUSION MODELS (LDDMS)

  • LAW propagates only the SC state between sampling steps and decodes tokens once at the end, whereas LDDMs sample discrete tokens at each step and feed their embeddings alongside the state.The two methods also differ in how the backbone uses the state and in the state map that computes it.Source: §A.3 · p. 20
  • LAW’s multi-pass training feeds the SC state alone to later passes at successive times, while LDDMs’ later pass combines token embeddings and the state at the same time.Both methods stop gradients through intermediate SC states and apply the loss to the last pass; LAW can use more than two passes.Source: §A.3 · p. 20
  • LAW propagates a probability-weighted linear combination of token embeddings as its state, while LDDMs propagate a layer-normalized hidden representation.Source: §A.3 · p. 20

A.4 GRADIENT FLOW BETWEEN PASSES

  • With a stop-gradient between passes, the loss updates parameters through the final forward pass only; without it, gradients also flow through SC states into earlier passes.The authors use stop-gradient by default and compare the variants on TinyGSM.Source: §A.4 · p. 20

A.5 MODE EMBEDDING

  • The mode embedding distinguishes token-embedding inputs from SC-state inputs, and zero initialization lets a Duo checkpoint compute the same function after loading.The corresponding learned vectors are added to the noise-level conditioning.Source: §A.5 · p. 20

B.1 FLOP MATCHING

  • A multi-pass update with stop-gradient costs (M + 2)F, while one without it costs 3MF; matching baseline compute therefore requires fewer training steps for LAW and LDDMs.For M = 2 and p = 0.9, the average update cost is 3.9F and the runs use 77% of baseline steps.Source: §B.1 · p. 21

B.2 TRAINING STEPS

This section describes the training budgets and shared configurations used for the experiments, including model and sampler choices.

B.2 TRAINING STEPS

  • The training-step tables give per-task budgets and run configurations, with E.E. denoting Expected Embedding and Hid. denoting the LDDM hidden representation.Rows listing multiple difficulties or regularization values represent separate runs with the same number of steps.Source: §B.2 · p. 21
  • By default, LAW and LDDMs use M = 2 passes with p = 0.9, costing 3.9F per update, and train for fewer steps to match baseline FLOPs.Source: Table 2 · p. 21

C.1 SHARED SETUP

  • The shared setup specifies original-implementation defaults for baselines and distinguishes their samplers, including ancestral sampling for MDLM and Duo and Euler integration for FLM.S-FLM uses top-1 velocity and decodes with argmax, so its sampling does not depend on temperature.Source: §C.1 · pp. 21–23
  • Diffusion and flow models share a DiT backbone, with stated exceptions; LAW uses the same DiT as Duo, allowing a Duo checkpoint to load without changes.The ARM uses the same Transformer with causal attention and no time conditioning.Source: §C.1 · p. 22

C.2 SUDOKU

The experimental appendices describe the datasets, training configurations, sampling, and evaluation measures for Sudoku, TinyGSM, Maze, and OpenWebText.

C.2 SUDOKU

  • Sudoku experiments use 48k training and 2k validation puzzles with unique solutions, and report exact-match accuracy on the 2k validation puzzles.LAW and LDDMs are trained from scratch, and LAW is sampled with 180 NFE.Source: §C.2 · pp. 23, 25

C.3 TINYGSM

  • TinyGSM pairs 11.8M synthetic grade-school questions with solving programs, while GSM8K evaluation counts a program correct when execution returns the reference answer.Examples are question–program sequences, with loss applied only to the program.Source: §C.3 · p. 25

C.4 MAZE

  • Maze evaluation uses 20 × 20 grids and measures valid-solution accuracy, requiring the generated grid to preserve maze cells and mark a shortest path.The reported evaluation set contains 1,000 mazes; the target paths are independently checked for shortest-path length.Source: §C.4 · pp. 25–26

C.5 OPENWEBTEXT

  • OpenWebText evaluation uses Gen. PPL and unigram entropy to characterize sample quality and diversity, with frontiers sampled at 64 and 1024 NFE across temperatures.Gen. PPL is computed with GPT-2 Large; unigram entropy helps detect repetitive, low-entropy samples.Source: §C.5 · p. 27

D.1 SUDOKU: ABLATIONS

The Sudoku ablations compare loss, multi-pass probability, pass count, and temperature regularization on hard puzzles. Results vary by configuration, with the strongest reported CE run reaching 96.5%.

Loss

  • 96.5% of hard puzzles are solved by the best CE run, against 88.9% for the best NELBO run.CE is more accurate than NELBO in every configuration.Source: §D.1 · p. 27

Multi-Pass Probability

  • p = 0.75 is best for both losses, reaching 94.5% with CE and 88.9% with NELBO.With two passes on only 10% of updates (p = 0.1), CE solves 84.6%.Source: §D.1 · p. 28, Table 7 · p. 28

Number of Passes

  • A third pass raises CE accuracy from 92.2% to 94.5% at p = 0.9, but lowers NELBO accuracy from 85.3% to 75.4%.Source: §D.1 · p. 28, Table 8 · p. 28

Temperature Regularization

  • With the best ρ, temperature regularization raises the best CE configuration to 96.5%, from 92.2% without regularization.For NELBO, it raises M = 3, p = 0.75 from 79.6% to 86.5%, but does not improve M = 2, p = 0.75.Source: §D.1 · p. 28, Table 9 · p. 28

D.2 TINYGSM: ABLATIONS

These supplementary results report TinyGSM ablations, Duo fine-tuning behavior, few-step comparison scope, and the difference between maze valid-solution and exact-match evaluation.

D.2 TINYGSM: ABLATIONS

  • The mode embedding brings no consistent gain: at 1024 NFE it lowers accuracy at T = 1 in every configuration and changes T = 0.1 accuracy by at most 3.3 points.ρ = 4 gives results similar to ρ = 3.Source: §D.2 · p. 28, Figure 7 · p. 29
  • At T = 0.1 with 1024 NFE, M = 2 reaches 48.8% for p = 0.9 and 49.4% for p = 0.75, while configurations with more passes reach at most 43.2%.At T = 1, M = 4 with p = 0.75 reaches 43.6%, against 42.2% for M = 2 with p = 0.9.Source: §D.2 · p. 29, Figure 8 · p. 29
  • Without the stop-gradient between passes, TinyGSM accuracy at T = 1 with 1024 NFE is 20.2% for p = 0.9 and 16.1% for p = 1, against 42.2% with it.The stop-gradient run reaches 38.7% after 120k steps, versus 20.9% (p = 0.9) and 13.7% (p = 1) without it at that checkpoint.Source: §D.2 · p. 30, Figure 10 · p. 30, Figure 9 · p. 30

D.3 TINYGSM: FINE-TUNING FROM DUO

D.4 TINYGSM: COMPARISON WITH FEW-STEP METHODS

  • Table 10 compares GSM8K accuracy for methods evaluated from 1 to 64 NFE on TinyGSM.Its caption identifies the methods and sampling conventions but supplies no cell values in this passage.Source: §D.4 · p. 32, Table 10 · p. 33

D.5 MAZE: EXACT MATCH VS. VALID SOLUTION

  • For S-FLM (S-arch.) at 128 NFE, valid-solution accuracy is 77.7%, while exact-match accuracy is 58.3%; many correct paths differ from the target.Every exact match is also a valid solution, so exact-match accuracy cannot exceed valid-solution accuracy.Source: §D.5 · p. 33, Figure 14 · p. 34, Figure 15 · p. 34

D.6 MAZE: ABLATIONS

The maze ablations find configuration-dependent effects of loss, pass count, and temperature regularization. OpenWebText results describe when regularization and mode embeddings stabilize training and how LAW compares with LDDMs.

D.6 MAZE: ABLATIONS

  • With NELBO, p = 0.75 gives the highest maze accuracy, 91.7% at 800 NFE; NELBO beats CE at every p except p = 0.9, where both solve about 90%.Source: §D.6 · p. 34, Figure 16 · p. 35
  • A third pass does not improve final maze accuracy overall; with CE and p = 0.9, M = 3 reaches 85.1% versus 90.0% for M = 2 at 800 NFE.With NELBO, M = 3 helps at 2 NFE (77.8% versus 65.2%) but not at many steps; with p = 0.75 it is below M = 2 for NELBO and close for CE.Source: §D.6 · p. 34, Figure 17 · p. 35
  • Temperature regularization does not consistently improve final maze accuracy; the best reported configuration reaches 92.2% at 800 NFE, versus 91.7% without regularization.The best configuration is NELBO, M = 2, p = 0.75, ρ = 2.Source: §D.6 · p. 34, Figure 18 · p. 36

D.7 OPENWEBTEXT: ABLATIONS

  • On OpenWebText, the ρ = 3 run collapses without the mode embedding: hidden-state correlation reaches 1 and entropy falls close to zero; adding the embedding prevents this collapse.The ρ = 4 run remains stable even without the embedding.Source: §D.7 · p. 37, Figure 20 · p. 38

D.8 OPENWEBTEXT: COMPARISON TO LDDMS

  • With temperature regularization, every non-collapsed LAW run has lower Gen. PPL than all LDDMs at both NFE settings; at 64 NFE, the best model uses ρ = 3 and the mode embedding.Without regularization, LAW reaches low entropy and higher Gen. PPL than LDDM-U; the Expected Embedding alone does not explain LAW’s gain over LDDMs.Source: §D.8 · p. 38, Figure 21 · p. 39

D.9 OPENWEBTEXT: SAMPLES

The section presents unconditional 1,024-token samples from LAW at T = 1, with reported Gen. PPL and unigram entropy for examples at two NFE settings.

D.9 OPENWEBTEXT: SAMPLES

Related papers

Loading 2610.30068v1…