PaperlayerView exact revision on arXiv

Source-linked AI summary

From Truncation to Commitment: Persistent Context in Uniform Discrete Diffusion

Satoshi Hayakawa

arXiv:2609.01043v1cs.LGcs.AImath.PRstat.ML

TL;DR

Uniform-state diffusion revises every position, so top-p decisions do not persist as explicit context across reverse steps. The paper introduces CRS, which stores selected argmax tokens for later denoiser inputs, and reports lower GenPPL than fixed top-p baselines across 8–64 NFE under a shared finalization rule. Its theory motivates warmup and persistent visibility, while experiments identify support restriction and persistent context as distinct controls of the GenPPL–entropy tradeoff.

  • Problem

    Top-p can make a current reverse-step decision without carrying its selected label into later denoiser inputs, motivating study of persistent context in revisable uniform-state diffusion.

  • Method

    CRS is a training-free sampler that stores selected argmax clean-token hypotheses and inserts them into later denoiser inputs while leaving other positions revisable.

  • Results

    CRS without top-p truncation has lower GenPPL than fixed p=0.95 and p=0.9 baselines across 8–64 NFE on Duo-distilled, with matched-entropy comparisons at 64 NFE also favoring CRS.

  • Takeaways & Limitations

    Support restriction and persistent context are distinct controls of the GenPPL–entropy tradeoff, with theory supporting later selection and keeping selected tokens visible.

Abstract

from arXiv · show

Uniform-state discrete diffusion models update all tokens in parallel while keeping every position revisable. Even when the commonly used top-$p$ rule leaves only one candidate at a position, that choice affects only the current reverse step and can be revised at the next sampling step. We ask what changes when selected hypotheses instead become persistent context for later predictions. We therefore propose committed reveal sampling (CRS), a training-free sampler that stores selected argmax tokens and inserts them into subsequent model inputs. Our analysis gives a rationale for selecting later and for keeping selected tokens visible. Under the exact forward process, the Bayes error of selecting a clean token cannot increase as noise decreases, while in a simple latent-mode model, keeping the selected token visible helps later parallel predictions agree on the same sequence-level choice. Empirically, paired experiments on Duo-distilled then separate this persistent effect from single-step top-$p$ restriction and scalar temperature scaling. Under the same finalization rule, CRS without top-$p$ truncation reaches lower generative perplexity (GenPPL) than fixed $p=0.95$ and $p=0.9$ baselines across budgets of 8--64 function evaluations (NFE). At 64 NFE, the comparison at matched unigram entropy also gives lower GenPPL for CRS, yielding a more favorable GenPPL--entropy tradeoff. Base Duo shows the same direction in a descriptive comparison, while other diversity and continuation metrics can rank these operating points differently. These results identify support restriction and persistent context as distinct controls of that tradeoff.

1 Introduction

Uniform-state diffusion keeps all positions revisable, so top-p choices constrain only individual reverse updates rather than becoming persistent context. The paper proposes CRS to store selected hypotheses and study how persistence affects sequence-level coordination and GenPPL–entropy tradeoffs.

  • Top-p can reduce clean-token support to a singleton, but the selected label is recomputed from a revisable noisy state at the next step.
  • CRS stores argmax clean-token hypotheses after warmup and inserts them into later denoiser inputs while leaving unselected positions revisable.
  • Persistent visible tokens can steer parallel predictions toward the same sequence-level alternative, although incorrect selections can steer them toward the wrong one.
  • The paper combines theoretical analysis with paired experiments measuring GenPPL and unigram entropy across compute budgets on base and distilled Duo checkpoints.
  • Across 8–64 NFE on Duo-distilled, CRS without top-p truncation has lower GenPPL than fixed p ∈{0.95, 0.9} baselines; matched-entropy comparisons at 64 NFE also favor CRS.

2 Related Work

The paper distinguishes CRS from methods that modify the learned reverse process or use masked commitment. CRS instead keeps the checkpoint and product denoiser fixed while exposing selected hypotheses as context across sampling steps.

  • CRS changes later denoiser context while leaving the checkpoint fixed, unlike joint-distribution methods that modify dependence, training, transitions, or state representation.
  • Masked diffusion makes reveal order part of sampling by turning an absorbing mask into visible clean tokens, whereas uniform-state models keep positions revisable.
  • Fast-dLLM commits confident tokens for parallel decoding and cache reuse, while deferred commitment postpones uncertain blockwise decisions.
  • LLaDA, Dream, GIDD, and Sumi extend masked, interpolating, or pure-uniform diffusion in complementary model settings to Duo’s explicitly time-indexed framework.

3 Uniform Reverse Sampling and Top-p Support Restriction

The native uniform reverse interface converts learned clean-label predictions into denoising posteriors and reverse transitions. Top-p performs hard support restriction at the clean-label interface for one step, without carrying the selected label explicitly into the next denoiser input.

  • 3.1 From the LOO law to the native reverse step: The Duo denoiser outputs a coordinatewise categorical law over clean labels, learned as an approximation to the leave-one-out conditional from other noisy coordinates.
  • 3.1 From the LOO law to the native reverse step: Bayes-rule bridges and uniform-channel composition provide the exact conditional ingredients used to form the native reverse update.
  • 3.1 From the LOO law to the native reverse step: The reverse pipeline separates the learned clean-label law, its local conversion to an ordinary denoising posterior, and the resulting reverse transition that samples the next noisy state.
  • 3.2 Top-p in the LOO distribution: Top-p sorts vocabulary probabilities and retains the smallest prefix whose cumulative probability reaches p, substituting that restricted law into the reverse transition.
  • 3.2 Top-p in the LOO distribution: At 64 NFE with p=0.9, top-p support is singleton for 27.5% of active coordinate–step pairs overall and 45.6% during the final ten reverse steps.
  • 3.2 Top-p in the LOO distribution: Even singleton top-p decisions affect only the current reverse step because the selected label is absent from the next denoiser input unless the noisy state preserves it.

4 From Point Hypotheses to Persistent Context

CRS turns selected clean-token hypotheses into persistent context while keeping other positions revisable. Its theoretical analyses motivate delayed selection and persistent visibility, and experiments separate these effects from within-step support restriction.

  • 4.1 Committed reveal sampling: CRS evaluates the denoiser on inputs containing stored labels, selects high-confidence spatially separated coordinates, and makes new argmax labels visible in subsequent evaluations.
  • 4.1 Committed reveal sampling: Argmax isolates contextual feedback from token-sampling randomness and coincides with the unique retained token whenever top-p support is a singleton.
  • 4.1 Committed reveal sampling: The shared finalization rule uses a coordinatewise argmax readout after N−1 stochastic reverse transitions, with NFE equal to N.
  • 4.2 Why persistent context can coordinate parallel updates: In a two-coordinate shared-mode example, persistent visibility makes the other coordinate match the selected value with probability 1, unlike the product law’s hybrid alternatives.
  • 4.2 Why persistent context can coordinate parallel updates: A mode-consistent visible label concentrates posterior mode weights and drives coordination error toward zero, whereas a poor label can favor the wrong mode.
  • 4.3 Why wait before selecting tokens?: Lower noise weakly reduces oracle clean-token selection error, supporting warmup before commitment, though warmup also leaves fewer later steps for selected context to influence generation.

5 Experiments

The experiments separate current-step top-p support restriction from CRS’s persistent context and evaluate their GenPPL–entropy tradeoffs under a common argmax final step. Across compute budgets and matched entropy, CRS generally achieves lower GenPPL, while other metrics can rank operating points differently.

  • GenPPL across NFE: CRS with top-p disabled (p = 1.0) has lower GenPPL than native sampling with p ∈ {1.0, 0.95, 0.9} throughout the tested NFE range on Duo-distilled.Base Duo shows the same ordering of point estimates in a descriptive, protocol-aligned comparison.
  • Persistent-context controls: Paired controls indicate that the path change depends on inserted token values and is not explained by position selection or scalar temperature alone.The best scalar-temperature fit leaves 79.4 ± 2.1% of the pooled KL shift unexplained, and top-ranked tokens change on 30.2 ± 0.8% of audited coordinates.
  • Selection timing: Later selection recovers unigram entropy while retaining a substantial likelihood gain, and all four warmup endpoints lie below the matched global top-p curve.The sweep varies how many steps precede selection at 64 NFE.
  • GenPPL–entropy tradeoff: At 64 NFE, CRS lies below global top-p across the overlapping entropy region on Duo-distilled, with base Duo showing the same direction descriptively.After entropy matching within each shard, CRS lowers GenNLL by 0.154 ± 0.031 nats per token.
  • GenPPL across NFE: Across 8–64 NFE, CRS with top-p disabled (p = 1.0) achieves lower GenPPL than fixed p ∈ {0.95, 0.9} baselines on both checkpoints.All methods use the same argmax final step.
  • Sampler design: CRS stores selected argmax tokens in later denoiser inputs, while unselected positions continue following the native sampler.Newly stored labels first enter the denoiser at the next step and can remain visible for a finite horizon or indefinitely.

B.1 Proof of Proposition 2

The proof establishes that, under the exact forward process, clean-token Bayes selection risk cannot increase as noise decreases. A separated latent-mode example then shows how lower noise and persistent evidence can make one coherent sequence mode identifiable.

  • Proposition 2: Under the exact forward process, the Bayes rule based on a less noisy state cannot have larger zero–one risk than one based on a noisier state.The Bayes rule selects the largest-probability leave-one-out clean label.
  • Latent-mode illustration: When the true mode is closer than half the minimum Hamming separation, it is the unique nearest codeword and therefore the unique MAP mode.The condition is d < ∆/2.
  • Latent-mode illustration: If the true mode has posterior weight above one half, the unique posterior-Hamming Bayes action is obtained by coordinatewise clean-token posterior argmax.Hamming risk separates by coordinate.
  • Latent-mode illustration: In the separated-mode model, lower noise makes one coherent sequence mode more likely to be uniquely identifiable.The codebook calculation provides a concrete sufficient condition, while the proposition gives the distribution-free exact-law comparison.

C.1 Proof of Theorem 1

The analysis models visible context as evidence about a shared latent mode and combines it with one-step support evidence. Under the stated conditional-independence assumptions, mode uncertainty decreases with consistent evidence, while the benefit of additional evidence exhibits diminishing returns.

  • Conditional mode model: Conditioning on visible context preserves a mixture-of-products form, with prior mode weights replaced by posterior weights w[u].This relies on Y ⊥⊥ U | M; without that assumption, arbitrary context can induce dependence among coordinates.
  • Combined evidence: Combining support evidence with persistent context yields a posterior-error bound controlled by the aggregate evidence S(A) + T(u1:H).The bound separates the largest mode from the remaining K − 1 modes and induces certificates for future product components.
  • Diminishing returns: Additional mode evidence has positive but diminishing benefit: Gδ(S,T) > 0, its partial derivatives are negative, and Gδ(S,T) → 0 as S + T → ∞.Thus both stronger support evidence and stronger persistent evidence reduce the marginal gain from adding more evidence.
  • Support evidence: A mode-consistent visible token becomes more informative as uniform corruption decreases toward the clean end of the reverse path.Its Bayes factor increases when the favored mode assigns that token higher probability than competing modes.
  • Reverse-step interpretation: A singleton clean-token proposal need not produce a point-mass next state because the stochastic bridge can still update the mutable noisy state.The point hypothesis is therefore distinct from deterministic state finalization.

D.2 NFE scaling on the two Duo checkpoints

Under common argmax finalization, CRS improves the GenPPL operating points relative to native top-p sampling across the tested NFE range, while the checkpoint audit shows that selection locations—not labels—are the main difference between Duo variants.

  • Path statistics: At 64 NFE, CRS cumulatively selects 991.23 ± 1.02 coordinates per sample on Duo-distilled and 963.73 ± 0.73 on Duo.The mean number of simultaneously visible stored labels is 508.23 ± 0.19 and 502.68 ± 0.16, respectively, showing gradual accumulation.
  • Low-NFE boundary: At 8 NFE, CRS commits 799.94 positions on Duo-distilled and 795.50 on Duo on average.With one-third warmup, selection begins after two steps, making the fixed path-total budget aggressive for this low-NFE setting.
  • Checkpoint differences: With selected positions fixed, the two checkpoints agree on argmax labels at least 99.75% of the time and differ in clean-label accuracy by at most 0.25 points.Their selection sets have Jaccard index 0.43–0.53, while self-selection accuracy differs by 0.9–2.1 percentage points.

D.4 Sampler-level warmup-by-duration ablation

This ablation varies warmup timing and persistent-context duration under full clean-token support, showing that delayed selection reduces both the likelihood gain and entropy cost of persistence.

  • Ablation design: At 64 NFE, the ablation varies warmup fractions 0.016, 0.17, 0.33, and 0.50 with visibility horizons H = 0, 8, and ∞.The corresponding warmup lengths are 1, 11, 21, and 32 steps; every persistent cell uses Dmax = 512 and final-step argmax decoding.
  • Schedule control: GenPPL ranges from 41.79 to 42.18 across H = 0 schedule-split controls, while unigram entropy ranges from 5.351 to 5.359.This stability localizes the interaction to persistent exposure rather than the artificial time-grid split.
  • Warmup effect: Delaying selection reduces both the likelihood gain and the entropy cost of persistent context.The reported ablation tests the qualitative warmup mechanism; oracle-risk calibration and optimal-warmup selection remain separate questions.
  • Duration effect: At warmup fraction 0.33, extending visibility from H = 8 to H = ∞ leaves unigram entropy nearly unchanged but lowers Dist-4 and raises Rep-4 and Self-BLEU.Dist-4 decreases from 0.9405 to 0.9359, Rep-4 increases from 0.0330 to 0.0367, and Self-BLEU increases from 0.2205 to 0.2248.

D.6 Fixed-prefix transfer

The fixed-prefix transfer evaluates late top-p schedules with persistent visibility on continuation generation using paired GPT-2-Large and Pythia-1.4B measurements.

  • Evaluator NLL: At late p = 1.0, H = 0 → 8 reduces GPT-2-Large NLL by 0.2165 ± 0.0061 nats per token, versus 0.1678 ± 0.0053 at late p = 0.8.The corresponding H = 8 → ∞ reductions are 0.2091 ± 0.0050 and 0.0198 ± 0.0021.
  • Evaluator NLL: The paired difference-in-differences are 0.0487 ± 0.0056 at late p = 1.0 and 0.1893 ± 0.0069 nats per token at late p = 0.8.Pythia-1.4B gives interaction estimates of 0.0451 ± 0.0064 and 0.1857 ± 0.0093.
  • Policy contrasts: Against every-step p = 0.8 with H = 0, late p = 0.8 changes GPT-2/Pythia NLL by +0.0361 ± 0.0105/+0.0208 ± 0.0111 for H = 8.For H = ∞, the changes are +0.0163 ± 0.0096/−0.0064 ± 0.0100; these are policy contrasts on different trajectories.
  • Diversity: Within-prompt continuation Self-BLEU is 0.0378 for every-step p = 0.8, H = 0, 0.0351 for H = 8, and 0.0357 for H = ∞.The table reports GPT-2 and Pythia conditional perplexity, with diversity summaries computed on continuation tokens.

D.7 Support-by-duration interaction

The support-by-duration factorial separates temporary support restriction from persistent visibility, finding that stronger restriction attenuates the marginal benefit of extending context duration.

  • Unconditional factorial: The complete unconditional factorial crosses post-warmup top-p ∈ {1.0, 0.9, 0.85, 0.8} with visibility duration H ∈ {0, 8, ∞}.The left and right Figure 4 panels report gains from eight-step and all-remaining-step visibility, respectively.
  • Unconditional factorial: Temporary eight-step context remains useful throughout the support-restriction sweep.Moving right in the figure denotes stronger restriction; blue lines are unconditional generation and orange squares are fixed-prefix endpoints at p = 1.0 and p = 0.8.
  • Duration interaction: The marginal gain from extending H = 8 to H = ∞ rapidly disappears as support restriction strengthens and becomes slightly negative at late p = 0.8.Fixed-prefix duration contrasts are complete only at p = 1.0 and p = 0.8, and the long-horizon gain remains positive but is much smaller at p = 0.8.
  • Interpretation: The sign test is qualitative because its theorem assumes exact, mode-consistent evidence, whereas the experiment uses a learned denoiser on free-running trajectories.The slight negative unconditional endpoint is therefore outside the theorem’s guaranteed regime rather than a contradiction.

E.2 Same-state temperature and token values

Same-state audits test whether persistent-context effects can be explained by scalar temperature or position selection alone, and directly intervene on stored token values and visibility.

  • Temperature control: The temperature audit follows full-support CRS paths at 64 NFE across 1792 primary coordinate–step pairs and fits a separate convex temperature at each coordinate.The reported residual fraction is pooled across four shards.
  • Temperature control: 79.4 ± 2.1% of the pooled KL shift remains unexplained by the best same-state scalar-temperature fit.No primary fitted optimum reaches the β = 0 uniform-reference boundary.
  • Ranking changes: The top-ranked token changes on 30.2 ± 0.8% of audited coordinates, with pairwise rank inversion in 99.89 ± 0.06% of top-64 unions.The initial-step empty-context control has zero top-rank changes and zero rank inversions.
  • Token-value intervention: At full support, argmax stored values yield 28.27 ± 0.53 GenPPL, compared with 77.75 ± 1.80 for current second-ranked values and 26.42 ± 0.39 for refreshed values.The intervention holds selected positions, transition streams, and final-step argmax decoding fixed.
  • Persistent visibility: Restoring one committed coordinate to its mutable noisy state leaves stored-label probabilities above paired fully revisable values at offsets 1, 2, 4, and 8.Restored-coordinate probabilities are 0.7483, 0.8323, 0.8230, and 0.8291, versus 0.4939, 0.5542, 0.5481, and 0.5630.
  • Matched comparison: At matched entropy, the CRS GenNLL residual is −0.154 ± 0.031, while matched Dist-4, Rep-4, and Self-BLEU residuals are −0.025 ± 0.048, +0.070 ± 0.036, and −0.144 ± 0.019.The comparison interpolates each shard’s global top-p curve at the observed CRS diversity without extrapolation.
Loading 2609.01043v1…