PaperlayerView exact revision on arXiv

Source-linked AI summary

Trust but Verify: Designing Region-Inhomogeneous Forward Dynamics for Discrete Diffusion

Gunmay Jhingran

arXiv:2610.22380v1cs.LG

TL;DR

Absorbing discrete diffusion fully trusts observed context, leaving corrupted tokens liable to contaminate infills. This paper combines region-dependent absorbing and mixing corruption and adds a heuristic sampler that revisits suspicious context. For uniform corruption, the method improves infill F1, while gains for plausible corruption are small.

  • Problem

    Absorbing corruption leaves observed context fully trusted, with no discrete mechanism for representing graded uncertainty or revisiting corrupted tokens.

    Source: §1 · p. 2

  • Method

    The method trains with absorbing dynamics on holes and low-rate uniform-mixing corruption on context, then re-noises suspicious context early during reverse sampling.

    Source: §1 · p. 2, §3.2 · p. 4, §3.4 · p. 4

  • Results

    For uniform corruption, arm D−A improves F1 by +3.84 at ρ = 0.1 and +6.42 at ρ = 0.2, while plausible-corruption gains are small.

    Source: §4.2 · p. 6, §5 · p. 8

  • Takeaways & Limitations

    The authors interpret the evidence as suggesting that corruption detectability and damage are jointly related to divergence from the data conditional.

    Source: §1 · p. 2, §4.2 · p. 7

    Generality remains untested beyond one 170M-scale model, one seed per arm, and English web text; plausible-corruption gains are small.

    Source: §5 · p. 8

Abstract

from arXiv · show

Continuous diffusion treats the forward process as a design space; discrete language diffusion often uses absorbing corruption, leaving context fully trusted throughout generation. This makes trust binary: corrupted context is copied as ground truth, propagating errors into generated spans. We reopen this design space by combining absorbing and uniform-mixing kernels with region-dependent rates. Missing positions follow absorbing dynamics, while context undergoes low-rate random substitution, exposing the model to observed but uncertain tokens during training. At inference, a heuristic sampler re-injects noise into suspicious context under an annealed intensity and freezes context updates late in the trajectory. An 8,000-step finetune of a pretrained 170M-parameter masked diffusion model takes under five hours on a 6 GB laptop GPU. The base model assigns roughly 50% probability to observed tokens even when they are uniform-random garbage. For uniform-random corruption, finetuning detects corrupted context with 0.96 recall and 0.91 precision, improves infill token-F1 by 3.8 points at 10% corruption and 6.4 points at 20%, and removes the accuracy dip next to corruptions. Clean conditioning shows no measurable regression, with a false-edit rate below 0.4%. Gains for plausible corruptions are weaker: about 0.5 F1 points at 10% corruption, without statistical significance. Our results suggest that detectability and damage depend jointly on departure from the data conditional: nearby noise is hard to detect and semantically mild, while distant noise is conspicuous and destructive. We interpret absorbing diffusion as the infinite-contrast limit of a trust axis, analogous to the signal-to-noise gap between known and unknown regions in diffusion inpainting. In our setting of a 170M model and a single text domain, intermediate trust is reached by changing forward dynamics rather than architecture.

1 Introduction

Discrete diffusion has largely settled on absorbing corruption, which makes observed context fully trusted and leaves no discrete representation of graded uncertainty. This paper reopens the design space with region-dependent corruption and evaluates whether it can detect and repair corrupted context.

1 Introduction

  • The design draws on the view of forward corruption as task-specific dynamics, while discrete diffusion practice has concentrated on the absorbing process.Absorbing corruption has practical advantages, including a simple weighted cross-entropy bound and a time-free optimal denoiser.Source: §1 · pp. 1–2
  • Absorbing dynamics treat observed tokens as certain, so corrupted context is copied as ground truth and can contaminate generated spans.The pretrained model assigns roughly 50% probability to observed tokens even when they are uniform-random garbage.Source: §1 · p. 2
  • The method combines absorbing and uniform-mixing kernels at region-dependent rates, training on uncertain context while holes follow standard absorbing dynamics.At inference, suspicious context is re-noised early and refilled; context updates freeze later in the trajectory.Source: §1 · p. 2
  • For uniform corruption, finetuning supports context detection, repair, and containment without measurable clean-conditioning regression, while gains for plausible corruptions are weaker.The ablations do not fully separate the effects of the η=0 mixture and trust annealing.Source: §1 · p. 2

2 Related work

Related work spans discrete corruption kernels, self-correcting hybrid processes, text editing, and continuous-diffusion conditioning. This paper focuses on applying mixed corruption selectively to user-supplied context.

2 Related work

  • Discrete diffusion began with structured corruption kernels [Austin et al. (2021)], while continuous-time treatments formalize Markov jump processes with learnable reverse rates [Campbell et al. (2022); Lou et al. (2024)].Later work specialized to absorbing corruption, enabling a weighted masked cross-entropy bound and a time-free optimal denoiser.Source: §2 · pp. 2–3
  • Hybrid-noise and self-correction methods include GIDD [von Rütte et al., 2025], SCDD [Wang et al., 2026], and ReMDM [Wang et al., 2025], but this work applies hybrid noise to context rather than model outputs.The comparison with homogeneous hybrid noise is indirect because this paper did not train a homogeneous hybrid-noise arm.Source: §2 · p. 3
  • Unlike separate text-editing or replaced-token detectors, this approach uses generative-model likelihoods for detection and the reverse dynamics for repair.The cited external-detector comparison uses a GPT-2 left-context surprisal detector; bidirectional and trained replaced-token detectors were not evaluated.Source: §2 · p. 3
  • The method frames context corruption as a discrete counterpart to lowering the known region from full noise to intermediate signal-to-noise ratio in continuous diffusion.Prior examples include renoised composition for Gaussian inpainting and per-position noise levels.Source: §2 · p. 3

3 Method

The method combines absorbing dynamics for infill holes with low-rate mixing noise on context, trains a copy-or-fix objective, and uses a heuristic reverse-time sampler to revisit suspicious context early.

3.1 Background: absorbing forward dynamics

  • Absorbing forward dynamics independently mask tokens with probability t, and conditional infilling preserves observed context with probability one.The standard training objective uses weighted masked cross-entropy over the masked positions.Source: §3.1 · p. 3

3.2 Region-inhomogeneous forward dynamics

  • Holes follow absorbing dynamics while context receives low-rate mixing corruption, with an η=0 component preserving clean conditioning and a separate hard-negative mixture for training noise.Sequences have length 512, with one to four contiguous hole spans covering 10–50% of positions; plausible evaluation corruptions come from a disjoint generator family.Source: §3.2 · pp. 3–4

3.3 Objective

  • The objective combines the standard masked loss over holes with a context-position term that teaches the model to copy or fix observed tokens.Both terms are normalized per position; λ is set to 0.25, and the objective is a surrogate rather than the exact variational bound of the composed process.Source: §3.3 · p. 4

3.4 Inference: reverse-time re-noising of the conditioning

  • At inference, observed-token probability scores context suspicion, and low-scoring context is re-noised and refilled during early reverse steps before context updates freeze.The sampler uses an annealed threshold and an edit budget; the re-noising step is a heuristic not derived from the composed process’s reverse-time dynamics.Source: §3.4 · p. 4, Figure 1 · p. 5

4 Experiments

The experiments test corrupted-context infilling across corruption types and show strong gains for uniform noise, but only small, nonsignificant gains for plausible noise. Detection and neighboring-token results likewise distinguish conspicuous, damaging corruption from plausible substitutions.

4.1 Setup

  • The paired evaluation uses 13 cells, 256 passages per cell, and plausible corruptions sampled from a generator family disjoint from all training-noise families.The study finetunes the 170M-parameter checkpoint for 8,000 steps; arms sharing weights differ only in sampler.Source: §4.1 · p. 5, Table 1 · p. 5

4.2 Results

  • For uniform ρ = 0.1 corruption, arm D reaches 0.959 recall at 0.909 precision; clean-context false-edit rates for mixture-trained arms range from 0.07% to 0.39%.On plausible corruptions, DP more than doubles recall over D (0.350 vs. 0.150), but two thirds remain undetected.Source: §4.2 · pp. 6–7
  • At ρ = 0.1, uniform-corruption infill F1 improves by +3.84 points for D−A, while plausible-corruption D−A is +0.38 points and not significant.At ρ = 0.2, uniform D−A is +6.42 points; plausible gains remain below the preregistered gate.Source: §4.2 · p. 6
  • Under uniform corruption, repair restores 97% of corruption-induced passage-quality damage: D/DP incur +0.03–0.04 nats/token versus A’s +1.52.For plausible corruption, DP scores 2.902 vs. 3.237 for A (−0.335 nats/token), though single-LM judge NLL is an indirect proxy.Source: Table 3 · p. 7, §4.2 · p. 7
  • On clean conditioning, treated-arm F1 deltas straddle zero, judge NLL is identical within 0.03 nats, and the held-out bound changes by +0.0036 nats/token.The comparison of arm E with D is confounded because E both drops the η=0 component and raises mean context-corruption rate.Source: §4.2 · p. 7
  • The annealed sampler raises plausible-corruption recall from 0.058 to 0.150 for C→D and from 0.197 to 0.350 for CP→DP, with higher false-edit rates.Because edit timing, threshold, budget, and refill procedure also change, this supports the sampler as a whole, not annealing specifically.Source: §4.2 · p. 7
  • On uniform corruption at ρ = 0.1, GPT-2 surprisal achieves 0.90 recall versus D’s 0.96, while D has F1 0.304 versus 0.297.Only one external detector was tested, so the results do not establish that external detectors generally underperform.Source: §4.2 · p. 7
  • At uniform ρ = 0.1, vanilla infill accuracy falls to 0.088–0.11 beside corruptions, while η-arms reach 0.20–0.24 and curves converge by distance ≥5.For plausible corruption, arms coincide within noise across distances.Source: §4.2 · p. 6, Figure 2 · p. 8

5 Discussion and limitations

The discussion concludes that reopening the forward-process design space substantially narrows the gap for conspicuous corruptions, but only slightly for plausible ones. It limits the evidence to a small, single-domain study and identifies unresolved evaluation and design questions.

5 Discussion and limitations

  • The method narrows the capability gap substantially for conspicuous corruptions, but only slightly for plausible ones.The authors frame the target as contamination from divergent corruptions and passage-level coherence, not token-F1 recovery for locally mild plausible noise.Source: §5 · p. 8
  • The authors frame repair as eliminating contamination from divergent corruptions and improving passage-level coherence where a judge detects it.They do not position it as a way to recover token-F1 for corruptions too close to the data conditional to matter locally.Source: §5 · p. 8
  • Generality remains untested beyond a 170M model, one seed per arm, and English web text; plausible-corruption gains are small.The plausible-corruption generator comes from one family, and the study does not cover code infilling with injected bugs or entity and number swaps.Source: §5 · p. 8
  • The mixture and annealing ablations are confounded, and the fixed ρ̂ = 0.15 budget caps recall at high corruption rates.The authors also describe links to reverse-time dynamics and continuous inpainting as motivating analogies rather than formal results.Source: §5 · p. 8

A Full token-F1 grid

Table 4 summarizes token-F1 across the paper's thirteen shared evaluation cells.

A Full token-F1 grid

  • Table 4 reports token-F1 for all 13 shared evaluation cells, with 256 paired passages per cell.The caption gives the table's scope but no cell values or comparisons.Source: Table 4 · p. 10

Related papers

Loading 2610.22380v1…