PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion Language Models Know the Answer Before Decoding

Pengxiang Li, Yefan Zhou, Dilxat Muhtar, Lu Yin, Shilin Yan, Li Shen, Soroush Vosoughi, Shiwei Liu

arXiv:2508.19982v5cs.CLcs.AI

TL;DR

DLM inference remains slower than autoregressive inference, while early answer convergence offers an overlooked opportunity to reduce refinement. Prophet uses confidence gaps to decide when to commit remaining tokens, achieving up to 3.4× reduction in decoding steps with negligible accuracy degradation across reasoning, code, and planning benchmarks.

  • Problem

    DLM inference remains slower than autoregressive inference, and whether early answer convergence can be exploited to reduce refinement remains underexplored.

    Source: §1 · pp. 1–2

  • Method

    Prophet uses the confidence gap between the top-two answer candidates to choose between continued refinement and committing all remaining tokens at once.

    Source: §1 · p. 2

  • Results

    Up to 3.4× reduction in decoding steps, with negligible accuracy degradation, across diverse reasoning, code, and planning benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings frame DLM decoding as an optimal-stopping problem rather than fixed-budget iteration.

    Source: §6 · p. 10

    Prophet is scoped to structured tasks with identifiable answer regions; open-ended generation may lack clear convergence until late in denoising.

    Source: A DISCUSSION · p. 15

Abstract

from arXiv · show

Diffusion language models (DLMs) have recently emerged as an alternative to autoregressive approaches, offering parallel sequence generation and flexible token orders. However, their inference remains slower than that of autoregressive models, primarily due to the cost of bidirectional attention and the large number of refinement steps required for high quality outputs. In this work, we highlight and leverage an overlooked property of DLMs early answer convergence: in many cases, the correct answer can be internally identified by half steps before the final decoding step, both under semi-autoregressive and random remasking schedules. For example, on GSM8K and MMLU, up to 97% and 99% of instances, respectively, can be decoded correctly using only half of the refinement steps. Building on this observation, we introduce Prophet, a training-free fast decoding paradigm that enables early commit decoding. Specifically, Prophet dynamically decides whether to continue refinement or to go "all-in" (i.e., decode all remaining tokens in one step), using the confidence gap between the top-2 prediction candidates as the criterion. It integrates seamlessly into existing DLM implementations, incurs negligible overhead, and requires no additional training. Empirical evaluations of LLaDA-8B and Dream-7B across multiple tasks show that Prophet reduces the number of decoding steps by up to 3.4x while preserving high generation quality. These results recast DLM decoding as a problem of when to stop sampling, and demonstrate that early decode convergence provides a simple yet powerful mechanism for accelerating DLM inference, complementary to existing speedup techniques. Our code is publicly available at https://github.com/pixeli99/Prophet.

1 INTRODUCTION

Many DLM instances reach correct answers before decoding ends, motivating Prophet, which uses a confidence-gap criterion to commit early and accelerate decoding while maintaining high generation quality.

1 INTRODUCTION

  • DLMs generate sequences by parallel iterative denoising and remasking, refining low-confidence predictions over successive rounds and allowing flexible token orders.Their stated strengths over autoregressive models include parallel decoding and flexible generation orders.Source: §1 · p. 1
  • Despite their potential speedup, DLM inference is slower than autoregressive inference in practice because it lacks KV cache mechanisms and fast parallel decoding can degrade performance.Prior efforts address KV caching and parallel-decoding quality.Source: §1 · p. 1
  • Up to 97% of GSM8K and 99% of MMLU instances are correctly decoded using only half the refinement steps, with early convergence observed under both remasking schedules.The trend is more significant with random remasking; the paper frames this as redundancy in conventional full-length decoding.Source: §1 · p. 2
  • Prophet monitors the confidence gap between the top two candidates to choose between continued refinement and decoding all remaining tokens at once.This training-free strategy calls the immediate finalization of the answer Early Commit Decoding.Source: §1 · p. 2
  • Up to 3.4× fewer decoding steps with negligible accuracy degradation, Prophet accelerates generation while preserving high quality across reasoning, code, and planning benchmarks.The reported benchmarks have identifiable answer regions.Source: §1 · p. 2

2 RELATED WORK

Prior work establishes DLMs’ diffusion formulations and their efficiency–accuracy tradeoff, while acceleration methods target caching, token pruning, or sampling. Prophet instead uses early answer convergence to reduce total decoding steps, complementing per-step optimizations.

2.1 DIFFUSION LARGE LANGUAGE MODEL

  • DLM research developed from discrete diffusion formulations, including D3PM’s noisy Markov-chain forward process and ELBO-trained reverse process, alongside continuous-time and likelihood-ratio approaches.Later analyses found multiple MDM parameterizations equivalent.Source: §2.1 · p. 2
  • DLMs face an efficiency–accuracy tradeoff: decoding more tokens per step can reduce steps but degrade quality, while decoding fewer tokens increases latency and limits KV caching.The passage also names commercial DLMs Mercury, Gemini Diffusion, and Seed Diffusion, and open-source LLaDA and Dream.Source: §2.1 · p. 2

2.2 ACCELERATION METHODS FOR DIFFUSION LANGUAGE MODELS

  • Acceleration methods include approximate or block-wise KV caching across denoising steps and token pruning, such as DPad’s training-free removal of distant suffix tokens.Caching approaches exploit similar hidden states across steps or reuse context and block states, sometimes with periodic refresh.Source: §2.2 · p. 3
  • Sampling and decoding methods reduce steps or decode more tokens, spanning training-based RL and distillation and training-free methods such as Fast-dLLM, SlowFast, and WINO.Other work studies denoising dynamics, alignment with small autoregressive models, and speculative decoding using a DLM draft model.Source: §2.2 · p. 3
  • Prophet reduces total denoising steps by committing when answer confidence supports stopping, unlike Fast-dLLM’s per-step optimization; the approaches are orthogonal and can be combined.Early Answer Convergence was independently identified by Wang et al. (2025a), whose work averages predictions across steps to improve accuracy rather than reducing computation through early commits.Source: §2.2 · p. 3

3 PRELIMINARY

The section defines DLM corruption and iterative reverse refinement, then examines when correct answers emerge during decoding and how prompting and remasking affect that timing.

3.1 BACKGROUND ON DIFFUSION LANGUAGE MODELS

  • DLMs progressively mask subsets of a clean sequence, then iteratively refine text through prediction and remasking; τ-leaping can recover multiple masked positions simultaneously.Remasking may be uniform at random or selective for low-confidence positions.Source: §3.1 · pp. 3–4

3.2 EARLY ANSWER CONVERGENCY

  • Under random remasking, 97.2% of samples are correctly predicted within the first half steps and 88.5% within the first 25% steps.Under low-confidence remasking, the corresponding proportions are 24.2% and 7.9%. The analysis uses LLaDA-8B on GSM8K and MMLU and includes only samples whose final output contains the ground-truth answer.Source: §3.2 · p. 4, Figure 1 · p. 5
  • Suffix prompting accelerates early correct-answer emergence across remasking strategies by semantically anchoring where the bidirectional model should locate the solution.With low-confidence remasking, the proportion rises from 7.9% to 59.7% by the 25% step and from 24.2% to 75.8% by the 50% step; with random remasking, it rises from 88.5% to 94.6% by the 25% step.Source: §3.2 · p. 4, Figure 1 · p. 5
  • Answer tokens tend to stabilize earlier and change less often, while most nonanswer chain-of-thought tokens fluctuate frequently before finalization.This pattern is shown in Figure 2.Source: §3.2 · p. 4

4 METHODOLOGY

Prophet is a training-free method for accelerating diffusion language model generation by monitoring prediction certainty and committing to all remaining tokens once the answer has stabilized. It uses an answer-region confidence gap and an adaptive threshold to decide when to stop refinement.

Early Commit Decoding

  • Prophet dynamically monitors prediction certainty and commits to all remaining tokens in one step when predictions have stabilized, rather than using fixed-step decoding.The method is designed to accelerate DLM generation without additional training.Source: §4 · p. 5

Confidence Gap as a Convergence Metric

  • Prophet measures certainty as the difference between the top two logits at each position and averages this confidence gap over the answer region.The answer region is a designated span within the current generation window.Source: §4 · p. 5

Early Commit Decoding

  • Prophet uses a progress-dependent threshold, demanding higher confidence early and progressively less later as predictions stabilize and the savings from another refinement step diminish.The staged thresholds apply when progress p is below 0.33, from 0.33 to below 0.67, and at least 0.67.Source: §4 · pp. 6–7

Algorithm Summary

  • When the exit condition is met, Prophet ends iterative refinement and fills remaining [MASK] tokens in parallel using the current logits’ argmax; otherwise, standard refinement continues.The algorithm checks the average answer-region confidence gap against the progress-dependent threshold at each step.Source: §4 · p. 7
  • Prophet adds negligible computational overhead, is model-agnostic, requires no retraining, and can wrap existing DLM inference code.Source: §4 · p. 7

5 EXPERIMENTS

Across benchmarks, Prophet often matches or improves full-budget quality while using fewer refinement steps, and it combines with other acceleration methods. Ablations show gains across step budgets, remasking strategies, and block sizes.

5.1 EXPERIMENTAL SETUP

  • Experiments use LLaDA-8B (Nie et al., 2025) and Dream-7B (Ye et al., 2025), compare full Tmax-step decoding with dynamic-threshold Prophet, and evaluate four capability domains using greedy decoding.Prophet’s thresholds are τhigh = 7.5, τmid = 5.0, and τlow = 2.5, with transitions at 33% and 67% of decoding progress.Source: §5.1 · pp. 7–8

5.2 MAIN RESULTS AND ANALYSIS

  • On Dream-7B MMLU, Prophet scores 66.1% accuracy versus 67.6% for full decoding, a 1.5% drop at 2.47× speedup, while LLaDA-8B matches or exceeds full decoding on reported general tasks.On LLaDA-8B, Prophet scores 54.0% on MMLU and 83.5% on ARC-C, both statistically on par with full-step decoding; on HellaSwag it scores 70.9% versus 68.7% for full decoding.Source: Table 1 · p. 8, §5.2 · p. 8
  • On GSM8K, LLaDA-8B Prophet reaches 77.9% accuracy versus 77.1%; on HumanEval it matches LLaDA (30.5%) and lifts Dream to 55.5% from 54.9%, with 1.20× speedup.Acceleration is more conservative on intricate tasks, as Prophet allocates more denoising steps when further refinement is needed.Source: §5.2 · pp. 8–9

5.3 COMPARISON WITH OTHER ACCELERATION METHODS

  • Against SDTT (Deschenaux & Gulcehre, 2025), Prophet gives 1.63× speedup and 77.9% accuracy without retraining; adding Prophet to the distilled model reaches 3.21× with negligible accuracy drop.The results indicate that SDTT and Prophet are complementary, and the distilled model retains early answer convergence.Source: Table 2 · p. 9, §5.3 · p. 9
  • Combining Prophet with Fast-dLLM (Wu et al., 2026) raises GSM8K speedup from 6.82× to 7.66× without quality degradation, as one reduces steps and the other per-step cost.The methods operate on orthogonal dimensions: Prophet reduces the number of steps, while Fast-dLLM reduces per-step computation.Source: Table 2 · p. 9, §5.3 · p. 9

5.4 ABLATION STUDIES

  • At L = 256, Prophet uses ≈160 of 256 steps (1.63×) and scores 77.9% versus 77.1%; at L = 128, ≈74 steps (1.73×) yields 72.7% versus 71.3%.Static step-cap accuracy rises with additional steps but remains below full-budget decoding or Prophet; Prophet adapts its stopping point to avoid late over-refinement.Source: Table 3 · p. 10, §5.4 · p. 10
  • Prophet beats static counterparts under random remasking (+2.8 points), low-confidence remasking (+1.4), and top-k margin remasking (+0.7), complementing rather than replacing token-selection policies.The largest gain occurs with random remasking, while improvements persist under the more informed heuristics.Source: Table 3 · p. 10, §5.4 · p. 10
  • At block lengths 64 and 128, Prophet improves accuracy by +9.9 and +19.1 points, respectively, attenuating the brittleness of static semi-autoregressive schedules.When coarse updates raise uncertainty, the threshold schedule defers early commitment; it exits once predictions settle.Source: Table 4 · p. 10, §5.4 · p. 10

6 CONCLUSION · APPENDIX

The conclusion presents early answer convergence as a property of diffusion language models and introduces Prophet, a training-free method that uses confidence gaps for early stopping. Experiments report fewer decoding steps with generation quality maintained, framing decoding as an optimal stopping problem.

6 CONCLUSION

  • On LLaDA-8B and Dream-7B, Prophet reduces decoding steps by up to 3.4× while maintaining generation quality.Source: §6 · p. 10
  • Up to 99% of instances can be correctly decoded using only half of the refinement steps.Source: §6 · p. 10
  • Prophet is a training-free early-commit decoding paradigm.Source: §6 · p. 10
  • Prophet monitors confidence gaps to determine when to terminate decoding.Source: §6 · p. 10
  • The conclusion recasts DLM decoding as an optimal stopping problem rather than fixed-budget iteration.Source: §6 · p. 10
  • The work suggests early convergence may support efficient inference on tasks with identifiable answer regions.Source: §6 · p. 10
  • The authors suggest early convergence reflects how DLMs resolve uncertainty across diverse tasks and settings.Source: §6 · p. 10

A DISCUSSION

Prophet’s early-termination benefits are strongest in structured tasks with identifiable answer regions, while persistent uncertainty on reasoning-heavy tasks can defer termination. The discussion also identifies ways to improve or combine its termination signal and system integration.

Scope and applicability

  • Prophet is designed for tasks with identifiable answer regions, such as mathematical reasoning, code generation, and planning, rather than as a universal accelerator for open-ended generation.When reasoning and answer boundaries are unclear, convergence may not emerge until late in denoising.Source: A DISCUSSION · p. 15

Conservative speedups on complex tasks

  • 3.21× speedup on GSM8K for SDTT + Prophet versus 1.63× for Prophet alone, as trajectory compression reduces reasoning-chain steps before Prophet exploits answer convergence.The combination is presented as a preliminary result.Source: A DISCUSSION · p. 15
  • 1.20× speedup on HumanEval versus 3.40× on Sudoku reflects Prophet’s conservative termination when answers depend on preceding reasoning and uncertainty persists.Deferring termination in these cases is intended to preserve correctness.Source: A DISCUSSION · p. 15

Learnable termination criteria

  • Prophet’s confidence gap is a simple, training-free, negligible-overhead criterion, while a future trained discriminator could provide a more robust termination signal when confidence poorly predicts correctness.The proposed “Judge Prophet” would replace the heuristic confidence gap with a trained discriminator, drawing on judge-based decoding (Bachmann et al.).Source: A DISCUSSION · p. 15

Integration with system-level optimizations

  • Prophet is orthogonal to distillation- and cache-based acceleration; in KV Cache frameworks, monitoring its termination signal can stop inference early and avoid cache updates for remaining steps.This system-level synergy is proposed as a direction for future exploration.Source: A DISCUSSION · p. 15

B ADDITIONAL RESULTS · B.1 CORROBORATING RESULTS ON EARLY ANSWER CONVERGENCE

Additional MMLU results complement the paper’s GSM8K findings by examining early correct-answer emergence under different remasking strategies and suffix prompting. The figure shows substantial early convergence with LLaDA 8B, accelerated by suffix prompting.

B ADDITIONAL RESULTS

  • These additional MMLU results complement the GSM8K results in the main paper’s Section 3.2.Source: §B.1 · p. 15

B.1 CORROBORATING RESULTS ON EARLY ANSWER CONVERGENCE

  • The analysis examines low-confidence remasking and random remasking on MMLU.Source: §B.1 · p. 15
  • The analysis investigates how suffix prompting affects the decoding steps at which correct answers first emerge.Source: §B.1 · p. 15
  • Suffix prompting dramatically accelerates early correct-answer convergence on MMLU with LLaDA 8B.The figure compares standard prompting with suffix prompting and shows correct answers stabilizing as top-1 candidates before full decoding.Source: Figure 4 · p. 16
  • The figure marks 50% and 70% completion thresholds and reports substantial early convergence.The supplied caption gives no exact statistics for those thresholds.Source: Figure 4 · p. 16
  • The figure’s histograms track when correct answers first emerge as a percentage of total decoding steps.The results use LLaDA 8B on MMLU.Source: Figure 4 · p. 16

B.2 QUALITATIVE ANALYSIS: DECODING TRAJECTORIES … C.1 ALGORITHM DESIGN

The analyses show that correct answers can stabilize before reasoning is complete, while incorrect answers tend to remain unstable until late decoding. Prophet’s suffix prompt and task-defined answer regions support early commitment, with fixed answer spans relying on task priors.

B.2 QUALITATIVE ANALYSIS: DECODING TRAJECTORIES

  • At 50% of steps, the model identifies the answer “3” while “2/2 = 1” remains masked, with no significant degradation in final-output fluency or grammatical correctness.The intermediate reasoning may remain noisy or less polished than a full-step generation.Source: §B.2 · p. 16, Table 5 · p. 17

B.3 DISTRIBUTION OF INCORRECT ANSWERS

  • Updates for incorrect GSM8K answers cluster in the last 20% of steps, so unstable predictions fail Prophet’s early-commit criterion and use the full step budget.This behavior helps avoid premature exits on incorrect answers while allowing early commitment on stable, correct ones.Source: Figure 5 · p. 17, §B.3 · p. 17

C.1 ALGORITHM DESIGN

  • Prophet inserts a suffix prompt before masked final-answer tokens, cueing the answer region and helping answer tokens stabilize earlier during bidirectional denoising.The sequence is structured as “[Question] [MASKs for Reasoning Chain] The answer is [MASKs for Final Result].”Source: §C.1 · pp. 17–18
  • The suffix prompt uses a deterministic answer position rather than ground-truth information, and Prophet and its baseline use identical configurations, attributing gains to convergence detection.The prompt tells the model where to place the answer, not what the answer is.Source: §C.1 · p. 18
  • Answer regions are task-defined: post-separator tokens for mathematical reasoning, a single prediction token for multiple-choice tasks, and the delimited function body for code generation.For multiple-choice tasks, Prophet monitors confidence over the answer token; code structure can converge early under syntactic dependencies.Source: §C.1 · p. 18
  • Fixed answer-region lengths rely on task priors; dynamic semantic extraction such as Wang et al. (2025a) could identify spans on the fly, while predefined regions were chosen for simplicity.The stated aim is to analyze Early Answer Convergence, not build a universally applicable acceleration tool.Source: §C.1 · p. 18

C.2 EVALUATION AND HYPERPARAMETER

Prophet uses a staged confidence schedule that lowers commitment thresholds as decoding progresses, and a light validation sweep found its fixed configuration worked consistently across tasks.

Evaluation protocol

  • The evaluation re-implemented LLADA and DREAM across reported datasets and extracted generated final answers rather than comparing multiple-choice log probabilities.This protocol can slightly lower benchmark scores when a model fails to produce an answer in the expected format; configurations are summarized in Table 6.Source: §C.2 · p. 18

Confidence schedule design

  • Three confidence stages start with a high threshold to avoid premature commitment, then lower it as decoding progresses and further computation yields less marginal gain.The schedule is designed around time-varying risk aversion.Source: §C.2 · p. 18

Hyperparameter robustness

  • A light GSM8K validation sweep found the three-stage thresholds worked consistently across tasks, though the authors did not conduct an extensive hyperparameter search.The authors interpret this robustness as evidence that the confidence gap generalizes without sensitive per-task tuning.Source: §C.2 · p. 18

Schedule ablation

  • The staged schedule and continuous linear decay over the same threshold range yielded comparable accuracy and speedup on GSM8K with LLaDA-8B.The tested linear schedule was τ(p) = 8.0 −4.5 × p over [8.0, 3.5]; the comparison is reported in Table 7.Source: §C.2 · p. 18, Table 7 · p. 19

Related papers

Loading 2508.19982v5…