PaperlayerView exact revision on arXiv

Source-linked AI summary

Time Is a Feature: Exploiting Temporal Dynamics in Diffusion Language Models

Wen Wang, Bozhen Fang, Chenchen Jing, Yongliang Shen, Yangyi Shen, Qiuyu Wang, Hao Ouyang, Hao Chen, Chunhua Shen

arXiv:2508.09138v3cs.CLcs.AI

TL;DR

Diffusion language models discard intermediate predictions, even though correct answers can be overwritten during later denoising. The paper uses temporal consistency for test-time voting and reinforcement-based post-training, reporting gains across several benchmarks. Its methods rely on the model producing useful intermediate predictions, which can be a limitation on tasks where those predictions are consistently inaccurate.

  • Problem

    dLLM decoding typically keeps only the final denoising-step output and discards intermediate predictions, although correct answers can emerge and later be overwritten.

    Source: §1 · p. 1, §3.2 · p. 4

  • Method

    The paper combines training-free voting across denoising steps with reinforcement post-training that uses negative Temporal Semantic Entropy as a reward for semantically consistent predictions.

    Source: §1 · p. 2, §4.2 · p. 6

  • Results

    2.0% on GSM8K, 4.3% on MATH500, 6.6% on SVAMP, and 25.3% on Countdown are reported gains when negative TSE reward is combined with accuracy reward.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support treating intermediate denoising predictions as useful signals for improving dLLM accuracy and stability.

    Source: §6 · p. 9

    The methods are limited on tasks where intermediate predictions are consistently inaccurate; on Sudoku, their average correctness across steps is below 5%.

    Source: §E.1 · p. 17

Abstract

from arXiv · show

Diffusion large language models (dLLMs) generate text through iterative denoising, yet current decoding strategies discard rich intermediate predictions in favor of the final output. Our work here reveals a critical phenomenon, temporal oscillation, where correct answers often emerge in the middle process, but are overwritten in later denoising steps. To address this issue, we introduce two complementary methods that exploit temporal consistency: 1) Temporal Self-Consistency Voting, a training-free, test-time decoding strategy that aggregates predictions across denoising steps to select the most consistent output; and 2) a post-training method termed Temporal Consistency Reinforcement, which uses Temporal Semantic Entropy (TSE), a measure of semantic stability across intermediate predictions, as a reward signal to encourage stable generations. Empirical results across multiple benchmarks demonstrate the effectiveness of our approach. Using the negative TSE reward alone, we observe a remarkable average improvement of 24.7% on the Countdown dataset over an existing dLLM. Combined with the accuracy reward, we achieve absolute gains of 2.0% on GSM8K, 4.3% on MATH500, 6.6% on SVAMP, and 25.3% on Countdown, respectively. Our findings underscore the untapped potential of temporal dynamics in dLLMs and offer two simple yet effective tools to harness them.

1 INTRODUCTION

The introduction identifies temporal oscillation: dLLMs can produce correct intermediate answers that later denoising steps overwrite. It proposes voting across steps and reinforcement with temporal consistency, reporting improvements across multiple benchmarks.

1 INTRODUCTION

  • Diffusion language models iteratively denoise and remask tokens in parallel, but standard decoding relies on the final-step sequence and discards intermediate predictions.This differs from autoregressive generation, which predicts one token at a time.Source: §1 · p. 1
  • Correct answers often appear during intermediate denoising steps but are overwritten later, a phenomenon the authors term temporal oscillation.One example changes from the correct answer 25 at step 55 to the incorrect answer 2 at final step 64.Source: §1 · pp. 1–2
  • Temporal Semantic Entropy measures semantic variation across intermediate answers, with higher values indicating fluctuation and lower values indicating convergence toward a stable meaning.The metric groups intermediate answers by semantic equivalence before quantifying uncertainty in their semantic content.Source: §1 · p. 2
  • Temporal Self-Consistency Voting aggregates predictions across denoising steps and selects the most consistent output, while Temporal Consistency Reinforcement uses negative TSE as a reward.The voting strategy is training-free and adds negligible computational overhead; negative TSE reward does not require ground-truth labels.Source: §1 · p. 2
  • 2.0% on GSM8K, 4.3% on MATH500, 6.6% on SVAMP, and 25.3% on Countdown are reported gains when temporal consistency reinforcement is combined with accuracy reward.Negative TSE reward alone yields an average improvement of 24.7% on Countdown; temporal self-consistency voting improves the LLaDA-8B-Instruct baseline by 1.5% on average.Source: §1 · p. 2

2 RELATED WORK

Related work covers diffusion language models, sampling strategies, test-time consistency and semantic entropy, and reinforcement-learning post-training. The paper positions its focus as temporal stability in dLLMs.

2 RELATED WORK

  • Diffusion language models span continuous approaches, simplex and flow-matching methods, discrete masked-token frameworks, and recent larger models including Dream and LLaDA.The cited flow-matching approaches remain limited to simpler sequences, while some discrete variants seek parallel sampling or scale.Source: §2 · p. 3
  • Prior sampling work examines token ordering and adaptive inference (Kim et al., 2025) or inference-time remasking (Wang et al., 2025), while this paper focuses on temporal stability.Source: §2 · p. 3
  • Self-Consistency uses majority voting over multiple outputs (Wang et al., 2022), while semantic entropy clusters equivalent outputs; this work adapts both ideas to dLLM denoising trajectories.The cited semantic-entropy work concerns uncertainty estimation and hallucination detection.Source: §2 · p. 3
  • Prior dLLM reinforcement-learning adaptations such as diffu-GRPO, UniGRPO, and coupled-GRPO rely on ground-truth rewards, unlike the paper’s unsupervised temporal-consistency approach.GRPO computes advantages directly from group rewards without a separately trained value function.Source: §2 · p. 3

3 EXPLORATIONS ON DLLMS

dLLMs iteratively denoise text, but intermediate predictions can contain correct answers that later steps overwrite. The analyses characterize this temporal oscillation through accuracy and entropy patterns, including semantic stability across decoding trajectories.

3.1 PRELIMINARIES ON DLLMS

  • The reverse generative process predicts clean text from a noisy sequence, then re-masks that prediction to produce a less-noisy sequence for the next step.The forward process progressively adds noise to clean text through a Markov chain.Source: §3.1 · pp. 3–4

3.2 TEMPORAL OSCILLATION

  • 68.5% final Pass @1 versus 80.5% EverPass @1 | t on GSM8K with length 128 reveals a 12.0% gap, showing correct intermediate answers can later be revised incorrectly.The gap appears across the evaluated settings and is termed temporal oscillation.Source: §3.2 · pp. 4–5

3.3 ANALYSES

  • Accuracy generally improves with more sampling steps, but a growing gap between Pass @1 and EverPass @1 | t shows early correctness does not ensure stable reasoning.Early correct predictions appear sooner on GSM8K than Countdown; simpler SVAMP starts high, while harder Countdown starts low and benefits from refinement.Source: Figure 2 · p. 4, §3.3 · p. 5
  • Token-level entropy decreases toward zero during sampling, and incorrect final answers have higher entropy than Finally-Correct answers on GSM8K and Countdown.On GSM8K, Intermediate-Correct questions begin with lower entropy than Always-Incorrect questions, suggesting initially confident but unstable predictions.Source: §3.3 · p. 5, Figure 3 · p. 5
  • Temporal Semantic Entropy (TSE) measures semantic variation across answers in a decoding trajectory, with higher values indicating more variation and lower values indicating convergence.It is computed by grouping intermediate answers with equivalent semantics into clusters.Source: §3.3 · pp. 5–6
  • Correctly answered questions tend to have lower TSE, while weaker-performing Countdown and MATH show higher TSE than GSM8K and SVAMP.The findings associate stable, semantically consistent trajectories with better performance.Source: §3.3 · p. 6

4 METHOD

The method uses intermediate denoising predictions as signals for selecting and training more temporally consistent outputs. It combines weighted voting at inference time with reinforcement fine-tuning that rewards semantic stability.

4.1 TEMPORAL SELF-CONSISTENCY VOTING

  • Weighted voting across timestep predictions selects the final answer, with decreasing timestep weights reflecting generally higher accuracy at later steps.The method compares constant, linear, and exponential weighting functions.Source: §4.1 · p. 6
  • A single diffusion sampling trajectory supplies intermediate predictions for temporal voting without the multiple full-length forward passes required by conventional self-consistency decoding.The approach obtains these predictions without additional model evaluations.Source: §4.1 · p. 6

4.2 TEMPORAL CONSISTENCY REINFORCEMENT

  • GRPO uses TSE as a self-supervised reward because correct answers generally have lower TSE than incorrect ones.The reward encourages temporally consistent outputs without requiring labeled data.Source: §4.2 · pp. 6–7
  • Each response receives reward −TSE, encouraging its intermediate predictions to remain semantically consistent throughout decoding.The training objective uses the TSE-based reward within the standard GRPO formulation.Source: §4.2 · pp. 6–7
  • When ground-truth answers are available, the method combines TSE with an accuracy reward; the formulation balances correctness and temporal consistency.The accuracy reward is binary, while TSE is normalized into a confidence score and transformed using a spherical scoring rule.Source: §4.2 · p. 7

5 EXPERIMENTS

Experiments evaluate temporal voting and reinforcement fine-tuning on four math benchmarks using two diffusion language models. The results report gains from voting, TSE-based rewards, and combined rewards, alongside changes in reward curves and model attributes after fine-tuning.

5.1 IMPLEMENTATION DETAILS

  • Experiments used LLaDA-8B-Instruct and LLaDA-1.5 on GSM8K, MATH500, SVAMP, and Countdown, with performance evaluated at different output lengths.LLaDA-8B-Instruct received 20-epoch SFT on s1K before RFT; SFT was omitted for LLaDA-1.5 because it often degraded performance.Source: §5.1 · p. 7

5.2 TEMPORAL SELF-CONSISTENCY VOTING

  • LLaDA-8B-Instruct improved by 1.6%, 1.2%, 1.0%, and 2.2% on GSM8K, MATH500, SVAMP, and Countdown with exponential weighting; α=5 peaked at a 1.5% average gain.Table 1 compares fixed, linear, and exponential weighting; exponential weighting was adopted by default.Source: Table 1 · p. 8, §5.2 · p. 8, Figure 5 · p. 8

5.3 TEMPORAL CONSISTENCY REINFORCEMENT

  • Reward curves rose across datasets during training, while post-RFT TSE and effective tokens decreased and the ever-pass rate remained above the final-pass rate.Figure 5 presents reward curves, and Figure 6 presents model attributes after reinforcement fine-tuning.Source: §5.3 · pp. 8–9, Figure 5 · p. 8, Figure 6 · p. 9
  • 2.0%, 4.3%, 6.6%, and 25.3% average improvements over the SFT baseline were achieved on GSM8K, MATH500, SVAMP, and Countdown, respectively, with combined TSE and accuracy rewards.TSE reward alone improved Countdown by 24.7% and matched or surpassed accuracy reward there; Table 2 reports the reinforcement fine-tuning results.Source: §5.3 · p. 8, Table 2 · p. 9

6 CONCLUSION

The conclusion presents temporal dynamics in dLLMs as useful signals rather than noise. It argues that the paper’s two methods demonstrate the value of using intermediate denoising predictions.

6 CONCLUSION

  • Temporal Self-Consistency Voting and Temporal Consistency Reinforcement improve accuracy and stability without additional inference passes or ground-truth supervision.The authors call for future work to treat intermediate denoising steps as a feature in diffusion-based text generation.Source: §6 · p. 9

ETHICS STATEMENT

The experiments use publicly available mathematical reasoning benchmarks and involve no human subjects, personal data, or sensitive information.

ETHICS STATEMENT

  • The study reports no human subjects, personal data, sensitive information, or foreseeable ethical concerns, and uses publicly available math benchmarks.The authors state that they conducted no data collection raising privacy, security, or fairness concerns.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The statement describes the paper’s reproducibility materials and clarifies that LLM assistance was limited to language polishing. The appendix overview lists where implementation details, analyses, limitations, and additional results are provided.

REPRODUCIBILITY STATEMENT

  • The paper says it provides dataset descriptions, training configurations and hyperparameters, sampling and evaluation procedures, and mathematical formulations and descriptions of its algorithms.It also states that models and training and inference code will be released upon acceptance.Source: REPRODUCIBILITY STATEMENT · p. 10

LLM USAGE

  • The authors state that an LLM was used only for language polishing, not research design, analysis, or conclusions.The appendix repeats this clarification in its LLM Usage section.Source: LLM USAGE · p. 14, B LLM USAGE · p. 14

A APPENDIX OVERVIEW

  • The appendix overview directs readers to sections on datasets, training settings, sampling and evaluation, and extended accuracy and entropy analyses.It also lists block-level token entropy analysis and analyses across varying generated lengths.Source: A APPENDIX OVERVIEW · p. 14
  • The appendix overview identifies limitations, Sudoku failure cases, additional experiments, and detailed examples of temporal oscillation as further material.It also lists studies of unified multi-task training, reward-scoring ablations, and voting on reinforcement-fine-tuned models.Source: A APPENDIX OVERVIEW · p. 14

C MORE IMPLEMENTATION DETAILS

The implementation details describe four evaluation datasets, reinforcement fine-tuning settings, and the sampling and evaluation procedure. Sampling uses semi-autoregressive blocks with low-confidence remasking, and results are selected from evaluations over a specified range of steps.

C.1 DATASETS

  • The four datasets cover grade-school math word problems, competition math, elementary word problems, and a combinatorial arithmetic game.GSM8K has 8.5K problems split into 7.5K training and 1K test examples; MATH500 is a 500-problem subset.Source: §C.1 · p. 15

C.2 TRAINING

  • Reinforcement fine-tuning uses 256-token sequences, batch size 6 per GPU, gradient accumulation over 2 steps, and LoRA with rank 128 and scaling factor 64.Answers that fail parsing are discarded for TSE computation, which uses answers from the second half of sampling steps.Source: §C.2 · p. 15

C.3 SAMPLING AND EVALUATION

  • Sampling splits sequences into left-to-right blocks and applies low-confidence remasking within each block; evaluation runs every 100 steps from 600 to 8,000, with the best result reported.Source: §C.3 · p. 15

D.1 ANALYSIS ON MATH500 AND SVAMP DATASETS

The supplementary analyses extend accuracy and entropy findings to MATH500 and SVAMP, test TSE patterns at longer generation lengths, and examine block-level entropy. The limitations discussion notes that the methods depend on useful intermediate predictions.

D.1 ANALYSIS ON MATH500 AND SVAMP DATASETS

  • MATH500 and SVAMP show a gap between Pass @1 and Pass @1 | t; SVAMP accuracy dips between steps 3 and 20 before recovering.Correct answers begin appearing near 4% of the way through sampling on MATH500 and 39% on SVAMP.Source: §D.1 · p. 15
  • For MATH500 and SVAMP, Finally-Correct questions have the lowest answer entropy throughout sampling, while Intermediate-Correct questions have lower early entropy than Always-Incorrect questions.SVAMP differs at the final step, where Intermediate-Correct questions retain relatively high entropy.Source: §D.1 · p. 15

D.2 TEMPORAL SEMANTIC ENTROPY OVER VARYING GENERATED LENGTH

  • At generation lengths 256 and 512, questions ending with incorrect answers retain relatively high TSE, consistent with the shorter-length pattern.The supplementary figure reports average TSE across datasets at both lengths.Source: §D.2 · p. 16, Figure S2 · p. 16

D.3 ANALYSIS ON THE AVERAGE ENTROPY IN BLOCKS

  • Within a block, average token entropy declines as sampling steps accumulate, then rises sharply when decoding moves to a new block.The paper attributes the decline to richer context and suggests simultaneous decoding of masked tokens likely contributes to the new-block spike.Source: §D.3 · pp. 16–17, Figure S3 · p. 17

E.1 DISCUSSIONS ON LIMITATIONS

  • The methods are limited on tasks where intermediate predictions are consistently inaccurate; the authors say they depend on the model’s ability to produce correct or near-correct trajectory answers.They note that combining TSE with accuracy reward may alleviate the reinforcement-fine-tuning issue to some extent.Source: §E.1 · p. 17

E.2 FAILURE CASE ANALYSIS

On Sudoku, temporal voting and negative-TSE reinforcement can underperform the base model. The analysis links this limitation to very low intermediate correctness, while combined TSE and accuracy rewards can outperform accuracy reward alone.

E.2 FAILURE CASE ANALYSIS

  • 2.8% accuracy with voting is 2.7% below the 5.5% base-model accuracy at generation length 512 on Sudoku.The table covers temporal self-consistency voting and temporal consistency reinforcement on Sudoku.Source: §E.2 · p. 17, Table S1 · p. 18
  • Below 5% average Temporal Accuracy across Sudoku’s intermediate steps leaves little signal for voting to identify correct answers or for reinforcement learning to guide the model.The figure compares temporal accuracy across datasets.Source: §E.2 · p. 18, Figure S4 · p. 18
  • Combining TSE with accuracy reward performs better than accuracy reward alone, whereas negative-TSE-only reinforcement fine-tuning performs worse on Sudoku.The authors hypothesize that TSE provides finer-grained rewards than binary correctness outcomes.Source: §E.2 · p. 18

F.1 TRAINING A UNIFIED MODEL ACROSS MULTIPLE TASKS

A jointly trained model using the combined reward outperforms supervised fine-tuning baselines across four reasoning benchmarks. The supplementary analyses compare scoring rules, test voting after reinforcement learning, and illustrate how flawed reasoning can turn initially correct answers into incorrect final outputs.

F.1 TRAINING A UNIFIED MODEL ACROSS MULTIPLE TASKS

  • Across GSM8K, MATH500, SVAMP, and Countdown, the combined reward method consistently outperforms supervised fine-tuning baselines.Examples include +3.9 points on GSM8K with LLaDA-1.5 at length 128 and +29.7 points on Countdown with LLaDA-1.5 at length 512.Source: Table S2 · p. 19, §F.1 · p. 19

F.2 ABLATIONS ON SCORING RULES FOR COMBINING TSE WITH ACCURACY REWARD

  • The spherical scoring rule is used by default because it demonstrates more superior results than the alternative reward-combination rules.The compared options include entropy, quadratic, spherical, and logistic scoring.Source: §F.2 · p. 19, Table S3 · p. 20

F.3 TEMPORAL SELF-CONSISTENCY VOTING AFTER RFT

  • Temporal Self-Consistency Voting improves performance for models trained with either negative TSE reward or combined TSE and accuracy reward.The authors describe voting and Temporal Consistency Reinforcement as complementary techniques that can be combined.Source: §F.3 · p. 20, Table S4 · p. 20

F.4 EXAMPLES OF TIME OSCILLATION

  • Flawed reasoning paths can make answers incorrect during temporal oscillation, even when the model initially reaches the correct answer.In one example, an erroneous cost deduction introduced during reasoning eventually leads to a wrong final response.Source: §F.4 · p. 21

G CHANGE LOG

  • The v2 change log reports correcting a LLaDA-1.5 LoRA evaluation bug and adding unified multi-task results; v3 added LLM Usage and Ethics Statement sections.The manuscript was also revised and condensed for readability.Source: G CHANGE LOG · p. 21

Related papers

Loading 2508.09138v3…