DiffCoT: Diffusion-styled Chain-of-Thought Reasoning in LLMs

1 Introduction

• Step-by-step CoT can propagate early mistakes into later reasoning, while teacher-forcing training exposes models to correct prefixes unlike error-containing inference prefixes.The introduction identifies reducing errors by guiding models toward reliable reasoning paths in long thought chains as the central challenge.Source: §1 · p. 1
• DiffCoT reframes multi-step reasoning as iterative denoising, progressively refining corrupted trajectories into coherent chains of thought.The approach aims to recover from diverse and compounding mistakes.Source: §1 · p. 2
• DiffCoT combines a sliding-window mechanism and causal noise schedule to revise earlier reasoning steps while preserving token-level autoregression and temporal dependencies.This design balances global error correction with the causal structure needed for coherent reasoning.Source: §1 · p. 2
• DiffCoT outperforms existing state-of-the-art preference-optimization methods on three mathematical reasoning benchmarks, with robust gains and improved error-correction capability.The framework is presented as an adaptive approach to multi-step CoT reasoning.Source: §1 · p. 2

2 Preliminaries

• A CoT trace consists of reasoning steps whose final step is the answer, with each step conditioned on the prompt and previously generated steps.The policy is trained on annotated CoT data by maximizing conditional likelihood.Source: §2 · p. 3
• Diffusion models gradually corrupt data toward a tractable noise distribution, then learn reverse transitions that reconstruct the sample through stepwise denoising.Inference iteratively applies the learned reverse transitions from x_T until x_0 is obtained.Source: §2 · p. 3

3.1 Diffusion-Styled Noising for CoT

• DIFFCOT assigns weaker noise to higher-reward step candidates and progressively stronger noise to lower-reward candidates, forming clean-to-corrupted reasoning states.The data construction gathers diverse responses across trajectories without requiring manual labels for correct, incorrect, or corrective steps.Source: §3.1 · pp. 3–4, Figure 2 · p. 4
• MCTS builds step-by-step search trees, collecting multiple candidate responses per step scored by a reward model or rollout-based success rates.The highest-reward candidate is treated as the lowest-noise state; other candidates are ordered by reward.Source: §3.1 · p. 4

3.2 Reformulating CoT Reasoning as Denoising

• A diffusion sliding window iteratively denoises previously generated steps while advancing autoregressively to generate the next step.This combines self-correction with forward generation and fine-tunes a pretrained LLM rather than training a separate diffusion language model from scratch.Source: §3.2 · pp. 4–5, Figure 2 · p. 4
• DIFFCOT’s preference training uses prefixes containing clean earlier steps and noisy steps inside the sliding window, supporting preference-consistent updates under partially corrupted context.The method adopts a DPO loss for preference pairs and aligns intermediate generations to the best sample as the winning target.Source: §3.2 · p. 5

3.3 Causal Diffusion Noise

• The causal noise schedule perturbs earlier reasoning steps less and later steps more, encoding step-wise dependencies within the sliding window.The method presents this schedule as a way to stabilize the causal chain and support self-correction in subsequent steps, even when a current step is erroneous.Source: §3.3 · p. 5

4.1 Settings

• Experiments use four backbones and three mathematical reasoning benchmarks, comparing DIFFCOT with CoT, search-based, supervised fine-tuning, and preference-optimization baselines.The backbones are Llama3-8B, Qwen3-8B, Ministral3-8B, and Qwen3-4B; the benchmarks are GSM8K, SVAMP, and MATH.Source: §4.1 · p. 6

4.2 Main Results

• DIFFCOT outperforms existing preference-optimization approaches on most benchmarks and evaluation settings, although it does not achieve the single best result in every case.Its gains are more pronounced on stronger backbones and harder MATH problems, where correct reasoning paths are sparser.Source: §4.2 · p. 7

4.3 Ablation Study

• Window size and stride extremes degrade performance, reflecting a trade-off between causal connectivity and the ability to correct earlier errors.A size and stride of 1 approximates autoregressive reasoning, while setting them to K reverts to diffusion; excessively long windows disrupt coherence.Source: §4.3 · p. 7, Table 2 · p. 7
• Randomly shuffling the noising order instead of following the accuracy-based progression substantially degrades performance across models and datasets.The ablation tests the causal noise schedule on GSM8K and SVAMP.Source: §4.3 · p. 7, Table 2 · p. 7

4.4 Analysis

• DIFFCOT progressively revises semantically unhelpful earlier steps rather than rigidly propagating their noise, allowing trajectories to move toward a correct solution.The case study contrasts this behavior with autoregressive reasoning, which carries early semantic noise forward without revision.Source: §4.4 · pp. 7–8, Figure 3 · p. 8
• On 300 GSM8K problems, DIFFCOT achieves substantially higher correction success rates than Full-Step-DPO across backbones under midpoint prefix corruption.Each preceding step is independently perturbed with probability ω.Source: §4.4 · p. 8

• Preference learning progressed from solution-level objectives to step-level signals and whole-trajectory optimization, but many step-wise methods train on clean prefixes and remain vulnerable to inference-time error accumulation.The passage contrasts this teacher-forcing setup with the paper’s framing of reasoning as a revisable trajectory.Source: §5 · p. 8
• Mathematical-reasoning research includes stronger pretraining or supervised fine-tuning, test-time generation and reranking, reward-guided decoding, and trajectory- or step-level preference optimization.The passage describes these approaches as improving mathematical reasoning or test-time performance.Source: §5 · pp. 8–9
• Prior search and alignment methods largely operate forward-only, motivating a diffusion-styled approach that can revise corrupted intermediate reasoning.The paper positions trajectory revision as a way to address a limitation not handled by forward-only approaches.Source: §5 · pp. 8–9

• DIFFCOT combines sliding-window diffusion and causal noise with autoregressive generation to refine intermediate steps, and future work will test scalability across more backbones and reasoning domains.Source: §6 · p. 9

• Off-policy preference data may create distribution shift, biased value estimates, and training instability, especially on harder datasets or longer reasoning chains.Source: Limitations · p. 9
• Revisiting historical steps breaks the local Markov property of prefix-conditioned generation, increasing uncertainty and controllability challenges and typically requiring more iterations and data for stable convergence.Source: Limitations · p. 9

A Data Statistics

• 300 training and at most 300 test instances are sampled from each dataset, with the same configuration used for all baselines.MATH test instances are sampled separately from each of its five difficulty levels.Source: A Data Statistics · p. 12
• The evaluation covers SVAMP, MATH, and GSM8K, with MATH organized into five difficulty levels.Source: A Data Statistics · p. 12
• Table 3 reports each dataset’s train and test counts, maximum reasoning-step budget K, and number of reasoning windows.Source: Table 3 · p. 12

B Implementation Details

• Thought search samples five candidate thoughts per prefix, scores each using eight Monte Carlo rollouts, then greedily selects the candidate with the highest success rate.Search repeats until the reasoning process terminates.Source: B Implementation Details · pp. 12–13
• Fine-tuning uses LoRA adapters with DPO loss, AdamW, a cosine learning-rate schedule, and one training epoch.Adapters are placed in each self-attention layer’s q_proj and v_proj projections.Source: B Implementation Details · p. 13
• Full-Step-DPO uses rollout-estimated success rates in place of PRM step rewards, while CPO and ToT preference pairs use LLM self-evaluation.Source: B Implementation Details · p. 13

C Discussion with Reinforcement Learning

• The discussion links trajectory-level optimization to error accumulation and presents DiffCoT’s sliding window as coupling reasoning generation with preference optimization.The authors suggest future work could integrate reinforcement learning to optimize the window’s denoising dynamics.Source: C Discussion with Reinforcement Learning · p. 13

D Additional Experiments

• Additional experiments cover β sensitivity, full test sets, alternative evaluation protocols, and AIME.Source: D Additional Experiments · p. 13

D.1 Sensitivity Analysis of Parameter β

• DiffCoT remains stable across a reasonable variation in β, without a sharp performance drop.Table 4 reports accuracy changes relative to the default β = 0.1 setting.Source: §D.1 · p. 14, Table 4 · p. 14

D.2 Results on the Full Test Set

• DiffCoT achieves the best performance on most model–dataset combinations in full-test evaluations, with substantial gains for Qwen models on MATH-L4.Table 5 covers the full SVAMP and MATH-L4 test sets.Source: Table 5 · p. 14, §D.2 · p. 14

D.3 Results under Different Evaluation Protocols

• Format-agnostic scoring ignores formatting, while contains_gt accepts a response if it includes the ground-truth answer anywhere.Source: §D.3 · p. 14
• DiffCoT consistently improves results under strict, format-agnostic, and contains_gt evaluation protocols.Tables 6 and 7 report Qwen3-4B and Qwen3-8B results under the different protocols.Source: Table 6 · p. 14, Table 7 · p. 15, §D.3 · p. 15

D.4 Results on AIME

• AIME results are reported only for Qwen3-4B and Qwen3-8B because Llama3-8B and Ministral3-8B produced zero correct answers on AIME 1983–2025.The authors state that this made preference-optimization training infeasible for those models.Source: §D.4 · p. 15
• On the AIME benchmark, the method remains competitive and achieves strong performance across different backbones, with results indicating generalization to harder mathematical reasoning benchmarks beyond SVAMP and MATH.Source: Table 8 · p. 15, §D.4 · p. 15