Sampling via Decision-Flow: Training-Free Extraction of Improved Latent Reasoning Paths in Large Language Models

Motivation

• The Distribution Sharpening hypothesis holds that RL reallocates probability toward high-reward trajectories already latent in base models.The section notes that RL can improve single-attempt metrics while reducing diversity and reasoning coverage.Source: §1 · p. 1

Research questions

• The section asks whether latent high-quality paths can be unlocked without costly RL fine-tuning and how to find them when local choices can yield globally wrong trajectories.Existing MCMC-inspired approaches correct reasoning locally, but local optimality does not guarantee global optimality.Source: §1 · p. 1

DF-Sample

• DF-Sample builds a hierarchical reasoning tree, scores complete chains, and propagates global quality signals backward to guide intermediate decisions.This training-free inference framework aims to recover high-quality paths that standard sampling overlooks.Source: §1 · pp. 1–2

Extensive empirical validation

• 81.8% accuracy on MATH500 with Qwen2.5-Math-7B exceeds GRPO (78.5%) and power sampling (74.8%).Figure 1 presents comparisons across MATH500, HumanEval, GPQA, and AlpacaEval 2.0.Source: Figure 1 · p. 2, §1 · p. 2

Correct reasoning paths are latent, not absent

• Correct reasoning paths may already be present in base models but receive lower probability than incorrect alternatives.For x^2 = 5x, dividing by x can omit the solution x = 0, whereas factoring preserves both solutions.Source: §2 · p. 2, Figure 2 · p. 2
• A higher-probability solution to x^2 = 5x can miss x = 0 because dividing by x implicitly assumes x ≠ 0.The passage says this path is more probable because it matches common patterns and takes fewer steps, not because it is more correct.Source: §2 · p. 2

Local sampling optimality does not guarantee global reasoning quality

• A locally plausible path can still be globally wrong, so single-step quality is not a reliable signal of complete-trajectory quality.For the injectivity example, a derivative-based path concludes incorrectly, while a symmetry-based path reaches the correct conclusion.Source: §2 · pp. 2–3, Figure 3 · p. 3

Motivation

• These examples motivate evaluating complete paths by path-level quality instead of selecting them by step-level probability.The mismatch between generation probability and reasoning quality is framed as a sampling problem rather than a need to train better reasoning.Source: §2 · pp. 2–3

Sequence notation

• A token sequence has length T and consists of tokens from a finite discrete vocabulary Y.The full sequence is denoted y0:T.Source: §3 · p. 3
• At time t, the prefix y<t contains earlier tokens and the suffix y>t contains future tokens.The prefix is the historical context; the suffix is the generation of future tokens.Source: §3 · p. 3

Language-model distribution

• An autoregressive language model factorizes sequence likelihood into conditional token probabilities and samples a complete sequence sequentially.The conditional distributions are pθ(yt | y<t).Source: §3 · p. 3

4.1 Hierarchical Reasoning Tree Construction

• DF-Sample expands the reasoning tree by sampling candidate continuations, then completes the tree before selection so paths can be evaluated globally.Each node records its generated reasoning text and accumulated path log-probability.Source: §4.1 · p. 3

4.2 Terminal Node Energy Evaluation

• Terminal-node energy scores complete chains using final-output quality and model likelihood, with lower energy indicating a higher-quality path.The final reasoning step is treated as an efficient proxy because it typically summarizes the chain's outcome.Source: §4.2 · pp. 3–4

4.3 Decision-Flow Backward Propagation

• Each non-leaf node receives the prior-weighted expected utility of its children, carrying trajectory-level quality information back through the tree.This gives intermediate decisions information about global path quality rather than only local next-step probabilities.Source: §4.3 · pp. 4–5

4.4 Posterior Path Selection

• Posterior selection combines generation priors with propagated utility, favoring children whose downstream paths tend to be better.A locally improbable step that consistently leads to correct conclusions can be favored over a probable step leading to wrong ones.Source: §4.4 · p. 5

4.5 Efficient Inference via Block-wise Sampling

• When the estimated reasoning depth exceeds threshold B, block-wise sampling selects partial trajectories and reuses them as context for subsequent blocks.This strategy addresses the substantial computational overhead from the exponential growth of candidate nodes as depth increases.Source: §4.5 · p. 5

5.1 Setup

• Experiments use three model families and four benchmarks spanning mathematics, code, science, and general helpfulness, with greedy decoding, low-temperature sampling, GRPO, and power sampling as baselines.MATH500 has 500 problems, HumanEval 164 tasks, GPQA-Diamond 198 questions, and AlpacaEval 2.0 805 prompts.Source: §5.1 · pp. 5–6

5.2 Main Results

• DF-Sample with Qwen2.5-Math-7B achieves 45.6% on GPQA-Diamond versus GRPO’s 39.9%, and 81.8% on MATH500 versus GRPO’s 78.5%, a 3.3-point gain without parameter updates.Source: §5.2 · p. 6

5.3 Analysis

• DF-Sample responses have higher average log-likelihood than power sampling while retaining a broader spread than GRPO, consistent with access to a wider region of the base model distribution.The analysis describes DF-Sample as favoring higher-quality paths while preserving broader likelihood coverage than GRPO.Source: §5.3 · pp. 6–7, Figure 6 · p. 7
• DF-Sample spans lower token-confidence regions than GRPO, indicating selection of locally uncertain paths that are globally correct.GRPO concentrates near the highest-confidence region, while DF-Sample shows the largest spread and extends furthest into low-confidence regions.Source: §5.3 · p. 7, Figure 7 · p. 7
• About 619 generated tokens for DF-Sample is comparable to the base model’s about 600; its MATH500 accuracy is 81.8% versus power sampling’s 74.8%.Power sampling takes about 340 seconds per question and DF-Sample around 384 seconds.Source: §5.3 · p. 7, Figure 8 · p. 7

5.4 Ablation Study

• On GPQA-Diamond, DF-Sample accuracy is about 1% above power sampling at K = 2 and about 18% higher at K = 4; larger K also increases inference latency.The experiments use K = 3 to balance accuracy and efficiency.Source: §5.4 · p. 7, Figure 9 · p. 7
• DF-Sample has the highest MATH500 pass@1 and pass@2 accuracy, while power sampling slightly surpasses it at a few larger k values.The methods become comparable as k increases, with differences remaining small.Source: §5.4 · pp. 7–8, Figure 10 · p. 8
• On GPQA-Diamond, moderate α values around 4 or 5 perform best; values that are too large or too small degrade path discrimination and accuracy.The ablation reports that overly large α favors high-probability paths, while overly small α makes the distribution too flat.Source: §5.4 · p. 8, Table 2 · p. 8

RL for LLM reasoning

• RLHF aligns outputs with human preferences, while RLVR directly optimizes task correctness through automated verifiers; GRPO uses group-relative policy updates (Shao et al., 2025).These post-training approaches require expensive fine-tuning and may reduce output diversity (Shao et al., 2025; He et al., 2025).Source: §6 · p. 8

Inference-Time Reasoning Search

• Tree of Thoughts (Yao et al., 2023), HyperTree (Gui et al., 2025), and ReST-MCTS (Zhang et al., 2024) explore tree-structured reasoning, whereas DF-Sample evaluates trajectories with global path-level utilities.Other work dynamically allocates branching compute during reasoning (Inoue et al., 2025); Power Sampling uses MCMC-style sampling (Karan and Du, 2025).Source: §6 · p. 8

Sampling-Based Methods

• Sampling methods reshape or approximate output distributions toward high-reward trajectories, including QUEST’s Metropolis–Hastings resampling and annealed sampling’s temperature scheduling.Decision Flow uses GFlowNet-style backward regression over sequence graphs (Chertkov et al., 2025); DF-Sample applies decision-flow-style evaluation to LLM reasoning trajectories.Source: §6 · p. 8

• DF-Sample combines terminal-node utility estimation and backward utility propagation in a hierarchical reasoning tree to identify high-quality, low-probability paths overlooked by standard decoding.Across reasoning benchmarks, it improves over base models and competitive sampling baselines and achieves performance comparable to RL-based approaches without parameter updates.Source: §7 · p. 8

• DF-Sample’s inference-only design leaves open whether its discovered reasoning patterns can be internalized through training or fine-tuning.The paper identifies integrating trajectory selection with learning-based approaches as a possible direction.Source: Limitations · p. 9