PaperlayerView exact revision on arXiv

Source-linked AI summary

Step-level Verifier-guided Hybrid Test-Time Scaling for Large Language Models

Kaiyan Chang, Yonghao Shi, Chenglong Wang, Hang Zhou, Chi Hu, Xiaoqian Liu, Yingfeng Luo, Yuan Ge, Tong Xiao, Jingbo Zhu

arXiv:2507.15512v3cs.CL

TL;DR

Training-free test-time scaling can avoid the computational costs of training-based approaches, but often underperforms on complex reasoning. The paper introduces conditional step-level self-refinement and combines it with parallel scaling under process verification. Across five instruction-tuned models, the hybrid approach consistently improves reasoning performance, with a reported maximum increase of 28.6%.

  • Problem

    Training-based TTS adds training and test-time computational overhead, while training-free methods generally underperform on complex reasoning, particularly sequential self-refinement.

    Source: §1 · p. 1

  • Method

    The paper proposes conditional step-level self-refinement, then combines fine-grained sequential scaling with parallel methods using a step-level verifier.

    Source: §1 · p. 2

  • Results

    Hybrid Test-Time Scaling consistently improves reasoning across five instruction-tuned LLMs on MATH500, AIME24, and GPQA Diamond, with a maximum performance increase of 28.6%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The experiments support that sequential and parallel scaling methods can be orthogonal and complementary, and that training-free TTS can enhance smaller instruction-tuned models.

    Source: §6 · p. 9

    The experiments perform only one inference run, so performance may fluctuate.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Test-Time Scaling (TTS) is a promising approach to progressively elicit the model's intelligence during inference. Recently, training-based TTS methods, such as continued reinforcement learning (RL), have further surged in popularity, while training-free TTS methods are gradually fading from prominence. However, the additional computation overhead of training amplifies the burden on test-time scaling. In this paper, we focus on training-free TTS methods for reasoning. We first design Conditional Step-level Self-refinement, a fine-grained sequential scaling method guided by process verification. On top of its effectiveness, we further combine it with other classical parallel scaling methods at the step level, to introduce a novel inference paradigm called Hybrid Test-Time Scaling. Extensive experiments on five instruction-tuned LLMs across different scales (3B-14B) and families demonstrate that hybrid strategy incorporating various training-free TTS methods at a fine granularity has considerable potential for expanding the reasoning performance boundaries of LLMs.

1 Introduction

Training-free test-time scaling avoids some costs of training-based approaches, but often underperforms on complex reasoning. The paper proposes fine-grained conditional self-refinement and combines it with parallel scaling at the step level; experiments report consistent improvements across models and benchmarks.

1 Introduction

  • 28.6% maximum performance increase is reported for Hybrid Test-Time Scaling, which consistently improves five instruction-tuned LLMs across MATH500, AIME24, and GPQA Diamond.Qwen2.5-3B-Instruct surpasses RL-enhanced DeepSeek-R1-Distill-Qwen-7B by 2.4% on GPQA Diamond.Source: Table 1 · p. 1, §1 · p. 2
  • Training-based TTS can add training cost and overthink simple problems, while training-free methods mitigate these burdens but often underperform on complex reasoning.The paper identifies sequential self-refinement as a case with marginal performance gains.Source: §1 · p. 1
  • Conditional step-level self-refinement is proposed to demonstrate the potential of fine-grained sequential scaling.Source: §1 · p. 2
  • Hybrid Test-Time Scaling combines parallel and sequential scaling at each reasoning step, using a verifier to navigate the model’s solution space.Source: §1 · p. 2

2 Related Works

Related work spans parallel, sequential, and hybrid training-free scaling, including solution-level and step-level search. The paper motivates precise process verification to evaluate intermediate steps and guide scaling.

2.1 Training-free Test-Time Scaling

  • Hybrid scaling combines parallel sampling and aggregation with sequential iterative refinement, drawing on their complementary benefits.Source: §2.1 · p. 2
  • Prior hybrid approaches include solution-level MCTS and step-level thought graphs, but solution-level exploration can be insufficient and self-feedback can inherit model hallucinations.SR-MCTS evaluates mini-steps using alternative-token log probabilities; the paper proposes a high-quality verifier to reduce error accumulation and guide intermediate steps.Source: §2.1 · p. 2

2.2 Verifiers

  • Outcome reward models evaluate final answers, whereas process reward models evaluate reasoning paths; the paper adopts Qwen2.5-Math-PRM-7B for step-level verification.Prior PRM work includes human-annotated and automatically constructed process-wise supervision, as well as consensus filtering.Source: §2.2 · pp. 2–3

3 Method

The method controls reasoning at step granularity, verifies each step before continuing, and combines parallel sampling with conditional refinement in an MCTS search. A PRM guides selection and refinement so search can assess intermediate steps without first completing the full solution.

3.1 Control of Generation Granularity

  • “Pause-then-Continue” uses “\n\n” to pause generation after each reasoning step, then appends “\n” to resume until “<eos>”.The approach relies on instruction-tuned LLMs with at least 3B parameters to follow this generation control.Source: §3.1 · p. 3

3.2 Conditional Step-level Self-refinement

  • Unconditional reflection risks revising correct reasoning and continuing past an optimal solution, motivating real-time verification.The method verifies each current step before generating the next, aiming to reduce error accumulation and unnecessary refinement.Source: §3.2 · p. 3
  • A PRM scores each reasoning step from 0 to 1 using the question and preceding steps; reflection is triggered below a predefined score threshold.A revised step is kept only when its PRM score exceeds the current step’s score; refinement stops at a target score, when gains plateau, or at the iteration limit.Source: §3.2 · pp. 3–4

3.3 Step-level Verifier-guided Hybrid Test-Time Scaling

  • Hybrid Test-Time Scaling combines Best-of-N sampling and conditional self-refinement within a step-level tree-search framework guided by a PRM.The PRM evaluates candidate steps, and the most promising is selected for expansion into the next reasoning-path node.Source: §3.3 · p. 4
  • At each search step, Best-of-N samples candidate reasoning steps in parallel and selects an action using a PUCT score combining PRM value, LLM prior, and visit count.The selected candidate undergoes conditional refinement when its PRM score is below a predefined threshold; an improved revision is adopted, otherwise the original is retained.Source: §3.3 · pp. 4–5
  • “Deep Scaling” applies multi-sampling and iterative refinement to intermediate steps, using PRM assessments rather than waiting for a complete solution to evaluate node quality.The selected step becomes the new leaf node for subsequent search expansion.Source: §3.3 · p. 5

4 Experimental Setup

Experiments span five instruction-tuned LLMs, three reasoning datasets, and three correctness metrics. The setup compares conditional step-level refinement and hybrid scaling with related scaling baselines under specified verifier and sampling conditions.

Models

  • The study evaluates five instruction-tuned models from Qwen2.5, LLaMA3.1, and Gemma3, spanning 3B to 14B; RL-enhanced LLMs are excluded because their implicit reasoning lacks clear step-by-step structure.Qwen2.5-Math-PRM-7B serves as the process verifier.Source: §4 · p. 5

Datasets

  • Evaluation uses MATH500, AIME24, and GPQA Diamond, covering mathematical computation, complex reasoning, and scientific questions.The passage characterizes the datasets as significantly difficult.Source: §4 · p. 5

Evaluation

  • Maj@k measures majority-vote correctness, RM@k measures correctness of the reward-model-selected solution, and Pass@k measures whether any of k outputs is correct.For RM@k, the selected path maximizes its minimum PRM score.Source: §4 · p. 5
  • Figure 2 compares four common training-free TTS methods using Qwen2.5-7B-Instruct on MATH500; “SR” denotes one iteration of basic self-refinement.Source: Figure 2 · p. 6

Baselines

  • Conditional step-level self-refinement is compared with parallel scaling and solution-level self-refinement under identical verifier and iteration settings; hybrid scaling is compared with pairwise combinations and solution-level OpenR under equivalent conditions.OpenR comparisons use equivalent sampling, search-path, and refinement conditions.Source: §4 · p. 6

5 Results and Analyses

Preliminary experiments favor conditional, fine-grained step-level refinement over solution-level refinement and show that verifier-based update and stopping rules matter. Across the main experiments, Hybrid Test-Time Scaling improves results across models and datasets, while ablations favor the complete strategy over pairwise combinations.

5.1 Preliminary Experiments for Conditional Step-level Self-refinement

  • Step-level self-refinement consistently outperforms solution-level self-refinement across four LLMs, showing an advantage for fine-grained sequential scaling.Basic one-iteration step-level refinement also outperforms other training-free TTS baselines reported in Ding et al. (2025).Source: §5.1 · p. 6, Figure 3 · p. 6
  • With more iterations, PRM_Cover yields steady gains, whereas unconditional Cover degrades performance, even below baseline.PRM_Cover overwrites a step only when its refined PRM score improves on the previous iteration.Source: §5.1 · p. 6, Figure 4 · p. 7
  • Refinement peaks at round five and plateaus afterward; the adopted rules cap iterations at five, skip steps scoring above 0.9, and stop when PRM improvement over two rounds is below 0.2.These conditions aim to balance performance and efficiency while abandoning steps deemed unlikely to be corrected.Source: §5.1 · pp. 6–7
  • PRM score sums correlate linearly with problem difficulty, supporting selective reflection that avoids overthinking easy problems and strategic giving up on excessively difficult ones.The authors describe these choices as improving efficiency.Source: §5.1 · pp. 6–7

5.2 Main Results for Hybrid Test-Time Scaling

  • Maj@k and RM@k generally trail Pass@k, with a gap reaching 20.7% for Gemma3-4B-IT on GPQA Diamond; accurate verification matters for answer selection and intermediate-step guidance.Using the weaker Math-Shepherd-Mistral-7B-PRM improves on the 75.2 baseline but remains behind Qwen2.5-Math-PRM-7B.Source: §5.2 · pp. 7–8
  • Hybrid TTS improves baselines across models and datasets, with peak gains of 28.6% for LLaMA3.1-8B-Instruct on MATH500, 13.4% for Qwen2.5-14B-Instruct on AIME24, and 21.5% for Qwen2.5-3B-Instruct on GPQA Diamond.The GPQA Diamond results also exceed the RL-enhanced DeepSeek-R1-Distill-Qwen-7B baseline for Qwen2.5-3B/7B-Instruct and Gemma3-4B-IT. Search-space expansion results on GPQA varied under identical settings, which the authors attribute to generation instability at higher temperatures.Source: §5.2 · pp. 7–8, Table 2 · p. 8
  • All models show performance growth with more samples and search paths on MATH500, while GPQA gains from expanded search are unstable across identical high-temperature runs.The authors attribute the variability to generation instability and argue that accurate external verification can support more precise exploitation of the solution space.Source: §5.2 · p. 8

5.3 Ablation Study

  • Every tested pairwise combination improves on baseline, but none matches the complete three-component Hybrid TTS strategy.The authors report that the maximum of five refinement rounds and conservative stopping conditions increase inference cost; they chose a strategy maximizing reasoning performance.Source: Table 4 · p. 9, §5.3 · p. 9

6 Conclusion

The paper proposes step-level verifier-guided Hybrid Test-Time Scaling, combining conditional self-refinement with parallel scaling methods. Experiments support the complementarity of these methods and the role of accurate step-level verifiers in improving smaller instruction-tuned LLMs.

6 Conclusion

  • The authors conclude that deeply exploring and precisely exploiting the LLM solution space is central to test-time scaling.They report that training-free TTS can enhance smaller instruction-tuned LLMs, underscoring the role of accurate step-level verifiers.Source: §6 · p. 9

Limitations

The experiments used a single inference run per setting, so performance may fluctuate. The appendix describes meta-prompts for step- and solution-level self-refinement as a performance-efficiency trade-off.

Limitations

  • The experiments run inference only once per setting, so performance may fluctuate.The authors frame the study as exploring the reasoning-performance boundaries of hybrid training-free TTS methods.Source: Limitations · p. 10

A Meta-prompts for Self-refinement

  • The meta-prompts for step- and solution-level self-refinement were designed to trade off performance and efficiency.The design follows efficient-prompting principles outlined in Chang et al. (2024).Source: A Meta-prompts for Self-refinement · p. 13
  • The step-level critic prompt requests a strict correction of one weak reasoning step in fewer than 150 tokens, without outputting the full solution.The solution-level critic prompt instead asks for a strict reflection correcting a weak solution.Source: A Meta-prompts for Self-refinement · p. 13

B The Algorithm for Hybrid Test-Time Scaling

The algorithm combines conditional step-level self-refinement with Best-of-N and Tree Search under a step-level verifier. The accompanying table is scoped to computational overhead across TTS combinations.

B The Algorithm for Hybrid Test-Time Scaling

C The Computational Overhead of Different Combinations of TTS Methods

Under comparable computational costs, Hybrid Test-Time Scaling outperforms pairwise combinations of parallel and sequential strategies. The comparison accounts for inference FLOPs through generated tokens across the framework’s generative steps.

C The Computational Overhead of Different Combinations of TTS Methods

  • With fixed model configurations, inference FLOPs are primarily determined by total generated tokens aggregated across the framework’s generative steps.The comparison includes Best-of-N sampling, Tree Search expansion, and self-refinement generation, reflection, and rewriting.Source: C The Computational Overhead of… · p. 15
  • Under comparable computational costs, Hybrid Test-Time Scaling consistently outperforms pairwise combinations such as MCTS+BoN and BoN+Self-Refinement.The authors report that only the synergistic combination of all three strategies lets the model fully realize its potential with training-free TTS.Source: C The Computational Overhead of… · p. 15
  • The reported comparisons include Best-of-16 for BoN+Self-Refinement, Best-of-8 for MCTS+BoN, and Best-of-4 for Hybrid TTS.Source: C The Computational Overhead of… · p. 15

D Verifier-indicated Difficulty Levels

The section examines whether solution-level PRM scores track problem difficulty, finding a linear correlation on MATH500 across five iterations of Qwen2.5-7B-Instruct.

E Instability of LLM during Training-free Test-Time Scaling

Under identical test-time-scaling settings, model performance can vary significantly. In a case study, step-level self-refinement corrected an error after one round, while solution-level refinement did not.

E Instability of LLM during Training-free Test-Time Scaling

  • Performance can vary significantly under the same setting; Figure 7 reports repeated Qwen2.5-7B-Instruct trials on MATH500 with Best-of-4 and 16-path search.Source: E Instability of LLM during… · p. 15

F Case Study of Step-level and Solution-level Self-refinement

Loading 2507.15512v3…