PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Evaluation-time Compute with Reasoning Models as Evaluators

Seungone Kim, Ian Wu, Jinu Lee, Xiang Yue, Seongyun Lee, Mingyeong Moon, Carolin Lawrence, Kiril Gashteovski, Julia Hockenmaier, Graham Neubig, Sean Welleck

arXiv:2503.19877v3cs.CL

TL;DR

The paper asks whether test-time compute can improve LM evaluation, then studies reasoning models prompted to judge whole responses and individual steps. It reports gains in evaluator performance and fixed-budget generation reranking.

  • Problem

    It is unclear whether evaluators, like generators, can improve by scaling test-time compute with long chains of thought, and whether improved evaluation can also improve generation.

    Source: §1 · p. 1

  • Method

    The paper prompts reasoning models to evaluate responses as a whole and assess their individual steps, then combines the judgments for evaluation and reranking.

    Source: §1 · p. 1, §2.2 · p. 3

  • Results

    Evaluation-time scaling improves evaluation performance and generation quality; with fixed compute, reasoning evaluators using Best-of-8 outperform direct evaluators using Best-of-64 by 4.30–6.63%.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Allocating more test-time compute to evaluation rather than sampling more responses can improve generator performance under a fixed compute budget.

    Source: §1 · p. 2, §4 · p. 6

    The study focuses on math and code, leaving tasks such as creative and scientific writing for future work.

    Source: Limitations · p. 9

Abstract

from arXiv · show

As language model (LM) outputs get more and more natural, it is becoming more difficult than ever to evaluate their quality. Simultaneously, increasing LMs' "thinking" time through scaling test-time compute has proven an effective technique to solve challenging problems in domains such as math and code. This raises a natural question: can an LM's evaluation capability also be improved by spending more test-time compute? To answer this, we investigate employing reasoning models-LMs that natively generate long chain-of-thought reasoning-as evaluators. Specifically, we examine methods to leverage more test-time compute by (1) using reasoning models, and (2) prompting these models to evaluate not only the response as a whole (i.e., outcome evaluation) but also assess each step in the response separately (i.e., process evaluation). In experiments, we observe that the evaluator's performance improves monotonically when generating more reasoning tokens, similar to the trends observed in LM-based generation. Furthermore, we use these more accurate evaluators to rerank multiple generations, and demonstrate that spending more compute at evaluation time can be as effective as using more compute at generation time in improving an LM's problem-solving capability.

1 Introduction

The paper asks whether spending more test-time compute can improve LM evaluators and, in turn, generation. It studies reasoning evaluators and reports gains in both evaluation and fixed-budget reranking.

1 Introduction

  • The work asks whether evaluators, like generators, can benefit from test-time scaling with long chains of thought, and whether improved evaluation can improve generation.The introduction frames generation and evaluation as interrelated: better evaluators can be integrated into inference-time algorithms.Source: §1 · p. 1
  • The paper examines reasoning evaluators—reasoning models prompted to evaluate responses—and prompts them to assess both each response step and the solution as a whole.This combines process-based and outcome-based evaluation approaches.Source: §1 · pp. 1–2
  • A 32B reasoning evaluator outperforms a 72B state-of-the-art PRM by 4.5% on ProcessBench without evaluator training.The paper also reports that evaluator performance improves monotonically as reasoning tokens increase.Source: §1 · p. 2
  • With a fixed compute budget, reasoning evaluators using Best-of-8 outperform direct evaluators using Best-of-64 by 4.30–6.63%.The evaluators rerank multiple solutions sampled by a generator.Source: §1 · p. 2

2 Methodology

The methodology prompts reasoning models to evaluate both final answers and individual response steps, then combines those judgments into a score. Its process-evaluation design uses stepwise assessment, model-based splitting, and score aggregation.

2 Methodology

  • Reasoning evaluators generate chain-of-thought before judging, unlike direct evaluators that predict scores without it; this work prompts off-the-shelf reasoning models to evaluate.The score for an outcome or response steps can be derived from answer-token logits rather than a trained reward-modeling head.Source: §2 · p. 2

2.1 Reasoning Outcome Evaluators

  • Outcome evaluators prompt for a “1” or “0” correctness judgment and use the logits for those tokens with a softmax to produce a scalar score.The judgment is made after the evaluator generates its chain-of-thought.Source: §2.1 · p. 3

2.2 Reasoning Process Evaluators

  • Multi-step process evaluation assesses each response step separately, using earlier steps as context; this avoids evaluating all steps at once and scales evaluation-time compute.An LM can insert split markers between steps, and the resulting step scores are aggregated; the mean_logit function performed better than min in the experiments.Source: §2.2 · p. 3

2.3 Combining outcome judgments and process scores

  • The method interpolates outcome and process scores: α = 0 uses only the process score, α = 1 only the outcome score, and experiments use α = 0.5.Outcome evaluation judges final-answer correctness, while process evaluation judges the correctness of each step.Source: §2.3 · p. 4

3 Evaluation-Time Scaling Trends for Process Evaluation

The experiments find that evaluating each solution step individually, especially when combined with self-consistency, improves ProcessBench error detection. Reasoning evaluators can outperform larger PRMs, while single-step reasoning evaluators do not match direct process evaluators.

3.1 Experimental setting

  • ProcessBench contains 3,400 instances from four math benchmarks, and the task is to identify the first paragraph with incorrect logic, if any.Performance is measured by F1; the evaluator is penalized for false error identification, a wrong error index, or failing to detect an error.Source: §3.1 · p. 4

3.2 Experimental results

  • DeepSeek-R1-Distill-Qwen-32B scores 75.5 average F1, outperforming Qwen2.5-72B-Instruct's 61.2 despite having 44% as many parameters.The authors report this association even though the reasoning model was not trained as an evaluator.Source: §3.2 · p. 5
  • Single-step reasoning did not match PRMs: DeepSeek-R1-Distill-Qwen-7B scored 54.5 F1 versus Qwen2.5-Math-7B-PRM800K's 56.2 and Qwen2.5-Math-PRM-7B's 73.5.The 32B reasoning models also fell behind Qwen2.5-Math-PRM-72B, 75.5 and 74.5 versus 78.3 F1.Source: §3.2 · p. 5
  • The experiment compares per-segment judgments with self-consistency over eight CoTs, at similar inference cost because ProcessBench responses average 7.56 steps.The reasoning process evaluator assesses each segment individually; self-consistency selects an error index by majority vote across trajectories.Source: §3.1 · p. 4, §3.2 · p. 5
  • Multi-step evaluation outperforms self-consistency at a similar compute budget, and also improves instruction-tuned LMs.DeepSeek-R1-Distill-Qwen-32B and QwQ-32B score 78.6 and 79.3 F1, respectively, versus 78.3 for Qwen2.5-Math-PRM-72B.Source: §3.2 · pp. 5–6
  • 82.8 F1 (DeepSeek-R1-Distill-Qwen-32B) and 82.0 (QwQ-32B) surpass Qwen2.5-Math-PRM-72B's 78.3 when multi-step evaluation and self-consistency are combined.The reasoning models were not explicitly trained as evaluators.Source: §3.2 · p. 6

4 Translating Improved Evaluation to Problem-Solving

The section tests whether spending compute on evaluation can improve Best-of-N problem solving instead of sampling more responses. At a comparable compute budget, reasoning evaluators using Best-of-8 match or outperform direct evaluators using Best-of-64, with additional gains from combining outcome and process scores.

4.1 Experimental setting

  • The evaluation uses three generators across seven benchmarks, with 64 responses generated per instance and 4,680 instances overall.LeetCode is reported with pass@1; the other six benchmarks use accuracy, averaged across the 21 generator-benchmark settings.Source: §4.1 · p. 6

4.2 Experimental results

  • At similar or lower total test-time compute, Best-of-8 reasoning evaluators match or outperform Best-of-64 direct evaluators.The x-axis combines generator and evaluation compute; Figure 3 compares direct and reasoning evaluators, with each dot representing a doubling of response count.Source: Figure 3 · p. 7, §4.2 · p. 7
  • Combining reasoning outcome and process scores can further improve Best-of-N performance.The authors hypothesize that the approaches provide complementary signals for assessing responses.Source: §4.2 · p. 7
  • Reasoning evaluators significantly outperform direct evaluators on code outputs with less compute, while direct evaluators show substantial reward-model over-optimization.The authors attribute this pattern partly to direct process evaluators' math-focused training and potentially suboptimal heuristic splitting for code.Source: §4.2 · pp. 7–8

5 Analyses on why reasoning process evaluators are effective

Reasoning process evaluators use varied multi-step checking patterns, including re-verification, edge-case search, and backtracking. They have higher precision but lower recall than reasoning outcome evaluators.

5 Analyses on why reasoning process evaluators are effective

  • Observed process-evaluation patterns include inspecting each step's implementation, re-verifying judgments, checking edge cases, and backtracking to the problem conditions.The examples are from evaluations of a LeetCode problem.Source: §5 · p. 8
  • The authors speculate that applying problem-solving strategies during evaluation is a key reason reasoning models evaluate better.They note similar patterns, such as verification between solution steps, in reasoning models' problem solving.Source: §5 · p. 8
  • Reasoning process evaluators have higher precision but lower recall than reasoning outcome evaluators; all-steps-correct predictions have false-positive rates of 3.8% and 3.5%.The rates are reported across different model sizes.Source: §5 · p. 8

6 Conclusion

The conclusion reports that scaling compute at evaluation improves evaluation performance, and can combine with generator test-time scaling to improve generation quality. Process evaluation tends to make more conservative predictions than outcome evaluation, and combining the two yields gains.

6 Conclusion

  • Evaluation-time scaling effectively enhances evaluator performance and can be combined with generator test-time scaling to further boost generation quality.Source: §6 · p. 8
  • Reasoning process evaluation tends to make more conservative predictions than reasoning outcome evaluation.Source: §6 · p. 8
  • Combining reasoning process and outcome evaluation yields performance gains.Source: §6 · p. 8

Limitations

The authors explain their dataset and model choices, and provide additional experimental results. Their study is bounded by the datasets and models they could test.

Limitations

  • ProcessBench is the sole dataset used to evaluate evaluator implementations; the authors favor its naturally occurring errors over synthetically perturbed examples, while noting REVEAL’s lower inter-annotator agreement (κ ∼0.5).ProcessBench covers reasoning traces across four benchmarks and 12 generators; REVEAL adds commonsense reasoning, which ProcessBench does not cover.Source: Limitations · p. 9

A.1 Model List for ProcessBench Experiments (section 3)

  • ProcessBench experiments cover direct PRMs, instruction-tuned models, and reasoning models; the authors exclude fine-tuned generative evaluators because they cannot readily detect process errors.The tested families include math-shepherd-mistral, Llama and Qwen instruction-tuned models, DeepSeek-R1-Distill-Qwen, and QwQ.Source: §A.1 · p. 14

B Additional Results for ProcessBench Experiments (section 3)

A.2 Model List for Best-of-N Experiments (section 4)

  • Best-of-N experiments include direct outcome and process evaluators, generative evaluators, reasoning outcome and process evaluators, and combined reasoning process-plus-outcome evaluators.The reasoning evaluator variants use DeepSeek-R1-Distill-Qwen-7B and -32B.Source: §A.2 · p. 14

C Additional Results and Discussion for Best-of-N Experiments (section 4)

C.1 Findings from Experiments

The findings report improved evaluator performance and Best-of-N results from reasoning evaluators, including outside math and code. They also discuss evaluation-time scaling, study constraints, and future directions.

C.1 Findings from Experiments

  • At fixed test-time compute, reasoning evaluators using Best-of-8 outperform or match direct evaluators using Best-of-64, and combining outcome and process scores can further improve results.The findings also report that reasoning evaluators are especially effective for coding.Source: Figure 5 · p. 15, §C.1 · p. 15
  • Evaluating each response step individually outperforms evaluating all steps at once, even when the evaluator LM is the same.Reasoning evaluators also outperform specially trained outcome evaluators such as CLoud-RM and Prometheus 2; model-based splitting helps direct process evaluators, but mean_logits aggregation does not.Source: §C.1 · p. 15

C.2 Checking statistical significance of results

  • Scores across N=1, 2, 4, and 8 are statistically significant at the 96% confidence level.Table 5 reports the statistical analysis of mean_logit scores for different sample sizes.Source: §C.2 · p. 16, Table 5 · p. 18

C.3 Testing on domains outside of math and code

  • Across economics, engineering, and law, reasoning evaluators using eight responses outperform the listed ORM and PRM baselines using 64 candidates.The reported comparisons include 7B against an 8B ORM and 7B PRM, and 32B against a 27B ORM and 32B PRM.Source: §C.3 · pp. 17–18, Table 6 · p. 18

D.1 Scaling Test-time Compute

  • Unlike prior test-time scaling work focused mainly on generation, this work scales evaluation compute by using reasoning models and assessing response steps.The authors describe evaluation-time scaling as a way to enhance evaluator capabilities.Source: §D.1 · p. 18, Table 7 · p. 19

D.2 Language Model Evaluators

  • Inaccurate evaluators can misrepresent model capabilities or yield diminishing returns as inference-time compute increases, motivating more robust evaluators.The work specifically focuses on reasoning models as process evaluators.Source: §D.2 · p. 19

D.3 Future Directions

  • The authors identify better training signals and training reasoning evaluators as future research directions.They propose investigating whether reasoning process evaluators can mitigate reward-model overoptimization during reinforcement learning.Source: §D.3 · p. 19

E Approximation of Test-Time Compute

The section estimates inference compute for comparing evaluator strategies, then explains why stronger evaluation can offset using fewer candidate responses.

E Approximation of Test-Time Compute

  • Inference compute is approximated using parameter count and token count; in the example, Best-of-8 with reasoning evaluation requires similar total compute to Best-of-64 with direct evaluation.The example allocates compute between generation and evaluation differently: reasoning evaluation uses more evaluation-time compute, while direct evaluation uses more generation-time compute.Source: E Approximation of Test-Time Compute · pp. 19–20

F Formal explanation of why evaluation-time scaling can improve the generator’s performance

  • As candidate count grows, imperfect evaluators are more likely to mis-select a response with an erroneously high score; stronger evaluation can reduce this risk.The explanation links this effect to reward-model over-optimization and describes improved evaluation error as having lower variance or higher fidelity to oracle quality.Source: F Formal explanation of why… · p. 20
  • With fewer candidates, improved evaluation quality can more than compensate for the loss in oracle score.At low budgets, additional sampling dominates; at 16–64 candidates, weaker evaluators become more vulnerable to mis-selection, while stronger evaluation benefits from reduced error.Source: F Formal explanation of why… · p. 20

G Self-Evaluation of Reasoning Models

The section examines whether reasoning models can improve their own outputs through Best-of-N and compares this approach with direct evaluators. It also analyzes process-evaluation errors and how outcome and process evaluators behave across precision–recall thresholds.

G Self-Evaluation of Reasoning Models

  • Using 8 responses with a reasoning evaluator achieved higher accuracy on both benchmarks than direct evaluators using 64 responses.The comparison uses DeepSeek-R1-Distill-Qwen-7B as the generator; Table 9 identifies the reasoning process evaluator as the proposed method.Source: Table 9 · p. 21, G Self-Evaluation of Reasoning Models · p. 22
  • Self-evaluation’s Gap Recovered was comparable to or larger than Best-of-N gains on instruction-tuned generators using the same evaluator.The self-evaluation experiment used DeepSeek-R1-Distill-Qwen-7B as both generator and evaluator.Source: G Self-Evaluation of Reasoning Models · p. 21, Table 8 · p. 21
  • Both summary-based and full-CoT outcome evaluation produced notable gains over Best-of-1 on AIME and AMC, with their relative performance reversing between datasets.Full-CoT evaluation improved over summary evaluation on AIME, while summary evaluation did better on AMC.Source: G Self-Evaluation of Reasoning Models · p. 22

G.1 Example of a reasoning model’s output

  • Figure 6 separates the example response’s chain of thought from its summary with a “</think>” token.The figure labels the chain-of-thought portion and summary separately.Source: §G.1 · p. 22, Figure 6 · p. 23

H.1 When does process evaluation fail?

  • In 90 false-negative cases analyzed across MATH-500, OlympiadBench, and GPQA, 44.4% of flagged steps contained genuine errors.The analysis distinguishes actual reasoning errors from correct steps incorrectly flagged by the evaluator; the examples are organized in Table 10.Source: §H.1 · p. 24, §H.3 · p. 24, Table 10 · p. 25

H.2 Precision-recall curve

  • Combining outcome and process scores followed process evaluators’ high-precision, low-recall behavior and outcome evaluators’ high-recall, high-precision behavior, achieving the best of both.Process evaluators were more conservative, while outcome evaluators achieved better overall accuracy in the high-recall region.Source: §H.2 · p. 24, Figure 8 · p. 25

I Analysis: How do mixing process and outcome evaluation lead to superior performance?

This analysis explains how combining process and outcome scores improves Best-of-N selection. Process scores can break ties between responses that outcome evaluators score indistinguishably.

I Analysis: How do mixing process and outcome evaluation lead to superior performance?

  • A grid search found the optimal mixing rate skewed toward outcome scores; weighting process scores above 0.5 reduced performance.The search varied α from 0.0 to 1.0 in steps of 0.1.Source: I Analysis: How do mixing process and… · p. 26
  • Process scores can rerank responses with indistinguishable outcome scores, helping filter outcome-evaluator false positives and improving Best-of-N accuracy.Outcome evaluators’ scores can be indistinguishable between responses labeled correct and wrong; process scores can break those ties by penalizing process errors.Source: I Analysis: How do mixing process and… · p. 26
  • Two-stage prompting performed significantly better than outcome evaluation alone and almost identically to the combined process-plus-outcome evaluator.In the two-stage approach, outcome evaluation filters responses before process evaluation reranks those that pass.Source: I Analysis: How do mixing process and… · p. 26
  • Two-stage prompting reduces inference cost by applying process evaluation only to responses that pass outcome evaluation; further compute-saving heuristics remain future work.The authors note that different heuristics could optimize compute while retaining Best-of-N performance.Source: I Analysis: How do mixing process and… · p. 26

J Analysis: How does problem difficulty affect outcome and process evaluation?

The analysis finds that evaluator performance varies with problem difficulty and that combining process and outcome scores can help rerank difficult cases. It also cautions that tests of reasoning evaluators have so far covered mathematics and coding, not socially sensitive or safety-related domains.

J Analysis: How does problem difficulty affect outcome and process evaluation?

  • Outcome evaluators outperform process evaluators on difficult problems, while process evaluators achieve higher Best-of-N accuracy on relatively easier problems.When enough correct responses exist, process evaluators’ conservative scoring can prevent selecting responses with incorrect steps.Source: J Analysis: How does problem… · pp. 26–27
  • The optimal mix is skewed toward outcome scores, and increasing the proportion of process scores reduces performance.Source: Figure 10 · p. 27
  • On difficult problems, process-plus-outcome evaluators outperform outcome evaluators more strongly, as false positives become more likely when few answers are correct.Outcome evaluators can assign scores above 0.99 to incorrect answers; process scores can rerank and filter these responses.Source: J Analysis: How does problem… · p. 27, Figure 11 · p. 28

N Potential Risks

  • The study tested reasoning evaluators on mathematics and coding, and calls for verification before applying them to socially sensitive or safety-related queries.The authors state that evaluator function should be verified, even partially.Source: N Potential Risks · p. 31
Loading 2503.19877v3…