PaperlayerView exact revision on arXiv

Source-linked AI summary

GroundedPRM: Tree-Guided and Fidelity-Aware Process Reward Modeling for Step-Level Reasoning

Yao Zhang, Yu Wu, Haowei Zhang, Weiguo Li, Haokun Chen, Jingpei Wu, Guohao Li, Zhen Han, Volker Tresp

arXiv:2510.14942v1cs.AI

TL;DR

PRMs supervise intermediate reasoning steps, but reliable annotations remain difficult to scale and existing signals can be noisy or unfaithful. GroundedPRM combines MCTS-structured paths, external step verification, and hybrid rewards. With 40K automatically labeled samples, it reports up to a 26% relative improvement on ProcessBench and surpasses human-labeled PRMs in reward-guided greedy search.

  • Problem

    PRMs need reliable, scalable step-level annotations, while human labeling is costly and LLM judgments or rollout-based estimates can be unfaithful or noisy.

    Source: §1 · p. 2

  • Method

    GroundedPRM combines MCTS-structured reasoning paths, external tool verification of intermediate steps, and hybrid reward aggregation.

    Source: §1 · p. 2

  • Results

    40K automatically labeled samples—10% of the best-performing auto-labeled PRM’s data—yield up to a 26% relative improvement in average performance on ProcessBench; reward-guided greedy search surpasses human-labeled PRMs.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    GroundedPRM offers a scalable, verifiable path toward high-quality process-level reasoning for training and inference.

    Source: §1 · p. 2

    Experiments focus on mathematics because its PRM benchmarks and baselines are established.

    Source: §6 · p. 10

Abstract

from arXiv · show

Process Reward Models (PRMs) aim to improve multi-step reasoning in Large Language Models (LLMs) by supervising intermediate steps and identifying errors. However, building effective PRMs remains challenging due to the lack of scalable, high-quality annotations. Existing approaches rely on costly human labeling, LLM-based self-evaluation that is prone to hallucination, or Monte Carlo (MC) estimation, which infers step quality solely from rollout outcomes and often introduces noisy, misaligned supervision due to credit misattribution. These issues result in three core limitations: noisy rewards, low factual fidelity, and misalignment with step-level reasoning objectives. To address these challenges, we introduce GroundedPRM, a tree-guided and fidelity-aware framework for automatic process supervision. To reduce reward noise and enable fine-grained credit assignment, we construct structured reasoning paths via Monte Carlo Tree Search (MCTS). To eliminate hallucinated supervision, we validate each intermediate step using an external tool, providing execution-grounded correctness signals. To combine both step-level validation and global outcome assessment, we design a hybrid reward aggregation mechanism that fuses tool-based verification with MCTS-derived feedback. Finally, we format the reward signal into a rationale-enhanced, generative structure to promote interpretability and compatibility with instruction-tuned LLMs. GroundedPRM is trained on only 40K automatically labeled samples, amounting to just 10% of the data used by the best-performing PRM trained with auto-labeled supervision. Nevertheless, it achieves up to a 26% relative improvement in average performance on ProcessBench. When used for reward-guided greedy search, GroundedPRM outperforms even PRMs trained with human-labeled supervision, offering a scalable and verifiable path toward high-quality process-level reasoning.

1 Introduction

GroundedPRM addresses noisy, unfaithful, and misaligned process supervision with tree-guided paths, external step verification, and hybrid rewards. It uses 40K automatically labeled samples and reports gains on ProcessBench and reward-guided search.

1 Introduction

  • Human labels are costly, LLM judges can reward fluent but incorrect reasoning, and Monte Carlo estimates can misassign credit based on rollout outcomes.A correct step can be penalized when a rollout fails, while a flawed step can be rewarded when the final answer is correct.Source: §1 · p. 2
  • GroundedPRM uses MCTS to build structured reasoning paths for step-level credit assignment and external math tools to verify intermediate steps.The tool-based correctness signals are grounded in executable logic rather than LLM-generated feedback.Source: §1 · p. 2
  • Rewards are formatted as binary decisions with rationale-enhanced justifications for interpretability and integration into downstream reasoning workflows.Source: §1 · pp. 2–3
  • 40K automatically labeled samples—10% of the best-performing auto-labeled PRM’s data—yield up to a 26% relative improvement in average performance on ProcessBench; reward-guided greedy search surpasses human-labeled PRMs.The comparison is reported for ProcessBench, and the greedy-search result is reported across multiple mathematical reasoning benchmarks.Source: §1 · p. 2
  • Its hybrid reward aggregation combines tool-based step validation with feedback from MCTS-guided reasoning paths.Source: §1 · p. 2

2 Related Work

Prior work addresses mathematical reasoning and step-level supervision, but the cited approaches leave challenges in correcting intermediate errors and obtaining reliable step annotations.

2.1 Mathematical Reasoning with LLMs

  • Output-level verification and selection strategies offer limited supervision for correcting internal reasoning steps, motivating process-level reward modeling.Source: §2.1 · p. 3

2.2 Process Reward Models for Step-Level Supervision

  • Expert-labeled PRM data is expensive to scale, while Monte Carlo supervision can misassign credit and depends on completion-model quality.Source: §2.2 · p. 3
  • GroundedPRM combines MCTS tree search, external math-engine verification, and fused value–correctness reward modeling for step-level supervision.Source: §2.2 · p. 3

3 Methodology

GroundedPRM combines MCTS-structured reasoning, external step verification, and hybrid local/global rewards to address noisy, unfaithful, and misaligned process supervision. It trains a rationale-generating reward model on verified reasoning traces.

3.1 Tree-Guided Reasoning Path Construction

  • MCTS builds structured reasoning paths, verifies simulated steps and final outcomes, and backpropagates rewards with stronger credit for steps nearer the outcome.UCT balances estimated value with an exploration bonus, while multiple rollouts balance path diversity with search efficiency.Source: §3.1 · pp. 4–5

3.2 Fidelity-Aware Step Verification with External Tool

  • External math tools check each intermediate step and assign binary labels, grounding step-level supervision in executable verification rather than LLM self-evaluation.The module is tool-agnostic; experiments use Wolfram Alpha, with alternatives including SymPy or domain-specific solvers.Source: §3.2 · p. 5

3.3 Hybrid Reward Aggregation

  • Hybrid aggregation combines tool-verified step correctness with final-answer correctness into a reward used to update MCTS values and guide exploration.A weighting coefficient adjusts the contribution of final-answer correctness relative to step-level reliability.Source: §3.3 · pp. 5–6

3.4 Generative Process Reward Model

  • GroundedPRM autoregressively generates correctness labels and rationales conditioned on the problem and intermediate reasoning trace, supporting interpretability and integration with instruction-tuned LLMs.Training instances pair reasoning steps with verification outcomes and explanations retained after consistency filtering.Source: §3 · p. 3, §3.4 · p. 6

3.5 Data Construction for GroundedPRM Training

  • The MATH-based training set contains approximately 40K verified samples built from multi-round MCTS rollouts, tool-checked steps, hybrid scores, and filtered traces.The final instances include a problem, reasoning trace, correctness labels, and explanations.Source: §3.5 · p. 6

4 Experiment

GroundedPRM is evaluated on step-error identification and reward-guided reasoning, with results across ProcessBench and six mathematical benchmarks. It leads auto-labeled PRMs on ProcessBench with 10% of the training data and achieves the highest average accuracy in reward-guided greedy search, including over human-labeled PRMs.

4.1 Experimental Setup

  • The evaluation measures first-error identification on ProcessBench and reward-guided greedy search across AMC23, AIME24, MATH, College MATH, OlympiadBench, and Minerva MATH.Greedy search samples eight candidate actions per generation step and selects the one with the highest predicted reward.Source: §4.1 · pp. 6–7

4.2 Results on ProcessBench

  • 26% relative improvement in average F1 over Math-Shepherd-PRM-7B on ProcessBench, using only 10% of its training samples.GroundedPRM also ranks first on MATH, OlympiadBench, and Omni-MATH.Source: Table 1 · p. 7, §4.2 · p. 7

4.1 Experimental Setup

  • All reward models use Qwen2.5-7B-Instruct as the base model and are fine-tuned on step-labeled trajectories with LoRA.The baselines cover human-labeled, automatically labeled, and hybrid supervision strategies.Source: §4.1 · p. 7

4.3 Analysis and Discussions

  • Across matched training sizes, GroundedPRM consistently achieves higher F1 scores than Qwen2.5-Math-7B-PRM800K despite using automatically labeled data.Both models use Qwen2.5-7B-Instruct, isolating differences in supervision sources.Source: Table 2 · p. 8, §4.3 · p. 8
  • GroundedPRM's generative supervision outputs a step-level reward with a rationale, which the authors report improves interpretability and distinguishes fluent-but-incorrect reasoning from valid logic.The reported gains include challenging benchmarks such as OlympiadBench and MATH.Source: §4.2 · p. 8

4.4 Results on Reward-Guided Greedy Search

  • GroundedPRM achieves the highest average accuracy among PRMs in reward-guided greedy search, despite training on only 40K automatically labeled examples.The comparison includes PRMs trained with automated, mixed, and large-scale human annotations.Source: Table 4 · p. 9, §4.4 · p. 9

4.3 Analysis and Discussions

  • Combining step verification with trajectory outcomes achieves the highest average F1; outcome-only labels misassign credit, while step-only labels lose precision on deeper logical errors.External tools can detect surface arithmetic errors but may miss deeper logical flaws.Source: §4.3 · pp. 8–9
  • Removing rationale generation consistently lowers F1, with larger gaps on MATH and OlympiadBench; rationales support consistency and error localization in long-horizon problems.The comparison is against a variant that predicts correctness labels without generating explanations.Source: §4.3 · p. 9

5 Conclusion

The conclusion presents GroundedPRM as a tree-guided, fidelity-aware framework that combines structured path exploration, tool verification, hybrid reward aggregation, and rationale-enhanced supervision. It reports strong ProcessBench performance with 40K automatically labeled samples and higher reward-guided greedy-search performance than PRMs trained with human labels.

5 Conclusion

  • GroundedPRM combines MCTS path exploration, tool-based step verification, hybrid reward aggregation, and rationale-enhanced supervision.The framework targets noisy rewards, low factual fidelity, and misalignment with step-level reasoning objectives.Source: §5 · p. 10
  • With 40K automatically labeled samples—10% of the best-performing auto-labeled PRM's data—GroundedPRM achieves up to 26% relative improvement in average ProcessBench performance.Source: §5 · p. 10
  • In reward-guided greedy search, GroundedPRM outperforms PRMs trained with human-labeled supervision.Source: §5 · p. 10

6 Future Work

The future-work discussion proposes broader verifiers, larger underlying language models, human preference signals, and integration into reinforcement-learning pipelines. The algorithm overview describes a unified data-generation procedure combining MCTS path construction, tool verification, and hybrid reward aggregation.

6 Future Work

  • Expanding external verifiers beyond the mathematical tool used here could improve flexibility across reasoning domains; proposed tools include self-checkers, retrieval-augmented verifiers, and rule-based evaluators.The authors also suggest scaling the underlying LLM and integrating human preference signals.Source: §6 · p. 10
  • Integrating GroundedPRM into reinforcement-learning pipelines is proposed as a way to use verifiable rewards for policy optimization in long-horizon tasks.The proposed study would examine interaction with on-policy updates, exploration, search, and credit assignment; current experiments focus on mathematics.Source: §6 · p. 10

A Algorithm Overview and Pseudocode

  • The data-generation algorithm unifies MCTS reasoning-path construction, tool-based step verification, and hybrid reward aggregation.It selects among generated children using tool verification and combines step rewards with final-answer correctness feedback.Source: A Algorithm Overview and Pseudocode · p. 15

B Prompt Template for Data Annotation

GroundedPRM uses separate prompts to generate intermediate reasoning steps during MCTS rollouts and verify each generated step with external tools. The outputs provide structured trajectories and fidelity-aware supervision signals.

B Prompt Template for Data Annotation

C Training Hyperparameters

GroundedPRM is fine-tuned in a sequence-to-sequence setup using LLaMA-Factory’s Trainer implementation on four A100 80GB GPUs.

C Training Hyperparameters

D Supplementary Evaluation Results

GroundedPRM achieves the highest average F1 across the ProcessBench math benchmarks among PRMs trained with automatically labeled data. The accompanying case studies illustrate its error detection and a comparison of supervision variants.

D Supplementary Evaluation Results

E Case Studies

  • The case studies cover arithmetic, algebraic, and constraint-based inconsistencies, and compare Step-Only with Dual-Signal supervision.The ablation case is described as showing enhanced data fidelity and accurate credit attribution with dual-signal fusion.Source: E Case Studies · p. 18

E.1 Fidelity- and Error-Type–Aware Reasoning Supervision

GroundedPRM’s qualitative cases illustrate how it checks quantities and constraints, localizes first errors, and gives error-specific explanations. In the described examples, a general-purpose LLM-as-judge misses inconsistencies that GroundedPRM detects.

E.1 Fidelity- and Error-Type–Aware Reasoning Supervision

  • GroundedPRM recomputes the quiz-score additions to obtain 630 and marks the solution’s total of 570 incorrect, while the LLM-as-judge accepts it.The case presents this as localization of the first wrong step.Source: §E.1 · p. 19
  • For the sum-of-pairs example, GroundedPRM identifies the unjustified division by 2 as the first error and confirms the correct aggregate of 210.The LLM-as-judge marks the resulting claim of 105 as correct.Source: §E.1 · p. 19
  • In the ratio example, GroundedPRM rejects (26, 41) for violating the upper bound and verifies (19, 34) against the stated constraints.The checks include the inequality, the difference of 15, and coprimeness.Source: §E.1 · p. 19
  • Across the cases, GroundedPRM recomputes quantities, localizes the first wrong step, names error types, and checks multi-constraint consistency.The paper says these behaviors support more reliable step-level judgments than generic LLM-as-judge baselines and produce actionable rationales.Source: §E.1 · p. 19

E.2 Dual-Signal Supervision Improves Data Fidelity and Credit Attribution

The examples show GroundedPRM combining step-level verification with trajectory-level feedback to catch errors that a Step-Only supervisor or GPT-4o validates. This supports more faithful error detection and credit attribution across arithmetic and factorization cases.

E.2 Dual-Signal Supervision Improves Data Fidelity and Credit Attribution

  • GroundedPRM catches an omitted factor in 168’s prime factorization and recovers the correct GCF, 12, where Step-Only validates the erroneous step.The example illustrates how dual-signal supervision can prevent surface-level false positives and yield faithful credit attribution.Source: §E.2 · p. 19, Figure 8 · p. 24, Figure 9 · p. 25
  • GroundedPRM detects that 570 is incorrect rather than validating the miscalculated reasoning step, as GPT-4o does.It re-evaluates the arithmetic and provides a rationale-supported supervision label.Source: Figure 4 · p. 20
  • After the sums total 210, GroundedPRM rejects the unjustified halving to 105 and localizes it as the first wrong step.GPT-4o incorrectly validates the halving; GroundedPRM verifies each sub-sum and outputs a rationale-supported negative label.Source: Figure 5 · p. 21
  • GroundedPRM rejects (26, 41) and identifies (19, 34) as the valid pair, while GPT-4o validates the rejected candidate.GroundedPRM re-evaluates the inequality, checks b−a = 15 and coprimeness, and gives a rationale-supported negative label.Source: §E.2 · pp. 22–23, Figure 7 · p. 23

Related papers

Loading 2510.14942v1…