Source-linked AI summary
A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Models
Congmin Zheng, Jiachen Zhu, Zhuoying Ou, Yuxiang Chen, Kangning Zhang, Rong Shan, Zeyu Zheng, Mengyue Yang, Jianghao Lin, Yong Yu, Weinan Zhang
TL;DR
Outcome reward models judge only final answers, while process reward models evaluate and guide reasoning at the step or trajectory level. This survey synthesizes how process data is generated, PRMs are built and used, and how they compare across reward-design dimensions.
Problem
Outcome reward models provide a coarse final-answer signal that struggles to capture progress, diagnose intermediate errors, or allocate computation adaptively across long reasoning chains.
Source: §1 · p. 1Method
Results
The survey characterizes PRMs as versatile for guiding search and reinforcement learning, while ORMs show stronger resistance to reward hacking and better generalization.
Source: §7 · pp. 8–9Takeaways & Limitations
The survey maps PRMs across data generation, model training, test-time scaling, reinforcement learning, applications, and benchmarks.
Source: §8 · p. 9The taxonomy simplifies hybrid methods, and benchmark and application summaries are selective rather than comprehensive.
Source: §9 · p. 9
Abstract
from arXiv · showhide
Although Large Language Models (LLMs) exhibit advanced reasoning ability, conventional alignment remains largely dominated by outcome reward models (ORMs) that judge only final answers. Process Reward Models(PRMs) address this gap by evaluating and guiding reasoning at the step or trajectory level. This survey provides a systematic overview of PRMs through the full loop: how to generate process data, build PRMs, and use PRMs for test-time scaling and reinforcement learning. We summarize applications across math, code, text, multimodal reasoning, robotics, and agents, and review emerging benchmarks. Our goal is to clarify design spaces, reveal open challenges, and guide future research toward fine-grained, robust reasoning alignment.
1 Introduction
The introduction frames PRMs as a shift from judging final answers to evaluating and guiding reasoning step by step. It presents a survey organized around the full PRM loop, from process-data generation through model building and use.
1 Introduction
- Outcome reward models judge only final answers, leaving long reasoning chains with a coarse signal that struggles to capture progress, diagnose intermediate errors, or allocate computation adaptively.Source: §1 · p. 1
- PRMs assess reasoning at the step or trajectory level and participate in a loop of generating data, training models, and using them to improve policies and produce new data.Source: §1 · p. 1, Figure 1 · p. 1
- The survey organizes PRM research across process-data generation, PRM building, and use for test-time scaling and PRM-guided reinforcement learning.Source: §1 · pp. 1–2
- The survey also covers applications and benchmarks, including math, code, multimodal reasoning, agents, and high-stakes domains.Source: §1 · p. 2
2 How to Generate Data
Process-data generation spans human annotation, automated supervision, and hybrid pipelines, each reflecting trade-offs between fidelity and scalability. Hybrid approaches combine selective human input with automated expansion across domains.
2.1 Human Annotation
- Human annotation validates intermediate reasoning steps; PRM800K demonstrated that human process judgments can improve PRM training and interpretable reasoning outcomes.Human-curated data is resource-intensive but provides high-fidelity signals, benchmarks, and seed material for scalable methods.Source: §2.1 · p. 2
2.2 Automated Supervision
- Automated pipelines generate process supervision through symbolic verification, consistency checks, execution feedback, and synthetic methods, but face error-propagation, verifier, and preference-alignment challenges.Source: §2.2 · pp. 2–3
2.3 Semi-automated Approaches
- Hybrid pipelines expand limited human-curated data through automated verification or synthesis in multimodal, medical, and practical task-oriented settings.Examples include medical reasoning seeded with around 8,000 human-curated examples and active learning that queries annotators when automated signals are uncertain.Source: §2.3 · p. 3
- Combining human anchors with automation can mitigate the weaknesses of either approach and support domains where neither alone is sufficient.Source: §2.3 · p. 3
3 How to Build PRMs
The survey organizes PRM construction into discriminative, generative, implicit, and architectural approaches, contrasting how they obtain and use process supervision. These designs range from explicit step labels to indirect signals and structured model innovations.
3 How to Build PRMs
- PRM training is grouped into discriminative, generative, implicit, and other architectural approaches.The survey discusses representative methods in each category.Source: §3 · p. 3
3.1 Discriminative PRMs.
- Discriminative PRMs score intermediate reasoning states using pointwise correctness or progress objectives, or pairwise preferences between candidate steps.Examples include uncertainty-weighted training, token-level feedback, and objectives for calibration or reasoning progress.Source: §3.1 · pp. 3–4
3.2 Generative PRMs.
- Generative PRMs first produce a verification or critique chain, then use it to score the original reasoning step.The survey connects this design to longer reasoning chains and semantic interpretation, with examples spanning inference-time computation and multimodal reasoning.Source: §3.2 · p. 4
3.3 Implicit PRMs
- Implicit PRMs infer fine-grained rewards from weaker signals, avoiding dependence on explicit step-level labels.The survey describes outcome-based pseudo-labeling, self-evaluation, consistency constraints, and transferred outcome-model knowledge as supervision sources.Source: §3.3 · p. 5
3.4 Other Architectural Innovations
- Other architectures change model structure or reasoning representation, including graph-based, adaptive-step, retrieval, multimodal, multilingual, and hierarchical designs.Examples also include domain-specific Text-to-SQL rewards and open preference trees.Source: §3.4 · p. 5
4 How to Use PRMs
PRMs are used both to allocate inference-time computation and to provide fine-grained feedback in reinforcement learning. The survey traces test-time methods from re-ranking toward generative verification, search, and adaptive control, alongside RL methods that incorporate process rewards into training.
4 How to Use PRMs
- The survey organizes PRM use into test-time scaling and reinforcement learning for policy learning, where step-level feedback guides inference, search, and training.Test-time methods allocate computation during inference rather than by enlarging the model.Source: §4 · p. 5, §4.1 · p. 5
4.1 Test-Time Scaling
- Test-time PRM use progresses from Best-of-N re-ranking to generative verification, search integration, adaptive granularity, and safety-aware scaling.Examples include verification-by-generation, beam search, beam annealing, code refinement, and confidence-based step partitioning.Source: §4.1 · pp. 5–6
4.2 RL for Policy Learning
- In RL, PRMs supply dense rewards for reasoning-heavy tasks, while distinct formulations shape credit assignment, stability, and policy updates.The survey covers stepwise rewards for math and coding, alternative reward objectives and token-level signals, domain-specific agent applications, and offline and online training infrastructure.Source: §4.2 · pp. 6–7
5 Downstream Application
The survey describes PRM applications across reasoning and decision-making domains, including math, code, multimodal and text tasks, robotics, agents, and high-stakes settings. Across these areas, PRMs assess intermediate steps, trajectories, or subgoals.
Math and Code
- In math and code, PRMs assess derivation steps and partial programs, supporting symbolic-error detection, execution feedback, and verification.The survey also describes math applications in grading, tutoring, and proof validation, and code applications in text-to-SQL and software engineering.Source: §5 · p. 7
Multimodal and Text
- For multimodal and text tasks, PRMs check visual-text coherence and score intermediate translation, question-answering, or retrieval reasoning steps.The survey describes selecting grounded explanations to support interpretability and factual consistency.Source: §5 · p. 7
Robotics, Agents, and Industry
- In robotics and interactive agents, PRMs provide subgoal rewards or trajectory criticism for long-horizon control and inference.The survey also describes evidence-based reasoning in medicine and finance for reliability and risk-sensitive decision-making.Source: §5 · p. 7
Multi-domain
- Cross-task studies explore PRMs that transfer process supervision across domains, pointing toward universal reasoning evaluators.The survey lists work on generalizable PRMs across multiple tasks and domains.Source: §5 · p. 7
6 Benchmark
The survey reviews benchmarks that evaluate process rewards across domains, error types, and evaluation goals. The examples range from step annotations and flawed trajectories to multimodal reasoning, web-agent decisions, and cross-distribution robustness.
Mathematical Reasoning
- PRMBench provides over 6,000 problems with 80,000 step annotations, while ProcessBench emphasizes earliest-error detection on competition-level tasks.PRMBench uses multidimensional labels including simplicity, soundness, and sensitivity.Source: §6 · p. 7
Reasoning Structure and Multimodal Tasks
- Socratic-PRMBench groups nearly three thousand flawed trajectories into six error patterns, while multimodal benchmarks assess errors, step correctness, answer aggregation, and guided search.The multimodal examples include comparisons between process and outcome models and human-labeled errors.Source: §6 · p. 7
Long-horizon Decision-making
- WebRewardBench evaluates web-agent clicks, form entries, and navigation using forty thousand step-level preference pairs.It is built on the WebPRM Collection.Source: §6 · p. 7
Robustness and Universality
- GSM-DC tests robustness with distractors, while UniversalBench evaluates trajectories across policy distributions for generalization and reproducibility.Source: §6 · pp. 7–8
7 Discussion
The survey compares rule-based rewards, ORMs, and PRMs across six dimensions, finding distinct trade-offs in cost, granularity, robustness, generalization, interpretability, and functionality. PRMs offer fine-grained and versatile supervision, while incurring higher annotation costs and facing greater hacking and generalization challenges.
Resource Efficiency
- Rule-based rewards are most economical, ORMs require moderate resources, and PRMs cost more because they need correctness labels for every intermediate step.The annotation workload is several times higher for PRMs than ORMs; the cited benchmarks average 7.1 steps for ProcessBench and 13.4 for PRMBench.Source: §7 · p. 8
Granularity
- PRMs provide step-specific error localization, whereas ORMs judge only final outcomes and rule-based granularity depends on handcrafted rules.PRM granularity is structural and scales with the number of steps in a solution.Source: §7 · p. 8
Anti-Hacking Robustness
- ORMs show the strongest resistance to reward hacking, while PRMs are more susceptible to verbosity bias and rule-based rewards to mis-specified rules.On PRM800K, step-level token-length standard deviation is 71.7 versus 50.6 at the trajectory level.Source: §7 · p. 8
Generalization
- ORMs generalize more readily across domains, while PRMs often need domain-specific step definitions and rule-based systems require environment-specific re-engineering.Mathematical derivations and code execution traces require different step definitions.Source: §7 · p. 8
Interpretability
- Rule-based rewards are most transparent, ORMs are black boxes, and PRMs localize errors through stepwise supervision and can provide natural-language justifications.ThinkPRM and GenRM produce justifications, while Socratic-PRMBench categorizes reasoning-error patterns.Source: §7 · pp. 8–9
Functionality
- PRMs are the most versatile, guiding search and supporting RL with stepwise credit assignment and trajectory-level reward shaping.ORMs provide only a final-outcome reward, while rule-based rewards lack adaptability beyond their original design.Source: §7 · p. 9
8 Conclusion
This survey organizes PRM research around the loop of process-data generation, model building, and use, covering applications and benchmarks across multiple domains. It identifies open challenges whose resolution could advance safer, more interpretable, and broadly applicable reasoning systems.
Conclusion
- The survey organizes PRMs around a closed loop of data generation, training, and use, and covers benchmarks and applications across math, code, multimodal tasks, robotics, and other domains.The usage stages include test-time scaling and reinforcement learning.Source: §8 · p. 9
- Key challenges include reducing annotation costs through robust automatic supervision, improving cross-domain generalization, integrating PRMs with agentic planning and memory, and standardizing evaluation.The paper states that addressing these challenges will advance safer, more interpretable, and broadly applicable reasoning systems.Source: §8 · p. 9
- The paper concludes that addressing these challenges will advance safer, more interpretable, and broadly applicable reasoning systems.Source: §8 · p. 9
9 Limitations
The survey presents a conceptual map of PRM research while noting that its taxonomy and coverage are necessarily selective.
Limitations
- The data–model–usage taxonomy simplifies some hybrid methods by discussing each under its primary aspect.Source: §9 · p. 9
- Benchmark and application summaries are selective and may omit task-specific datasets or proprietary evaluation suites.Source: §9 · p. 9
- The authors position the synthesis as a starting point for exploring and systematizing PRM research, despite these boundaries.Source: §9 · p. 9
A Paper Structure and Taxonomy Overview
The survey maps PRM research across data construction, training, usage, and evaluation, then discusses challenges in human oversight, automated supervision, and step granularity.
A Paper Structure and Taxonomy Overview
- The taxonomy follows the PRM loop from data process through training and usage to benchmarks, with finer categories for each research area.Data methods include human, automated, and semi-automated approaches; training includes discriminative, generative, implicit, and other architectures; usage covers test-time scaling and RL.Source: A Paper Structure and Taxonomy Overview · pp. 14–15
B.1 The Cognitive Scalability of Human Annotation
- As reasoning tasks grow more complex, verifying intermediate steps can outstrip human cognitive capacity, risking a human ceiling when annotators mistake valid novel reasoning for errors.The discussion proposes AI-assisted verification tools, with humans overseeing intent and alignment rather than labeling raw correctness.Source: §B.1 · pp. 15–16
B.2 Echo Chambers and Goodhart’s Law in Automated Supervision
- Automated supervision can create echo chambers when reward and policy models share blind spots, while optimization against fixed rewards can encourage reward hacking.The proposed directions include heterogeneous model supervision and dynamically updated or adversarially tested reward models.Source: §B.2 · pp. 16–17
B.3 The Tension of Granularity: Defining a “Step”
- Rigid heuristic segmentation can clash with reasoning structure: frequent evaluation may disrupt context, while sparse evaluation weakens dense supervision.The discussion proposes learnable semantic step boundaries, multi-scale evaluation, or continuous assessment of reasoning trajectories.Source: §B.3 · p. 17
B.4 The Proxy-Reward Gap: Proxy Metrics vs. Actual Utility
Static classification and ranking accuracy may not predict a PRM’s utility during dynamic inference, where calibration and out-of-distribution robustness matter. The paper recommends on-policy evaluation, iterative training, and calibration-first objectives that recognize uncertainty.
B.4 The Proxy-Reward Gap: Proxy Metrics vs. Actual Utility
- Static classification or ranking accuracy may not translate to downstream success because active search exposes PRMs to policy-generated paths outside their training distribution.Such paths can include diverse, often erroneous branches that a statically optimized PRM may not be calibrated to prune effectively.Source: §B.4 · p. 17
- The paper recommends on-policy evaluation on the target policy’s inference trajectories and iterative training on the challenging errors that policy currently generates.This shifts evaluation and updates away from static datasets that may be outdated relative to the policy’s trajectory distribution.Source: §B.4 · p. 17
- Calibration error and out-of-distribution robustness should be primary metrics, with PRMs signaling uncertainty so fallback mechanisms can avoid confidently misdirecting reasoning.Suggested fallbacks include human supervision or tool invocation.Source: §B.4 · p. 17