Source-linked AI summary
Beyond Outcome Verification: Verifiable Process Reward Models for Structured Reasoning
Massimiliano Pronesti, Anya Belz, Yufang Hou
TL;DR
The paper addresses the lack of fully verifiable process supervision for intermediate reasoning. It introduces VPRMs, which use deterministic rule-based verifiers within reinforcement learning, and reports improved accuracy, coherence, and process soundness on risk-of-bias assessment. The approach is strongest for structured tasks with explicit domain rules, while broader generalization remains open.
Problem
Existing outcome-level verification does not ensure valid intermediate reasoning, while neural process judges remain opaque and vulnerable to bias and reward hacking.
Method
VPRMs use deterministic, domain-guideline-based verifiers to reward each reasoning step alongside the final outcome during reinforcement learning.
Results
VPRM-trained models achieve substantially higher accuracy and more coherent reasoning traces across risk-of-bias domains than outcome-only, neural process-reward, and prompted baselines.
Takeaways & Limitations
Verifiable process supervision is reported as a practical and robust approach for inducing reliable reasoning behavior in structured scientific and decision-making tasks.
Takeaways & Limitations
VPRMs rely on deterministic domain-specific rules, and their evaluation is focused on risk-of-bias assessment, leaving generalization to other domains and open-ended reasoning unestablished.
Abstract
from arXiv · showhide
Recent work on reinforcement learning with verifiable rewards (RLVR) has shown that large language models (LLMs) can be substantially improved using outcome-level verification signals, such as unit tests for code or exact-match checks for mathematics. In parallel, process supervision has long been explored as a way to shape the intermediate reasoning behaviour of LLMs, but existing approaches rely on neural judges to score chain-of-thought steps, leaving them vulnerable to opacity, bias, and reward hacking. To address this gap, we introduce Verifiable Process Reward Models (VPRMs), a reinforcement-learning framework in which intermediate reasoning steps are checked by deterministic, rule-based verifiers. We apply VPRMs to risk-of-bias assessment for medical evidence synthesis, a domain where guideline-defined criteria and rule-based decision paths enable programmatic verification of reasoning traces. Across multiple datasets, we find that VPRMs generate reasoning that adheres closely to domain rules and achieve substantially higher coherence between step-level decisions and final labels. Results show that VPRMs achieve up to 20% higher F1 than state-of-the-art models and 6.5% higher than verifiable outcome rewards, with substantial gains in evidence grounding and logical coherence.
1 Introduction
The paper identifies a gap in outcome-only and neural process supervision: intermediate reasoning is not fully verifiable. It introduces VPRMs and evaluates them on rule-structured risk-of-bias assessment, reporting stronger accuracy and reasoning coherence.
- Outcome-only RLVR verifies terminal labels but does not guarantee valid intermediate reasoning, while neural process judges introduce opacity, bias, and reward-hacking opportunities.
- Verifiable process rewards address this gap by checking every reasoning step against explicit, domain-defined rules during reinforcement learning.
- VPRMs provide fine-grained rewards that guide models toward reasoning traces that are both correct and aligned with domain logic.
- Risk-of-bias assessment is a suitable test case because clinical guidelines prescribe structured reasoning steps and deterministic decision rules.
- Across models and risk-of-bias domains, VPRM-trained models achieve higher accuracy and more coherent reasoning traces than outcome-only, neural process-reward, and prompted baselines.
- The paper contributes a deterministic step-level verification framework, theoretical support for its optimization behavior, and empirical validation in medical systematic reviews.
2 Preliminaries
The preliminaries describe group-based policy optimization and deterministic rule-based rewards, then situate the application in systematic reviews and structured risk-of-bias assessment.
- Group Relative Policy Optimization (GRPO): GRPO optimizes a policy by comparing multiple candidate completions generated for the same input.
- Group Relative Policy Optimization (GRPO): GRPO normalizes group rewards and uses a clipped, KL-regularized objective to favor high-reward behavior while remaining near a reference policy.
- Dynamic Sampling Policy Optimization (DAPO): DAPO extends GRPO with dynamic sampling, clip-higher, token-level policy gradients, and overlong reward shaping.
- Dynamic Sampling Policy Optimization (DAPO): Dynamic sampling retains groups containing both correct and incorrect answers so sampled prompts provide effective gradients.
- Rule-based Reward Modeling: Rule-based reward modeling computes rewards deterministically from explicit task constraints rather than learned preference models.
- Systematic Reviews: Systematic reviews aggregate evidence through predefined searches, explicit inclusion criteria, and reproducible synthesis pipelines.
- Risk of Bias Assessment: Risk-of-bias assessment evaluates methodological flaws across fixed bias domains and maps relevant trial features to low, medium, or high-risk judgments.
3 Verifiable Process Reward Models
VPRMs combine deterministic verification of intermediate reasoning with outcome verification and reinforcement learning. Under stated assumptions, the resulting optimization favors rule-consistent trajectories in expectation.
- VPRMs evaluate intermediate reasoning steps with deterministic, externally checkable verifiers grounded in domain guidelines.
- Verifiable process rewards verify each reasoning step and associated label, whereas outcome rewards evaluate only the final risk label.
- A reasoning trajectory consists of ordered outputs whose step identifiers and labels are checked against rule-derived guidance for each prefix.
- Step-level correctness is scored with bounded functions, and a terminal reward evaluates whether the final risk value matches the gold value.
- The full process reward is computable with deterministic rule-based checks, making all components of the reasoning trajectory verifiable.
- Under finite-variance, reward-separation, and sufficiently large-sample assumptions, correct trajectories receive positive expected advantage while incorrect trajectories receive negative expected advantage.
- The framework's optimization guarantee applies to both GRPO and DAPO and is supported by a proof for both objectives.
4 Experiments
The experiments evaluate VPRMs for risk-of-bias assessment using multiple datasets, rule-based coherence measures, and comparisons with pretrained, outcome-reward, and neural process-reward models. VPRMs consistently improve predictive performance and reasoning consistency, while ablations show that combining outcome and process rewards is strongest.
- Datasets: 2,946 paper-risk instances from 104 systematic reviews form the main corpus, with full-text studies aligned to risk-of-bias labels from forest plots.Evaluation includes COCHRANEFOREST and RoBBR Cochrane and Non-Cochrane datasets.
- Metrics: Accuracy and macro-F1 measure discrete risk-label prediction, while Coherence measures consistency between final risk predictions and intermediate reasoning under a fixed decision function.The decision function uses the macros of the RoB2 tool.
- Main results: VPRM-trained Qwen2.5-7B models consistently outperform pretrained models and outcome-only training across COCHRANEFOREST and both RoBBR test sets.The reported improvements extend to RoBBR Non-Cochrane, indicating generalisation beyond the training distribution.
- Main results: VPRMs consistently outperform neural process-reward baselines when both receive the same verifiable outcome reward.The comparison isolates the effect of process-level supervision and attributes the gap to cleaner deterministic, guideline-based verification.
- Ablations: Removing the outcome reward substantially degrades performance, while the full VPRM still outperforms steps-only supervision.The strongest results combine outcome verification with correctness-aware, fine-grained process rewards.
- Reasoning analysis: VPRM-trained models achieve substantially higher coherence and high coherent accuracy, unlike pretrained models whose apparently self-consistent reasoning rarely yields correct final judgments.Reward trajectories also show process and correctness rewards rising together, whereas thought-format rewards saturate quickly.
5 Related Work
Related work extends RLVR with structural signals and uses neural judges for process supervision, but these approaches remain non-verifiable or outcome-centered. Reward dynamics show format learning saturates early while accuracy and process rewards continue improving.
- Prior RLVR extensions add structural signals or self-verification, but remain centered on terminal-outcome verification.
- Format rewards plateau early, while accuracy and process rewards improve gradually.
- Neural process supervision provides dense intermediate feedback but relies on model-based evaluations prone to bias and reward hacking.
6 Conclusion
The paper introduces verifiable process rewards that combine deterministic step-level verification with reinforcement learning and theoretical guarantees. It reports substantial empirical gains on medical risk-of-bias assessment and presents the approach as a practical route to reliable reasoning behaviour.
- The framework integrates deterministic step-level verification with reinforcement learning and provides theoretical guarantees under mild assumptions.
- VPRMs achieve substantial empirical gains on risk-of-bias assessment in medical systematic reviews.
- The results indicate that verifiable process supervision is practical and robust for inducing reliable reasoning behaviour in LLMs.
Limitations
VPRMs depend on deterministic domain rules and compatible reasoning-trace formats, while the empirical evaluation is currently limited to risk-of-bias assessment. Broader generalisation and resilience to incomplete guidelines remain unresolved.
- The approach relies on deterministic, domain-specific rules, so tasks without well-defined intermediate reasoning steps may not benefit directly.
- The empirical evaluation focuses on risk-of-bias assessment, leaving generalisation to other domains, especially open-ended reasoning, unestablished.
- Verifier effectiveness can decline when model outputs are incompatible with expected reasoning-trace formats, especially for smaller models.
- VPRMs do not fully eliminate model bias or errors arising from incomplete guidelines.
A Theoretical Analysis
The theoretical analysis extends prior RLVR results to verifiable process rewards under assumptions about reward variance, reward separation, and concentration of group statistics. It describes group-based policy optimization using trajectory-level and token-level advantages.
- The proof of Theorem 1 extends prior reinforcement-learning-with-verifiable-rewards results to base LLMs.
- The analysis assumes verifiable rewards have finite mean and nonnegative finite variance.
- The reward-gap assumption requires correct reasoning trajectories to have higher probabilities of inducing correct answers than incorrect trajectories.
- As group size G →∞, empirical reward means and standard deviations are assumed to concentrate.
- GRPO uses trajectory-level normalised advantages, while DAPO scales the same trajectory advantage into token-level advantages.
A.3 Proof of Theorem 1
The proof establishes sign separation for the advantage estimates: correct traces receive positive expected weight, while incorrect traces receive negative expected weight. It then shows that outcome-only rewards are a special case of VPRMs without intermediate labels.
- The proof uses conditional expectations and Assumption A3 with Slutsky’s theorem to establish sign separation.
- E[ ˆA(Y ) | C] > 0 and E[ ˆA(Y ) | Cc] < 0, separating correct from incorrect traces.
- The same positive-versus-negative separation holds for step-level estimates when scaled by nonnegative constants c_i,t.
- Thus, both GRPO and DAPO assign positive expected weight to correct traces and negative weight to incorrect ones.
- With no intermediate verifiable labels, a VPRM reduces to a Verifiable Outcome Reward Model whose scalar reward depends only on final outcome statistics.
E. Incomplete outcome data
The incomplete-outcome-data domain assesses missingness, its causes, and whether analyses appropriately account for it. The broader assessment also considers selective reporting, baseline comparability, and contamination between study arms.
- E. Incomplete outcome data: Reviewers assess how much data is missing, whether missingness is plausibly explained, and whether analyses handle it appropriately.High or unexplained attrition and inadequate handling can produce biased effect estimates.
- E. Incomplete outcome data: Selective reporting is evaluated by comparing planned and reported outcomes when a study protocol is available.Omissions or additions raise concern, while clear correspondence indicates low risk.
- E. Incomplete outcome data: Supplementary domains assess baseline outcome similarity, baseline-characteristic similarity, and contamination between study arms.These domains address imbalances at baseline and unintended cross-group intervention exposure.
D Silver Steps and Labels Manual Verification
The study manually verifies automatically generated reasoning traces and silver labels used for VPRM training. In a random sample of 20 instances, all inspected traces followed the correct decision structure and step-level labels were almost always accurate.
- D Silver Steps and Labels Manual Verification: 20 randomly sampled instances were manually evaluated for valid decision paths and accurate step-level labels.Two NLP master’s students inspected each complete reasoning trace.
- D Silver Steps and Labels Manual Verification: The annotation prompt asks the model to produce a structured reasoning trace that justifies the ground-truth value for a risk-of-bias domain.The study identifies separate figures for synthetic-data annotation, steps and labels, and VPRM training and inference.
- D Silver Steps and Labels Manual Verification: All inspected traces followed the correct decision structure, and step-level labels were almost always accurate.The manual verification judged the generated training traces and silver labels to be consistently high quality.
- D Silver Steps and Labels Manual Verification: Process-supervision research scores chain-of-thought steps with neural judges, while other RLVR extensions add structural or self-verification signals.These approaches include auxiliary coherence signals and progress-estimation or critique modules.
- D Silver Steps and Labels Manual Verification: Automated risk-of-bias assessment has used supervised models, prompted LLMs, retrieval, and auxiliary decision heads.Examples include machine-reading-comprehension classifiers such as RoBIn and retrieval-enhanced systems such as RoBGuard.
- D Silver Steps and Labels Manual Verification: Training uses a temperature of 0.7 and a maximum output length of 2,048 tokens for the annotation and training prompts.The experiments use API access or an internal cluster, with open-source models trained on a distributed cluster.