Learning Process Rewards via Reasoning State Propagation

• Process annotations are costly, whereas outcome labels are cheaper and scalable, but how to integrate both signals in PRMs remains underexplored.Source: §1 · pp. 1–2
• Existing PRMs often classify prefixes independently, while errors may be corrected later and a correct final answer does not establish that every earlier step was valid.Source: §1 · p. 2
• RSP propagates binary validity states across successive reasoning steps using break and repair probabilities to model error introduction and recovery.Source: §1 · pp. 2–3
• Process annotations ground intermediate states, while outcome labels supervise the final propagated state and send learning signals to earlier updates.Source: §1 · pp. 2–3
• RSP consistently improves over representative PRMs in experiments across test-time scaling and reinforcement learning.Source: §1 · p. 3

2.1 ENHANCING REASONING IN LRMS

• Lightman et al. (2024) use process reward models for Best-of-N ranking, and REST-MCTS* (Zhang et al., 2024) uses them in Monte Carlo search; verifiable-outcome reinforcement learning strengthens reasoning, while PURE densifies intermediate-step supervision.Source: §2.1 · p. 3

2.2 PROCESS REWARD MODELING

• PRMs use human, rollout-based, or automated step labels, but existing approaches do not jointly exploit process and outcome supervision while explicitly modeling flexible state transitions.Examples include PRM800K, Math-Shepherd, OVM, PQM, and CRM (Zhang et al., 2025a).Source: §2.2 · p. 3

3.1 PRELIMINARIES

• The task is to estimate prefix validity for autoregressively generated reasoning trajectories using process-annotated and outcome-annotated data.The process labels indicate prefix validity at annotated steps; outcome labels indicate whether the complete trajectory reaches a correct answer.Source: §3.1 · p. 4
• Independent prefix classification imposes no explicit relation between successive validity predictions and cannot naturally use outcome-only trajectories.Source: §3.1 · p. 4

3.2 REASONING STATE PROPAGATION

• RSP represents each prefix as valid (G) or unresolved-error (B), then propagates its state through break and repair transitions across steps.The propagated state distribution tracks error introduction, persistence, and recovery.Source: §3.2 · p. 4

3.3 LEARNING FROM PROCESS AND OUTCOME SUPERVISION

• Process loss supervises propagated states at annotated steps, while terminal outcome loss backpropagates through state propagation to train intermediate break and repair predictions.Source: §3.3 · p. 5
• Outcome loss favors lower break and higher repair when the remaining trajectory is more likely to reach its target from G than from B, with the reverse effect when B is more likely.Source: §3.3 · p. 5

4.1 EXPERIMENTAL SETTINGS

• Experiments use Qwen3 backbones, PRM800K process annotations, filtered AceMath-RM outcome labels, and beam search, Best-of-N, and reinforcement-learning evaluations.Unless otherwise specified, 20% of PRM800K is used for process training, with process- and outcome-annotated examples at a default 1:3 batch ratio.Source: §4.1 · p. 6, §4.2 · p. 6, §4.3 · pp. 6–7

4.2 PROCESS-GUIDED BEAM SEARCH

• At e = 12, RSP outperforms OVM by 4.9% on both MATH500 and Gaokao, and leads overall across Qwen3 backbones and expansion widths.Beam width is fixed at 4; Table 1 reports beam-search accuracy.Source: §4.2 · p. 6, Table 1 · p. 7

4.3 BEST-OF-N SELECTION

• RSP reaches 74.4% average Best-of-N accuracy across six generators, versus 73.4% for CRM with outcome supervision and 72.1% for CRM.RSP slightly exceeds Qwen2.5-Math-PRM despite the latter's additional large-scale process supervision; Table 2 reports average accuracy over N = 8, 16, 32, 64, and 128.Source: Table 2 · p. 7, §4.3 · p. 7

4.4 REINFORCEMENT LEARNING WITH PROCESS REWARDS

• RSP improves average Avg@16 accuracy by 2.1% over PRM-guided reinforcement learning with Qwen2.5-Math-PRM, the strongest overall result among compared methods.RSP is the only PRM-guided method that substantially improves average accuracy over GRPO; Table 3 reports Avg@16 after reinforcement learning.Source: Table 3 · p. 8, §4.4 · p. 8

4.5 ANALYSIS OF REASONING STATE PROPAGATION

• Break scores are higher for G → B than G → G, and repair scores are higher for B → G than B → B; adding outcome supervision generally increases gains over Process-Only.The outcome-supervision analysis varies the process-to-outcome data ratio from 1:1 to 1:9 and finds gains for both beam search and reinforcement learning.Source: §4.5 · p. 8, Figure 2 · p. 9

4.6 ABLATION STUDY

• Combining process and outcome supervision gives the best performance in evaluated settings, while process-only and outcome-only models show distinct weaknesses.Process-Only drops on beam search and ProcessBench; Outcome-Only nearly preserves Best-of-N performance but drops 9.6% on ProcessBench.Source: §4.6 · pp. 8–9
• Removing repair degrades all four settings, with the largest drop of 3.3% on beam search; using only the current boundary representation also consistently degrades performance.No Repair prevents an invalid state from returning to valid; using representations before and after a step provides more informative transition estimates.Source: §4.6 · p. 9
• Sharing break and repair tokens drops performance by 2.5% on beam search and 2.3% on ProcessBench; blocking outcome-loss backpropagation also degrades results.The largest reported drops from blocking propagation are 6.7% on beam search and 4.0% on ProcessBench, despite unchanged forward propagation.Source: §4.6 · p. 9

5 CONCLUSION

• RSP models break and repair probabilities to link reasoning states across successive steps and capture error introduction, persistence, and recovery.The propagated state connects intermediate validity with the final trajectory state.Source: §5 · p. 10
• Process annotations supervise intermediate states, while outcome labels supervise the final propagated state and can provide learning signals to earlier updates.The conclusion describes the two supervision sources as complementary for learning reliable process rewards.Source: §5 · p. 10
• Generative AI tools assisted mainly with language editing and clarity, and were also used as an LLM judge in a limited experimental analysis.The authors state that they reviewed and verified AI-assisted content and outputs.Source: AI USE STATEMENT · p. 10
• The paper provides implementation and experimental details in Section 4 and Appendix A, with code in the supplementary material.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 RSP TRAINING DETAILS

• RSP initializes from Qwen3-4B and trains the backbone, special-token embeddings, and transition heads on a 1:3 mixture of selected PRM800K and AceMath-RM trajectories.The setup uses 20% of PRM800K for two epochs; each head takes adjacent boundary representations through a two-layer MLP.Source: §A.1 · p. 12

A.2 REINFORCEMENT LEARNING DETAILS

• The RL setup trains Qwen3-4B for one epoch on an 8K DAPO-MATH-17K subset, using RSP step scores only when all sampled responses receive zero reward.Otherwise, policy updates use outcome rewards; RSP step scores are normalized within the sampled group and assigned to corresponding step tokens without accumulating later-step scores.Source: §A.2 · p. 13

A.3 IMPLEMENTATION DETAILS OF ADDITIONAL BASELINES

• Additional baselines share RSP’s backbone, training mixture, and optimization settings, but differ in how they supervise outcome-only trajectories and intermediate steps.Pseudo-Label PRM uses confidence thresholds of 0.9 and 0.1; Joint-Supervised PRM supervises only the final position on outcome-labeled data; CRM† models first-error probabilities and outcome likelihoods.Source: §A.3 · p. 13

• Outcome trajectories retain the original binary correctness label after question-response extraction and filtering of multiple-choice, malformed, or unidentifiable examples.Source: §A.4 · p. 13
• AceMath-RM uses answer matching, reward-model ranking, and score-sorted filtering to retain high-confidence responses and reduce noise from heuristic answer matching.Source: §A.4 · pp. 13–14
• A correct final answer is not assumed to certify every reasoning step; outcome labels supervise only the final propagated state, while intermediate validity comes from propagation and process supervision.Source: §A.4 · p. 14

B PROCESS ERROR IDENTIFICATION

• 9.2%: RSP improves over Supervised PRM on ProcessBench; it also outperforms Qwen2.5-Math-PRM on MATH by 4.2%, though that model has the strongest overall result with additional large-scale process supervision.The comparison indicates that process annotations remain particularly important for ProcessBench’s fine-grained error-identification task.Source: B PROCESS ERROR IDENTIFICATION · p. 14

C.1 PROCESS SUPERVISION OF BREAK AND REPAIR

• Adjacent process labels constrain break after a valid prefix and repair after an invalid one, approximately extending to uncertain preceding states without separate labels for the two heads.The proposition states that well-fitted adjacent states constrain the corresponding transition predictions.Source: §C.1 · p. 15

C.2 OUTCOME EVIDENCE FOR INTERMEDIATE REASONING STATES

• Terminal outcomes increase support for intermediate states that make the observed result more likely; a correct outcome may follow uninterrupted validity or an error followed by recovery.Source: §C.2 · p. 15

C.3 LEARNING BREAK AND REPAIR FROM THE FINAL OUTCOME

• Outcome loss supplies break and repair heads with posterior-weighted targets tied to subsequent reasoning; a correct outcome after an invalid prefix requires later recovery, whose location is learned from reasoning context.When the remaining reasoning makes both states equally likely, the local outcome-loss derivatives vanish.Source: §C.3 · pp. 16–17

C.4 STRENGTH OF STATE PROPAGATION

• κt controls both how later states depend on earlier states and how outcome supervision propagates backward to earlier transitions.The learned intervening transitions modulate terminal evidence, so earlier states whose effects are weakly preserved receive weaker outcome signals.Source: §C.4 · p. 17
• 91.5% of transitions have κt > 0, with median κt 0.092 and mean 0.124; a positive tail indicates stronger dependence for some steps.A higher valid-state probability before the step generally increases the propagated valid-state probability after it.Source: §C.4 · p. 17
• 51.1% of transitions have |κt| < 0.1, while 79.0% have |κt| < 0.2; propagation strength varies across steps.Only 3.3% have |κt| < 0.01, so near-complete removal of preceding-state dependence is relatively uncommon.Source: §C.4 · p. 17, Figure 3 · p. 18
• Selective propagation lets RSP vary both state dependence and backward outcome supervision across the trajectory rather than imposing equal strength at every step.Later reasoning can preserve an earlier state, introduce an error, or recover from one.Source: §C.4 · pp. 17–18

Scaling with Model Size

• RSP consistently improves over Process-Only across four evaluation settings as Qwen3 backbone size increases from 1.7B to 8B.Training data and other settings were held fixed.Source: D SCALING ANALYSIS · p. 18

Scaling with Outcome Supervision

• With process supervision fixed, increasing the process-to-outcome batch ratio from 1:1 to 1:9 raises beam search accuracy from approximately 69% to above 72%.Reinforcement learning improves from about 63.2% to 65.8%; Best-of-N selection and ProcessBench remain relatively stable.Source: D SCALING ANALYSIS · p. 18

E TRAINING DATA AND EVALUATION OVERLAP

• Process supervision uses a deduplicated PRM800K version and selects the 20% subset used in the main experiments.Deduplication removes duplicated full trajectories while retaining distinct trajectories that share partial prefixes.Source: E TRAINING DATA AND EVALUATION OVERLAP · p. 18
• MATH500’s 500 problems are held out from the PRM800K process-training data, which uses the MATH split’s other 4,500 problems from the original test set.The held-out evaluation separation is by construction.Source: E TRAINING DATA AND EVALUATION OVERLAP · p. 19
• Outcome supervision uses released AceMath-RM training data, filtered and converted into trajectories without adding evaluation-benchmark questions.The underlying AceMath math training data had test-data filtering applied before model training.Source: E TRAINING DATA AND EVALUATION OVERLAP · p. 19
• RL training uses 8K prompts sampled from public DAPO-MATH-17K, with the same prompts, policy initialization, and optimization budget across compared methods.Process, outcome, and reinforcement-learning experiments use their corresponding public training data, and preprocessing adds no evaluation-benchmark examples.Source: E TRAINING DATA AND EVALUATION OVERLAP · p. 19

F QUALITATIVE EXAMPLES

• A fully valid trajectory keeps break probability near zero and the propagated process score near one across successive steps.This example illustrates validity preservation.Source: F QUALITATIVE EXAMPLES · p. 21
• An incorrect reduction of f(3/2) to f(1/2) sharply lowers the process score; the following step has low repair probability and preserves the invalid state.The example illustrates error introduction followed by persistence.Source: F QUALITATIVE EXAMPLES · p. 22
• After an intermediate error, a subsequent valid derivation raises repair probability and gradually restores the propagated process score.The examples include recovery after an invalid step.Source: F QUALITATIVE EXAMPLES · pp. 20, 23
• In another recovery example, the response first includes invalid k values, then excludes them and returns to the correct set before producing the answer.The invalid values yield c = 22, which is excluded because c ≤20.Source: F QUALITATIVE EXAMPLES · p. 25