Entropy-Regularized Process Reward Model

1 Introduction

• Because reward quality sets an upper bound on algorithm performance, the paper treats reward construction as central to improving reasoning RL.The introduction describes both training-time alignment and inference-time selection as relying on external reward models.Source: §1 · pp. 1–2
• PRMs score each reasoning step, whereas ORMs score the final answer; prior work reports stronger performance and more interpretable feedback from PRMs.The introduction cites a result where PRM selection reaches 78.2% test accuracy, versus 72.4% for ORM and 69.6% for majority voting.Source: §1 · p. 2
• Math-Shepherd estimates a step’s potential to reach a correct answer from the ratio of correct sampled continuations, avoiding expensive human process labeling.The passage reports that this approach can outperform human-labeled data.Source: §1 · p. 2
• The paper identifies a mismatch between Math-Shepherd’s traditional MDP and LLM RL practice, which uses entropy-regularized rewards to balance optimization with staying near the initial policy.The authors formulate multi-step mathematical reasoning under an entropy-regularized MDP.Source: §1 · pp. 2–3
• The authors introduce entropy-regularized process reward labeling and train ER-PRM, reporting gains on MATH and GSM8K under best-of-N evaluation and policy improvements with rejection sampling.They also state that they will release the data, code, and checkpoints.Source: §1 · p. 3

Mathematical Reasoning and Synthetic Data Generation

• Synthetic-data pipelines use teacher models to generate and filter correct responses, including through rejection or best-of-n sampling, and emphasize stepwise reasoning.The passage lists datasets including Meta-MathQA, MAmmoTH, MMIQC, ToRA, Open-MathInstruct, and Dart-Math.Source: §2 · p. 3

Applying RL to Mathematical Reasoning

• Mathematical-reasoning approaches include deep RL, direct and iterative preference learning, and inference-time search or sampling, all of which depend on reward models and can benefit from this study.The passage names PPO, GRPO, DPO, KTO, best-of-n sampling, and MCTS.Source: §2 · p. 3

Reward Modeling in Mathematical Reasoning

• Reward models include ORMs for whole responses and PRMs for individual reasoning steps; Math-Shepherd estimates process labels using correctness ratios from parallel completions.The passage notes that PRMs require extensive human annotation and describes Math-Shepherd as effective as human annotation for generating high-quality datasets.Source: §2 · p. 3

3 Approach

• ER-PRM derives process-reward calculation from an entropy perspective using KL regularization, then relates the formulation to obtaining an optimal policy through RLHF.The authors contrast their approach with aggregation-based calculations that depend on the initial policy model.Source: §3 · p. 4

3.1 Task Formulation and Optimal Policy

• The task formulation seeks a policy for an L-step reasoning chain under a KL-regularized loss, with the initial policy serving as the reference model.The initial policy is a pretrained LLM, while the policy being optimized is fine-tuned.Source: §3.1 · p. 4

3.2 Multi-step Reasoning

• For a partial chain, the intermediate reward aggregates exponentiated final rewards over continuations; using the reference policy enables reward computation without simultaneous reward-and-policy learning.The formulation also has an equivalent optimal-policy version described as soft pessimism; when completions come from the initial policy, the reward is soft-max, while optimal-policy completions give soft-min.Source: §3.2 · pp. 4–5

Process Reward Construction

• Step labels estimate each reasoning step’s potential to reach the correct answer from sampled continuations, using Equation 2 and the initial policy.The process-reward score is based on how many sampled paths reach the correct final answer.Source: §4 · p. 5
• For each problem, the labeling setup samples 15 solutions and generates 16 possible paths from each step; datasets are GSM8K and MATH.The approach is described as automatic labeling with entropy regularization, similar to Math-Shepherd and inspired by MCTS.Source: §4 · p. 5

Process Reward Sampling

• Mistral-MetaMath-7B and DeepSeek-math-7B-instruct collect about 260K data points and 1.5 million step annotations, used to train reward models with Llama-3.1-8B.Mistral-data and DeepSeek-data use their respective models as generator and completer; reward models are also trained at 1B and 3B scales in Section 6.1.Source: §4 · p. 5

Process Reward Model Evaluation

• Best-of-N evaluation samples N solutions per test problem, scores each response by its lowest-scoring step, and selects the response with the highest overall score.Evaluation uses the full GSM8K test set and MATH500.Source: §4 · p. 6

Baseline Settings and Training Details

• The baselines are Hard-label PRM, Soft-label PRM, and ORM; autoregressive PRMs predict a special token after each step, while ORM predicts it at response end.Models are trained with cross-entropy against ground-truth rewards; experiments are repeated with five seeds and report average accuracy and standard deviation.Source: §4 · p. 6

Policy Optimization with PRM-Guided Reinforcement Learning

• Rejection-sampling fine-tuning ranks 16 responses per question with a PRM, selects top-scoring responses, and evaluates the updated policies with zero-shot CoT Top-1 accuracy.Fine-tuning uses MATH training data and OpenMathInstruct-2, and reports average accuracy and standard deviation on GSM8K and MATH.Source: §4 · p. 6

5.1 Auto-Regressive Training Results

• With Mistral-MetaMath-7B as generator, ER-PRM consistently beats all baselines, PRMs outperform ORM, and the gap widens on harder MATH problems.The figure reports best-of-N evaluation on GSM8K and MATH500 using Mistral-data.Source: §5.1 · p. 6, Figure 2 · p. 7

DeepSeek-math-7b-instruct as the Generator, Reward Models trained on DeepSeek-data

• With DeepSeek data and generator, ER-PRM beats Soft-Label and Hard-Label PRMs on GSM8K and MATH500; it also beats Math-Shepherd in nearly all settings.Hard-Label PRM often surpasses Soft-Label PRM, especially at large N; on MATH500, gains from N=4 to N=64 are less pronounced, and large N brings reward hacking.Source: DeepSeek-math-7b-instruct as the… · pp. 7–8, Figure 3 · p. 7

5.2 RLHF Results

• The RLHF comparison uses RAFT with Mistral-MetaMath-7B and DeepSeek-math-7B-instruct policies, pairing each with reward models trained on its corresponding data.Source: §5.2 · p. 8

5.2.1 Manifestation and Mitigation of Overfitting

• All reported evaluations use GSM8K and MATH test sets; because the policies had seen these datasets during supervised fine-tuning, RLHF blends in 12,500 novel OpenMathInstruct-2 questions.The novel questions are training-only, and 16 candidate rationales per question are generated for reward-model selection.Source: §5.2.1 · p. 8

5.3 Performance Evaluation

• For Mistral, ER-PRM surpasses Soft-Label PRM by 0.8% and 2.7% on GSM8K and MATH, respectively, and Hard-Label PRM by 0.5% and 0.8%.Across in-domain and out-of-domain data, ER-PRM also has 1–2% higher best-of-N accuracy when selecting RLHF data.Source: §5.3 · p. 8, Table 1 · p. 8

6.1 Reward Models with Various Sizes

• ER-PRM outperforms Soft-label and Hard-label PRMs at both 1B and 3B, while the 3B model outperforms 1B and is comparable to 8B on MATH.For 1B, accuracy drops about 2% from best-of-16 to best-of-64 on MATH, whereas reward hacking is largely mitigated at 3B; the 1B model's gain from N=4 to N=64 is 2–3%, versus 4–6% for 8B.Source: §6.1 · p. 9

6.2 Out-of-Distribution Performance of Process Reward Models

• On OpenMath2-Llama3.1-8B, all three reward models improve MATH500 accuracy by more than 4% over Top 1, and ER-PRM improves the policy model most.With DeepSeek-Math-7B-RL, best-of-N gains accrue slowly and ORM performance drops; the authors attribute smaller gains partly to reward models trained on weaker Mistral traces. On OpenMath2, N=64 is not best, indicating reward hacking.Source: §6.2 · pp. 10–11, Figure 4 · p. 10

6.3 Impact of Entropy Regularization

• η has a dataset-dependent peak: η=1 or 2 on MATH and η=15 on GSM8K, while both very small and large settings reduce performance.The authors conclude that fine-grained reward scores outperform hard labels, which may add training noise; vanilla soft labeling at η=0.1 also performs relatively poorly.Source: §6.3 · pp. 11–12, Figure 5 · p. 11

6.4 Regression Training Strategy

• Regression-trained reward models improve as N grows and outperform the hard- and soft-label settings, but remain consistently below autoregressively trained models on both datasets.The gap is especially significant on MATH; the authors hypothesize that integrated, coherent steps in autoregressive training reduce forgetting and ease learning longer reasoning traces.Source: Figure 6 · p. 11, §6.4 · p. 12

7 Conclusion

• ER-PRM outperforms baseline outcome and process reward models on the MATH and GSM8K benchmarks.The conclusion reports a large margin but gives no numerical values in this passage.Source: §7 · p. 12
• The method derives process reward scores through a new aggregation method grounded in an entropy perspective and uses entropy-regularized reinforcement learning.Source: §7 · p. 12
• Rejection sampling with the process reward model significantly improves policy-model performance in RLHF experiments.Source: §7 · p. 12

A Ethical Statements and Limitations

• The authors report using publicly available preference datasets under their licenses and flag potential bias, reasoning-style overfitting, and misuse risks.They recommend rigorous testing, diverse datasets, safety evaluations, and responsible deployment.Source: A Ethical Statements and Limitations · p. 18
• ER-PRM’s stated limitations include high computational cost, overfitting risk, untested generalization beyond mathematics, and limited interpretability of reward aggregation.The paper highlights tuning concerns particularly for DeepSeek-Math-7B-Instruct.Source: A Ethical Statements and Limitations · p. 18

B Technical Lemma

• The technical lemma identifies the loss-functional minimizer as a Gibbs distribution.The supplied passages do not provide a complete expression for the minimizer.Source: B Technical Lemma · p. 18

C Best-of-N Evaluation Results in Tables

• Tables 5 and 6 list best-of-N evaluation performance for ER-PRM and other baselines; Table 5 shows results for reward models with policy model Mistral-MetaMath-7b, and Table 6 for Deepseek-Math-7B-Instruct.Source: C Best-of-N Evaluation Results in Tables · p. 18
• Tables 7 and 8 cover different η values on GSM8K and MATH, while Table 9 reports results for regression-trained reward models.The section maps Tables 7 and 8 to Section 6.3 and Table 9 to Section 6.4.Source: C Best-of-N Evaluation Results in Tables · p. 18

D Definition of Step, Partial Chain, and Chain

• Tables 5 and 6 compare best-of-N reward models on GSM8K and MATH using Mistral-MetaMath-7b and DeepSeek-Math-7B-Instruct policy models.The reward models are trained autoregressively from Llama-3.1-8B on Mistral data for Table 5 and DeepSeek data for Table 6.Source: Table 5 · p. 19, Table 6 · p. 19
• Tables 7 and 8 evaluate different η values on GSM8K and MATH; Table 9 uses regression training, and Table 10 evaluates DeepSeek-Math-7B-RL.Tables 7–10 specify different training or evaluation settings, not outcome values in the supplied captions.Source: Table 7 · p. 19, Table 8 · p. 19, Table 9 · p. 19
• A step is one operation, a chain is the complete sequence of steps, and a partial chain is an intermediate sequence that may not reach the final answer.Partial chains are used to compute intermediate rewards that guide reasoning step by step.Source: D Definition of Step, Partial Chain,… · pp. 18, 20
• “Step-level derivation” refers to the last-step reward for a partial chain, evaluating intermediate contributions rather than the full chain’s final success.The distinction matters because step-level scores guide the model during reasoning.Source: D Definition of Step, Partial Chain,… · p. 20

E.1 Intuitive Perspective on Entropy-Regularized Process Rewards

• Under the initial policy π0, the soft-max view treats a continuation’s reward as an optimistic assessment of its potential to reach a correct answer.Higher η makes the assessment more focused on high-reward paths while still weighting multiple possible futures.Source: §E.1 · p. 20
• Under the optimal policy π∗, the soft-min view estimates the smallest penalty needed to make a step comparable to alternatives, focusing cautiously on worst-case scenarios.The passage describes this as aligning with the preferences of a stronger model.Source: §E.1 · p. 20
• In the Figure 1 example, step a1 has three continuations with correctness values 1, 1, and 0, which are combined into an expected reward score.The first two paths reach correct final answers; the third does not.Source: §E.1 · p. 20

E.2 Intuitive Understanding and Theoretical Advantages

• Sampling from π0 yields a soft-max reward formulation, while sampling from π∗ yields a soft-min formulation.The soft-max is described as optimistic for a weaker initial policy; the soft-min is conservative for a stronger optimal policy.Source: §E.2 · p. 21
• The exponential transformation amplifies high-reward trajectories before expectation, while a subsequent logarithm preserves reward ordering and controls scale.The paper contrasts this with simple expectations that can yield weak signals when the initial policy rarely samples high-reward paths.Source: §E.2 · p. 21
• The method tunes one hyperparameter, η, to control the entropy penalty and balance exploitation of high-reward trajectories against policy-space breadth.Source: §E.2 · p. 21
• The formulation can compute process rewards using only the initial policy π0, avoiding dependence on the optimal policy and complex weighting schemes.The passage links this property to equation 2.Source: §E.2 · p. 21

• Figure 7 is a figurative illustration of the experimental setting.Source: Figure 7 · p. 22