Source-linked AI summary
Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations
Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, Zhifang Sui
TL;DR
LLMs struggle with complex multi-step mathematical reasoning, while training process reward models commonly depends on costly human annotations. Math-Shepherd automatically constructs step-wise supervision and applies it to verification and step-by-step PPO, producing strong results across GSM8K and MATH and improving Mistral-7B from 77.9% to 84.1% on GSM8K and from 28.6% to 33.0% on MATH.
Problem
Complex multi-step mathematical reasoning remains difficult for LLMs, and training process reward models requires costly human annotations.
Method
Math-Shepherd automatically constructs process-wise supervision by scoring intermediate steps according to their potential to yield correct final answers, then uses the model for verification and step-by-step PPO.
Results
Math-Shepherd demonstrates effectiveness in verification and reinforcement learning, including improving Mistral-7B from 77.9%→84.1% on GSM8K and 28.6%→33.0% on MATH with step-by-step PPO.
Takeaways & Limitations
Automatic process supervision can support process reward modeling and improve mathematical reasoning without relying on labor-intensive human annotation.
Takeaways & Limitations
Generating multiple subsequent reasoning processes for each step requires substantial computing resources, although it remains significantly cheaper than human annotation.
Abstract
from arXiv · showhide
In this paper, we present an innovative process-oriented math process reward model called \textbf{Math-Shepherd}, which assigns a reward score to each step of math problem solutions. The training of Math-Shepherd is achieved using automatically constructed process-wise supervision data, breaking the bottleneck of heavy reliance on manual annotation in existing work. We explore the effectiveness of Math-Shepherd in two scenarios: 1) \textit{Verification}: Math-Shepherd is utilized for reranking multiple outputs generated by Large Language Models (LLMs); 2) \textit{Reinforcement Learning}: Math-Shepherd is employed to reinforce LLMs with step-by-step Proximal Policy Optimization (PPO). With Math-Shepherd, a series of open-source LLMs demonstrates exceptional performance. For instance, the step-by-step PPO with Math-Shepherd significantly improves the accuracy of Mistral-7B (77.9\%$\to$84.1\% on GSM8K and 28.6\%$\to$33.0\% on MATH). The accuracy can be further enhanced to 89.1\% and 43.5\% on GSM8K and MATH with the verification of Math-Shepherd, respectively. We believe that automatic process supervision holds significant potential for the future evolution of LLMs.
1 INTRODUCTION
Math-Shepherd addresses the cost of human process annotation by automatically constructing step-wise supervision for math reasoning. It supports verification and step-by-step reinforcement learning, improving open-source LLM performance on GSM8K and MATH.
- Motivation: Process reward models evaluate reasoning paths step-by-step, enabling precise error feedback but requiring costly human annotations for complex problems.This feedback is useful for reinforcement learning and automatic correction, yet advanced multi-step tasks make human supervision expensive.
- Method: Each intermediate step is scored by its potential to produce the correct final answer, estimated from multiple decoded continuations and their final-answer correctness.A step yielding more correct answers receives a higher correctness score.
- Evaluation: Math-Shepherd is evaluated for reranking multiple LLM outputs and for step-by-step PPO reinforcement learning on GSM8K and MATH.Experiments cover open-source LLMs ranging from 7B to 70B parameters.
- Results: 77.9%→84.1% on GSM8K and 28.6%→33.0% on MATH for Mistral-7B after step-by-step PPO with Math-Shepherd.Verification further raises accuracy to 89.1% on GSM8K and 43.5% on MATH.
- Contribution: Math-Shepherd automatically constructs process supervision datasets without human annotations for math reasoning tasks.The framework is proposed to address the annotation bottleneck in process reward model training.
- Contribution: The paper empirically analyzes factors involved in training high-performing process reward models with automatic step-by-step verification and supervision.The analysis is presented as guidance for improving reasoning capability.
2 RELATED WORKS
Prior work improves mathematical reasoning through pre-training, fine-tuning, prompting, and verification. Verification methods use either outcome-level or process-level reward models to select among candidate solutions.
- Improving Mathematical Reasoning: Mathematical reasoning improvement methods include pre-training on math-related data, fine-tuning, prompting, and verification.These approaches target the difficulty of mathematical reasoning tasks for LLMs.
- Reasoning Verification: Outcome reward models score an entire solution, whereas process reward models score each individual reasoning step.The related work describes process reward models as outperforming outcome reward models in recent findings.
3 METHODOLOGY
The methodology defines process-reward training and evaluates reward models through best-of-N verification and step-by-step reinforcement learning. It automatically labels reasoning steps by completing them and estimating their potential to reach the correct answer.
- Verification: Best-of-N verification scores sampled candidate solutions with a reward model and selects the highest-scoring solution.The evaluation measures whether improved scoring increases the likelihood of selecting a candidate with the correct answer.
- Reward models: PRM assigns scores to individual reasoning steps, providing more detailed feedback than ORM’s single score for an entire solution.PRM training is treated as binary classification in this paper, while prior work also formulated it as three-class classification.
- Automatic process annotation: The framework defines a step’s quality by its potential to deduce the correct answer, while acknowledging that this criterion introduces some noise.The authors report that the definition remains beneficial for training a PRM.
- Automatic process annotation: A completer decodes multiple subsequent reasoning paths from an intermediate step, and the decoded answers are used to estimate that step’s quality.The framework uses hard estimation and soft estimation, then trains the PRM with cross-entropy loss.
- Reinforcement learning: Step-by-step PPO supplies rewards at the end of each reasoning step instead of only at the response end.The automatically constructed PRM supervises LLMs, with accuracy evaluated using greedy decoding.
- Experimental setup: The verification experiments use 256 outputs and reward models trained on LLama2-70B for GSM8K and LLemma-34B for MATH.The ranking procedure can also combine self-consistency with reward models by grouping solutions according to their final answers and aggregating group scores.
4 EXPERIMENTS
Experiments evaluate MATH-SHEPHERD for best-of-N verification and step-by-step reinforcement learning across GSM8K and MATH. The method improves model performance, while combined reinforcement learning and verification are reported as complementary.
- Experimental Setup: The study uses GSM8K and MATH with LLaMA2, LLemma, Mistral, and DeepSeek models spanning approximately 7B to 70B parameters.MATH verification uses the 500-problem MATH500 subset because of computation cost.
- Verification: MATH-SHEPHERD consistently outperforms self-consistency and ORM as a verifier across GSM8K and MATH with all tested generators.DeepSeek-67B reaches 93.3% on GSM8K and 48.1% on MATH with MATH-SHEPHERD.
- Reinforcement Learning: 84.1% on GSM8K and 33.0% on MATH are achieved by Mistral-7B with step-by-step PPO supervised by MATH-SHEPHERD.Step-by-step PPO significantly improves two supervised fine-tuned models and outperforms vanilla PPO with ORM.
- Combined Reinforcement Learning and Verification: 7.2% accuracy is the MATH improvement of step-by-step PPO Mistral-7B over supervised fine-tuning with self-consistency as verifier.The corresponding greedy-decoding performance gap is 4.4%.
- Combined Reinforcement Learning and Verification: Reinforcement learning and verification are complementary, but vanilla reward-model verification after reinforcement learning is inferior to self-consistency.The authors attribute this to the initial reward model being insufficient to supervise the more powerful post-PPO model.
5 ANALYSIS
The analysis examines annotation quality, verifier scaling, model-size effects, data efficiency, and out-of-distribution behavior. Results generally favor automatically supervised PRMs, while annotation quality depends on completer strength, training data, and sampling choices.
- Candidate Solutions: The automatically annotated dataset outperforms human-annotated PRM800K on MATH in the reported candidate-count comparison.The authors cite distribution alignment with open-source LLaMA outputs and a dataset four times larger than PRM800K.
- Quality of Automatic Process Annotations: 86% annotation accuracy is reached by a LLaMA2-70B completer at N=4 in hard estimation.Further increases in N reduce accuracy, which the authors associate with false positives.
- Quality of Automatic Process Annotations: SE aligns progressively closer to the human-annotated distribution as N increases, while HE does not show the same trend.Despite this, verifier performance does not substantially differ between SE- and HE-trained models.
- Annotation Methods: The proposed annotation strategy substantially outperforms NLI- and rule-based automatic process annotation methods in the Table 4 comparison.The comparison is between methods from Li et al. (2023b) and MATH-SHEPHERD.
- Quality of Automatic Process Annotations: Larger completers and higher-quality training sets produce better automatic annotation quality; the Weak training set yields markedly larger loss.The authors suggest that prior exposure to questions helps LLMs act as completers.
- Influence of Model Sizes: PRM consistently outperforms self-consistency and ORM across 7B, 13B, and 70B generator settings.Larger reward models are more robust as candidate counts increase.
- Influence of Model Sizes: A larger reward model improves verification of a smaller generator, whereas a smaller reward model can harm verification of a larger generator relative to self-consistency.This comparison supports matching verifier capacity to generator capacity.
- Data Efficiency: PRM exceeds ORM by approximately 4% accuracy with a 10k-instance training set, indicating higher data efficiency and a higher apparent performance ceiling.The comparison is reported in Figure 6(a).
6 LIMITATIONS
The paper identifies computational cost and noise in automatically constructed process annotations as limitations, while noting that both remain compatible with promising empirical results.
- Computational cost: The completion process requires substantial computing resources because each reasoning-step label uses N decoded subsequent reasoning paths.Higher N improves automatic annotation quality, but efficient inference techniques may mitigate the cost; it remains lower than human annotation.
- Annotation noise: Automatic process annotations contain noise, although the resulting PRM outperforms the human-annotated PRM800K dataset.The paper states that the effect of this potential noise on PRM performance remains undetermined.
- Future work: A comprehensive comparison between human and automated process annotations remains future work.The paper suggests that integrating both annotation types could help construct robust and efficient process supervision.
7 CONCLUSION
The paper introduces MATH-SHEPHERD, a process-oriented verifier trained with automatically constructed process-wise supervision rather than labor-intensive human annotation. Experiments in verification and reinforcement learning demonstrate its effectiveness.
- 7 CONCLUSION: MATH-SHEPHERD assigns reward scores to individual steps of math solutions and is trained using automatically constructed process-wise supervision data.The approach removes the necessity for labor-intensive human annotation and correlates strongly with human annotations.
- 7 CONCLUSION: Experiments in verification and reinforcement learning scenarios demonstrate the effectiveness of MATH-SHEPHERD.The reported evaluations cover both selecting among generated outputs and reinforcing language models.