PaperlayerView exact revision on arXiv

Source-linked AI summary

Let's Verify Step by Step

Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, Karl Cobbe

arXiv:2305.20050v1cs.LGcs.AIcs.CL

TL;DR

Large language models can make logical errors in complex multi-step reasoning, motivating a comparison of outcome and process supervision. This paper finds that process supervision produces more reliable reward models on MATH, solving 78.2% of a representative test subset, while active learning improves data efficiency.

  • Problem

    Large language models remain prone to logical errors in multi-step reasoning, while evidence comparing outcome and process supervision remains limited beyond grade-school mathematics.

  • Method

    The paper compares outcome and process supervision on MATH using step-level human labels and strategically surfaces convincing wrong solutions for active learning.

  • Results

    78.2% of representative MATH test problems were solved using process supervision, and active learning improved process-supervision data efficiency by 2.6×.

  • Takeaways & Limitations

    Process supervision is underexplored in mathematical reasoning, while active learning can reduce human data-collection costs and PRM800K supports related research.

  • Takeaways & Limitations

    Subtle contamination of the MATH test set cannot be ruled out and may have slightly inflated performance.

Abstract

from arXiv · show

In recent years, large language models have greatly improved in their ability to perform complex multi-step reasoning. However, even state-of-the-art models still regularly produce logical mistakes. To train more reliable models, we can turn either to outcome supervision, which provides feedback for a final result, or process supervision, which provides feedback for each intermediate reasoning step. Given the importance of training reliable models, and given the high cost of human feedback, it is important to carefully compare the both methods. Recent work has already begun this comparison, but many questions still remain. We conduct our own investigation, finding that process supervision significantly outperforms outcome supervision for training models to solve problems from the challenging MATH dataset. Our process-supervised model solves 78% of problems from a representative subset of the MATH test set. Additionally, we show that active learning significantly improves the efficacy of process supervision. To support related research, we also release PRM800K, the complete dataset of 800,000 step-level human feedback labels used to train our best reward model.

1 Introduction

Large language models can perform complex multi-step reasoning but remain prone to logical errors and hallucinations. This study compares outcome and process supervision on the challenging MATH dataset, finding stronger reliability from process supervision and improved data efficiency through active learning.

  • Motivation: Large language models solve complex multi-step tasks with chain-of-thought, yet state-of-the-art models still produce falsehoods and hallucinations.These failures are especially problematic for domains requiring multi-step reasoning.
  • Supervision methods: Outcome supervision evaluates only final results, whereas process supervision provides feedback for each chain-of-thought step.Process supervision offers more precise feedback by identifying where a solution goes wrong.
  • Study design: The study compares outcome and process supervision using a more capable base model, significantly more human feedback, and the more challenging MATH dataset.Earlier work found similar final performance for both methods on grade school math.
  • Main contributions: 78.2% of problems from a representative MATH test subset were solved using the study’s state-of-the-art process-supervised reward model.The result supports process supervision as a way to train more reliable reward models than outcome supervision.
  • Main contributions: 2.6× improvement in data efficiency resulted from active learning for process supervision.The study also found that large reward models can approximate human supervision for smaller reward models and released PRM800K, its full process supervision dataset.

2 Methods

The study compares outcome and process supervision by training reward models and evaluating their ability to select correct solutions from generator samples. Process labels come from human judgments of individual steps, with active selection and iterative retraining used to make limited feedback more valuable.

  • Supervision data: Outcome supervision uses automatically checkable final answers, whereas process supervision relies on human labels of each model-generated solution step.Labels classify steps as positive, negative, or neutral, indicating correctness and reasonableness, incorrectness or unreasonableness, or ambiguity.
  • Evaluation: Reward models are evaluated by best-of-N search: they rank uniformly sampled generator solutions, whose selected final answers are automatically graded for correctness.The generator remains fixed and is not improved with reinforcement learning; supervision refers specifically to training the reward model.
  • Supervision data: 800K step-level labels cover 75K solutions to 12K problems in PRM800K, collected exclusively from the large-scale generator.The evaluation uses the remaining 500 MATH test problems after including data from 4.5K test problems in training.
  • Supervision data: Active data collection surfaces solutions more likely to fool the best reward model, rather than uniformly sampling obvious errors.The process also iteratively retrains the PRM and applies top-K filtering either per problem or globally across the dataset.
  • Reward models: PRMs predict each step’s correctness after its final token, enabling standard language-model training and a single forward pass to score an entire solution.Process supervision deliberately labels only through the first incorrect step, revealing its location while preserving the shared information about whether a solution contains a mistake.

3 Large-scale Supervision

The study trains a large-scale PRM on PRM800K step-level labels and compares it with a substantially larger ORM training set. The PRM outperforms both the ORM and majority voting across best-of-N evaluations, while RM-weighted voting adds no noticeable improvement.

  • Training setup: The large-scale PRM is trained on PRM800K step-level labels, whereas the ORM uses 100 uniform generator samples per problem.The ORM training set has no overlap with PRM800K and is an order of magnitude larger, so the datasets are not directly comparable.
  • Best-of-N evaluation: The PRM strongly outperforms both the ORM and majority voting for every evaluated best-of-N value.The ORM performs slightly better than majority voting, while the PRM’s performance gap widens as N increases.
  • Evaluation and voting: RM-weighted voting does not noticeably improve performance when combining PRM and majority voting.Evaluation uses a specific subset of the MATH test set, with further results broken down by problem difficulty in Appendix G.

4 Small-scale Synthetic Supervision

Using PRMlarge to simulate human feedback, the study directly compares process supervision with two outcome-supervision baselines on otherwise identical datasets. Process supervision significantly outperforms both outcome methods, while active-learning benefits are limited by insufficient diversity and iterative retraining is unstable.

  • Synthetic supervision: PRMlarge supervises smaller models to ablate outcome-versus-process supervision factors at modest cost despite the high cost of human feedback.The authors use the large-scale process-supervision model from Section 3, called PRMlarge, to simulate extensive data collection.
  • Direct comparison: For each dataset, reward models receive process supervision from PRMlarge, outcome supervision from PRMlarge, or final-answer supervision, with identical training data otherwise.The experiment samples between 1 and 200 solutions per problem from a small-scale generator.
  • Direct comparison: Process supervision significantly outperforms both outcome-supervision forms at all data-collection scales under best-of-500 evaluation.Across best-of-N evaluations, PRMlarge-based outcome supervision is more effective than final-answer checking.
  • Baseline choice: Outcome supervision by PRMlarge is considered the more relevant baseline for domains less susceptible to false positives than MATH.Final-answer supervision is more explicitly outcome based, but false positives may be over-emphasized in MATH.
  • Active learning: Active learning selects N samples per problem with 80% convincing wrong-answer samples and 20% remaining convincing samples, scored by PRMlarge.PRMselector scores 1000 samples per problem after training on a single sample from each problem.
  • Active learning: Iteratively retraining PRMselector produced instability, and the resulting reward models performed no better than the non-iterative models.The authors were unable to diagnose the instability and have no concrete evidence that iterative retraining helps.

5 OOD Generalization

The study evaluates the large-scale ORM and PRM for out-of-distribution generalization on 224 recent STEM exam questions likely unseen during pre-training, reporting best-of-100 results for both models and majority voting.

  • Evaluation setup: 224 STEM questions from recent AP Physics, AP Calculus, AP Chemistry, AMC10, and AMC12 exams form the held-out evaluation set.These exams were released after the pre-training dataset was compiled, providing high confidence that the model had not seen the problems.
  • Evaluation setup: The evaluation reports best-of-100 performance for the ORM, PRM, and majority voting.The results are presented in Table 1.
  • Findings: The authors state that the observed results are similar to those from prior evaluations.The supplied passage truncates before specifying which results are being compared.

6 Discussion

The discussion attributes process supervision’s strong performance to more precise credit assignment and highlights advantages for interpretability and safety. It also acknowledges possible MATH contamination while presenting evidence that it did not significantly affect the reported comparisons.

  • Advantages of process supervision: Process supervision specifies which initial steps are correct and identifies the incorrect step, easing credit assignment compared with outcome supervision.Outcome supervision must infer where an incorrect solution went wrong, while negative labels are less informative when most generated solutions contain errors.
  • Advantages of process supervision: Process supervision encourages interpretable reasoning and directly rewards aligned chains of thought, whereas outcome supervision relies on outcomes as a proxy for aligned behavior.The discussion also characterizes outcome supervision as harder to scrutinize.
  • Test-set contamination: MATH test problems may overlap with model pretraining data despite string-matching removal attempts, because online rephrasings are difficult to detect.The authors therefore cannot guarantee that MathMix and MATH are free of overlap.
  • Test-set contamination: Low-single-digit percentage generator solve rates and qualitatively similar results on guaranteed-uncontaminated problems support the claim that contamination did not significantly impact the work.Manual inspection found no clear memorization, but subtle memorization cannot be ruled out; any contamination would be expected to affect methods similarly.

7 Related Work

Related studies connect this work to comparisons of outcome and process supervision, reward-model supervision, and methods that improve multi-step reasoning. Differences in supervision scale help reconcile prior findings with this paper’s stronger results for process supervision.

  • Outcome and process supervision: Uesato et al. (2022) found similar final-answer error rates for outcome and process supervision, with process supervision achieving them using less data.Their study examined grade school mathematics and used a methodology closely related to this work.
  • Outcome and process supervision: Supervision scale may explain differing results: small amounts of process supervision and large amounts of outcome supervision can yield similar performance.This interpretation is supported by the data-scaling trend reported in Figure 4a.
  • Reward-model supervision: Gao et al. (2022) used a large reward model to supervise smaller models while studying RLHF over-optimization with extensive human preference data.They replaced human feedback with a gold-standard reward model, paralleling this work’s use of large-scale reward models to supervise smaller reward models.
  • Reasoning methods: Prior work improved multi-step reasoning through technical-content finetuning, self-consistency, explicit intermediate reasoning, and zero-shot prompting.These studies reported improved MATH performance, strong reasoning-benchmark results without finetuning, and the effectiveness of chain-of-thought or scratchpad reasoning, including zero-shot elicitation.

8 Conclusion

The paper concludes that process supervision produces more reliable reward models than outcome supervision for mathematical reasoning. It also shows that active learning reduces human data-collection costs and releases PRM800K to support related research.

  • 8 Conclusion: Process supervision trains much more reliable reward models than outcome supervision for mathematical reasoning.The conclusion characterizes this as a demonstrated result within the mathematical-reasoning domain.
  • 8 Conclusion: Active learning lowers human data-collection costs by surfacing only the most valuable model completions for feedback.The method focuses human review on completions judged most valuable for feedback.
  • 8 Conclusion: The authors release PRM800K, the full human-feedback dataset used to train their state-of-the-art reward model.The release is intended to remove a significant barrier to entry for related research.

A MathMix · B PRM800K

MathMix is a smaller, aggressively filtered math-pretraining dataset used before fine-tuning, while PRM800K provides roughly 800,000 filtered step-level human-feedback labels from 75,000 solutions. The sections also describe contamination checks and quality-control procedures for the data.

  • A MathMix: MathMix is a smaller, aggressively filtered dataset of math problem-solving content that omits explicitly mixed general-language data.It contains individual problems and solutions, free-form mathematical discussions, and synthetic data.
  • A MathMix: 1.5B tokens comprise MathMix, compared with Minerva’s 38.5B-token pretraining dataset.MathMix contains math problems, solutions, mathematical discussions, and synthetic data, whereas Minerva’s dataset included 5% general natural language.
  • A MathMix: Smaller-model experiments use a 1B-token MathMix variant without critiques, while large-model experiments train on roughly 3B tokens.Small models train for 6 epochs, roughly 6.6B tokens; large models train for 2 epochs.
  • A MathMix: MathMix undergoes decontamination checks against the MATH test split, but the authors make no strong guarantees about their efficacy.Checks include stripping LaTeX and searching for matching n-grams; the authors do not expect contamination to significantly affect relative comparisons.
  • B PRM800K: 1,085,590 step-level labels were collected over 101,599 solution samples, producing a filtered dataset of about 800,000 labels over 75,000 solutions.Quality-control labels and labels from incomplete tasks were discarded during training, and the complete PRM800K dataset was released.
  • B PRM800K: PRM800K data collection used two phases, with phase 1 gathering labels for multiple alternative completions and yielding more repetitive labels.Phase 1 accounted for about 5% of PRM800K and was cumbersome because alternatives were often repetitive and solutions were sometimes uninteresting.
  • B PRM800K: Labeler quality was screened with 30 quality-control questions requiring at least 75% agreement with gold labels before phase 2.Continuous quality control removed labelers whose performance declined and supported educational material addressing common mistakes.

C Evaluation · D Labelling Instructions

The evaluation uses 500 uniformly sampled held-out MATH test problems after expanding training with 4,500 test-split problems. Labelers classify solution steps by correctness, contextual appropriateness, and progress, with procedures revised after phase-1 trajectories became inefficient or unstable.

  • C Evaluation: 4,500 MATH test-split problems were added to training to reduce overfitting on the original 7,500 MATH training problems.The models are evaluated only on the remaining 500 held-out problems.
  • C Evaluation: 500 held-out MATH test problems were selected uniformly at random for evaluation.The subset’s difficulty and subject distributions were compared with those of the original MATH test set.
  • C Evaluation: Figure 5 compares histograms of difficulty levels and subjects between the original MATH test set and the 500-problem evaluation subset.
  • D Labelling Instructions: Labelers classified each solution step as positive, negative, or neutral.Neutral steps were contextually appropriate, reasonable, correct, and easily verifiable; positive steps additionally progressed toward the solution, while all others were negative.
  • D Labelling Instructions: Labelers received ground-truth final answers but no reference solutions when assigning step labels.
  • D Labelling Instructions: In phase 1, labelers could enter replacement steps when all candidate steps were negative, after which trajectories continued from randomly selected positive or neutral steps.
  • D Labelling Instructions: Phase-1 labeling often produced endless neutral-step sequences or negative steps requiring constant human supervision, motivating phase-2 pre-generation of whole solutions.

E ORM Training Details · F PRM Details

The outcome-supervised reward model is trained for one epoch without dropout or a joint language-modeling objective. It scores solutions using the final token’s prediction after assigning each solution-wide label to every token.

  • E ORM Training Details: ORM training uses a single epoch, no dropout, and no jointly learned language-modeling objective.Performance is not sensitive to most other hyperparameters within a reasonable range.
  • E ORM Training Details: Model samples are drawn uniformly from the generator at temperature 1.0 without rebalancing positive or negative examples.This describes the sampling procedure used to collect training samples.
  • E ORM Training Details: At training time, the reward model predicts every context token, assigning each solution token the same target based on solution correctness.At test time, the final completion token’s score serves as the overall solution score.

F.1 Training … I.3 False Positives

The paper details PRM training and scoring choices, showing how process rewards behave across problem difficulties and under synthetic supervision. Visual examples illustrate both successful error detection and recurring false positives.

  • F.1 Training: PRMs predict positive, negative, and neutral labels for solution steps, with low learning rates stabilizing fine-tuning after the shift from language modeling to classification.All PRMs are trained for 2 epochs; this helps smaller datasets more than larger ones.
  • F.2 Scoring: The solution score reduces step-level positive-label probabilities using either a minimum or product, while treating neutral labels as positive or negative.The chosen strategy is the product with neutrals treated as positives; differences among the four strategies are minor.
  • G Difficulty Breakdown: ORM and PRM performance differ across all MATH difficulty quintiles, while additional samples help most on the hardest problems.On the easiest problems, adversarial examples can fool the ORM, whose performance slightly decreases as sample count rises.
  • H Synthetic Supervision Details: PRMlarge supplies synthetic step labels by marking steps incorrect when negative probability exceeds 20%, reflecting its slight positive-label miscalibration.Process supervision returns labels through the first negative step, whereas outcome supervision marks a solution correct only if every step is correct.
  • I PRM Visualizations: Visualization examples use GPT-4 generator solutions, report generator pass-rates, and select best-of-1860 solutions ranked by the large-scale PRM.True-positive examples include difficult trigonometry, guess-and-check verification, and successful polynomial factorization.
  • I.1 True Positives: True-positive visualizations show the PRM recognizing valid chains of reasoning, including insightful identity use and successful multi-step algebraic transformations.The examples demonstrate verification of potentially hallucinated guess-and-check steps as well as correct trigonometric simplification.
  • I.2 True Negatives: The reward model catches incorrect simplifications, calculation errors, invalid factorizations, and other mistakes in several generator solutions.These true-negative examples include problems with generator pass-rates from 4.5% to 93.5%.
  • I.3 False Positives: The reward model is fooled by subtle counting, algebraic, decimal-notation, and sequence-period mistakes in several solutions.The illustrated false-positive examples have generator pass-rates of 18.5%, 17.6%, 13.4%, and 9.1%.
Loading 2305.20050v1…