PaperlayerView exact revision on arXiv

Source-linked AI summary

Solving math word problems with process- and outcome-based feedback

Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, Irina Higgins

arXiv:2211.14275v1cs.LGcs.AIcs.CL

TL;DR

The paper asks whether language models producing reasoning traces should be supervised on final answers or on each reasoning step, a distinction relevant when correct answers can hide flawed reasoning. It comprehensively compares these approaches on GSM8K using supervised and reward-model-based training, finding similar final-answer error but much lower trace error with process-based feedback or reward models that emulate it.

  • Problem

    Correct final answers can conceal incorrect reasoning traces, creating a need to compare supervision of final outcomes with supervision of reasoning processes.

  • Method

    The paper compares outcome- and process-based supervision on GSM8K using reasoning-trace evaluation, supervised learning, reinforcement learning, and outcome- or process-supervised reward models.

  • Results

    Similar final-answer error rates are achieved by outcome- and process-based approaches, while low trace error requires process-based feedback or a reward model that emulates it.

  • Takeaways & Limitations

    Outcome supervision can suffice for final-answer accuracy on GSM8K, but reliable reasoning traces require process-based supervision or learned feedback that captures process correctness.

  • Takeaways & Limitations

    The conclusions may be specific to math word problems because incorrect traces are typically harmful for reaching correct answers, unlike in some other domains.

Abstract

from arXiv · show

Recent work has shown that asking language models to generate reasoning steps improves performance on many reasoning tasks. When moving beyond prompting, this raises the question of how we should supervise such models: outcome-based approaches which supervise the final result, or process-based approaches which supervise the reasoning process itself? Differences between these approaches might naturally be expected not just in final-answer errors but also in reasoning errors, which can be difficult to detect and are problematic in many real-world domains such as education. We run the first comprehensive comparison between process- and outcome-based approaches trained on a natural language task, GSM8K. We find that pure outcome-based supervision produces similar final-answer error rates with less label supervision. However, for correct reasoning steps we find it necessary to use process-based supervision or supervision from learned reward models that emulate process-based feedback. In total, we improve the previous best results from 16.8% $\to$ 12.7% final-answer error and 14.0% $\to$ 3.4% reasoning error among final-answer-correct solutions.

1. Introduction

The paper compares outcome-based supervision of final answers with process-based supervision of reasoning steps for language models solving GSM8K math word problems. It finds similar final-answer performance, but substantially lower reasoning-trace error when using process-based feedback or reward models that emulate it.

  • Motivation and approach: Human-comprehensible reasoning matters because correct final answers can accompany incorrect traces, especially in educational and safety-relevant settings.The paper notes that incorrect or difficult-to-understand strategies may produce superficially appealing outcomes.
  • Motivation and approach: The study compares outcome-based supervision of final answers with process-based supervision of individual reasoning steps on GSM8K.It evaluates supervised fine-tuning, reinforcement learning, expert iteration, reward modeling, reranking, and human-provided process annotations.
  • Key findings: The paper’s best approach reduces final-answer error from 16.8% to 12.7% and trace error from 14.0% to 3.4%.Final-answer error falls to 2.7% when the model can abstain on 30% of questions.
  • Key findings: 23.5% vs. 22.3% without reward models and 16.6% vs. 14.8% with reward models show similar final-answer error rates for outcome- and process-supervised models.The comparison indicates that final-answer correctness supervision can achieve nearly the same final-answer error rate as imitation of human solutions.
  • Key findings: Outcome-supervised reward models agree more closely with process-based labels than with outcome-based labels themselves.The paper cautions that this effect may be dataset-specific but links it to reward models’ effectiveness for improving trace error.
  • Key findings: 12.4% versus 3.8% trace error shows that direct reinforcement learning on final-answer correctness does not achieve the low trace error of the best process-based method.Using a reward model instead of final-answer correctness reduces trace error to 5.5%.

2. Problem and methods

The paper evaluates outcome- and process-based supervision for generating reasoning traces on GSM8K math word problems, using final-answer and trace error rates. It compares supervised finetuning, reward models, decoding strategies, and expert iteration.

  • Dataset: GSM8K contains grade-school math word problems with natural-language reasoning traces; the experiments use 7118 training and 1319 test examples after creating a 256-example validation split.The dataset choice reflects its benchmark status and inclusion of natural-language reasoning traces.
  • Evaluation metrics: Final-answer error rate measures incorrect final answers, while trace error rate measures incorrect reasoning steps among problems with correct final answers.Final answers are evaluated by exact string matching, while trace correctness is estimated from human annotations.
  • Task formulation: The task maps a problem statement to a newline-separated reasoning trace whose last step provides the integer final answer.The model is trained as a sequence-to-sequence system that generates the answer text.
  • Supervised finetuning: Supervised finetuning is process-based when the complete GSM8K reasoning trace is the target, rather than only the final answer.The paper contrasts trace-token targets with outcome-based final-answer targets.
  • Reward models: Outcome-supervised reward models label each step by whether the completed sample reaches the reference answer, whereas process-supervised reward models label whether the steps so far are correct.Process labels require human annotations because reliable programmatic checks for intermediate-step correctness are unavailable.
  • Decoding: At test time, the system generates K=96 full solutions and selects among them by majority voting without a reward model or RM-weighted decoding otherwise.RM-weighted decoding first aggregates estimated correctness probabilities by final answer, then selects the highest-scoring sample for that answer.
  • RL via expert iteration: Expert iteration alternates policy improvement, which combines a policy with search to produce expert samples, and distillation, which trains the policy on those samples.The paper evaluates Final-answer RL, ORM-RL, and PRM-RL policy-improvement procedures.

3. Results

The results show that outcome-based supervision can achieve competitive final-answer accuracy with less label supervision, but low trace error requires process-based feedback or a reward model that emulates it. Reward models improve both metrics, and selective prediction further reduces error when models can abstain.

  • Outcome-based and process-based approaches achieve similar final-answer error rates, both without reward models and with reward models.The reported comparisons are 23.5% versus 22.3% without reward models and 16.6% versus 14.8% with reward models.
  • 1–4 tokens of label supervision per question can yield competitive performance when final-answer correctness is sufficient, compared with hundreds for supervised fine-tuning.Few-shot plus Final-Answer RL requires only a final answer from demonstrators, whereas SFT uses full reasoning traces.
  • ORM predictions agree more with process-based labels than outcome-based labels, reaching 85% versus 77% agreement across all steps.RM reranking also reduces SFT trace error from 11.4% to below 5%, suggesting reward models evaluate reasoning steps rather than only final answers.

4. Discussion

The discussion contrasts process- and outcome-based feedback across accuracy, human understanding, and tampering incentives, emphasizing that the appropriate choice depends on the evaluation goal and domain.

  • Final-answer vs. trace error: Outcome-based feedback is label-efficient when low final-answer error is sufficient, whereas low trace error benefits from process-based feedback or an approximation of it.The appropriate metric depends on whether final answers are easy to validate and whether reasoning steps matter directly.
  • Human understanding: Process-based approaches require greater human understanding because annotators must evaluate or demonstrate individual reasoning steps.This requirement can make process-based supervision more demanding than outcome-based evaluation using metrics such as power consumption or chip area.
  • Human understanding: Process-based feedback may facilitate human understanding by selecting reasoning steps that people can follow, while outcome optimization may find hard-to-understand strategies.In GSM8K, adding Final-Answer RL decreased final-answer error but increased trace error, though not significantly.
  • Tampering incentives: Process-based feedback evaluates individual actions rather than overall satisfaction, so changes in future user preferences do not affect rewards for the corresponding actions.The paper presents this as a way to reduce incentives for optimizing such preference changes, though it does not directly prevent them.
  • Scope: The outcome-versus-process distinction applies beyond GSM8K to supervising any sequence of actions either by resulting outcomes or by individual actions.GSM8K uses reasoning steps as its actions, but the supervision distinction is presented more generally.
  • Generalizability: The results may generalize less strongly beyond mathematics because incorrect traces are usually harmful to correct answers in math, unlike domains where undesirable behaviors can improve rated outcomes.The authors therefore expect outcome optimization to induce correct processes more strongly for math than for domains involving behaviors such as manipulation.

5. Related work

Related work spans prompting, fine-tuning, reward models, process-based reasoning, and algorithmic tasks; this paper distinguishes itself by directly comparing process- and outcome-based feedback on a natural-language task.

  • Solving math problems with LMs: The paper focuses on fine-tuning because it studies feedback procedures and finds that fine-tuning significantly outperforms prompting alone for its base language model.It uses the original GSM8K reward-model approach and compares it with step-aware and outcome-based methods.
  • Multistep reasoning in LMs: Unlike work focused exclusively on one supervision type, this study directly compares outcome-based and process-based techniques and analyzes trace error rates.Prior process-based work includes recursive book summarization and synthetic reasoning settings, while other work applies outcome-based methods to multistep questions.
  • Multistep reasoning in LMs: WebGPT provides a prior comparison in which browser interactions receive supervised or outcome-based reinforcement-learning feedback, but this paper explores both approaches more comprehensively.Both studies observe gains from outcome-based reward-model reranking, with limited additional effect from full-solution reinforcement learning.
  • Comparisons between process- and outcome-based approaches to algorithmic tasks: Earlier head-to-head comparisons of process- and outcome-based approaches primarily used algorithmic tasks such as sorting lists of numbers.The paper positions its contribution as the first comprehensive comparison on a natural-language task.
  • Solving math problems with LMs: GSM8K was selected because its natural-language reasoning traces enable detailed comparison of process- and outcome-based approaches without collecting new traces.Other trace-containing datasets include EntailmentBank, StrategyQA, Proof Writer, and CLUTTR.

6. Conclusion

The conclusion reports similar final-answer performance for process- and outcome-based supervision but finds that low trace error requires process-based feedback or a reward model that emulates it.

  • Conclusion: 13.8% final-answer error with outcome-based supervision and 12.9% with process-based supervision improve the GSM8K state of the art, while the two approaches remain similar overall.The comparison directly evaluates both supervision types on a natural-language task.
  • Conclusion: 3.4% trace error from process-based SFT with PRM reranking improves the previous 14.0% state of the art, whereas its outcome-based analogue reaches 12.7%.These results indicate that low trace error depends on process-based supervision or an approximation of it.
  • Conclusion: Outcome-labeled reward models agree more closely with process-based labels than with outcome-based labels themselves, and using them during RL reduces trace error from 12.7% to 5.5%.The authors note that some conclusions may be specific to math word problems and require testing in other domains.

A. Example GSM8K problems and solutions

The examples present randomly sampled model solutions and incorrect traces, illustrating both correct-answer solutions and cases where a correct final answer coexists with reasoning errors.

  • Random problems and model solutions: Table 4 contains 10 randomly sampled problems and model solutions from the SFT+ORM-RL model with ORM reranking.The model has 12.7% final-answer error overall.
  • Incorrect problems and solutions: Table 5 contains five randomly selected solutions with correct final answers but incorrect reasoning traces.The first step rated incorrect is marked, and one example was marked incorrect because of a blank step before the final answer.
  • Random problems and model solutions: The examples include a final-answer-incorrect bird-distance solution that adds distances traveled in opposing directions as if they were all cumulative.The model reports 446 miles after calculating 300, 36, and 110 miles for the three legs.
  • Random problems and model solutions: A housekeeping example shows a correct final answer of 20 after computing expenses for eight clients and subtracting $72 from $92 income.The solution calculates $32 for bleach and $40 for cloths before deriving the $20 profit.

B. Data annotation details

The study used qualified human annotators and quality filtering to label reasoning traces, achieving high training-set agreement but lower agreement on evaluation labels.

  • Quality assurance: 21 of 91 candidates passed the qualification requirement for mathematical-solution annotation.Participants had to correctly annotate at least 3 of 4 qualification solutions.
  • Quality assurance: Duplicate annotations covered 20% of training problems, and low-agreement annotators were removed.Four of 21 annotators were removed, eliminating 21% of originally labelled training data and leaving 530 annotated problems.
  • Agreement: 92% inter-rater agreement and Cohen’s κ=.915 were achieved for identifying the first incorrect step in training data.The estimate is slightly upward-biased because the same duplicated problems were used for filtering raters.
  • Agreement: Evaluation labels reached 87% agreement but only Cohen’s κ=.34 for binary full-trace correctness.Intermediate errors are harder to judge when the final answer is correct.

C. Additional RM analysis

Last-step agreement between reward models and labels is similar to all-step agreement, with the outcome reward model aligning more closely with process labels despite its training target.

  • Reward-model agreement: The outcome reward model has higher last-step agreement with process labels than with outcome labels.On the last step, the two label types coincide except when the final answer is correct but the trace is incorrect.

D. OOD evaluation details

The paper evaluates out-of-distribution generalization on the pre-algebra split of MATH after restricting and transforming the dataset for compatibility with the GSM8K-oriented setup.

  • Dataset preparation: The evaluation retained 633 of 871 MATH pre-algebra problems after removing problems with diagrams.The study also removed some LaTeX formatting before evaluation.
  • Results: Final-answer error was fairly similar across approaches on MATH pre-algebra.Table 7 reports majority-voting results unless reward-model reranking is specified.
  • Dataset preparation: A regex-based transformation converted common LaTeX expressions into plain text rather than covering every formatting edge case.The transformation targeted commonly used commands to emphasize cross-problem rather than cross-format generalization.
  • Evaluation procedure: The final answer was extracted from the argument of each solution’s boxed expression.MATH solutions place the final answer inside the boxed command.

E. Negative and preliminary results

Several additional experiments exposed training costs, reranking failures, and limitations of preliminary observations and attempted annotation procedures.

  • Scope of evidence: The appendix’s experimental observations are less carefully checked and may depend on details of the training setup.They are presented as potentially useful observations for future researchers rather than as equally established main-paper results.
  • Negative and preliminary results: Training the outcome reward model required roughly 20 times more steps than supervised fine-tuning.The authors associate this with sparser token supervision, dropout regularization, and multiple generated samples per problem.
  • Negative and preliminary results: Step-level reranking increased final-answer error by about 1% with the process reward model and 3% with the outcome reward model.Manual inspection linked these failures largely to insufficient policy entropy.
  • Negative and preliminary results: Retraining the outcome reward model between expert iterations did not help in preliminary SFT+ORM-RL experiments.Few-shot expert-iteration implementations retrained it every iteration partly to avoid imitating early trace errors.
  • Annotation limitations: Corrected-step annotations were difficult to specify precisely and lacked an easy agreement measure.Annotators sometimes reordered calculations or overfit corrections to a reference solution despite alternative valid approaches.
Loading 2211.14275v1…