PaperlayerView exact revision on arXiv

Source-linked AI summary

GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinements

Alex Havrilla, Sharath Raparthy, Christoforus Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, Roberta Raileanu

arXiv:2402.10963v2cs.CLcs.LG

TL;DR

LLMs can struggle to identify when and where to refine reasoning without external feedback. GLoRe trains synthetic-data SORMs and global and local refiners, combining them with ORM reranking; the combined strategy raises greedy GSM8K accuracy for RL fine-tuned LLaMA-2 13B from 53% to 65%.

  • Problem

    LLMs can struggle to identify when and where to refine reasoning without external feedback.

    Source: §1 · p. 1

  • Method

    The method uses an ORM to decide when to refine, a synthetic-data SORM to identify where, and global and local refiners to produce corrections.

    Source: §1 · p. 2

  • Results

    53% to 65%: greedy-sampled GSM8K accuracy for an RL fine-tuned LLaMA-2 13B model using the combined strategy.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Global and local refinements are complementary, solving largely disjoint sets of problems, and are combined using the ORM as a reranker.

    Source: §1 · p. 2

    An ORM’s intermediate-step judgments depend on its data-generating policy and can underestimate solvability when that policy often fails.

    Source: §4 · p. 4

Abstract

from arXiv · show

State-of-the-art language models can exhibit impressive reasoning refinement capabilities on math, science or coding tasks. However, recent work demonstrates that even the best models struggle to identify \textit{when and where to refine} without access to external feedback. Outcome-based Reward Models (\textbf{ORMs}), trained to predict correctness of the final answer indicating when to refine, offer one convenient solution for deciding when to refine. Process Based Reward Models (\textbf{PRMs}), trained to predict correctness of intermediate steps, can then be used to indicate where to refine. But they are expensive to train, requiring extensive human annotations. In this paper, we propose Stepwise ORMs (\textbf{SORMs}) which are trained, only on synthetic data, to approximate the expected future reward of the optimal policy or $V^{\star}$. More specifically, SORMs are trained to predict the correctness of the final answer when sampling the current policy many times (rather than only once as in the case of ORMs). Our experiments show that SORMs can more accurately detect incorrect reasoning steps compared to ORMs, thus improving downstream accuracy when doing refinements. We then train \textit{global} refinement models, which take only the question and a draft solution as input and predict a corrected solution, and \textit{local} refinement models which also take as input a critique indicating the location of the first reasoning error. We generate training data for both models synthetically by reusing data used to train the SORM. We find combining global and local refinements, using the ORM as a reranker, significantly outperforms either one individually, as well as a best of three sample baseline. With this strategy we can improve the accuracy of a LLaMA-2 13B model (already fine-tuned with RL) on GSM8K from 53\% to 65\% when greedily sampled.

1 Introduction

The paper decomposes self-refinement into deciding when, where, and how to refine, using reward models and global and local refiners. Its combined strategy improves greedy-sampled LLaMA-2 13B accuracy on GSM8K from 53% to 65%.

1 Introduction

  • The study targets self-refinement on reasoning tasks without external feedback beyond ground-truth answers for training problems.The paper frames refinement as deciding when, where, and how to revise a solution.Source: §1 · pp. 1–2
  • SORMs use synthetic labels from repeated sampling of the student policy at an intermediate step to estimate its chance of reaching the correct final answer.The paper interprets this as approximating a lower bound of the optimal value function; SORMs improve intermediate-step feedback over ORMs.Source: §1 · p. 2
  • ORMs can underestimate whether an intermediate solution is solvable, producing high false-negative rates when classifying erroneous steps.PRMs directly predict step correctness but require extensive human annotations.Source: §1 · pp. 1–2
  • 53% to 65%: the combined refinement strategy improves greedy-sampled accuracy on GSM8K for an RL fine-tuned LLaMA-2 13B model.The ORM reranks the initial draft and global and local refinements.Source: §1 · p. 2
  • The method assigns when-refinement decisions to an ORM, where-refinement decisions to a SORM, and how-refinement decisions to global and local refiners.The authors report that global and local refinements solve largely different problems, making them complementary.Source: §1 · p. 2

2 Background

The background establishes notation for reasoning tasks, reward models, and refinement. It distinguishes outcome-level evaluation from step-level evaluation and defines global and local refinement inputs.

2 Background

  • A reasoning task is a distribution of natural-language question/answer pairs, with an answer represented as a sequence of intermediate steps ending in the final answer.In this paper, a step boundary corresponds to a newline token.Source: §2 · p. 2
  • A reward model approximates reward for an action in a reinforcement-learning environment; in language settings, it estimates reward for an LLM response.Language-model rewards are generally sparse and assigned at the end of generation, including in RLHF.Source: §2 · pp. 2–3
  • An ORM estimates final-answer correctness and can evaluate prefixes by the likelihood that continuations sampled from its student policy lead to a correct answer; a PRM predicts correctness of each step.The ORM is trained from repeated samples of a student model, while PRMs directly supervise step correctness.Source: §2 · p. 3
  • A global refiner conditions on the question and draft, whereas a local refiner also receives an error-location signal.They predict a new solution conditioned on their respective inputs.Source: §2 · p. 3
  • The paper uses Dtrain and Dtest for task-specific training and test splits, and calls the downstream fine-tuned pretrained LLM the base model.The student model is produced by fine-tuning the base model on supervised data or with reinforcement learning.Source: §2 · p. 3

3 Related Works

Prior work studies prompted self-refinement, synthetic and human-sourced training data, and process-based verification. This paper compares ORM and SORM step verification and uses synthetic stepwise labels to train refiners.

3 Related Works

4 Method

The method separates refinement into deciding when to refine, locating the first error, and correcting the draft. It builds reward-model and refinement training data synthetically from a fine-tuned student policy.

4 Method

  • ORM intermediate scores estimate success under the data-generating policy and can mistake likely student failure for an error in the current prefix.Source: §4 · p. 4

Step 2: Training the ORM/SORM

  • SORM approximates V* by labeling a prefix positive when any of eight student continuations reaches a correct answer, unlike ORM's rollout-outcome labels.Labels are post-processed by propagating positives backward, checking verification consistency, and balancing labels by prefix length; rejection sampling may under-approximate V*.Source: §4 · p. 6, Step 2: Training the ORM/SORM · pp. 6–7

4.1 Training pipeline

  • Training data for reward models and both refiners is generated synthetically because the pipeline assumes no human or better-model fine-tuning data.Source: §4.1 · p. 6

Step 1: Fine-tuning a student model

  • Expert Iteration samples 96 rollouts per question, filters incorrect answers, deduplicates correct ones, and repeats fine-tuning until maj@1 converges.Each iteration's fine-tuning data combines newly generated rollouts with prior training data.Source: Step 1: Fine-tuning a student model · p. 6

Step 3: Training refinement models

  • Local-refinement examples mark the first error and pair the incorrect trace with a correct verification from the preceding step, while global examples pair incorrect and correct rollouts for one question.The SORM locates the first step below a correctness threshold to create the local error annotation.Source: Step 3: Training refinement models · pp. 7–8

4.2 Evaluation

  • Evaluation samples global and local refinements alongside the draft, then uses the ORM to rerank them because the two refiners solve complementary subsets of problems.Test drafts are greedily sampled from the student; local refiners receive error locations from an ORM or SORM. The main body evaluates process-based local refinement, while value-based refinement is explored in the appendix.Source: §4.2 · p. 8

5 Results

SORM improves intermediate-step evaluation on the harder GSM8K benchmark, while ORM is more accurate on final-answer prediction. Combining global and local refinements with ORM reranking improves on the best-of-three baseline.

5.1 Evaluting the ORM and SORM

  • SORM raises GSM8K intermediate-step accuracy from 73% to 81%, an improvement of up to 8% over ORM.SORM and ORM have comparable step-level accuracy on SVAMP.Source: §5.1 · p. 8, Table 1 · p. 9
  • ORM is more accurate than SORM at predicting final-answer correctness, with the 13B SORM lagging by 5% on GSM8K.This pattern holds on both GSM8K and SVAMP.Source: Table 2 · p. 9, §5.1 · p. 9

5.2 Evaluating global and local refinements

  • Across all drafts, refinements improve overall accuracy by at most 1%, but refining only incorrect drafts yields average gains of 10% on GSM8K and 8% on SVAMP.All refinement models struggle to identify correct drafts that do not need refinement.Source: §5.2 · p. 10, Figure 5 · p. 11
  • Combining global and local refinements fixes 41% of incorrect GSM8K drafts from the 13B student, compared with 28% for either alone.The two refinement types solve partially disjoint sets of problems.Source: Table 3 · p. 11, §5.2 · p. 11
  • Reranking drafts and refinements improves accuracy by 8% on average over draft accuracy; on GSM8K, gains over Bo3 reach 9% with ORM and 13% with a perfect reranker.The reported ORM advantage over Bo3 is around 8% on GSM8K.Source: §5.2 · pp. 11–12, Figure 6 · p. 12

6 Conclusion and Future Work

The conclusion reports that ORM/SORM-guided global and local refinements solve largely unique sets of problems and perform best together with ORM reranking. It also identifies unresolved questions about critique quality and tool-assisted verification, while the appendices describe model preparation and refinement training.

6 Conclusion and Future Work

  • Global and local refinements solve largely unique sets of problems, and combining them through ORM reranking gives the best performance.The conclusion also reports that SORM generalizes better than ORM on intermediate test steps, at the cost of final-answer accuracy.Source: §6 · p. 12
  • Future work targets more reliable, more informative local critiques and broader information for local refiners.The authors also note that GPT-4's use of tools leaves unclear how much it uses them to understand why a step is wrong.Source: §6 · p. 12
  • Iterative refinement is proposed as in-context exploration to reduce the model rollouts needed to determine how to refine.The authors suggest that local self-refinement strategies might enable behaviors inaccessible through repeated iid sampling.Source: §6 · p. 12

A Hyperparamters

  • The appendix lists training-job hyperparameters in Table 4.The supplied passage does not report the values.Source: A Hyperparamters · p. 18

B RL for Reasoning

  • The appendix describes RL fine-tuning the base model with Expert Iteration and points to Table 5 for its accuracy scores.The cited passage names maj@1, maj@96, Rerank@96, and pass@96, without giving their values.Source: B RL for Reasoning · p. 18

C RL for (global) refinement

  • For global refinement, Expert Iteration is trained on incorrect/correct solution pairs, while PPO receives a sparse reward based on correctness changes.The EI setup samples 96 SFT2 solutions per prompt; PPO is initialized from SFT2 and tested greedily on GSM8K drafts.Source: C RL for (global) refinement · p. 18

C.0.1 Results for global refinement

Global refinement models make only small gains because they struggle to decide when to refine, and PPO can collapse to copying the draft. The paper proposes assigning when, where, and how decisions to separate reward models and refinement models.

C.0.1 Results for global refinement

  • EI improves over the SFT baseline by only 3%, while PPO shows no improvement, as both models struggle to decide when to refine.EI often incorrectly refines correct drafts.Source: §C.0.1 · p. 19
  • PPO collapses to returning the draft final answer, and alternative reward setups do not overcome the copy bias that limits exploration.The prompted baseline also exhibits copying behavior, accounting for most of its nontrivial accuracy.Source: §C.0.1 · p. 19
  • The proposed decomposition assigns when-to-refine decisions to the ORM, where-to-refine decisions to the SORM for local refinement, and how-to-refine to the refinement model.The evaluation found that refinement models alone struggled to identify when to refine, motivating the ORM's role.Source: §C.0.1 · p. 19

D Misc. Objectives for Reranking

The reranking experiments favor final-answer ORM estimates over alternative SORM step-score weightings. Additional tests report ORM cross-generalization problems and stronger reranking by a classifier ORM than a contrastive ORM.

D Misc. Objectives for Reranking

E ORM and SORM extra-model generalization

  • Both ORMs underperform when evaluated on test data generated by the opposite student model, and a 7B ORM labels 42% of correct GSM8K training solutions incorrect.Table 7 is identified as the cross-generalization evaluation.Source: E ORM and SORM extra-model… · p. 20, Table 7 · p. 20

F Contrastive vs. Classifier RMs

  • Classifier ORM reranking accuracy is 0.56, versus 0.47 for the contrastive ORM.The experiment concludes that the classifier is the better reranker in this comparison.Source: F Contrastive vs. Classifier RMs · p. 20

G Accuracy of the SORM data generating method

The SORM data-generation process can produce both false-positive and false-negative step labels. In a hand-labeled GSM8K check, generated labels agreed with ground truth 94% of the time.

G Accuracy of the SORM data generating method

H Mixing other sources of PRM data

The section tests adding MATH PRM data and reports strong performance on its PRM test set but weaker transfer to final-answer correctness.

H Mixing other sources of PRM data

I Self-supervised learning with the SORM

The section filters noisy SORM training data using SORM-generated labels, then retrains models on the filtered datasets.

I Self-supervised learning with the SORM

J Value refinement

Value-based local refinement inserts feedback at the model’s highest-value step, but fixes fewer incorrect drafts than SORM refinement. On the reported GSM8K evaluation, global refinement also outperforms value-based refinement.

J Value refinement

  • Unlike process-based feedback, the value-based signal is student-specific, and the following step need not itself contain an error.Source: J Value refinement · p. 21
  • Value-based feedback places a [BAD] token before the step after the highest-value step, using the ORM’s estimate of the student policy’s chance of solving from each prefix.The highest-value step is selected using verifying rollouts, and a differing rollout supplies the refinement.Source: J Value refinement · p. 21
  • The downstream models and datasets are generated using Llama-2 7B EI on GSM8K as the student policy.Source: J Value refinement · p. 22
  • 14% of incorrect drafts are fixed by value-based refinement, compared with 21% by SORM refinement; global refinement outperforms value-based refinement by 6%.Evaluation uses incorrect greedily sampled drafts from the GSM8K test set.Source: J Value refinement · p. 22

Related papers

Loading 2402.10963v2…