PaperlayerView exact revision on arXiv

Source-linked AI summary

Recursive Introspection: Teaching Language Model Agents How to Self-Improve

Yuxiao Qu, Tianjun Zhang, Naman Garg, Aviral Kumar

arXiv:2407.18219v2cs.LGcs.AIcs.CL

TL;DR

LLMs often fail to improve answers sequentially even when told about their mistakes, limiting their ability to tackle challenging reasoning prompts. RISE reframes single-turn fine-tuning as a multi-turn MDP and iteratively trains on on-policy attempts with reward-based supervision. On GSM8K and MATH, RISE enables Llama2, Llama3, and Mistral models to improve over turns and outperform several single-turn strategies, while the paper notes computational and task-scope limitations.

  • Problem

    Prompting and existing models often fail to make LLMs correct their own mistakes sequentially on complex reasoning tasks.

  • Method

    RISE reframes a single-turn problem as a multi-turn MDP and iteratively fine-tunes on on-policy rollouts with expert or self-generated supervision.

  • Results

    RISE produces monotonically increasing performance across turns and improves mathematical reasoning on GSM8K and MATH across Llama2, Llama3, and Mistral models.

  • Takeaways & Limitations

    RISE can instill self-improvement in a single model without requiring multiple models, while preserving one-turn abilities according to the paper’s analysis.

  • Takeaways & Limitations

    RISE was limited to two training iterations, or one with learner-generated supervision, and fine-tunes models on one task at a time.

Abstract

from arXiv · show

A central piece in enabling intelligent agentic behavior in foundation models is to make them capable of introspecting upon their behavior, reasoning, and correcting their mistakes as more computation or interaction is available. Even the strongest proprietary large language models (LLMs) do not quite exhibit the ability of continually improving their responses sequentially, even in scenarios where they are explicitly told that they are making a mistake. In this paper, we develop RISE: Recursive IntroSpEction, an approach for fine-tuning LLMs to introduce this capability, despite prior work hypothesizing that this capability may not be possible to attain. Our approach prescribes an iterative fine-tuning procedure, which attempts to teach the model how to alter its response after having executed previously unsuccessful attempts to solve a hard test-time problem, with optionally additional environment feedback. RISE poses fine-tuning for a single-turn prompt as solving a multi-turn Markov decision process (MDP), where the initial state is the prompt. Inspired by principles in online imitation learning and reinforcement learning, we propose strategies for multi-turn data collection and training so as to imbue an LLM with the capability to recursively detect and correct its previous mistakes in subsequent iterations. Our experiments show that RISE enables Llama2, Llama3, and Mistral models to improve themselves with more turns on math reasoning tasks, outperforming several single-turn strategies given an equal amount of inference-time computation. We also find that RISE scales well, often attaining larger benefits with more capable models. Our analysis shows that RISE makes meaningful improvements to responses to arrive at the correct solution for challenging prompts, without disrupting one-turn abilities as a result of expressing more complex distributions.

1. Introduction

RISE addresses the difficulty of making LLMs improve their responses across sequential attempts, where prompting and ordinary fine-tuning often fail on complex reasoning tasks. It trains models recursively and reports improving mathematical performance over turns.

  • LLM agents need interactive, goal-directed behavior that seeks task information and verifies or improves decisions at inference time.
  • Prompting with feedback can elicit better responses but often fails to teach models to correct mistakes on complex reasoning tasks.
  • RISE iteratively trains models on on-policy rollouts, using best-of-N revision candidates and reward-weighted regression to teach self-improvement.
  • RISE improves LLaMA3-8B by 8.2% and Mistral-7B by 6.6% using their own data, while five-turn gains reach 17.7% for LLaMA2-7B and 23.9% for Mistral-7B on GSM8K.
  • RISE also improves five-turn performance on MATH by 4.6% for LLaMA2-7B and 11.1% for Mistral-7B, with gains generalizing to out-of-distribution prompts.

2. Related Work

Prior work improves reasoning through prompting, critique, search, verifiers, and fine-tuning, but RISE focuses on teaching one model to improve its own answers over multiple turns. Its design combines iterative on-policy data generation with learning from both successful and unsuccessful rollouts.

  • Prior approaches use tools, reflection, verbalized thoughts, self-critique, or other models’ critiques to improve reasoning and interaction.
  • Fine-tuning methods train on self-generated responses using verifiers, search, contrastive negative data, or iterated supervised and reinforcement learning.
  • Unlike GLoRE and Self-Correct, RISE trains a single model to answer and improve responses over more than two turns.
  • RISE requires iterative on-policy data generation and an objective that learns from both successful and unsuccessful rollouts.
  • On-policy distillation queries experts on partial responses, whereas RISE uses supervision on sequential attempts.

3. Problem Setup and Preliminaries

The paper formulates sequential response improvement as optimizing an LLM conditioned on the problem, prior attempts, and auxiliary instructions. This differs from standard single-response fine-tuning by explicitly training reactions to response history.

  • The model conditions on the problem, previous attempts, and auxiliary instructions such as correction prompts or environment feedback.
  • Unlike standard supervised fine-tuning, the objective trains the policy to react appropriately to its own previous response history.

4. RISE: Recursive Introspection for Self-Improvement

RISE converts single-turn problems into multi-turn MDPs and trains models to improve responses across sequential attempts. It collects on-policy trajectories with improved responses, then learns from them using weighted supervised training, with oracle and no-oracle inference modes.

  • 4.1. Converting Single-Turn Problems into a Multi-Turn MDP (MDP): RISE converts each single-turn prompt into a multi-turn MDP whose state includes prior attempts and optional environment feedback.The model's response is the action, and correctness supplies a sparse binary reward.
  • 4.2. Learning in the Multi-Turn MDP: Data collection unrolls the current model for sequential attempts, records feedback and rewards, and requires trajectories to show likely mistakes followed by non-degrading improvements.The collection strategy is designed to keep responses relevant to the prompt and prior attempts.
  • 4.2. Learning in the Multi-Turn MDP: Improved responses come either from a stronger teacher model or from the learner's own multiple sampled candidates selected by reward.The former is the distillation variant; the latter is self-distillation and does not require a teacher model.
  • 4.2. Learning in the Multi-Turn MDP: RISE trains on the collected data with weighted supervised regression, using exponentiated rewards and a reward-centering modification to reduce bias toward already-easy problems.The temperature parameter controls the contrast between higher- and lower-reward actions.
  • 4.3. Inference at Deployment Time: At inference, the model either checks each revision with an oracle and stops when correct, or revises without an oracle and uses majority voting across turns.When turns exceed the training iteration count, both modes retain only the most recent k interactions to avoid distribution shift.

5. When and Why is Self-Improvement Over Turns Possible?

RISE’s sequential self-improvement is supported by the idea that conditioning on prior responses can represent more flexible target distributions than one-shot prediction. Evidence indicates this helps models reduce loss and solve problems missed by larger first-turn sampling budgets, while knowledge-based question answering remains a boundary.

  • Why self-improvement is possible: RISE’s sequential conditionals can represent a more flexible marginal distribution than directly predicting the final response from the prompt.The hypothesis attributes this flexibility to conditioning on intermediate responses before marginalization.
  • Empirical evidence: RISE-trained models assign higher probability to the true answer than the comparison model.Figure 4 compares the probability of the true answer given the prompt.
  • Empirical evidence: RISE reduces training loss more than classic fine-tuning that fits only the oracle answer, attaining lower perplexity.The comparison is reported for average negative log-likelihood over training samples.
  • Scope boundary: RISE is not expected to produce meaningful improvements on knowledge-based questions when critical features for the correct mapping are absent.The authors instead associate the method’s applicability with reasoning problems where sampling can expose correct responses.
  • Empirical evidence: RISE solves a significant fraction of problems left unsolved by much larger first-turn pass@B budgets.The sequential procedure therefore differs from simply converting larger single-turn sampling budgets into first-turn performance.

6. Experimental Evaluation

RISE improves sequential self-improvement across math reasoning evaluations, including when trained with self-generated data, while preserving first-turn performance. Its gains increase with iterative training, transfer across datasets, and depend on multi-turn training and reward-weighted data collection.

  • Main results: RISE achieves the largest performance improvements over competing approaches on GSM8K and MATH, with gains increasing across training iterations.On Llama2, five-turn performance improves by 15.1% and 17.7% on GSM8K and 3.4% and 4.6% on MATH after two iterations.
  • Sequential versus parallel strategies: Sequential RISE outperforms parallel sampling and single-turn SFT under equal sampling budgets, indicating that its algorithmic protocol is important.The five-turn procedure improves GSM8K by 4% to 8% and MATH by 6.5% over first-turn majority voting, while single-turn SFT performs substantially worse.
  • Main results: RISE preserves first-turn performance while improving five-turn performance, including when multiple training iterations are applied.This addresses the concern that learning sequential improvement might reduce one-turn ability.
  • Self-distillation: Self-distillation enables RISE to improve five-turn performance using entirely self-generated data and supervision, though one iteration remains insufficient to surpass first-turn best-of-N performance.On GSM8K, Mistral-7B and Llama-3-8B improve by more than 1% from turn 1 to turn 5 without an oracle.
  • Generalization: RISE generalizes sequential improvement across MATH, GSM8K, and SVAMP when trained on one dataset and evaluated on another.The cross-dataset model generally improves the base model over five sequential turns, whereas the base Llama2 model often degrades from turn 1 to turn 5.
  • Data and training strategy: Multi-turn on-policy training and reward-weighted use of correction data outperform expert-only, offline, or filtered-successful-data alternatives.Training on intermediate failed attempts improves performance, while offline imitation reduces loss without generalizing to new problems.

7. Discussion, Future Directions, and Limitations

RISE fine-tunes LLMs to improve their own responses across multiple turns, outperforming related refinement approaches while using a single model. The method achieves strong reasoning-task gains, but its training process remains computationally limited and manually iterative.

  • RISE fine-tunes LLMs to improve their own responses over multiple sequential turns.It uses an iterative RL recipe over on-policy rollouts with expert or self-generated supervision.
  • RISE significantly improves the self-improvement abilities of 7B models on GSM8K and MATH reasoning tasks.The paper reports improvement over turns that previous work did not observe in strong proprietary models.
  • RISE outperforms related refinement approaches while requiring only one end-to-end model.This simplifies the system relative to approaches that run multiple models.
  • Training was limited to two RISE iterations, including only one iteration when supervision came from the learner itself.The authors identify additional computation and iterations as open requirements for self-generated supervision.
  • RISE requires manual training iterations, motivating a more online variant for scaling on-policy learning efficiently.The authors also leave integration into general instruction-tuning and post-training pipelines as future work.

A.1. Inclusion of Correct-to-Correct Data

The paper tests whether RISE should include examples where correct responses remain correct across turns. Adding a novel correct response performs best, while repeating an existing correct response reduces performance.

  • RISE originally trains on conversions from incorrect responses to correct responses, without showing how to act on correct responses.The analysis investigates whether correct-to-correct examples can further improve self-improvement.
  • Repeating the same correct response at the end of a successful rollout largely reduces performance.The authors suspect spurious correlations from duplicated data contribute to the weaker result.
  • Appending a successful rollout with a novel correct response produces the highest performance gains on GSM8K.All tested mechanisms improve over the base model, but this design performs best.
  • Figure 10 measures the percentage of correct responses that become incorrect between consecutive turns.A decreasing trend indicates better self-improvement, while lower absolute values indicate greater resistance to noisy feedback.

A.2. Weak-to-Strong Generalization: RISE on Weak Model Data Improves Strong Models

RISE-generated rollouts from a weaker Llama2-7B model transfer effectively to training Mistral-7B. The reverse direction performs worse, consistent with the difficulty of learning from a stronger model’s mistakes.

  • RISE-generated data from Llama2-7B improves Mistral-7B more than on-policy Mistral-7B rollouts.Mistral-7B trained on Llama2-generated multi-turn rollouts also outperforms Llama2-7B trained on those same data.
  • The transfer benefit is especially significant for Mistral-7B’s multi-turn reasoning performance at m1@t5.The comparison covers all metrics reported in Table 5.
  • Training Llama2-7B on Mistral-7B’s mistakes performs worse than the reverse transfer direction.The authors attribute this likely to the difficulty of internalizing errors from a stronger model, producing hallucinations and memorization.
  • Training on weak-model RISE data does not degrade single-turn performance.The paper identifies this as a potential practical benefit when generating rollouts from stronger models is expensive.

B.1. Complete Comparisons and Discussion: Extended Version of Table 1

The extended comparisons show RISE attaining the largest one-turn-to-five-turn gains on GSM8K and MATH without an oracle, while Self-Refine degrades performance and GLoRE uses multiple models. RISE also achieves substantial sequential gains relative to comparable baselines.

  • Comparison with other approaches: RISE attains the biggest performance improvements from one-turn to five-turn performance without an oracle on both GSM8K and MATH.The performance gap becomes larger when oracle early termination is allowed.
  • Comparison with Self-Refine: Self-Refine degrades performance across models without oracle or human feedback, providing only a slight boost with oracle-assisted early termination.The baseline prompts the model to locate errors and refine its initial answer.
  • Comparison with Self-Refine: RISE provides consistent improvement without an oracle, unlike prompting-only Self-Refine.Its training enables iterative refinement and self-correction over multiple turns.
  • Comparison with GLoRE: GLoRE’s refinement models show little or no overall accuracy improvement without an oracle, while oracle-guided GLoRE improves the 7B GSM8K model by 10%.GLoRE uses separate models for drafting, error detection, and refinement.
  • Comparison with GLoRE: RISE improves Llama2-7B by 13.4% in three turns without oracle feedback and by 23.4% with an oracle on GSM8K.These sequential gains are reported despite GLoRE using three independent models.
  • Summary of comparisons: RISE offers end-to-end error correction with one model, fewer samples, and solutions unavailable through first-turn random sampling.Its five-turn performance surpasses GLoRE’s best absolute performance in oracle and non-oracle settings, despite a lower maj@1 comparison.
  • Comparison with other base models: RISE achieves comparable performance to Mistral-7B and Eurus-7B-SFT despite fine-tuning on a single domain.The method also generalizes to better base models.

B.2. Qualitative Examples from RISE

RISE qualitatively corrects earlier mathematical responses through both small edits and substantial rewrites, while sometimes localizing errors even when correction remains incomplete. Examples also illustrate tolerance of noisy feedback and successful correction across turns.

  • Small and large edits: 9 points corrects the ping-pong example after the first response incorrectly computes the second-period score.The second response changes the percentage calculation from 8 to 5 points and then totals 9.
  • Small and large edits: RISE can make either small edits or broad rewrites when correcting a previous mathematical answer.Broad rewrites occur when an early step is wrong; corrected and mistaken steps are highlighted separately.
  • Small and large edits: 25 flowers is correctly obtained by subtracting the 5 flowers that did not grow from the 30 planted over 15 days.The example presents the intermediate total and the final corrected count.
  • Noise tolerance: With uniformly negative verifier feedback, RISE still outputs the correct answer in 3 out of 4 turns from turn 2 through turn 5.One example shows a response that is mostly correct except for its final step.
  • Error localization: Some five-step attempts fail to finish correction but repeatedly preserve correct intermediate work, including reaching 12000 before the erroneous final step.The example demonstrates meaningful error localization despite remaining unsolved.

D.3. Datasets

The evaluation uses GSM8K and MATH training splits to generate RISE trajectories, holds out their test splits, and evaluates generalization on SVAMP. Prompts require step-by-step mathematical reasoning with a boxed final answer, alongside one-shot and self-critique examples.

  • Datasets: GSM8K contains 7,473 training problems and 1,319 testing problems, while MATH contains 7,500 training problems and 1,000 testing problems.Training portions generate trajectories during RISE iterations, whereas testing portions are held out for evaluation.
  • Datasets: SVAMP contributes 1,000 problems used solely for evaluation to test the method’s generalizability.Unlike GSM8K and MATH, SVAMP is not used for trajectory generation.
  • Trajectory generation: At iteration i, the current model rolls out for at most i steps, after which an expert samples five responses and the highest-reward response is selected.Ties are broken arbitrarily when multiple responses receive the same highest reward.
  • Prompting examples: The evaluation prompt asks the model to solve math problems step-by-step and present the final answer in boxed form.The supplied examples include one-shot problem solving and self-critique with error correction.
Loading 2407.18219v2…