PaperlayerView exact revision on arXiv

Source-linked AI summary

Recursive Introspection: Teaching Language Model Agents How to Self-Improve

Yuxiao Qu, Tianjun Zhang, Naman Garg, Aviral Kumar

arXiv:2407.18219v2cs.LGcs.AIcs.CL

TL;DR

LLMs often fail to improve answers by correcting prior mistakes on difficult reasoning problems. RISE trains models for sequential self-correction, and experiments report gains over turns on math reasoning tasks.

  • Problem

    Prompting and domain-specific fine-tuning have not taught LLMs a reliable test-time strategy for correcting mistakes on complex reasoning tasks.

    Source: §1 · p. 1

  • Method

    RISE casts iterative problem solving as a multi-turn MDP and fine-tunes on learner rollouts with improved next-turn responses using reward-weighted regression.

    Source: §1 · p. 2, §4.1 · p. 5

  • Results

    4% to 8% improvement in maj@5 on GSM8K and 6.5% on MATH (Mistral-7B) from turn 1 to turn 5 showed sequential improvement unavailable from equal-sample parallel sampling.

    Source: §6.1.1 · p. 11

  • Takeaways & Limitations

    The findings show that self-improvement over turns is achievable on reasoning tasks, while parallel sampling alone does not confer the same ability.

    Source: §6.1.1 · p. 11

    Computational constraints limited RISE to at most two training iterations, and to one iteration when using learner-generated supervision.

    Source: §7 · p. 16

Abstract

from arXiv · show

A central piece in enabling intelligent agentic behavior in foundation models is to make them capable of introspecting upon their behavior, reasoning, and correcting their mistakes as more computation or interaction is available. Even the strongest proprietary large language models (LLMs) do not quite exhibit the ability of continually improving their responses sequentially, even in scenarios where they are explicitly told that they are making a mistake. In this paper, we develop RISE: Recursive IntroSpEction, an approach for fine-tuning LLMs to introduce this capability, despite prior work hypothesizing that this capability may not be possible to attain. Our approach prescribes an iterative fine-tuning procedure, which attempts to teach the model how to alter its response after having executed previously unsuccessful attempts to solve a hard test-time problem, with optionally additional environment feedback. RISE poses fine-tuning for a single-turn prompt as solving a multi-turn Markov decision process (MDP), where the initial state is the prompt. Inspired by principles in online imitation learning and reinforcement learning, we propose strategies for multi-turn data collection and training so as to imbue an LLM with the capability to recursively detect and correct its previous mistakes in subsequent iterations. Our experiments show that RISE enables Llama2, Llama3, and Mistral models to improve themselves with more turns on math reasoning tasks, outperforming several single-turn strategies given an equal amount of inference-time computation. We also find that RISE scales well, often attaining larger benefits with more capable models. Our analysis shows that RISE makes meaningful improvements to responses to arrive at the correct solution for challenging prompts, without disrupting one-turn abilities as a result of expressing more complex distributions.

1. Introduction

RISE trains language models to improve their responses across successive attempts, addressing the failure of prompting or answer-focused fine-tuning to reliably teach test-time self-correction. Its iterative training uses supervision on the learner’s own responses, and the introduction reports improvements on GSM8K and MATH.

Introduction

  • Prompting with feedback and domain-specific question-answer fine-tuning do not reliably teach complex self-correction, while imitating another model’s revisions may not address the learner’s errors.The introduction also notes that proprietary models may produce low-quality multi-turn improvements, despite providing useful responses to the original problem.Source: §1 · pp. 1–2
  • RISE’s key insight is to supervise improvements to the learner’s own responses iteratively, aiming to teach a general procedure for tackling hard prompts rather than only what answer to give.The introduction draws inspiration from online imitation learning [36] and reinforcement learning (RL), and allows supervision from oracle responses sampled from more capable models or from the learner itself.Source: §1 · p. 2
  • RISE iteratively builds on-policy rollouts from the learner, selects better next-turn responses through best-of-N revisions, and fine-tunes with reward-weighted regression (RWR [34]).Revision candidates come from the learner or a more capable model, and selection uses a task success indicator; training learns from both high- and low-quality rollout parts.Source: §1 · p. 2
  • On GSM8K, RISE improves LLaMa3-8B by 8.2% and Mistral-7B by 6.6%, with five-turn gains of 17.7% for LLaMa2-7B and 23.9% for Mistral-7B.The reported five-turn improvements outperform parallel sampling from the first turn for LLaMa2-7B; performance increases monotonically after each turn.Source: §1 · pp. 2–3
  • On MATH, five-turn RISE improves LLaMa2-7B by 4.6% and Mistral-7B by 11.1%; GPT-3.5 improves by 4.6% over five turns on GSM8K.The introduction says RISE’s mathematical-reasoning results generalize to out-of-distribution prompts.Source: §1 · p. 3

2. Related Work

Prior work improves reasoning through prompting, self-correction, and fine-tuning, but intrinsic self-correction is often infeasible for off-the-shelf LLMs. RISE instead trains a single model to improve answers over sequential turns, using on-policy rollouts and supervision for successful and unsuccessful attempts.

2. Related Work

  • Prompting methods use tools, reflection, verbalized thoughts, or critique, but self-correction often depends on detailed error traces; intrinsic self-correction is often infeasible for off-the-shelf LLMs [5] [7] [14] [32] [54] [56] [15] [63] [33] [52] [65] [31] [40] [2] [12] [20] [21].The paper states that fine-tuning with RISE induces this capability.Source: §2 · p. 3
  • Prior fine-tuning combines model-generated responses with verifiers, search, contrastive prompting, or iterative supervised learning or RL; RISE instead targets improvement across sequential turns, not single-turn performance alone [39] [62] [30] [46] [57] [58] [60] [28] [47] [50] [13] [26] [33] [38] [9] [48] [8] [37] [59].Unlike generic multi-turn RL [41] [66], this setting has deterministic dynamics characterized by prompts and oracle answers and must generalize to novel test prompts; RISE’s self-improvement objective is orthogonal to whether training uses RL.Source: §2 · p. 3
  • Unlike GLoRE [17] and Self-Correct [53], which train separate models and study at most two turns, RISE trains one model to improve answers over more turns.RISE uses iterative on-policy data generation and an objective that learns from successful and unsuccessful rollouts; it resembles online imitation learning [36] [44], while on-policy distillation [1] [4] requests completions on partial responses rather than sequential attempts.Source: §2 · pp. 3–4

3. Problem Setup and Preliminaries

The setup frames self-improvement as learning a policy that uses prior attempts and auxiliary instructions to solve problems, rather than producing only a one-shot answer. It treats this objective as multi-turn learning and distinguishes parameter updates from prompt-only methods.

3. Problem Setup and Preliminaries

  • The goal is to improve performance across sequential attempts by learning a policy that uses the problem, previous responses, and auxiliary instructions to solve it correctly.Auxiliary instructions may ask the model to find and fix a mistake or provide environmental feedback, such as compiler feedback.Source: §3 · p. 4
  • Unlike single-response supervised fine-tuning, the objective trains the model to react to its prior attempts and is framed as an RL-like objective converted into a multi-turn MDP.Source: §3 · p. 4
  • Prompting methods such as Self-Refine [31] optimize the objective only by modulating instructions, and unchanged model parameters prevent full optimization.Source: §3 · p. 4

4. RISE: Recursive Introspection for Self-Improvement

RISE trains language models to revise earlier responses by turning single-turn problems into multi-turn interactions and learning from improved attempts. Its procedure combines rollout data collection, policy improvement, deployment-time inference modes, and practical training stages.

4.2. Learning in the Multi-Turn MDP

  • Rollouts pair the learner’s attempts with improved responses from a stronger teacher or the learner’s best sampled candidate, illustrating how to correct earlier mistakes.The collection strategy is designed to show likely learner errors and relevant improvements without including trajectories that degrade on later turns.Source: Figure 2 · p. 4, §4.2 · pp. 5–6

4.1. Converting Single-Turn Problems into a Multi-Turn Markov Decision Process (MDP)

  • RISE formulates each prompt as an MDP state, model responses as actions, and subsequent states as prior context plus an introspection prompt, with correctness determining reward.The state may also include environment feedback.Source: Figure 2 · p. 4, §4.1 · p. 5

4.2. Learning in the Multi-Turn MDP

  • Policy improvement uses reward-weighted supervised regression, centering exponentiated rewards around each prompt’s mean to avoid prioritizing already-easy problems.The paper describes this centering as an advantage-style modification that addresses a “rich-gets-richer” effect.Source: §4.2 · p. 6

4.3. Inference at Deployment Time

  • At deployment, RISE either checks each revised response with an oracle and stops when correct, or retries without an oracle and uses majority voting across turns.The paper reports that most evaluations use the no-oracle mode.Source: §4.3 · p. 6, Figure 3 · p. 7

4.4. Practical Algorithm and Implementation Details

  • For 7B models, an initial knowledge-boosting stage can teach response style and instruction following before on-policy data collection; iterative training uses data from all rounds starting from the base model.The authors found this approach more beneficial than continued fine-tuning from the previous iteration’s checkpoint.Source: §4.4 · p. 7

5. When and Why is Self-Improvement Over Turns Possible?

RISE can improve reasoning answers over multiple turns by using previous responses to model solutions that prompt-only prediction may not represent. The evidence also identifies a limit: iterative refinement cannot supply missing knowledge or features.

When and Why is Self-Improvement Over Turns Possible?

  • When the target distribution exceeds prompt-only model capacity, learning successive conditionals and marginalizing can yield a more flexible answer distribution; RISE uses the base model to induce iterative updates.The paper likens this to diffusion models fitting intermediate distributions for greater flexibility than VAEs [25], while RISE uses the base model rather than hand-designed noise schedules; the cited comparison supports the flexibility claim.Source: §5 · pp. 7–8
  • RISE has lower oracle-answer NLL than Classic across training checkpoints and reduces training loss more, attaining lower perplexity than standard fine-tuning.The checkpoint comparison uses 256 prompts and oracle responses with 95% confidence intervals; Figure 5 compares fitting oracle answers with fitting a sequence of answers.Source: Figure 4 · p. 8, §5 · p. 8, Figure 5 · p. 8
  • RISE is not expected to fix knowledge-based questions when critical features are absent and may instead incentivize hallucinations; reasoning differs because turn-one maj@K exceeds pass@1, indicating improvement through sampling [27].The paper argues that prior attempts can expose spurious correlations in knowledge-based tasks, whereas Figure 6 shows sequential RISE solving some problems that first-turn sampling misses.Source: §5 · p. 8, Figure 6 · p. 9
  • RISE’s sequential procedure solves a significant fraction of problems unsolved by much larger first-turn pass@B budgets, suggesting it indexes pre-trained knowledge rather than merely converting pass@K performance to pass@1.Figure 6 reports the fraction solved by sequential five-turn RISE sampling for B = 5 × k.Source: §5 · pp. 8–9, Figure 6 · p. 9

6. Experimental Evaluation

The evaluation tests whether RISE enables multi-turn self-improvement, improves with iterative training, generalizes to unseen prompts, and benefits from particular training-data compositions.

6.1. Does RISE improve performance over multiple turns compared to other approaches?

  • RISE raises Llama2 five-turn performance by 15.1% and 17.7% on GSM8K and 3.4% and 4.6% on MATH across two iterations, with larger gains than compared approaches.Prompting-only self-refinement often degrades performance, while single-turn SFT on RISE-generated data does not instill self-improvement, underscoring the importance of RISE’s training procedure.Source: Table 1 · p. 10, §6.1 · pp. 10–11

6.1.1. Can RISE Effectively Make Use of Mistakes and Correct Them?

  • With equal sample counts, comparing maj@5 at the end of 1 and 5 turns, RISE improves performance by 4–8% on GSM8K and 6.5% on MATH with Mistral-7B.Multiple RISE iterations preserve first-turn performance while improving five-turn performance. Mistral-7B + RISE outperforms Eurus-7B-SFT at five turns.Source: §6.1.1 · p. 11, §6.1.2 · p. 11

6.1.3. Self-Distillation Version of RISE

  • Self-distillation with self-generated data improves five-turn performance by more than 1% over turn one for Llama-3-8B and Mistral-7B on GSM8K.This one-iteration variant does not outperform the fine-tuned model’s m5@t1 performance; the authors suggest multiple iterations or online RL may be needed to close that gap.Source: §6.1.3 · pp. 11–12, Table 2 · p. 12

6.2. Does the Performance of RISE Improve with Iterative Training?

  • RISE’s five-turn performance improves across training iterations, both with and without oracle feedback.The authors suggest iterative self-training can be combined with RISE and point to full online reinforcement learning as a potential direction.Source: §6.2 · p. 12

6.3. Does RISE Also Improve Sequential Performance on Out-of-Distribution Prompts?

  • Training RISE on one math dataset enables positive five-turn performance improvement on prompts from other datasets, unlike the base Llama2 model’s turn-one-to-turn-five degradation.The reported cross-dataset evaluations include MATH, GSM8K, and SVAMP.Source: §6.3 · p. 12, Table 3 · p. 13

6.4. What Data Compositions and Data Quantity are Crucial for RISE?

  • Multi-turn interaction histories outperform one-turn oracle-answer training, while reward-weighted training and using all rollouts outperform filtering to successful data.These choices let RISE learn from intermediate, potentially suboptimal responses and improve performance on later turns.Source: Figure 7 · p. 13, §6.4 · pp. 13–14
  • On-policy DAgger-style training improves performance over expert data alone, while offline imitation reduces loss but does not generalize to new problems.The authors attribute the advantage to aligning training contexts with the learner’s on-policy distribution.Source: §6.4 · p. 14, Figure 8 · p. 14

6.6. Error Analysis of RISE over Turns

  • Without oracle feedback, RISE increases the fraction of problems staying correct and decreases those staying incorrect or changing from correct to incorrect over turns.Qualitative examples show full rewrites or targeted edits to errors, and tolerance of noisy environmental feedback without oracle-assisted early termination.Source: Figure 9 · p. 15, §6.6 · p. 15

7. Discussion, Future Directions, and Limitations

RISE uses iterative fine-tuning to help language models improve their responses over multiple turns, with strong reasoning-task results. The discussion highlights limited training iterations and proposes more online training and integration into general post-training as future directions.

Discussion, Future Directions, and Limitations

  • RISE’s iterative RL recipe improves 7B models over turns on GSM8K and MATH, surpassing prior refinement approaches and achieving gains not observed in strong proprietary models [21].It uses expert or self-generated supervision and can work with one model rather than requiring multiple models.Source: §7 · p. 16
  • Computational constraints limited RISE to two training iterations, or one when using learner-generated supervision, which may require more computation and iterations than expert supervision.The approach currently requires manual iterations; the authors suggest a more online variant for data-efficient scaling of on-policy learning.Source: §7 · p. 16
  • Although models were fine-tuned on one task at a time, the authors propose incorporating RISE data into general instruction tuning and post-training to teach sequential self-improvement.They hypothesize this would not harm first-turn performance, based on their finding that RISE fine-tuning did not hurt that performance in any trained model.Source: §7 · p. 16

Appendices · A. Additional Ablations on Data Composition and Weak-to-Strong Generalization · A.1. Inclusion of Correct-to-Correct Data

This ablation tests how adding correct-to-correct examples affects RISE. Appending a novel correct response performs best, while repeating the same correct response reduces performance; turn-by-turn analysis also examines response stability.

A.1. Inclusion of Correct-to-Correct Data

  • The ablation uses Llama2-7B RISE data from GSM8K Iteration 1, comparing duplicated correct responses with rollouts followed by a different correct response.Both variants add one extra turn after a successful rollout; the former repeats its final correct response, while the latter supplies a novel correct response.Source: §A.1 · p. 22
  • The authors motivate correct-to-correct training by arguing that self-improvement requires verifying prior responses and deciding whether to change them, not merely converting incorrect answers to correct ones.The prior RISE data had only shown conversions from incorrect to correct responses.Source: §A.1 · p. 22
  • All methods beat the base model, but appending a novel correct response performs best, Default RISE is a close second, and repeating the correct response reduces performance.The authors suspect duplication-induced spurious correlations explain the repeated-response variant’s poorer performance.Source: Table 4 · p. 22, §A.1 · p. 22
  • Default RISE and the novel-correct-response variant steadily reduce correct-to-incorrect changes, from 12.3% to 3.9% and from 9.8% to 3.3%, respectively.The passage treats decreasing change rates as indicating better self-improvement, with lower absolute rates indicating greater resistance to noisy feedback.Source: §A.1 · p. 22, Figure 10 · p. 23
  • Boost has the highest overall correct-to-incorrect change percentages and rises from turn 4 to 5, while repeating a correct response starts at 6.3% but increases from turn 3 onward.These patterns suggest difficulty maintaining correct responses in Boost and possible instability across extended interactions for the repeated-response variant.Source: §A.1 · p. 22, Figure 10 · p. 23

A.2. Weak-to-Strong Generalization: RISE on Weak Model Data Improves Strong Models · B. Additional Results

RISE rollouts generated by the weaker Llama2-7B can train Mistral-7B to improve its self-improvement performance, while the reverse transfer is less effective. Training on these data does not degrade single-turn performance.

A.2. Weak-to-Strong Generalization: RISE on Weak Model Data Improves Strong Models

  • Mistral-7B trained on Llama2-7B RISE rollouts outperforms training on on-policy Mistral-7B rollouts, with transfer benefits particularly significant for multi-turn reasoning.Mistral-7B + Iteration 1 data from Llama2 also outperforms Llama2-7B trained on its own Iteration 1 data on all reported metrics, especially m1@t5.Source: Table 5 · p. 23, §A.2 · p. 23
  • Training Llama2-7B on Mistral-7B multi-turn rollouts performs worse than training it on its own on-policy rollouts.The authors suggest that a weaker model may struggle to internalize a stronger model’s mistakes, leading to hallucinations and memorization.Source: §A.2 · pp. 23–24
  • Llama2-7B’s mistakes can be more informative for training self-improvement than Mistral-7B’s, underscoring the importance of error quality, not just generator capability.The authors suspect this asymmetry makes reverse transfer a harder learning problem.Source: §A.2 · pp. 23–24
  • RISE training on weak-model rollouts does not degrade single-turn performance, and may be useful when generating stronger-model rollouts is expensive.The section frames its comparison as using Llama2-7B-generated RISE data to train Mistral-7B.Source: §A.2 · pp. 23–24

B.1. Complete Comparisons and Discussion: Extended Version of Table 1

This extended comparison finds that RISE improves over multiple-turn refinement and baseline approaches, with gains that do not depend on oracle feedback. It also contrasts RISE’s sequential self-correction with methods that rely on external refinement models, verifiers, or more samples.

B.1. Complete Comparisons and Discussion: Extended Version of Table 1

  • RISE achieves the largest 1-to-5-turn gains on GSM8K and MATH without an oracle, and its performance gap grows when oracle early termination is allowed.Table 6 also reports that Self-Refine largely degrades performance, while GLoRE performs worse than RISE.Source: Table 6 · p. 24, §B.1 · p. 25
  • Self-Refine prompts the model to identify errors and revise its initial answer, but without oracle hints it degrades performance across models; oracle-assisted early termination yields only a slight boost.RISE, by contrast, produces consistent multi-turn improvements without oracle assistance.Source: Figure 11 · p. 25, §B.1 · p. 25
  • RISE improves GSM8K performance by 13.4% in 3 turns without oracle feedback and 23.4% with an oracle, while GLoRE’s 7B model gains 10% with oracle guidance [17].RISE starts from Llama2 7B and has slightly lower absolute performance than GLoRE, but its 5-turn performance surpasses GLoRE’s best in both oracle and non-oracle scenarios; the comparison is apples-to-oranges because GLoRE uses multiple models while RISE uses one end-to-end model.Source: §B.1 · pp. 25–26
  • RISE improves the base model by 44.5% using 5 samples after 3 fine-tuning iterations, compared with V-STaR’s 35.2% improvement using 64 samples [19].V-STaR does not revise responses sequentially and is bounded by its candidate generator; RISE can solve problems that majority voting misses, while verifier filtering could complement RISE.Source: §B.1 · p. 26
  • RISE achieves maj@5 performance comparable to Mistral-7B and Eurus-7B-SFT while using single-domain fine-tuning, and fine-tuning RISE can further improve Mistral-7B.Those comparison models were fine-tuned on much larger datasets, including math-specific data; the authors also report that RISE can generalize to stronger base models.Source: §B.1 · p. 26

B.2. Qualitative Examples from RISE … D.2. Inference Hyperparameters

RISE’s qualitative examples show both successful correction and meaningful but incomplete error localization, including resilience to noisy feedback. The accompanying pseudocode and experimental details describe iterative loops, reward-weighted fine-tuning, and inference setups.

B.2. Qualitative Examples from RISE

  • In a small-edit example, RISE revises a wrong total of 12 to the correct answer of 9 by correcting the second-period calculation.The model changes the mistaken calculation of 4 + 4 points to 4 + 4 × 25% = 5 points, then adds 4 + 5.Source: §B.2 · p. 28, Figure 12 · p. 28
  • With feedback that always says an answer is incorrect, RISE still produces the correct answer in 3 of 4 turns from turns 2–5, using new paraphrases.The turn-4 response gets half the solution right but makes an error in the final step.Source: Figure 13 · p. 29
  • In a house-flipping example, RISE does not reach the correct final answer within five turns, but preserves correct intermediate work and makes localized changes.The figure notes that turn 2 reaches a correct intermediate step before adding a mistaken step; subsequent turns do not fix the error.Source: Figure 14 · p. 30, §B.2 · p. 30

C. Pseudocode

  • The pseudocode includes a loop over dataset indices and repeated step loops indexed by T−1 or N.Source: C. Pseudocode · p. 31

D.1. Hyperparameters for Fine-Tuning with RISE

  • RISE fine-tuning uses FastChat with a reward-weighted loss and loads Llama-2-7b-chat-hf and Mistral-7B-Instruct-v0.2 from Hugging Face.Source: §D.1 · p. 32

D.2. Inference Hyperparameters

  • For inference, GPT-3.5 is queried through OpenAI’s API, while open-source models are served with FastChat; serving a 7B model requires one A100 or A40 GPU.Source: §D.2 · p. 32

D.3. Datasets · D.4. Prompts used for seeding RISE on the GSM8K and MATH tasks · D.5. One-shot Example

The appendices specify the GSM8K and MATH data splits and RISE’s trajectory-generation procedure, then provide the math prompt and one-shot examples of solving and self-correction.

D.3. Datasets

  • GSM8K has 7,473 training and 1,319 test problems, while MATH has 7,500 training and 1,000 test problems.Source: §D.3 · p. 32
  • Training portions generate RISE trajectories, test portions evaluate models, and the 1,000-problem SVAMP dataset tests generalizability.Source: §D.3 · p. 33
  • At iteration i, the current model is rolled out for a maximum of i steps to generate training trajectories.Source: §D.3 · p. 33
  • An expert model such as GPT-3.5 samples five responses from the conversation history, and the highest-reward response is selected, with ties broken arbitrarily.Source: §D.3 · p. 33

D.4. Prompts used for seeding RISE on the GSM8K and MATH tasks

  • The GSM8K and MATH prompt asks the model to solve step by step and put its final answer in a box; the pipeline retries up to a maximum number of turns.Source: §D.4 · p. 33, Figure 15 · p. 33

D.5. One-shot Example

  • The one-shot example asks how many of 60 drama-club students take neither mathematics nor physics, given the subject and overlap counts.Source: §D.5 · p. 34, Figure 16 · p. 34
  • A separate Self-Refine one-shot example demonstrates self-critique and error correction based on that critique.Source: §D.5 · p. 35, Figure 17 · p. 35

Related papers

Loading 2407.18219v2…