Source-linked AI summary
Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning
Shelly Bensal, Umar Jamil, Christopher Bryant, Melisa Russak, Kiran Kamble, Dmytro Mozolevskyi, Muayad Ali, Waseem AlShikh
TL;DR
When task-specific training data and synthetic data from stronger models are unavailable, improving a model’s task performance is difficult. This paper trains models to reflect on failed attempts using binary verifier feedback, and reports average performance gains on function calling and Countdown equation solving.
Problem
Models can have task-specific blind spots, and retraining may be infeasible when relevant data or synthetic data from stronger models is unavailable.
Source: §1 · p. 1Method
After a verified failure, the model reflects and retries, and GRPO rewards reflection tokens when the second attempt succeeds.
Source: §1 · p. 2Results
Models trained for self-reflection with GRPO improved performance by an average of 9.0% on function calling and 16.0% on Countdown equation solving.
Source: §6 · p. 9Takeaways & Limitations
The approach is suited to tasks where a validator can detect whether responses are correct, such as formatted JSON, executable code, or equations meeting constraints.
Source: §6 · p. 8The method requires models with some basic ability to perform the task, self-reflect, and learn; Llama3.2-3B Instruct did not learn to self-correct on function calling.
Source: §7 · p. 9
Abstract
from arXiv · showhide
We explore a method for improving the performance of large language models through self-reflection and reinforcement learning. By incentivizing the model to generate better self-reflections when it answers incorrectly, we demonstrate that a model's ability to solve complex, verifiable tasks can be enhanced even when generating synthetic data is infeasible and only binary feedback is available. Our framework operates in two stages: first, upon failing a given task, the model generates a self-reflective commentary analyzing its previous attempt; second, the model is given another attempt at the task with the self-reflection in context. If the subsequent attempt succeeds, the tokens generated during the self-reflection phase are rewarded. Our experimental results show substantial performance gains across a variety of model architectures, as high as 34.7% improvement at math equation writing and 18.1% improvement at function calling. Notably, smaller fine-tuned models (1.5 billion to 7 billion parameters) outperform models in the same family that are 10 times larger. Our novel paradigm is thus an exciting pathway to more useful and reliable language models that can self-improve on challenging tasks with limited external feedback.
1 Introduction
LLMs can fail on tasks where task-specific training data or synthetic data from stronger models is unavailable. This paper trains models to improve their self-reflections, using reinforcement learning and binary success feedback to target downstream tasks.
1 Introduction
- Models can have task-specific blind spots, while fine-tuning may be infeasible when relevant training data or synthetic data from stronger models is unavailable.The effectiveness of prompting-based reflection also depends on the reasoning or reflection prompt.Source: §1 · p. 1
- After an incorrect first attempt, the model reflects and retries; if the retry succeeds, GRPO rewards the reflection tokens to improve future reflections.The approach aims to improve tasks without requiring task-specific data, relying instead on binary success or failure feedback from a response verifier.Source: §1 · p. 2
- Experiments evaluate the approach on APIGen function calling and the Countdown equation task, where responses can be verified.The method is intended for challenging tasks that can be checked by a response verifier.Source: §1 · p. 2
2 Related Work
Prior work describes self-reflection as a way to critique and revise model answers, while noting dependence on reliable feedback and the limits of teacher-generated training data. This paper combines failed-case reflection with GRPO, an outcome-based reinforcement-learning method.
2.1 Self-Reflection
- Self-reflection analyzes a model’s reasoning to identify and correct potential mistakes, and commonly proceeds from an initial answer to critique and refinement.Related work notes challenges including unreliable self-error detection, diminishing returns, and possible deterioration on easier prompts or strong base models.Source: §2.1 · p. 2
- Prior training-based methods learn from self-correction trajectories or multi-turn reinforcement learning, but typically rely on larger teacher models for data or supervision.These approaches report persistent improvements even without test-time reflection.Source: §2.1 · p. 2
- The proposed approach applies reflection to failed cases identified by an external verifier and trains from the model’s own outputs without external LLMs.The approach is presented as reducing test-time overhead by applying corrections conditionally.Source: §2.1 · p. 2
2.2 Reinforcement Learning for Language Models
- GRPO estimates advantages by comparing sampled completion outcomes, without a separate value network, for settings with sparse end-of-generation feedback.Its reward can reflect whether a completed solution is correct.Source: §2.2 · p. 3
- The paper uses GRPO as its sole reinforcement-learning mechanism, without an additional supervised fine-tuning stage.Related work cited in the passage describes reward modifications for retries and applications to tool use and mathematical problem solving.Source: §2.2 · p. 3
3 Reflect, Retry, Reward
Reflect, Retry, Reward uses a verifier to identify failed attempts, then asks the model to reflect and retry. GRPO rewards reflection tokens only when the retry succeeds.
3 Reflect, Retry, Reward
- A binary validator checks the initial response; successful responses stop, while failures trigger a self-reflection prompt.Examples of potentially verifiable tasks include API calls, equations, and code, though some tasks may require gold-standard answers.Source: §3 · p. 3
- The model retries with its reflection in the conversation history, and a successful retry leads GRPO to reward only the reflection tokens.Other generated tokens receive zero advantage so training targets general reflection rather than task-specific answer production.Source: §3 · p. 4
- If the second attempt still fails, no reward is given because the reflection did not turn the failure into a success.The overall sequence is illustrated as completing a task, reflecting after failure, and retrying with the reflection included.Source: §3 · pp. 3–4
4 Experiments
The experiments assess function calling and equation writing with task-specific datasets and verifiers. Training uses failure examples and multi-step GRPO; the supplied passages describe setup and implementation but do not report final benchmark comparisons.
4.1 Function Calling
- Function-calling experiments use APIGen, with correctness requiring the right tool and exact parameter values in the generated call.The dataset contains 60,000 function calls and 4,211 unique tools; queries average 2.8 tool options.Source: §4.1 · p. 4
4.2 Countdown Math Equations
- Countdown asks models to form an equation from three or four numbers that reaches a target while using every number once.The experiments use a task-specific validator to check the equation against the target.Source: §4.2 · p. 5
4.3 A Dataset of Failures
- Training data contains only queries on which the model failed, identified by generating multiple responses and applying task-dependent verifiers.The authors say this saves time, captures nondeterministic failures, and helps measure training samples needed for convergence; they describe it as functionally equivalent to learning from successful and failed responses.Source: §4.3 · p. 5
4.4 Multi-Step GRPO
- The multi-step GRPO implementation uses successive completions and a mask to reward reflection tokens based on later performance.Training ran for up to 1,750 steps with an effective batch size of 256 failures, and most models converged sooner.Source: §4.4 · pp. 5–6
- Table 1 reports APIGen accuracy on 12,000 test samples for first and second attempts, with and without GRPO self-reflection training.The supplied caption identifies the comparisons but does not provide the table’s values.Source: Table 1 · p. 6
- Final reported experiments used models from 1.5 billion to 8 billion parameters because of GRPO computational-efficiency and scalability concerns; smaller models were tested but their results were not reported.Source: §4.4 · p. 6
5 Experimental Results
GRPO training improved self-reflection performance on APIGen and Countdown, with gains from reflection after training. The trained models also produced more concise reflections and showed little degradation on other benchmarks.
5 Experimental Results
- After GRPO, self-reflection raised APIGen performance by 4.7% on average and Countdown performance by 8.6% on average.For APIGen, the 4.7% gain came from allowing a second reflective attempt.Source: §5 · pp. 6–7
- A GRPO-trained Qwen-2-7B outperformed vanilla Qwen-2-72B on APIGen with two attempts, despite being 10 times smaller.After training, almost all models also outperformed their two-attempt vanilla counterparts after just one attempt.Source: §5 · p. 6
5.1 Better Self-Reflections
- GRPO reflections were more concise than vanilla reflections, contrasting with verbose chain-of-thought outputs; when concision helps remains an open question.Vanilla reflections were described as longer, more verbose, and repetitive.Source: Figure 2 · p. 7, §5.1 · p. 8
5.2 Low Catastrophic Forgetting
- In most cases, benchmark degradation after training was less than 1%; Qwen-2.5-1.5B improved 0.6% on MMLU-Pro and 0.8% on MATH.The authors treat stable performance on these benchmarks as evidence of robustness to catastrophic forgetting.Source: §5.2 · p. 8, Table 3 · p. 8
6 Conclusion
The paper trains models to improve self-reflection rather than a particular task, using only a validator’s correct-or-incorrect signal. On APIGen and Countdown, GRPO-trained models improved performance and could outperform much larger untrained models.
6 Conclusion
- The approach trains models to improve self-reflection rather than a particular task and requires only a validator that detects correct or incorrect responses.Source: §6 · p. 8
- GRPO-trained self-reflection models improved average performance by 9.0% on APIGen function calling and 16.0% on Countdown equation solving.Source: §6 · p. 9
- Trained 7B models outperformed untrained 72B models on both evaluated tasks, and the models were robust to catastrophic forgetting.Source: §6 · p. 9
- Models also performed significantly better on first attempts, even when they did not need to self-reflect.The authors hypothesize this may reflect more general reasoning improvement and identify cross-task generalization as future work.Source: §6 · p. 9
7 Limitations
The method depends on a usable binary success/failure validator and does not work for every model-task combination. It requires basic task ability, self-reflection ability, and capacity to learn.
7 Limitations
- A binary success/failure validator may not be straightforward to define for every task.Ground-truth labels, when available, or a larger model as judge are possible alternatives.Source: §7 · p. 9
- The approach requires models to have basic ability to perform the task, self-reflect, and learn.Llama3.2-3B Instruct, for example, could not learn to self-correct on function calling.Source: §7 · p. 9
A Prompt Templates
The appendix documents task-specific prompts used to establish reasonable baselines and elicit a first attempt, reflection after failure, and a second attempt. Function-calling prompts vary by model format, while Countdown prompts specify equation constraints and answer formatting.
A Prompt Templates
- Prompts were iterated to achieve reasonable baselines for each model and task, following provider recommendations to the best of the authors’ ability.The appendix supplies the training prompt templates for reproducibility and clarity.Source: A Prompt Templates · p. 13
A.1 Function Calling
- The function-calling second attempt follows the reflection, with one template repeating the system prompt and user query and another providing just the query.The two documented templates differ in how the second attempt is set up.Source: §A.1 · pp. 13–14
- For function calling, the appendix gives model-specific templates, including the recommended Llama 3.1 format for Llama 3.1 and Phi 3.5.Phi performed better with the Llama tool-calling template than with the Qwen 2 template.Source: §A.1 · p. 13
- The function-calling setup presents tools and a dataset query, elicits a first attempt, and prompts reflection when the attempt is incorrect.The reflection prompt asks the model to identify what went wrong and help it do better next time.Source: §A.1 · p. 13
A.2 Countdown Math Equations
- The Countdown prompt asks the model to form a target equation from provided numbers using basic arithmetic, each number at most once, and a boxed final answer.It also requests step-by-step reasoning.Source: §A.2 · p. 14
- For Countdown, a failed first attempt triggers a short self-reflection, after which the user problem is repeated for a second attempt.The appendix also includes APIGen and Countdown first-attempt error-analysis tables, covering test sets of 12,000 and 15,000 samples, respectively.Source: §A.2 · p. 14, Table 4 · p. 15, Table 5 · p. 15
B Error Analysis
The error analysis examines only first-attempt failures, comparing error types before and after training.
B Error Analysis
- The analysis categorizes errors before and after training using only first-attempt task errors (pass@1).Its aim is to understand error types and which may be mitigated by self-reflection training.Source: B Error Analysis · p. 15
B.1 Function Calling
Function-calling errors are grouped into tool-choice, parameter, and formatting errors. Training patterns differ by model size and error type.
B.1 Function Calling
- Function-calling errors are categorized as tool-choice errors, parameter-name or value errors, and format errors; parameter choice is considered harder than tool choice.Source: §B.1 · p. 15
- The 1.5B Qwen-2 and Phi-3.5-mini models struggle with tool choice and show little or no parameter-value improvement, while 7–8B models mainly improve parameter selection.The larger models are already comparatively good at tool choice without training.Source: §B.1 · p. 15
B.2 Math Countdown Equations
Math Countdown equation errors are classified by validity, number use, and whether the equation reaches the target. Training generally shifted models toward using the provided numbers, with Qwen-2.5-7B Instruct as an exception.
B.2 Math Countdown Equations
- Countdown equation errors fall into invalid equations, use of numbers outside the provided set, and equations that miss the target.Source: §B.2 · p. 15
- Training reduced wrong-number errors for every model except Qwen-2.5-7B Instruct, and most models learned to use the correct numbers even when missing the target.Source: §B.2 · p. 16
- Qwen-2.5-7B Instruct instead learned to hit the target, even when doing so required using the wrong numbers.Source: §B.2 · p. 16