Reflexion: Language Agents with Verbal Reinforcement Learning

• Existing LLM agents generate text and executable actions, but model scale makes gradient-based reinforcement learning costly, motivating reliance on in-context examples.Source: §1 · p. 1
• Reflexion converts environmental feedback into reflective text for subsequent episodes, parameterizing its policy through an agent’s memory encoding paired with chosen LLM parameters.These reflections provide concrete directions for improvement—a semantic gradient—and experiments show self-reflection helps agents learn complex tasks over a handful of trials.Source: §1 · p. 2
• Feedback can come from binary environmental signals, predefined failure heuristics, or self-evaluation, including LLM classification and self-written unit tests.Producing useful reflections requires identifying mistakes and generating actionable insights, addressing the credit assignment problem. Evaluation signals become natural-language experience summaries that can be stored in long-term memory.Source: §1 · p. 2
• Reflexion avoids LLM fine-tuning and provides nuanced feedback, interpretable episodic memory, and explicit action hints, but depends on self-evaluation or heuristics without guaranteed success.Source: §1 · p. 2
• 22% absolute improvement over strong baselines on AlfWorld [24] is achieved in 12 iterative learning steps.Reflexion also improves HotPotQA [28] reasoning by 20% and HumanEval [6] Python programming by as much as 11%. The authors report state-of-the-art results on various code-generation benchmarks.Source: §1 · p. 2
• 40 hard-level Leetcode questions across 19 programming languages constitute LeetcodeHardGym, the introduced code-generation reinforcement-learning environment.Source: §1 · p. 2

Reasoning and decision-making

• Self-Refine [15] and Pryzant et al. [21] address single-generation tasks, while Paul et al. [20] fine-tune critics and Xie et al. [27] search actions.Self-Refine conditions self-evaluation and improvement on supplied task constraints. Xie et al. [27] use stochastic beam search with self-evaluation to provide foresight.Source: §2 · p. 2
• Reflexion extends prior reasoning approaches with persistent self-reflective memory, allowing agents to identify their errors and suggest lessons from mistakes over time.The paper presents self-reflection as an enhancement to existing evaluation and improvement mechanisms.Source: §2 · p. 3
• Yoran et al. [31] and Nair et al. [16] use deciders across generations; Kim et al. [10] retry without evaluation, while Goodman proposes optimizations through qualitative evaluation.Kim et al. [10] use a fixed number of retry steps.Source: §2 · pp. 2–3

Programming

• AlphaCode, Self-Debugging [7], and CodeRL [12] require ground-truth tests that invalidate pass@1 eligibility and lack self-reflection connecting error identification with implementation improvement.CodeT [5] avoids hidden tests but lacks a self-learning step to improve code writing.Source: §2 · p. 3
• AlphaCode evaluates generations using hidden tests, CodeT [5] scores implementations with self-generated tests, and Self-Debugging [7] and CodeRL [12] debug using execution feedback.CodeRL [12] formulates debugging as reinforcement learning with an actor-critic setup.Source: §2 · p. 3

Actor

• The LLM Actor generates text and actions conditioned on observations and memory, with Chain of Thought [26] and ReAct [30] explored as Actor models.The added memory supplies context to the Actor, inspired by the in-context policy iteration approach suggested by Brooks et al. [3].Source: §3 · p. 4

Evaluator

• The Evaluator scores trajectories using exact-match grading for reasoning, predefined heuristics for decision-making, or an LLM for decision-making and programming.These alternatives address the difficulty of defining effective reward functions in semantic spaces.Source: §3 · p. 4

Self-reflection

• The LLM Self-Reflection model converts sparse rewards, the current trajectory, and persistent memory into specific verbal feedback stored for future trials.Feedback can identify an action that caused subsequent errors and propose an alternative action for later trials.Source: §3 · p. 4

The Reflexion process

• Reflexion repeats acting, evaluation, and reflection until the Evaluator judges a trajectory correct, appending each trial’s verbal feedback to memory.The Actor first interacts with the environment; the Evaluator scores its trajectory, and the Self-Reflection model summarizes the trajectory and reward into feedback.Source: §3 · pp. 3, 5

Memory

• 1-3 experiences is the usual reflection-memory limit; the Actor uses trajectory history as short-term memory and stored reflections as long-term memory.The storage limit accommodates LLM context constraints.Source: §3 · pp. 4–5

4 Experiments

• 22% on AlfWorld, 20% on HotPotQA, and 11% on HumanEval are Reflexion’s reported performance improvements over strong baselines.The experiments span sequential decision-making, question answering, and code generation.Source: §4 · p. 5, Figure 3 · p. 6, Table 1 · p. 7

4.1 Sequential decision making: ALFWorld

• 130 out of 134 ALFWorld tasks are completed by ReAct + Reflexion using heuristic detection of hallucinations and inefficient planning, significantly outperforming ReAct.Reflexion continues learning across 12 trials, whereas ReAct-only improvement stalls between trials 6 and 7. ReAct-only converges at a 22% hallucination rate without long-term recovery.Source: §4.1 · pp. 5–6, Figure 3 · p. 6
• Self-reflection turns failed ALFWorld trajectories into reusable self-hints that correct imagined object possession, revise plans, and guide searches across trials.ReAct [30] uses LLM classification or a heuristic that triggers after more than three identical action-response cycles or over 30 actions. Each retry resets the environment, and Reflexion retains only the last three reflections.Source: §4.1 · pp. 5–6

4.2 Reasoning: HotpotQA

• 14% accuracy improvement with ground-truth context and an 8% absolute boost over episodic memory demonstrate Reflexion’s reasoning benefits on HotPotQA.CoT [26] tests reasoning with supplied context, while ReAct [30] retrieves Wikipedia evidence; exact-match feedback and three-experience memory guide retries. Baselines never recover first-trial failures, whereas Reflexion retries until three consecutive failures.Source: §4.2 · pp. 6–7, Figure 4 · p. 7

4.3 Programming

• Programming evaluates Python and Rust on MBPP [2], HumanEval [6], and the new LeetcodeHardGym benchmark using self-generated unit tests.LeetcodeHardGym contains 40 hard-rated questions released after October 8, 2022; Rust benchmark subsets are translated with MultiPL-E [4]. Tests use Chain-of-Thought prompting [26], undergo AST-validity filtering, and are capped at six, while memory retains one experience.Source: §4.3 · p. 7
• 91% overall accuracy on HumanEval Python accompanies state-of-the-art Python and Rust results across the evaluated programming benchmarks, except MBPP Python.Reflexion underperforms the GPT-4 baseline on MBPP Python.Source: Table 1 · p. 7, §4.3 · p. 8
• 16.3% false-positive test execution on MBPP Python, versus 1.4% on HumanEval Python, helps explain Reflexion’s MBPP Python underperformance.These rates measure incorrect submissions among those passing internal tests; false positives cause premature submission, whereas false negatives can be corrected through reflection. Baseline pass@1 accuracies are relatively similar: 82% for HumanEval Python and 80% for MBPP Python.Source: Table 2 · p. 8, §4.3 · p. 8
• 52% accuracy without test generation and execution, versus 60% for the baseline, shows harmful effects of unguided reflection on the hardest HumanEval Rust subset.Without tests, the agent cannot determine correctness or stop early and makes harmful edits. Removing natural-language self-reflection also fails to improve over baseline despite detecting syntax and logic errors, undermining blind trial-and-error debugging on complex Rust programs.Source: Table 3 · p. 8, §4.3 · pp. 8–9

5 Limitations

• Reflexion optimizes policies through natural language but may still become trapped in non-optimal local minima.Learning from experience can improve action choice without guaranteeing an optimal solution.Source: §5 · p. 9
• Long-term memory is restricted to a sliding window with maximum capacity, motivating future exploration of more advanced memory structures.Suggested extensions include vector embedding databases and traditional SQL databases.Source: §5 · p. 9
• Test-driven code generation faces practical difficulties specifying accurate input-output mappings for nondeterministic, API-interacting, hardware-dependent, and parallel or concurrent functions.These include nondeterministic generators, impure functions interacting with APIs, hardware-dependent outputs, and difficult-to-predict parallel or concurrent behavior.Source: §5 · p. 9

6 Broader impact

• Reflexion could improve automation and work efficiency while amplifying misuse risks, motivating greater attention to safety and ethics.These implications concern language agents interacting with external environments and humans.Source: §6 · p. 9
• Verbal reinforcement learning may make autonomous agents more interpretable and diagnosable, addressing challenges associated with reinforcement learning’s black-box policies and optimization.The authors identify both interpretability and alignment as challenges in conventional reinforcement learning.Source: §6 · p. 9
• Self-reflections could be monitored to check agents’ intent before tool use, particularly when that tool use is difficult for humans to understand.The authors present this as a possible safeguard, not a demonstrated guarantee.Source: §6 · p. 9

7 Conclusion

• Reflexion agents significantly outperform widely used decision-making approaches by utilizing self-reflection.Source: §7 · p. 9
• Reflexion leverages verbal reinforcement to teach agents to learn from past mistakes.Source: §7 · p. 9
• Future work could extend Reflexion with natural-language value learning or off-policy exploration techniques studied in traditional reinforcement learning.Source: §7 · p. 9

8 Reproducibility

• Use isolated execution environments for autonomous code-writing experiments, since generated code is not validated before execution.Source: §8 · p. 9

A Evaluation with additional models

• The ability to specify self-corrections emerges in stronger, larger models.Source: A Evaluation with additional models · p. 12
• The additional-model investigation tests the applicability of trial-and-error problem-solving across models of varying strengths.Source: A Evaluation with additional models · p. 12
• The HumanEval Python evaluation measures Pass@1 accuracy using starchat-beta [13].The supplied table caption contains no accuracy values.Source: Table 4 · p. 12
• The HotPotQA evaluation measures Pass@1 accuracy on 100 HotPotQA using various models.The supplied table caption contains no model scores or comparisons.Source: Table 5 · p. 12

B Decision-making

• The initial decision-making trial plans to find and take the mug before using the desklamp, beginning with a drawer-by-drawer mug search.Source: B Decision-making · p. 13
• An AlfWorld agent corrects inefficient planning by reflecting that it should find the desklamp before the mug, then executes a concise action sequence.Source: Figure 5 · p. 13

B.1 WebShop Limitation

• 100 customer requests on WebShop show no significant advantage for ReAct + Reflexion over ReAct; runs stop after four trials without improvement.Failed attempts produced unhelpful reflections, exposing difficulty escaping local minima requiring diverse, creative exploration. AlfWorld exposes permissible actions, while HotPotQA permits less precise searches than WebShop.Source: Figure 6 · p. 14, §B.1 · p. 14

C Programming / C.1 Programming function implementation example (HumanEval Python)

• The HumanEval Python example seeks a minimum-sum nonempty subarray using nested loops and a minimum comparison, while programming prompts require function bodies only.Strict output instructions counter the models' extensive dialogue training.Source: §C.1 · pp. 14–15, C Programming · p. 14

C.2 Reflexion Actor instruction

• The Reflexion Actor uses previous code, unit-test feedback, and self-reflection to produce an improved function body without its signature.The response must begin with four spaces of indentation; the generation sequence ends with an instruction for the next implementation.Source: §C.2 · p. 15

C.3 Reflexion Self-reflection instruction and example

• The self-reflection generation template lists an instruction, function implementation, and unit-test feedback.The accompanying prompt requests an improved function body without a signature, beginning with four spaces.Source: §C.3 · p. 15

C.4 Reflexion programming no Self-Reflection ablation example / C.5 Reflexion programming no test generation ablation example

• The no-self-reflection and no-test-generation ablation templates both list implementation, unit-test feedback, self-reflection, and an instruction for the next implementation.Despite their ablation labels, both supplied sequences explicitly retain the self-reflection field.Source: §C.4 · p. 15, §C.5 · pp. 15–16

D Reasoning / D.1 Full example

• The full HotPotQA example shows a Reflexion + ReAct agent using self-reflection to choose a better search method for its next trial.Figure 7 depicts two trials within the same environment and task.Source: Figure 7 · p. 17

D.2 Chain-of-Thought + Reflexion

• Chain-of-Thought + Reflexion corrects the shared profession of John Lanchester and Alan Dean Foster from 'novelist and screenwriter' to 'novelist'.The first answer receives incorrect feedback; the revised answer receives correct feedback.Source: §D.2 · p. 18

D.3 HotPotQA Chain-of-Thought (GT) + Reflexion

• The ground-truth-context example corrects 'Battle of White Plains' to 'The New York and New Jersey campaign' by distinguishing a single battle from a campaign.The reflection identifies the question's request for a series of battles as the crucial distinction.Source: §D.3 · p. 18

D.4.1 (EPM) Chain-of-Thought + Reflexion

• Six bands for Jonny Craig versus seven for Pete Doherty underpin the initial comparison, whose answer of Pete Doherty receives incorrect feedback.The question asks which musician has been a member of more bands.Source: §D.4.1 · p. 19
• Chain-of-Thought + Reflexion answers Jonny Craig on its second trial and receives correct feedback, despite an internally inconsistent comparison.The retry assigns seven bands to each musician yet concludes Craig has more.Source: §D.4.1 · p. 19
• The band-comparison reflection attributes failure to overlooking past memberships and recommends researching both musicians’ past and current bands.The second-trial reasoning explicitly adopts this research intention.Source: §D.4.1 · p. 19

D.4.2 (EPM) Chain-of-Thought (GT) + Reflexion

• Chain-of-Thought (GT) + Reflexion answers Engineering correctly on its second trial by identifying Basnet’s specific field of study.The supplied context states that Basnet holds an M.Sc. in Engineering.Source: §D.4.2 · p. 19
• The GT reflection attributes the initial mistake to interpreting the question as requesting a degree category and emphasizes identifying the requested field of study.The second-trial prompt includes the previous trajectory.Source: §D.4.2 · p. 19
• The initial GT trajectory answers Sciences, Engineering, and Medicine rather than a specific field, and receives incorrect feedback.It reasons from fields typically associated with a Master of Science rather than selecting Basnet’s stated field.Source: §D.4.2 · p. 19