PaperlayerView exact revision on arXiv

Source-linked AI summary

Reflexion: Language Agents with Verbal Reinforcement Learning

Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, Shunyu Yao

arXiv:2303.11366v4cs.AIcs.CLcs.LG

TL;DR

Language agents struggle to learn efficiently from trial-and-error without costly model fine-tuning. Reflexion uses verbal reflection stored in memory to improve subsequent decisions, outperforming baselines across decision-making, reasoning, and programming tasks.

  • Problem

    LLM-based agents have limited means to learn from trial-and-error because conventional reinforcement learning requires substantial compute and time.

  • Method

    Reflexion converts task feedback into verbal summaries stored in episodic memory, guiding an LLM agent’s decisions in subsequent trials.

  • Results

    Reflexion improves performance across decision-making, reasoning, and programming tasks, including a 22% absolute gain on AlfWorld over strong baselines.

  • Takeaways & Limitations

    Self-reflection helps LLM agents learn complex tasks over a handful of trials without updating model weights.

  • Takeaways & Limitations

    Reflexion may reach non-optimal local minima, and its long-term memory is limited to a fixed-capacity sliding window.

Abstract

from arXiv · show

Large language models (LLMs) have been increasingly used to interact with external environments (e.g., games, compilers, APIs) as goal-driven agents. However, it remains challenging for these language agents to quickly and efficiently learn from trial-and-error as traditional reinforcement learning methods require extensive training samples and expensive model fine-tuning. We propose Reflexion, a novel framework to reinforce language agents not by updating weights, but instead through linguistic feedback. Concretely, Reflexion agents verbally reflect on task feedback signals, then maintain their own reflective text in an episodic memory buffer to induce better decision-making in subsequent trials. Reflexion is flexible enough to incorporate various types (scalar values or free-form language) and sources (external or internally simulated) of feedback signals, and obtains significant improvements over a baseline agent across diverse tasks (sequential decision-making, coding, language reasoning). For example, Reflexion achieves a 91% pass@1 accuracy on the HumanEval coding benchmark, surpassing the previous state-of-the-art GPT-4 that achieves 80%. We also conduct ablation and analysis studies using different feedback signals, feedback incorporation methods, and agent types, and provide insights into how they affect performance.

1 Introduction

Reflexion is a lightweight framework that uses verbal feedback and episodic memory to help LLM agents learn from prior failures without updating model weights. Experiments across decision-making, reasoning, and programming tasks show improvements over strong baselines and state-of-the-art code-generation results.

  • Method: Reflexion converts binary or scalar environmental feedback into verbal summaries added as context in the agent’s next episode.The textual feedback provides a concrete direction for improvement and acts as a semantic gradient signal.
  • Advantages: Reflexion avoids LLM fine-tuning while supporting nuanced feedback and explicit, interpretable episodic memory over prior experiences.These properties distinguish it from traditional policy- or value-based reinforcement learning approaches.
  • Evaluation: Reflexion agents improve as decisionmakers, reasoners, and programmers across sequential decision-making, reasoning, and programming experiments.The programming tasks involve using external tools such as compilers and interpreters.
  • Contributions: Reflexion parameterizes a policy as an agent’s memory encoding paired with a choice of LLM parameters.This defines verbal reinforcement as a new paradigm for agent learning.
  • Contributions: Self-reflection enables LLM agents to learn complex tasks over a handful of trials.The paper presents this as an empirically useful emergent property of self-reflection in LLMs.
  • Contributions: LeetcodeHardGym contains 40 challenging hard-level Leetcode questions across 19 programming languages.It is introduced as a code-generation reinforcement-learning gym environment.

2 Related work

Prior work improves generation through self-evaluation, retries, decider models, or debugging, but is often limited in task scope or lacks persistent self-reflective memory. Reflexion combines self-reflection with persisting memory across reasoning, decision-making, and programming settings.

  • Reasoning and decision-making: Self-Refine iteratively improves generations through self-evaluation but is limited to single-generation reasoning tasks.Its self-evaluation and self-improvement are conditioned on task constraints.
  • Reasoning and decision-making: Reflexion is the only listed reasoning approach combining self-refinement, hidden constraints, decision-making, binary reward, and memory.The comparison table marks all five capabilities for Reflexion, while Self-refine lacks hidden constraints, decision-making, binary reward, and memory, and Beam search lacks memory.
  • Programming: Reflexion is the only listed programming approach combining testing, debugging, self-generated tests, multiple languages, and self-reflection.The comparison table marks all five capabilities for Reflexion, while the other listed approaches each omit at least one capability.
  • Reasoning and decision-making: Prior reasoning approaches use decider models, fixed-step retries, or qualitative evaluation, whereas Reflexion adds self-reflection and persistent memory of experiences.The memory helps agents identify errors and self-suggest lessons to learn from them.
  • Programming: Programming research spans hidden-test evaluation, self-generated unit tests, execution-feedback debugging, and reinforcement-learning-based code improvement.AlphaCode evaluates generations on hidden tests, CodeT scores implementations with self-generated tests, Self-Debugging uses execution feedback, and CodeRL frames the problem in reinforcement learning.

3 Reflexion: reinforcement via verbal reflection

Reflexion reinforces language agents through a modular Actor–Evaluator–Self-Reflection architecture that converts task feedback into verbal guidance rather than weight updates. The agent stores these reflections alongside trajectory history, using both short- and long-term memory to improve decisions across trials.

  • Modular components: Reflexion comprises an Actor that generates actions, an Evaluator that scores outputs, and a Self-Reflection model that produces verbal reinforcement cues.The Actor is an LLM prompted to generate text and actions from state observations, while the Evaluator assesses generated trajectories.
  • Verbal reinforcement: The Self-Reflection model transforms sparse feedback, trajectories, and persistent memory into nuanced feedback that guides future trials.With binary success or failure signals, it can articulate alternative actions and store the resulting experience for later use.
  • Memory: The Actor conditions decisions on short-term trajectory history and long-term stored reflections, combining recent task context with lessons accumulated across trials.This iterative cycle of trial, error, reflection, and persistent memory is intended to improve decision-making in varied environments.
  • Iterative process: Reflexion iteratively generates a trajectory, computes an Evaluator score r_t = M_e(τ_0), and analyzes the trajectory–reward pair to create feedback for improvement.The Evaluator’s scalar reward reflects task-specific performance, while Self-Reflection amplifies it into a form usable by an LLM.

4 Experiments

Experiments evaluate Reflexion across decision-making, reasoning, and code-generation tasks, finding substantial gains over baselines while showing that reflective feedback and episodic memory improve difficult multi-step behavior. Coding analyses also expose dependence on reliable self-generated tests and natural-language reflection.

  • Experimental scope: Reflexion is evaluated on search-based question answering, interactive household tasks, and code generation across HotPotQA, AlfWorld, HumanEval, MBPP, and LeetcodeHard.The experiments cover sequential decision-making, language reasoning, and programming environments.
  • AlfWorld: 130 of 134 AlfWorld tasks are completed with ReAct + Reflexion using a simple heuristic to detect hallucinations and inefficient planning.Reflexion also learns additional tasks over 12 consecutive trials, whereas ReAct-only improvement halts between trials 6 and 7.
  • AlfWorld: Reflexion converts long failed AlfWorld trajectories into self-hints that identify early mistakes, propose revised actions or plans, and reduce search over many containers or surfaces.Episodic experience memory helps the agent recover from errors and exploit prior search experience.
  • HotPotQA: On HotPotQA, Reflexion outperforms all baseline approaches by significant margins over several learning steps, while ReAct-only and CoT-only variants do not probabilistically improve.The setup uses binary exact-match feedback between trials and allows Reflexion to retry failed tasks until three consecutive successes.
  • Code generation: Reflexion exceeds baseline accuracies and establishes new state-of-the-art results on Python and Rust benchmarks except MBPP Python 1.The coding evaluation covers MBPP, HumanEval, and LeetcodeHardGym, with Python-to-Rust translation through MultiPL-E.
  • Code generation: Removing natural-language explanations after failed unit-test evaluations prevents improvement over baseline, indicating that test generation alone is insufficient for harder Rust programs.The analysis also notes that flaky or incorrectly written test suites can produce false-positive or false-negative feedback.

5 Limitations

Reflexion uses natural language for policy optimization, but this approach may still succumb to non-optimal local minima. Its long-term memory is limited to a fixed-capacity sliding window, motivating more advanced memory structures.

  • Optimization: Reflexion’s natural-language policy optimization may still succumb to non-optimal local minima solutions.The paper characterizes policy optimization as improving action choice through experience while retaining this limitation.
  • Memory: Long-term memory is limited to a sliding window with maximum capacity.The authors encourage future work on vector embedding databases or traditional SQL databases as more advanced memory structures.

6 Broader impact

Reflexion could make autonomous agents more interpretable and diagnosable through verbal reinforcement and monitorable self-reflections, while also increasing misuse risks and requiring further safety and ethical work.

  • Risks and ethics: Greater agent automation and work efficiency could also amplify misuse risks, motivating more research on safety and ethics.These risks arise as language models interact with environments and humans, including the Internet, software, and robotics.
  • Interpretability and diagnosis: Verbal reinforcement may make autonomous agents more interpretable and diagnosable than black-box reinforcement-learning policies.The paper identifies interpretability and alignment as challenges in conventional reinforcement learning.
  • Interpretability and diagnosis: Monitoring self-reflections could help verify an agent’s intent before it uses tools that humans may struggle to understand.The passage gives tool usage as an example application of monitorable self-reflections.

7 Conclusion

Reflexion uses verbal reinforcement and self-reflection to help agents learn from past mistakes, outperforming widely used decision-making approaches. Future directions include adapting value learning in natural language and off-policy exploration techniques from traditional reinforcement learning.

  • Reflexion leverages verbal reinforcement to teach agents to learn from past mistakes.
  • Reflexion agents significantly outperform currently widely-used decision-making approaches by utilizing self-reflection.
  • Future work could apply value learning in natural language and off-policy exploration techniques studied in traditional reinforcement learning.

8 Reproducibility

The paper strongly recommends isolated execution environments for autonomous code-writing experiments because generated code is not validated before execution.

  • Reproducibility: Use isolated execution environments when running autonomous code-writing experiments.This is presented as a strong recommendation for reproducibility and safety.
  • Reproducibility: Generated code is not validated before execution.The lack of pre-execution validation motivates the recommendation for isolation.
  • Reproducibility: The isolation recommendation specifically applies to experiments involving autonomous code writing.

A Evaluation with additional models · B Decision-making

Reflexion’s trial-and-error benefits depend on model capability, with self-correction emerging in stronger, larger models. In decision-making, reflective feedback helps agents identify inefficient action orders and execute more concise plans.

  • A Evaluation with additional models: Self-correction is an emergent capability of stronger, larger models.The evaluation investigated trial-and-error problem-solving across models of various strengths.
  • A Evaluation with additional models: HumanEval evaluation reports Pass@1 accuracy for starchat-beta.Table 4 specifically evaluates HumanEval Python using starchat-beta.
  • A Evaluation with additional models: HotPotQA evaluation reports Pass@1 accuracy across various models.Table 5 evaluates 100 HotPotQA examples using multiple models.
  • B Decision-making: In AlfWorld, the agent initially searches for the mug before locating the desklamp.The trajectory shows the agent taking the mug from desk 1 and then checking locations for a desklamp.
  • B Decision-making: The reflection identifies a more efficient order: search for the desklamp before the mug.This correction addresses the inefficient planning order shown in the failed trajectory.
  • B Decision-making: After reflection, the agent corrects its reasoning trace and executes a concise action sequence.Figure 5 contrasts the failed inefficient trajectory with the corrected execution.

B.1 WebShop Limitation · C Programming

Reflexion struggles on WebShop when success requires escaping local-minimum choices through highly creative behavior. Its programming setup instead relies on strict instructions and templates to ensure LLMs produce function bodies only.

  • B.1 WebShop Limitation: WebShop tests agents navigating an e-commerce website to locate and purchase products for client requests.The experiment used 100 customer shopping requests.
  • B.1 WebShop Limitation: After four trials, the WebShop runs were terminated because the agent did not show sufficient progress.The passage reports testing a two-shot ReAct + Reflexion agent in 100 environments.
  • B.1 WebShop Limitation: Reflexion struggles to overcome local-minimum choices that require extremely creative behavior to escape.This shortcoming was observed in the WebShop experiment.
  • B.1 WebShop Limitation: ReAct + Reflexion fails to significantly outperform ReAct on WebShop across 100 customer shopping requests.Figure 6 directly compares Reflexion and ReAct performance.
  • C Programming: Programming LLM calls require strict instructions to produce function bodies only because of the models’ extensive dialogue training.Programming examples reportedly highlight instructions in blue and include templates.
  • C Programming: The programming section reports examples with highlighted instructions and templates, while directing readers to the full implementation online.The implementation link is provided in the passage.

C.1 Programming function implementation example (HumanEval Python) … D.3 HotPotQA Chain-of-Thought (GT) + Reflexion

The examples show how Reflexion uses implementation feedback and self-reflection to improve programming and reasoning across subsequent trials. In HotPotQA, reflection corrects answers by identifying overly narrow interpretations and guiding more appropriate responses.

  • C.1 Programming function implementation example (HumanEval Python): The HumanEval example asks for the minimum sum of any non-empty sub-array of nums and illustrates nested-loop implementation with a conditional minimum update.The implementation includes iteration over starting and ending indices and checks whether current_sum is less than min_sum.
  • C.2 Reflexion Actor instruction: The Reflexion Actor receives a prior function implementation, unit-test results, and self-reflection, then returns only an improved function body with syntactically appropriate indentation.Its generation format sequences instruction, implementation, test feedback, self-reflection, and the next implementation instruction.
  • C.3 Reflexion Self-reflection instruction and example: The self-reflection instruction likewise frames programming revision as applying necessary changes to a previous implementation using test results and reflective feedback.The supplied passage specifies that the response should contain only the improved function body and omit the signature.
  • C.4 Reflexion programming no Self-Reflection ablation example: The programming ablations separately examine actor generations without self-reflection and without test generation, retaining the implementation-revision format in the latter case.The no-self-reflection format is listed alongside the standard sequence, while the no-test-generation example contains instruction, implementation, unit-test feedback, self-reflection, and the next instruction.
  • D.1 Full example: In the HotPotQA full example, the Reflexion + ReAct agent uses self-reflection to determine a better search method for the next trial.The figure describes two trials within the same environment and task.
  • D.2 Chain-of-Thought + Reflexion: The Chain-of-Thought example changes an incorrect answer of “novelist and screenwriter” to the correct shared profession, “novelist,” in Trial #2.The first trial incorrectly assumes both authors share the same professions, whereas the second identifies novelist as the common profession.
  • D Reasoning: The accompanying reflection attributes the error to incorrectly assuming identical professions and recommends researching both authors’ backgrounds more carefully.It also advises considering that the authors may have multiple professions in common.

D.4 HotPotQA episodic memory (EPM) ablation prompts

The HotPotQA episodic-memory prompts show Reflexion correcting an initially incorrect comparison through research and reflection, while a separate degree-field question remains incorrect after a broad interpretation of MS abbreviations.

  • D.4.1 (EPM) Chain-of-Thought + Reflexion: Trial #1 concluded that Pete Doherty had been a member of more bands, based on counts of six for Jonny Craig and seven for Doherty.
  • D.4.1 (EPM) Chain-of-Thought + Reflexion: Action 1: Finish[Pete Doherty] received the observation Answer is INCORRECT.
  • D.4.1 (EPM) Chain-of-Thought + Reflexion: In Trial #2, the agent researched both musicians’ past and current bands and identified seven bands for each.
  • D.4.1 (EPM) Chain-of-Thought + Reflexion: The agent then answered Jonny Craig and received Answer is CORRECT.
  • D.4.1 (EPM) Chain-of-Thought + Reflexion: The reflection attributed the initial failure to overlooking Jonny Craig’s past bands while considering Pete Doherty’s current bands.
  • D.4.2 (EPM) Chain-of-Thought (GT) + Reflexion: A separate prompt asks which field corresponds to the degree held by the department head of the Rastriya Janashakti Party’s Foreign Relations Department.
  • D.4.2 (EPM) Chain-of-Thought (GT) + Reflexion: The previous trajectory interpreted the MS, M.S., and ScM abbreviations as referring broadly to sciences, engineering, and medicine.
  • D.4.2 (EPM) Chain-of-Thought (GT) + Reflexion: The answer Sciences, Engineering, and Medicine was marked INCORRECT.
Loading 2303.11366v4…