Source-linked AI summary
ReAct: Synergizing Reasoning and Acting in Language Models
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao
TL;DR
LLM reasoning and acting have largely been studied separately, limiting their integration in task solving. ReAct interleaves verbal reasoning traces with task-specific actions and outperforms prior methods across language and interactive decision-making benchmarks, while producing interpretable decision traces.
Problem
Reasoning and acting in language models have primarily been studied as separate capabilities, despite their complementary roles in task solving.
Method
ReAct prompts a frozen language model with few-shot examples to generate reasoning traces and domain-specific actions in an interleaved trajectory.
Results
ReAct achieves superior performance across question answering, fact verification, and interactive decision-making tasks, with improved interpretability over methods lacking reasoning or acting components.
Takeaways & Limitations
Interleaving reasoning with acting lets models update plans, incorporate external information, and produce interpretable task-solving trajectories.
Takeaways & Limitations
Human feedback is not incorporated in the presented approach and is left for future work.
Abstract
from arXiv · showhide
While large language models (LLMs) have demonstrated impressive capabilities across tasks in language understanding and interactive decision making, their abilities for reasoning (e.g. chain-of-thought prompting) and acting (e.g. action plan generation) have primarily been studied as separate topics. In this paper, we explore the use of LLMs to generate both reasoning traces and task-specific actions in an interleaved manner, allowing for greater synergy between the two: reasoning traces help the model induce, track, and update action plans as well as handle exceptions, while actions allow it to interface with external sources, such as knowledge bases or environments, to gather additional information. We apply our approach, named ReAct, to a diverse set of language and decision making tasks and demonstrate its effectiveness over state-of-the-art baselines, as well as improved human interpretability and trustworthiness over methods without reasoning or acting components. Concretely, on question answering (HotpotQA) and fact verification (Fever), ReAct overcomes issues of hallucination and error propagation prevalent in chain-of-thought reasoning by interacting with a simple Wikipedia API, and generates human-like task-solving trajectories that are more interpretable than baselines without reasoning traces. On two interactive decision making benchmarks (ALFWorld and WebShop), ReAct outperforms imitation and reinforcement learning methods by an absolute success rate of 34% and 10% respectively, while being prompted with only one or two in-context examples. Project site with code: https://react-lm.github.io
1 INTRODUCTION
The paper motivates combining language-model reasoning with task-oriented actions because existing approaches study them separately and each has important limitations. ReAct interleaves reasoning traces and actions, then evaluates this paradigm across language and decision-making benchmarks.
- Human problem solving combines verbal reasoning with actions to track progress, adjust plans, handle exceptions, and seek external information.
- Chain-of-thought reasoning is static and ungrounded, which can cause fact hallucination and error propagation.
- Prior action-generation systems interact with environments but generally lack abstract reasoning about high-level goals and working memory for acting.
- ReAct interleaves verbal reasoning traces and task-specific actions so models can create, maintain, and adjust plans while incorporating external information.
- ReAct is evaluated on HotPotQA, Fever, ALFWorld, and WebShop against state-of-the-art baselines.
- On ALFWorld and WebShop, one- or two-shot ReAct improves absolute success rates over imitation or reinforcement learning methods by 34% and 10%, respectively.
2 REAC T: SYNERGIZING REASONING + ACTING
ReAct augments an agent’s action space with language-based thoughts that update context without changing the external environment. A frozen language model generates thoughts and domain-specific actions from few-shot human trajectories, with thought frequency adapted to the task.
- ReAct adds language thoughts to the action space; thoughts do not affect the environment but update context for future reasoning and acting.
- Thoughts can decompose goals, create plans, inject commonsense, extract observations, track progress, and handle exceptions.
- The method prompts frozen PaLM-540B with few-shot human trajectories containing actions, thoughts, and environment observations.
- For reasoning-heavy tasks, ReAct alternates thoughts, actions, and observations across multiple steps.
- The unlimited language action space makes learning difficult and requires strong language priors.
- For decision-making tasks with many actions, the model determines when thoughts occur, allowing them to appear asynchronously at relevant trajectory positions.
3 KNOWLEDGE-INTENSIVE REASONING TASKS
ReAct combines Wikipedia interaction with interleaved reasoning for knowledge-intensive question answering and fact verification. Its strongest results come from combining ReAct with CoT-SC and from fine-tuning ReAct trajectories, while retrieval quality and structural constraints remain important limitations.
- Methods: ReAct uses search, lookup, and finish actions to retrieve Wikipedia evidence while interleaved thoughts decompose questions, extract information, reformulate searches, and synthesize answers.The Wikipedia API returns entity-page sentences, supports string lookup, and terminates with a final answer.
- Results and Observations: ReAct outperforms Act on both HotpotQA and Fever, showing the value of reasoning traces for guiding actions and synthesizing final answers.Fine-tuning results also confirm the benefit of reasoning traces for more informed acting.
- Results and Observations: 60.9 vs. 56.3: ReAct outperforms CoT on Fever, but 27.4 vs. 29.4: ReAct slightly lags CoT on HotpotQA.The paper attributes Fever’s pattern to the importance of retrieving accurate, up-to-date knowledge for closely differing claims.
- Results and Observations: 14% vs. 6%: CoT has a higher false positive rate than ReAct in success-mode examples, while hallucination constitutes 56% of CoT’s major failure mode.The authors characterize ReAct trajectories as more grounded because they access an external knowledge base.
- Results and Observations: ReAct →CoT-SC and CoT-SC →ReAct are the best prompting methods on HotpotQA and Fever respectively, reaching 21-sample CoT-SC performance with 3-5 samples.The results support combining internal model knowledge with externally retrieved knowledge.
- Results and Observations: With only 3,000 examples, fine-tuned ReAct becomes best among the four methods: PaLM-8B and PaLM-62B ReAct outperform larger prompting baselines.The paper notes that all prompting methods remain significantly below domain-specific state-of-the-art approaches and suggests more human-written fine-tuning data.
4 DECISION MAKING TASKS
ReAct is evaluated on interactive decision-making tasks requiring long-horizon planning, exploration, and state tracking. It outperforms action-only and prior learning-based methods on ALFWorld and WebShop, while remaining below expert-human performance on WebShop.
- ALFWorld: ALFWorld tasks require agents to pursue high-level goals across environments with more than 50 locations and potentially more than 50 steps.The benchmark tests subgoal planning, state tracking, systematic exploration, and commonsense-based item-location reasoning.
- WebShop: WebShop contains 1.18M products and 12k human instructions, requiring purchases from noisy structured and unstructured product information.The task differs from ALFWorld by emphasizing varied real-world product text and instruction matching.
- ALFWorld: 71% average success was achieved by the best ReAct trial on ALFWorld, versus 45% for Act and 37% for BUTLER.Even the worst ReAct trial reached 48%, exceeding the best trials of both comparison methods.
- ALFWorld: ReAct’s ALFWorld advantage over Act was consistent across six controlled trials, with relative gains from 33% to 90% and a 62% average.The reported qualitative explanation is that action-only prompting fails to decompose goals or track the environment state.
- WebShop: 10% absolute success-rate improvement over the previous best was achieved by ReAct on WebShop with additional sparse reasoning.ReAct more often identified instruction-relevant products and options by bridging noisy observations and actions.
- WebShop: ReAct remains far from expert-human performance on WebShop, where experts conduct more product explorations and query reformulations.These exploration and reformulation behaviors remain challenging for prompting-based methods.
- Reasoning and feedback: ReAct is described as the first LLM demonstration combining reasoning and action in an interactive environment within a closed-loop system.Its sparse reasoning traces are more flexible than Inner Monologue’s feedback limited to environment state and goal completion.
- Reasoning and feedback: 71 versus 53 overall success rate favored ReAct over IM-style prompting, with advantages on five of six ALFWorld tasks.The comparison attributes IM-style errors to difficulty identifying completed subgoals and selecting the next one.
5 RELATED WORK
Related work covers language models for reasoning and for interactive decision making. ReAct is positioned alongside chain-of-thought methods and LLM-based planning systems such as WebGPT, SayCan, and Inner Monologue.
- Language model for reasoning: Chain-of-Thought and related prompting methods use language models to formulate multi-step thinking procedures for problem solving.The cited follow-up approaches include least-to-most prompting, zero-shot-CoT, and self-consistency.
- Language model for decision making: WebGPT uses a language model to navigate web pages and infer answers to complicated questions from ELI5.The passage positions WebGPT within language-model decision-making work and contrasts it with ReAct’s explicit reasoning traces.
- Language model for decision making: SayCan prompts LLMs to predict robot actions and reranks them with an affordance model grounded in visual environments.Inner Monologue extends this direction by injecting feedback from the environment.
- Language model for decision making: Inner Monologue is presented as prior closed-loop work, while ReAct argues that its feedback does not constitute flexible inner thoughts.ReAct distinguishes its reasoning traces as flexible and sparse rather than limited to environment observations and goal completion.
6 CONCLUSION
ReAct combines reasoning and acting for multi-hop question answering, fact checking, and interactive decision-making, while producing interpretable decision traces. Its scaling remains constrained by demonstration length, and stronger results may require more human annotations.
- ReAct combines reasoning and acting in large language models across question answering, fact checking, and interactive decision-making tasks.
- ReAct produces interpretable decision traces alongside its reported performance gains.
- Complex tasks with large action spaces may require more demonstrations than fit within in-context learning limits.
- Human feedback is described as complementary to ReAct, but its incorporation is left for future work.
- Initial fine-tuning results are promising, but more high-quality human annotations are identified as desirable for further improvement.
REPRODUCIBILITY STATEMENT
The paper reports experiments primarily on a non-open PaLM model and provides prompts, GPT-3 experiments, and prompting code to improve reproducibility. Its external interactions were restricted to low-risk benchmark settings.
- The main experiments use PaLM, which is not openly accessible.
- The authors provide all used prompts, additional GPT-3 experiments, and associated GPT-3 ReAct prompting code to increase reproducibility.
- Experiments limit interactions to Wikipedia and WebShop, excluding private information and dangerous actions such as purchasing products or editing Wikipedia.
A.1 GPT-3 EXPERIMENTS
Additional GPT-3 experiments examine ReAct across models, while analyses illustrate its behavior on outdated labels and human-edited trajectories. These results support cross-model effectiveness and inspectability within the reported settings.
- Table 5 compares PaLM-540B with GPT-3 text-davinci-002 using greedy decoding on HotpotQA and ALFWorld validation instances.
- ReAct prompting is effective across different large language models on different tasks.GPT-3 consistently outperforms PaLM-540B on HotpotQA and ALFWorld in the reported comparison.
- ReAct can obtain an up-to-date HotpotQA answer through real-world web interaction and reasoning when the original dataset label is outdated.
- Editing two ReAct thoughts can change a failed ALFWorld trajectory into one that succeeds.The example removes a hallucinating thought and adds hints.
- Thought editing can modify model behavior beyond changing actions, including internal beliefs or reasoning styles within the flexible thought space.
B EXPERIMENT DETAILS
Fine-tuning uses different training-step schedules across model sizes and method families, while an ablation restricts ReAct-IM to a narrow set of externally guided thoughts.
- On PaLM-8B, ReAct and Act are fine-tuned for 4,000 steps, while Standard and CoT are fine-tuned for 2,000 steps.
- On PaLM-62B, ReAct and Act are fine-tuned for 4,000 steps, while Standard and CoT are fine-tuned for 1,000 steps.
- ReAct and Act generally benefit from more training steps and data, whereas Standard and CoT degrade soon after fine-tuning.
- ReAct-IM is limited to thoughts that decompose the current goal and identify the current subgoal.
C PROMPTS
The prompts interleave ReAct thoughts with environment actions, using external observations to refine reasoning and complete diverse tasks. Examples span HotPotQA question answering and FEVER fact verification.
- C.1 HOTPOTQA: HotPotQA trajectories use Wikipedia searches and lookups to connect entities before producing answers such as shared professions, publication order, and namesakes.Examples include Nicholas Ray and Elia Kazan, Arthur’s Magazine and First for Women, and Milhouse’s namesake.
- C.1 HOTPOTQA: The Act-style HotPotQA examples search entities and finish directly, while chain-of-thought examples reason internally without new observations.The supplied trajectories show action-only entity searches and standalone step-by-step answers.
- C.1 HOTPOTQA: ReAct prompts alternate free-form thoughts, searches, lookups, observations, and final actions to refine task-solving trajectories.Thoughts can decompose goals, extract information, and identify missing evidence.
- C.1 HOTPOTQA: ReAct’s Colorado-orogeny trajectory searches the topic, looks up the eastern sector, retrieves the High Plains elevation range, and finishes with 1,800 to 7,000 ft.The High Plains observation states an east–west rise from around 1,800 to 7,000 ft.
- C.1 HOTPOTQA: The prompts use multiple thought-action-observation steps, with free-form thoughts supporting decomposition, evidence extraction, and decisions about further searches.For HotPotQA and Fever, the paper manually composes few-shot ReAct-format trajectories from training cases.
- C.2 FEVER: A Nikolaj Coster-Waldau FEVER trajectory begins by searching the person and uses the retrieved biography to investigate the claim about Fox Broadcasting Company.Both supplied passages show the search and observation sequence for the same claim.
D.1 FEVER TRAJECTORIES
The trajectories illustrate how interleaved reasoning helps ReAct recover task structure and use observations, whereas action-only or feedback-only variants can repeat errors or lose track of subgoals.
- FEVER: FEVER ReAct examples use observations to classify claims as REFUTES or SUPPORTS, including claims about the Bermuda Triangle and Princess Mononoke.The trajectories explicitly state the retrieved evidence and final labels.
- ALFWorld: ReAct correctly finds, cleans, and places the knife on a countertop in the ALFWorld example.The trajectory is presented as a successful ReAct run for the specified knife-cleaning task.
- ALFWorld: Act finds the knife but attempts cleaning before reaching the sinkbasin, then becomes stuck repeating commands.The failure is linked to missing explicit subgoal tracking in the trajectory.
- ALFWorld: ReAct’s existing thought identifies that the knife was taken and that going to a sinkbasin is the next cleaning subgoal.This connects the immediate action sequence to a concrete high-level subgoal.
- ALFWorld: ReAct-IM finds the knife but cannot clean it and then becomes stuck trying to place it on a countertop.The supplied description attributes the issue partly to a thought suggesting that the knife was already clean.
- WebShop: ReAct uses reasoning to find WebShop products that satisfy all target attributes, unlike the Act comparison shown in the trajectory table.The table caption frames the comparison as Act versus ReAct trajectories.
E.1 SUCCESS AND FAILURE MODES ANALYSIS
The success and failure examples show that ReAct can connect searches and intermediate reasoning, but still encounters search, reasoning, hallucination, and label-ambiguity errors.
- Success: A successful ReAct trajectory searches David Chanoff, identifies William J. Crowe, and answers Bill Clinton from Crowe’s ambassadorship.The example contrasts this with a separate chain-of-thought answer about the Arizona State Sun Devils.
- False positive: A false-positive ReAct trajectory concludes San Marco came first after failing to find About Time and inferring it was a 2013 film.The example’s paired chain-of-thought answer correctly compares Vanderbilt and Emory founding dates.
- Reasoning error: A reasoning-error example shows ReAct failing on a question asking for an actress’s middle name after searching a long cast list.The supplied passage identifies the failure category but truncates the trajectory.
- Search error: A search error occurs when the query “goddess frigg” is not found and the trajectory must redirect to “Frigg.”The search result provides similar entries, including Frigg and related figures.
- Hallucination: A chain-of-thought hallucination answers 1916 for a journal-year question whose label is 1909.The example explicitly marks the generated answer as incorrect against the label.
- Label ambiguity: A label-ambiguity example searches for the developer of the dictator game and identifies Amos Tversky as Israeli, but the supplied trajectory is truncated.The passage explicitly categorizes this case as label ambiguity.