PaperlayerView exact revision on arXiv

Source-linked AI summary

SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks

Bill Yuchen Lin, Yicheng Fu, Karina Yang, Faeze Brahman, Shiyu Huang, Chandra Bhagavatula, Prithviraj Ammanabrolu, Yejin Choi, Xiang Ren

arXiv:2305.17390v2cs.CLcs.AIcs.LGcs.MAcs.RO

TL;DR

Complex interactive tasks require planning, memory, subgoal decomposition, and exception handling beyond the strengths of many existing agents. SwiftSage combines a fast behavior-cloning module with a deliberate LLM-based planning and grounding module, achieving state-of-the-art ScienceWorld performance across 30 task types while reducing inference cost. Its evidence is limited to a textual simulator with restricted actions and tasks, and LLM-based SAGE may pose scalability challenges.

  • Problem

    Complex interactive tasks require long-horizon planning, long-term memory, subgoal decomposition, commonsense knowledge, and exception handling.

  • Method

    SwiftSage combines SWIFT, a small behavior-cloned encoder-decoder LM, with SAGE, an LLM module for subgoal planning, grounding, and exception correction.

  • Results

    84.7 average score across 30 ScienceWorld task types outperforms SAYCAN at 33.8, REACT at 36.4, and REFLEXION at 45.3.

  • Takeaways & Limitations

    SwiftSage provides state-of-the-art performance with increased efficiency and reduced cost relative to baseline agents.

  • Takeaways & Limitations

    Evaluation is limited to the textual ScienceWorld simulator, and SAGE's LLM use may create scalability challenges due to substantial computational requirements.

Abstract

from arXiv · show

We introduce SwiftSage, a novel agent framework inspired by the dual-process theory of human cognition, designed to excel in action planning for complex interactive reasoning tasks. SwiftSage integrates the strengths of behavior cloning and prompting large language models (LLMs) to enhance task completion performance. The framework comprises two primary modules: the Swift module, representing fast and intuitive thinking, and the Sage module, emulating deliberate thought processes. The Swift module is a small encoder-decoder LM fine-tuned on the oracle agent's action trajectories, while the Sage module employs LLMs such as GPT-4 for subgoal planning and grounding. We develop a heuristic method to harmoniously integrate the two modules, resulting in a more efficient and robust problem-solving process. In 30 tasks from the ScienceWorld benchmark, SwiftSage significantly outperforms other methods such as SayCan, ReAct, and Reflexion, demonstrating its effectiveness in solving complex interactive tasks.

1 Introduction

Complex interactive tasks require long-horizon planning, memory, subgoal decomposition, and exception handling, capabilities that existing approaches often lack. SWIFTSAGE combines fast behavior cloning with deliberate LLM reasoning and achieves strong ScienceWorld performance with greater efficiency.

  • Motivation: ScienceWorld tasks require agents to navigate dynamic environments, acquire items, perform experiments, and interpret results over long horizons.The benchmark includes tasks such as determining an unknown object's electrical conductivity.
  • Limitations of prior methods: Existing RL and behavior-cloning approaches struggle with subgoal decomposition, long-term memory, unseen tasks, and environment-specific exceptions.Prompted LLMs can reason and plan, but prior methods remain costly because they infer a new action at every time step.
  • Approach: SWIFTSAGE integrates behavior cloning as fast System 1 thinking with LLM prompting as deliberate System 2 reasoning.The framework is inspired by dual-process theory and targets both task-completion performance and efficiency.
  • Approach: The SWIFT module is a small encoder-decoder LM trained on oracle trajectories, while SAGE uses LLMs for planning, grounding, subgoals, and exception correction.SAGE converts planned subgoals into action sequences rather than generating only the next immediate action.
  • Results: 84.7 average score across 30 ScienceWorld task types establishes SWIFTSAGE as state of the art against SAYCAN at 33.8, REACT at 36.4, and REFLEXION at 45.3.The framework also requires fewer tokens per action for LLM inference than prior methods.

2 Background and Related Work

Complex interactive reasoning differs from simpler object-manipulation tasks through larger action spaces, longer planning horizons, and unforeseen obstacles. Prior RL, imitation-learning, and prompting methods provide complementary capabilities but face limitations in generalization, cost, annotation, or recoverability.

  • Interactive reasoning: Interactive reasoning involves accomplishing goals within interactive environments such as AI2Thor, TextWorld, and ScienceWorld.The paper focuses on ScienceWorld's textual environment and complex tasks.
  • Complex interactive reasoning: ScienceWorld contains 10 locations, 200 object types with varying states, and 25 action types, alongside random obstacles requiring adaptation and replanning.Examples include broken stoves and missing soil that force agents to seek alternative resources.
  • Reinforcement and imitation learning: DRRN learns separate observation and action representations and selects candidate actions using feedback from the simulated environment.CALM combines DRRN with a causal language model trained on oracle transcripts.
  • Reinforcement and imitation learning: KG-A2C represents environment states as dynamically updated graphs that constrain combinations of action templates and objects.The approach uses OpenIE to construct these graph representations.
  • Reinforcement and imitation learning: Behavior cloning trains offline seq2seq Transformers on oracle action transcripts to predict the next action from prior actions and observations.This approach learns from similar training tasks generated by oracle agents.
  • Prompting LLMs: SAYCAN, REACT, and REFLEXION use LLMs for action planning, but their deployment has limitations involving repeated inference, annotations, or irreversible trial and error.All three require a new LLM inference at each time step, while REACT and REFLEXION require subgoal annotations for unseen task types.
  • Dual-process theory: Dual-process theory motivates combining a small seq2seq model for fast associative reasoning with prompted LLMs for slower analytical reasoning.The paper uses this complementary design to address diverse interactive challenges.

3 SWIFTSAGE: A Generative Agent with Fast and Slow Thinking

SWIFTSAGE combines a fast imitation-learned action module with a slower LLM-based module for planning, grounding, and exception handling in complex interactive tasks. The framework uses longer action history, visited-room context, and buffered multi-action outputs to improve efficiency and robustness.

  • Integration and Exception Handling: The framework switches from SWIFT to SAGE when exceptions occur, addressing SWIFT’s difficulty with unseen situations and meaningless repeated actions.SWIFT is more accurate initially but can fail when learned plans encounter environmental exceptions, such as a broken stove.
  • SWIFT: The Module for Intuitive and Associative Thinking via Imitation Learning: SWIFT is a T5-large encoder-decoder module trained on oracle trajectories to encode recent context and decode the next individual action.Its input includes previous actions, observations, rewards, visited rooms, and the current environment state.
  • SWIFT: The Module for Intuitive and Associative Thinking via Imitation Learning: SWIFT uses a sliding window of K = 10 recent actions and a deduplicated visited-room field to provide longer context and reduce unnecessary navigation.The history includes observations and rewards associated with recent actions.
  • SAGE: The Module for Deliberate and Analytical Thinking via Prompting LLMs: SAGE separates LLM-based planning from grounding, translating higher-level recommendations into specific action sequences for planned subgoals.The planning stage asks about needed objects, uncollected objects, subgoals, progress, and exceptions.
  • SAGE: The Module for Deliberate and Analytical Thinking via Prompting LLMs: SAGE incorporates planning outputs and the recent 10-step action history, then generates an action buffer containing multiple actions rather than only one.Answers to Q1-Q3 improved short-task performance by about 2 points on average in a small-scale ablation.

4 Evaluation

Evaluation on ScienceWorld compares SWIFTSAGE with conventional and LLM-based agents across task lengths, efficiency, exception handling, and trajectory progress. SWIFTSAGE combines strong performance with lower inference cost and more efficient progress, while baseline behavior varies by task length and method.

  • Evaluation Setup: ScienceWorld evaluation compares agents across 30 task types grouped by short, medium, and long oracle trajectory lengths.The benchmark setup also includes prompting baselines such as SAYCAN, REACT, and REFLEXION.
  • Results and analysis: 49.22 is the overall performance of the 770m SWIFT-only agent, which the authors attribute to balanced training data and a sliding window for longer action histories.TDT performs on par with DRRN while using a larger 11b behavior-cloning model.
  • Results and analysis: REACT improves over SAYCAN on short and medium tasks, while REFLEXION surpasses REACT on shorter tasks but receives up to four rounds instead of one.The authors note that this makes REFLEXION comparisons less fair, especially for multiple-choice tasks.
  • Results and analysis: SWIFTSAGE’s plan-and-ground prompts handle environment exceptions more effectively than independently used SWIFT and rarely grounded corrective actions from REACT or REFLEXION.The example concerns a broken stove that causes some agents to repeat meaningless actions or move aimlessly.
  • Results and analysis: 757.07 tokens per action is SWIFTSAGE’s cost, versus 1,855.84 for SAYCAN, 1,971.03 for REACT, and 2,983.46 for REFLEXION.The lower cost is associated with producing action sequences, invoking LLMs only when needed, and using an action buffer.
  • Results and analysis: SWIFTSAGE reaches scores similar to oracle trajectories more efficiently, especially on longer tasks, whereas REACT often plateaus below 100.SWIFTSAGE still falls short of a perfect score on some tasks, including 9-2 and 1-3.

5 Conclusion

The paper concludes that SWIFTSAGE combines a fast specialized smaller LM with LLM-based deliberate planning for complex interactive reasoning. Its ScienceWorld results support this collaborative design, while the authors identify simulator scope, safety, scalability, and switching decisions as limitations and future directions.

  • Contributions: SWIFTSAGE combines SWIFT for fast thinking with SAGE for slow thinking and outperforms baseline agents on 30 ScienceWorld tasks.The framework is also reported to improve efficiency and reduce cost.
  • Implications: Smaller LMs can learn task- and environment-specific patterns, while LLMs provide zero-shot generalization and deliberate thinking.The authors frame these capabilities as complementary within collaborative agent frameworks.
  • Limitations: The evaluation is limited to the textual ScienceWorld simulator, whose actions and tasks are narrower than real-world situations.The paper also reports absent safeguards for potentially hazardous real-world actions and scalability concerns from SAGE’s LLM requirements.

A Dataset Statistics

The dataset statistics describe ScienceWorld’s task splits, trajectory-length categories, sampled evaluation variations, and the data used to train SWIFT. The authors down-sample selected tasks and less informative actions to reduce imbalance and improve imitation learning.

  • Dataset Statistics: Task 9-x, 10-x, and 3-3 are down-sampled because their large datasets caused significant training-data imbalance.Less informative actions, such as closing a kitchen door, are also down-sampled for imitation learning.
  • Dataset Statistics: 270 variations are evaluated after limiting tasks with more than 10 test variations to their first 10 variations.If a final action produces an irrecoverable negative score, evaluation uses the last non-negative score.
  • Dataset Statistics: Table 2 reports oracle trajectory length, down-sampled variation counts for each split, and action-prediction seq2seq training-data counts for SWIFT.These statistics characterize both evaluation coverage and the supervised training set.

B.1 Training Details of SWIFT

SWIFT is trained as a compact action-prediction model, while SAGE uses staged prompting and memory augmentation to plan and ground actions. Heuristics control when SAGE activates and how invalid actions are handled.

  • B.1 Training Details of SWIFT: SWIFT uses flan-t5-large (770m), fine-tuned on 62k seq2seq action-prediction examples for 500 training steps.The reported learning rate is 1e-4 and batch size is 128; larger models produced only marginal gains at higher training cost.
  • B.2 Prompting in SAGE: SAGE uses a two-stage prompting framework comprising planning and grounding.The LLM output from planning can be reused as grounding input, with Q1-Q3 optionally omitted to reduce computation.
  • B.2 Prompting in SAGE: Memory augmentation supplies previously observed objects and location-tagged action history to SAGE during planning and grounding.This supplements the agent’s current-location perception with information from visited locations.
  • B.2 Prompting in SAGE: SAGE grounds plans through permitted action templates such as TELEPORT, PICK, OPEN, ACTIVATE, EXAMINE, and MOVE.The templates convert planned operations into executable action forms.
  • B.2 Prompting in SAGE: Invalid LLM-generated actions are discarded, and the system can revert to SWIFT when necessary.SAGE activation also restricts SWIFT from performing critical “focus on” actions, while consecutive invalid actions or exceptions trigger reversion.

C Additional Results and Analysis

Replacing GPT-4 with GPT-3.5-turbo reduces competing methods’ performance substantially, whereas SWIFTSAGE retains stronger performance and robustness. The paper notes input and output length limits as a practical constraint for current open-source LLMs.

  • C.1 Sensitivity to LLMs: GPT-3.5-turbo vs GPT-4: SWIFTSAGE scores 62.22 with GPT-3.5-turbo, while ReAct falls from 36.43 to 19.76.The reported contrast indicates greater robustness for SWIFTSAGE under the less expensive LLM.
  • C.1 Sensitivity to LLMs: GPT-3.5-turbo vs GPT-4: Current open-source LLMs impose more restrictive input and output length limits, constraining planned extensions to open-source models.The authors plan to investigate open-source LLMs and distillation from closed-source models.

C.2 Efficiency Analysis

Trajectory visualizations show SWIFTSAGE reaching higher scores more quickly than oracle agents, while ReAct often plateaus before a perfect score. Oracle agents still consistently achieve 100.0.

  • C.2 Efficiency Analysis: Most SWIFTSAGE trajectories lie near the top-left, indicating higher scores than oracle agents at a faster rate.The figure’s efficiency interpretation uses score progression over time steps.
  • C.2 Efficiency Analysis: ReAct is competitive on shorter tasks but typically plateaus at intermediate scores without reaching 100.The comparison is based on the trajectories shown in Figure 5.
  • C.2 Efficiency Analysis: Oracle agents consistently achieve a perfect score of 100.0, but SWIFTSAGE often has better efficiency on longer tasks.The passage distinguishes final score from the speed of reaching scores.

C.3 Cost-effectiveness

SWIFTSAGE is presented as more cost-effective than other LLM-based methods because it uses fewer tokens per action while achieving higher scores. Its cost analysis reports token and score efficiency across ScienceWorld tasks.

  • C.3 Cost-effectiveness: 757.07 tokens per action is reported for SWIFTSAGE, versus 1,855.84 for SAYCAN, 1,971.03 for REACT, and 2,983.46 for REFLEXION.The analysis covers tokens per action and scores per action across all tasks.
  • C.3 Cost-effectiveness: SWIFTSAGE remains lower-cost despite SAGE invoking LLM APIs twice because each invocation produces an action sequence of about 5 actions.The action buffer and selective LLM invocation are identified as contributors to efficiency.
  • C.3 Cost-effectiveness: Table 3 compares SAYCAN, REACT, REFLEXION, and SWIFTSAGE using GPT-3.5-turbo and additionally reports SWIFT-only results.The table is an additional ScienceWorld benchmark analysis distinct from the GPT-4 evaluation.
  • C.3 Cost-effectiveness: Long tasks have lower SWIFTSAGE tokens per action than medium and short tasks, associated with longer action buffers and more effective SWIFT behavior.SWIFTSAGE also achieves higher scores per action while using fewer tokens.
Loading 2305.17390v2…