PaperlayerView exact revision on arXiv

Source-linked AI summary

Executable Code Actions Elicit Better LLM Agents

Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, Heng Ji

arXiv:2402.01030v4cs.CLcs.AI

TL;DR

Existing text- and JSON-based action formats constrain LLM agents’ action spaces and flexibility, especially for composing tools. CodeAct uses executable Python actions in multi-turn interactions, and evaluation across 17 LLMs found higher success for most models, while CodeActInstruct and CodeActAgent extend this approach to agent training and sophisticated tasks.

  • Problem

    Text- and JSON-based actions typically constrain LLM agents’ action spaces and restrict flexible composition of multiple tools.

  • Method

    CodeAct consolidates agent actions into executable Python code within a multi-turn framework, supported by CodeActInstruct trajectories and a Python-integrated CodeActAgent.

  • Results

    CodeAct achieved higher task success for 12 of 17 evaluated LLMs; gpt-4-1106-preview improved success by 20.7% over text while using 2.1 fewer interaction turns on average.

  • Takeaways & Limitations

    CodeActAgent can perform sophisticated tasks with existing Python packages and autonomously rectify errors through self-debugging in multi-turn interaction.

  • Takeaways & Limitations

    CodeActAgent may hallucinate, such as imagining a variable’s contents without printing it, indicating a need for further alignment.

Abstract

from arXiv · show

Large Language Model (LLM) agents, capable of performing a broad range of actions, such as invoking tools and controlling robots, show great potential in tackling real-world challenges. LLM agents are typically prompted to produce actions by generating JSON or text in a pre-defined format, which is usually limited by constrained action space (e.g., the scope of pre-defined tools) and restricted flexibility (e.g., inability to compose multiple tools). This work proposes to use executable Python code to consolidate LLM agents' actions into a unified action space (CodeAct). Integrated with a Python interpreter, CodeAct can execute code actions and dynamically revise prior actions or emit new actions upon new observations through multi-turn interactions. Our extensive analysis of 17 LLMs on API-Bank and a newly curated benchmark shows that CodeAct outperforms widely used alternatives (up to 20% higher success rate). The encouraging performance of CodeAct motivates us to build an open-source LLM agent that interacts with environments by executing interpretable code and collaborates with users using natural language. To this end, we collect an instruction-tuning dataset CodeActInstruct that consists of 7k multi-turn interactions using CodeAct. We show that it can be used with existing data to improve models in agent-oriented tasks without compromising their general capability. CodeActAgent, finetuned from Llama2 and Mistral, is integrated with Python interpreter and uniquely tailored to perform sophisticated tasks (e.g., model training) using existing libraries and autonomously self-debug.

1. Introduction

CodeAct uses executable Python as a general-purpose action format, addressing the limited scope and flexibility of text or JSON actions. Experiments and the resulting agent show benefits on complex tool-use tasks, multi-turn interaction, and software-assisted problem solving.

  • Text and JSON actions often constrain tool scope and prevent composing multiple tools in one action.
  • CodeAct lets LLMs generate executable Python that can dynamically revise actions from observations during multi-turn interaction.The framework integrates a Python interpreter and supports code execution against environmental feedback.
  • Python control and data flow support intermediate variables, tool composition, loops, and conditional logic within one action.
  • Up to 20% absolute success-rate improvement and 30% fewer actions were achieved over baselines on complex multi-tool tasks.The benchmark contains 82 human-curated tasks requiring multiple tool calls across multi-turn interactions.
  • CodeActInstruct contains 7k multi-turn trajectories, and CodeActAgent improves agent-task performance without compromising general capabilities.The agent is finetuned from LLaMA-2 and Mistral-7B.
  • CodeActAgent can use existing Python packages for sophisticated tasks and autonomously self-debug from software error messages.

2. CodeAct Makes LLMs Better Agents

The paper evaluates CodeAct as a Python-based action format in atomic and complex tool-use settings, then situates it within a multi-turn agent framework. CodeAct generally matches or exceeds alternatives, with larger gains on complex tasks and fewer interactions.

  • Evaluation setup: The evaluation examines CodeAct against text and JSON across atomic API calls and complex multi-tool, multi-turn tasks.The study analyzes 17 off-the-shelf LLMs and asks whether code familiarity and control/data flow provide advantages.
  • CodeAct framework: CodeAct represents each environment action as Python code and returns execution results or errors as observations for subsequent turns.
  • Atomic tool use: CodeAct achieves comparable or better atomic API-call performance for most LLMs, with stronger gains among open-source models.The comparison uses API-Bank correctness by matching ground-truth and executed API outputs.
  • M3ToolEval: M3ToolEval contains 82 human-curated instances requiring multiple calls to multiple tools in multi-turn interactions.
  • Complex tool use: CodeAct has higher task success rates for 12 of 17 LLMs and lower average turns for 12 of 17 on M3ToolEval.The benchmark reports success rate and average turns per instance.
  • Complex tool use: gpt-4-1106-preview gains 20.7% absolute success rate over text while requiring 2.1 fewer interaction turns on average.
  • Agent capabilities: CodeActAgent integrates Python libraries such as Pandas, Scikit-Learn, and Matplotlib, using error messages to self-debug across turns.

3. Empowering Open-source LLM Agent to be Better at CodeAct

The authors build CodeActAgent by instruction-tuning open-source LLMs on CodeActInstruct and general conversations, targeting multi-turn agent-environment interaction. The resulting agents perform strongly on CodeAct tasks, generalize to text actions, and largely preserve general LLM capabilities.

  • 3.1. CodeActInstruct: Agent-Environment Interactions: CodeActInstruct contains 7k multi-turn agent-environment interaction trajectories, combined with general conversation data for training CodeActAgent.The dataset covers computer-oriented interactions, physical-world robot planning, and selected challenging instances.
  • 3.1. CodeActInstruct: Agent-Environment Interactions: The training data promotes improvement from interaction by retaining trajectories where agents encounter errors and later rectify them through self-debugging.This selection targets behavior based on prior observations, including reflection after execution errors.
  • 3.1. CodeActInstruct: Agent-Environment Interactions: CodeActInstruct contains 3.8x and 3.5x more trajectories, and 5x and 19x more tokens, than AgentInstruct and FireAct, respectively.The comparison also emphasizes broader domains, quality data, and direct interaction with Python interpreters and open-source toolkits.
  • 3.2. CodeActAgent: CodeActAgent is fine-tuned from Llama-2 7B and Mistral 7B using mixtures of CodeActInstruct and general conversations.The evaluation uses MINT, MiniWob++, ScienceWorld, and general LLM tasks, with MINT success rates measured at interaction turn k = 5 unless otherwise specified.
  • 3.2. CodeActAgent: CodeActAgent variants outperform evaluated open-source LLMs on in-domain and out-of-domain MINT subsets, while the Mistral variant also performs strongly on M3ToolEval.The Llama-2 variant shows no improvement on M3ToolEval, and CodeActAgent generalizes to text actions despite not being optimized for them.
  • 3.2. CodeActAgent: Both CodeActAgent variants maintain or improve performance on tested general LLM tasks, except for slight MMLU degradation with CodeActAgent (Mistral, 7B).Ablations indicate that CodeActInstruct and general conversations both contribute to agent-task performance, while general conversations help maintain general-task performance.

4. Related Work

Prior work improves LLM agents through prompting and instruction tuning, while agent architectures also emphasize profiles, memory, reasoning, planning, and action modules. These approaches include reflection, structured reasoning, and task-specific expert demonstrations.

  • 4. Related Work: LLM-based autonomous agents commonly combine customized profiles, long-term memory, reasoning and planning algorithms, and action modules.Action modules facilitate interaction with external entities such as humans and tools.
  • 4. Related Work: Prompt engineering methods improve reasoning through chain-of-thought, self-consistency, tree-based approaches, and reflection on previous plans.Reflection can support refining initial actions through trial and error.
  • 4. Related Work: Instruction tuning enhances models intrinsically, including their agent capabilities, often using human-curated demonstrations for specific tasks such as web browsing.The cited prior work contrasts broad instruction tuning with task-focused expert demonstrations.

5. Conclusions

The paper introduces CodeAct, CodeActInstruct, and CodeActAgent to support executable-code actions, multi-turn interaction, software reuse, and autonomous error correction. The approach is presented as advantageous for sophisticated agent tasks.

  • 5. Conclusions: CodeAct uses executable Python code for LLM-agent actions, while CodeActInstruct provides multi-turn interaction trajectories for instruction tuning.CodeActAgent is designed to integrate with Python, use existing packages, perform sophisticated tasks, and rectify errors autonomously.

Impact Statement

The paper aims to advance autonomous agents that communicate naturally and perform tasks in environments, while acknowledging prototype limitations and safety concerns.

  • CodeActAgent is an initial autonomous-agent prototype with practical limitations.
  • Hallucinations, such as imagining a variable’s contents without printing it, motivate further alignment.
  • CodeActAgent demonstrates limited self-improvement through error-message-based self-debugging and can interact with environments.
  • Freely executable code in a sandbox could potentially escape sandbox restrictions and cause harm through cyber-attack, motivating stronger safety mechanisms.

A. Comparison with Work that Uses Code Generation for Problem-solving

The paper distinguishes CodeAct from prior code-generation approaches by enabling multi-turn adjustment based on observations and supporting diverse tasks with less prompt engineering.

  • Dynamic interaction: Most prior code-generation approaches produce static action sequences in one turn and cannot adjust actions after new observations.
  • Dynamic interaction: CodeAct supports multi-turn action adjustment or new actions using textual environmental observations such as execution output and error feedback.
  • Dynamic interaction: CodeActInstruct collects data specifically for multi-turn self-improvement, including learning from interaction feedback.
  • Task generality: Prior approaches often require heavy prompt engineering and few-shot demonstrations for domain-specific tasks.
  • Task generality: CodeAct uses executable Python to unify actions, while CodeActInstruct covers diverse tasks to support scaling across domains with minimal human effort.
  • Related approaches: Voyager iteratively fixes code errors within constrained function definitions, whereas CodeAct allows dynamic adjustment of atomic or composed actions during execution.
  • Related approaches: OpenCodeInterpreter improves iterative debugging for competitive code-generation questions, but its applicability to general LLM-agent tasks remains unknown.

B. Comparison with TaskWeaver

Compared with TaskWeaver, this work moves beyond conceptual demonstrations by rigorously evaluating code actions and releasing resources designed to improve practical LLM-agent capabilities.

  • TaskWeaver’s exploration relied on limited qualitative examples and closed-source backbone models, remaining primarily conceptual.
  • This work conducts extensive analysis that quantitatively evaluates the benefits of code actions in LLM agents.
  • The paper introduces CodeActInstruct and the open-source CodeActAgent to strengthen code-based action execution and support future open-source research.

C. General Data Down-sample

The paper describes dataset-specific down-sampling procedures for ShareGPT, OpenOrca, and CapyBara resources.

  • ShareGPT removes single-turn conversations and randomly sub-samples to the desired final size.
  • ShareGPT (GPT-4) is not sub-sampled.
  • OpenOrca selects its chain-of-thought subset and then randomly sub-samples to the desired final size.
  • CapyBara is not sub-sampled.

D. CodeActAgent Training Details

The training setup uses LLaMA-2 and Mistral with specified optimization settings, while deployment prompts define Python-interpreter interaction and alternative tool-call formats.

  • Training configuration: SFT runs use a fork of Megatron-LLM on one 4xA100 40GB SXM node, with approximately 9k tokens per second throughput.Training uses ChatML, assistant-response-only loss, instance packing, and flash attention.
  • Training configuration: LLaMA-2 and Mistral are trained for five epochs with learning rate 1e-5, batch size 32, and the third-epoch checkpoint used for experiments.Tensor parallelism is 4, with 50 warmup steps, cosine decay, and an ending learning rate of 1e-6.
  • Deployment prompt: The deployment prompt gives the assistant access to an interactive Python environment and requires executable code to be enclosed in execute tags.It also instructs the assistant to install packages when needed, define imports and variables first, and stop executing after obtaining the answer.
  • Deployment prompt: Tool-use prompts may provide tool descriptions and in-context examples, with placeholders substituted by corresponding task information.Different action formats use different formatting instructions, including Python code, JSON objects, or textual calls.
  • Action formats: The Python action format permits for-loops, if-statements, and other constructs, whereas JSON and text formats restrict calls to one tool at a time.Python actions begin with Action: and end with End Action, matching the surrounding prompt protocol.

G.2. Data Selection Heuristic

CodeActInstruct trajectories are selected with heuristics targeting valid code actions, self-improvement after errors, and instruction following, yielding 7,139 retained examples.

  • Data selection heuristic: Selection begins with successful trajectories containing more than two interaction turns.The heuristic is designed to promote code-as-actions, self-improvement, and instruction-following capabilities.
  • Code-as-Actions: Code-as-actions filtering removes trajectories with incorrect API invocations or action formats unsuitable for parsing and execution.This retains examples that adhere to the executable-code interaction framework.
  • Self-Improving: Self-improving filtering preserves trajectories where initial errors are corrected later and removes trajectories producing errors in every execution.The goal is to avoid demonstrations that teach persistent erroneous execution despite correct answers.
  • Instruction-Following: Instruction-following filtering removes rare trajectories with an odd number of interaction turns, indicating failure to follow instructions and respond to the user.This heuristic targets complete interaction trajectories rather than isolated execution outcomes.
  • Final dataset: 7,139 trajectories remain after filtering: 6,728 from GPT-3.5 and Claude, plus 411 from GPT-4-0613.The retained counts are reported after applying all three heuristics.

H. CodeActAgent Anomaly on M3ToolEval

On M3ToolEval, the same CodeActInstruct and general-conversation mixture produced divergent outcomes: LLaMA-2 did not improve, while Mistral improved by more than 10% absolute.

  • Observed anomaly: More than 10% absolute improvement was obtained by Mistral, whereas the LLaMA-2-backed CodeActAgent failed to improve performance.Both models were fine-tuned with the same mixture of CodeActInstruct and general conversations.
  • Possible explanations: The authors found no match for the generated artifacts in the training mixture and hypothesize that they may originate from inaccessible pre-training data.They also suggest weaker fundamental capability in the LLaMA-2 backbone as another possible explanation.
Loading 2402.01030v4…