PaperlayerView exact revision on arXiv

Source-linked AI summary

CoAct: A Global-Local Hierarchy for Autonomous Agent Collaboration

Xinming Hou, Mingming Yang, Wenxiang Jiao, Xing Wang, Zhaopeng Tu, Wayne Xin Zhao

arXiv:2406.13381v1cs.CL

TL;DR

LLMs continue to struggle with complex real-world tasks despite strategies such as CoT and ReAct. CoAct assigns global planning and local execution to collaborating agents; on WebArena, it outperforms ReAct across five tasks. The authors also identify global-planning deficiencies and the lack of an efficient memory mechanism as limitations.

  • Problem

    Existing LLMs struggle with complex real-world tasks despite strategies such as CoT and ReAct, and current explorations predominantly focus on a single LLM and memory stream.

    Source: §1 · p. 1

  • Method

    CoAct pairs a global planning agent that scopes and decomposes tasks with a local execution agent that carries out specific subtasks within the global plan.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    CoAct outperforms ReAct across all five WebArena tasks, improving ReAct by over 40% success rate and up to 70% with force stop intervention.

    Source: §3.2 · p. 3

  • Takeaways & Limitations

    On long-horizon WebArena tasks, CoAct can rearrange its process trajectory when facing failures.

    Source: §4 · p. 4

    Approximately 60% of CoAct’s failures stem from global-planning deficiencies, and the lack of an efficient memory mechanism may limit operational-efficiency improvements.

    Source: §5 · p. 5

Abstract

from arXiv · show

Existing LLMs exhibit remarkable performance on various NLP tasks, but still struggle with complex real-world tasks, even equipped with advanced strategies like CoT and ReAct. In this work, we propose the CoAct framework, which transfers the hierarchical planning and collaboration patterns in human society to LLM systems. Specifically, our CoAct framework involves two agents: (1) A global planning agent, to comprehend the problem scope, formulate macro-level plans and provide detailed sub-task descriptions to local execution agents, which serves as the initial rendition of a global plan. (2) A local execution agent, to operate within the multi-tier task execution structure, focusing on detailed execution and implementation of specific tasks within the global plan. Experimental results on the WebArena benchmark show that CoAct can re-arrange the process trajectory when facing failures, and achieves superior performance over baseline methods on long-horizon web tasks. Code is available at https://github.com/xmhou2002/CoAct.

1 Introduction

LLMs still struggle with complex real-world reasoning tasks, while current approaches predominantly rely on a single model and memory stream. CoAct proposes a global-local hierarchy and reports improved WebArena performance over ReAct.

  • LLMs struggle with complex reasoning tasks despite strategies such as CoT, ReAct, and self-refine, while many approaches still focus on one LLM and memory stream.Prior work cited in the passage links single-LLM performance constraints to finite attention and hierarchical capacity.Source: §1 · p. 1
  • CoAct pairs a global planner, which scopes problems and provides subtask descriptions, with a local agent that executes tasks within the global plan.The framework transfers hierarchical planning and collaboration patterns from human society to LLM systems.Source: §1 · p. 1
  • CoAct can rearrange its process trajectory when failures occur and achieves superior performance over ReAct on real-world tasks.The introduction reports empirical validation on WebArena across diverse website environments.Source: §1 · pp. 1–2

2 Framework

CoAct organizes web-task work as a hierarchy: a global planner decomposes and assigns subtasks, while a local agent executes them and reports feedback. The agents can replan when execution falters.

2 Framework

  • The framework decomposes tasks, assigns and communicates subtasks, executes them, collects feedback, evaluates progress, and replans when necessary.The global agent divides work into phased subtasks for the local execution agent.Source: §2 · p. 2

2.1 Global Planning Agent

  • The global planning agent constructs phased plans and reviews potential replanning when the local agent requests guidance or adjustments.It maintains the global plan and checks that the final output aligns with the initial strategy.Source: §2.1 · p. 2

2.2 Local Execution Agent

  • The local execution agent implements specific subtasks, verifies actions against the global plan, and uses feedback to proceed, revise its plan, or request a new global plan.Execution results are reported to support alignment with global objectives.Source: §2.2 · pp. 2–3

3 Experiment

On WebArena, CoAct outperforms ReAct across five tasks and can redirect execution after failures. The experiments also examine task difficulty, failure types, and preliminary search-engine enrichment.

3.1 Setting

  • The evaluation uses 100 randomly sampled examples from each of WebArena’s five tasks and measures task-completion success rate.The compared approaches use gpt-3.5-turbo-16K-0613; CoAct is compared with ReAct and a force-stop variant.Source: §3.1 · p. 3

3.2 Main Results

  • CoAct improves ReAct by over 40% success rate, and by up to 70% with force-stop intervention, outperforming it across all five WebArena tasks.The largest highlighted advantage is on Shop.Source: §3.2 · p. 3
  • In Shop tasks that do not fit predefined categories, ReAct can persist in category search, while CoAct’s global plan and local subtasks support redirection and macro-level replanning.The global agent can adjust the plan when the local agent requests it; Figure 4 presents a Shop example.Source: §3.2 · pp. 3–4, Figure 4 · p. 3

3.3 Task Analysis

  • For manually categorized Shop examples, success rates for ReAct and CoAct were 34.0% and 52.0% on Easy, 5.0% and 16.0% on Medium, and 0.0% for both on Hard.The Easy:Medium:Hard distribution was 30%:50%:20%.Source: §3.3 · p. 4
  • Planning inadequacies account for about 40% of CoAct failures, while iterative and repetitive actions account for about 60% and can exceed the interaction-round limit.The paper proposes web-page-specific knowledge for planning deficiencies and memory mechanisms for repetitive actions.Source: §3.3 · p. 4
  • Search-engine information raises CoAct success rates from 24.0% to 31.0% on Shop and from 10.0% to 19.0% on Gitlab in preliminary experiments.The authors say broader applications require further investigation and fine-tuning.Source: §3.3 · p. 4, Table 2 · p. 4

4 Conclusion

CoAct combines global planning with local execution and, on WebArena, can rearrange its process trajectory after failures while outperforming ReAct on long-horizon web tasks.

  • CoAct combines a global planning agent with a local execution agent in a hierarchical framework.Source: §4 · p. 4
  • On WebArena, CoAct can rearrange its process trajectory when facing failures.Source: §4 · p. 4
  • CoAct achieves superior performance over ReAct on long-horizon web tasks.Source: §4 · p. 4

5 Limitations

The reported limitations include planning deficiencies, repetitive actions without an efficient memory mechanism, and limited evidence for generalizing web page-specific knowledge across application contexts.

  • Approximately 40% of CoAct’s failures are attributed to global-planning deficiencies that cause inaccurate initial plans.Source: §5 · p. 5
  • Separately, about 60% of failures are attributed to global-planning deficiencies, while the lack of an efficient memory mechanism may limit operational efficiency.Source: §5 · p. 5
  • Initial experiments found promising performance improvements from web page-specific knowledge, but its generalization across application contexts needs further refinement.Source: §5 · p. 5
  • Future research avenues include refining knowledge integration, optimizing interaction protocols, and safeguarding training-dataset integrity.Source: §5 · p. 5

6 Ethical Considerations

The paper identifies fairness, safety, transparency, and multi-agent interaction as ethical concerns, and says it uses only publicly accessible datasets.

  • The ethical concerns named include fairness and inclusivity, safety, transparency for accountability and trustworthiness, and multi-agent interactions.Source: §6 · p. 5
  • The research says it exclusively uses publicly accessible datasets, avoiding proprietary or confidential information.Source: §6 · p. 5
  • The paper presents this dataset choice as supporting its ethical integrity.Source: §6 · p. 5

A Algorithm

The algorithm initializes the planner and local agents, delegates the task for phased planning and execution, replans after failed validation when accepted, and returns a task with a validation summary.

  • The algorithm takes a task, global planner, and local agents as input, and returns a completed task with a validation summary.Source: A Algorithm · p. 6
  • It initializes the planner and local agents, delegates the task to the planner, then iterates over planned phases and assigned local subtasks.Source: A Algorithm · p. 6
  • After local execution, a failed validation prompts the planner to replan the subtask, and an accepted replacement plan updates the global plan.Source: A Algorithm · p. 6
  • The agents execute tasks, after which the global planner validates the completed task before the algorithm returns it with the validation summary.Source: A Algorithm · p. 6

B Task Categorization

The Shop task is divided into Easy, Medium, and Hard classes according to the processing required and whether ReAct and CoAct can address the example.

C Prompts

The section presents prompts for the global planning and local execution agents, alongside a table analyzing Shop-task difficulty.

C Prompts

  • The section presents detailed prompts for the two agents.Source: C Prompts · p. 6
  • Table 3 analyzes task difficulty for the Shop task.Source: Table 3 · p. 7
  • Table 4 contains the prompt for the global planning agent.Source: Table 4 · p. 8
  • Table 5 contains the prompt for the local execution agent.Source: Table 5 · p. 9
Loading 2406.13381v1…