PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive In-conversation Team Building for Language Model Agents

Linxin Song, Jiale Liu, Jieyu Zhang, Shaokun Zhang, Ao Luo, Shijian Wang, Qingyun Wu, Chi Wang

arXiv:2405.19425v3cs.CL

TL;DR

The paper asks how to build effective LLM-agent teams for a given task and proposes Captain Agent, which adaptively forms teams and uses nested conversation and reflection. Across six real-world scenarios, it reports improved average accuracy compared with existing methods.

  • Problem

    The paper asks how to build an effective team of LLM agents for a given task, noting that static teams can be difficult to manage and adapt as requirements change.

    Source: §1 · p. 1

  • Method

    Captain Agent forms and manages teams for task-solving steps, using nested group conversations and reflection to guide its next decisions.

    Source: §1 · p. 2, §2.2 · p. 3

  • Results

    Across six real-world scenarios, Captain Agent outperforms single- and multi-agent methods using the same prompts, with a 21.94% improvement on average accuracy.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The adaptive team-building paradigm helps ensure diversity and prevent limited knowledge extraction.

    Source: §5 · p. 8

    Real-world interaction experiments are limited to GAIA tasks involving web search and browsing; agents cannot access publicly available information or publish web content.

    Source: Impact Statement · p. 9

Abstract

from arXiv · show

Leveraging multiple large language model (LLM) agents has shown to be a promising approach for tackling complex tasks, while the effective design of multiple agents for a particular application remains an art. It is thus intriguing to answer a critical question: Given a task, how can we build a team of LLM agents to solve it effectively? Our new adaptive team-building paradigm offers a flexible solution, realized through a novel agent design named Captain Agent. It dynamically forms and manages teams for each step of a task-solving process, utilizing nested group conversations and reflection to ensure diverse expertise and prevent stereotypical outputs, allowing for a flexible yet structured approach to problem-solving. A comprehensive evaluation across six real-world scenarios demonstrates that Captain Agent significantly outperforms existing multi-agent methods with 21.94% improvement in average accuracy, providing outstanding performance without requiring task-specific prompt engineering. Our exploration of different backbone LLM and cost analysis further shows that Captain Agent can improve the conversation quality of weak LLM and achieve competitive performance with extremely low cost, which illuminates the application of multi-agent systems.

1. Introduction

The section asks how to build effective LLM-agent teams for a given task, noting that static teams can be difficult to manage and adapt. It introduces Captain Agent’s adaptive team-building approach and reports strong results across six real-world scenarios.

1. Introduction

  • The central question is how to build an effective team of LLM agents for a given task.Source: §1 · p. 1
  • Static teams must retain all required expertise throughout a task, making larger teams harder to manage and less adaptable to changing requirements.The introduction frames adaptive team formation as a way to leverage different expertise as task demands evolve.Source: §1 · pp. 1–2
  • Captain Agent forms and manages agent teams for each problem-solving step, using nested group conversation and reflection to guide whether to adjust the team or conclude.The workflow separates team activity from the main conversation and bases next-step decisions on nested-conversation results.Source: §1 · p. 2, Figure 1 · p. 2
  • Captain Agent achieves a 21.94% improvement on average accuracy compared with single- and multi-agent methods using the same task prompts.The evaluation covers six real-world scenarios and requires no task-specific prompt engineering.Source: §1 · p. 2

2. Captain Agent

Captain Agent builds and manages teams for successive task subtasks, combining agent-and-tool selection with nested group conversations and reflection. Its adaptive teams are intended to reduce redundant participation and prompting without sacrificing team diversity.

2. Captain Agent

  • Captain Agent decomposes a user task into subtasks, assigns roles, creates a team, and repeats the process until a round limit or terminal condition is reached.The team solves each subtask in a nested conversation, whose summarized history is analyzed by a reflector and returned to Captain Agent.Source: §2 · p. 2

2.1. Adaptive Multi-agent Team Building

  • For each suggested role, Captain Agent retrieves similar agents and tools, filters for a suitable agent or abstains, and generates and stores an agent when none fits.Generated agents receive a profile, task-solving and interaction instructions, and retrieved tools; retrieved agents and tools are selected using embedding similarity.Source: §2.1 · p. 3

2.2. Nested Conversation and Reflection

  • In nested conversations, team members share conversation history while an LLM-based reflector summarizes it, checks disputes, and informs Captain Agent’s next decision.The reflector’s report supports a decision to continue or conclude the task.Source: §2.2 · p. 3

2.3. Benefits over Static Team

  • Compared with static teams, adaptive teams aim to reduce prompting load and irrelevant agents’ redundant output without sacrificing team diversity.Source: §2.3 · pp. 3–4

3. Evaluation

Captain Agent is evaluated across real-world scenarios against multi-agent baselines, with analyses of adaptive team building, libraries, reflection, backbone choice, and selected agents. The reported comparisons favor adaptive teams in most scenarios and find that libraries and reflection contribute to performance.

3.1. Experimental Setup / 3.2. Evaluation Protocol

  • The evaluation covers six real-world scenarios and compares Captain Agent with multi-agent baselines; accuracy is used for math, data analysis, and science, while programming is scored by passing all tests.Information-retrieval baselines are taken from the GAIA leaderboard; the main comparison uses GPT-4-0125-preview.Source: §3.1 · p. 4, §3.2 · p. 5, Table 2 · p. 5

3.3. Main Results

  • Captain Agent achieves outstanding results across scenarios; it outperforms AutoAgents in all scenarios and both outperform Assistant+Executor on chemistry and physics.The authors attribute the comparison to domain-specific expert roles and Captain Agent’s high-level planning with adaptive instructions and teams.Source: §3.3 · p. 5

3.4. Analysis and Ablation Studies

  • The analysis samples problems from MATH, HumanEval, DABench, and SciBench, using the evaluation protocol from Section 3.3.The MATH and HumanEval subsets are selected from the AutoGenBench GPT-4 failure set; the SciBench subset is split between chemistry and physics.Source: §3.4 · p. 6

3.4.1. STATIC VS. ADAPTIVE TEAM-BUILDING

  • Adaptive team-building outperforms static building in all reported scenarios except physics, with a greater-than-30.43% accuracy gain on data analysis.The comparison changes team members and instructions across rounds for the adaptive setting, while the static setting fixes the team after round one.Source: §3.4.1 · p. 6, Figure 2 · p. 6

3.4.2. TOOL LIBRARY AND AGENT LIBRARY

  • Removing either the agent or tool library impairs GAIA performance, and the best results require both libraries; the agent library matters especially for Level 2 tasks.The authors describe Level 2 tasks as requiring substantial web navigation, reasoning, and coordination of multiple tools.Source: §3.4.2 · p. 6, Table 4 · p. 6

3.4.3. ABLATION ON REFLECTOR

  • Reflection improves Captain Agent across scenarios by detecting disputes and helping prevent errors from propagating through multi-round problem solving.The reflector is counted as successful when it detects a dispute and highlights it to Captain Agent; physics remains the lowest-performing scenario despite reflection’s role there.Source: Figure 3 · p. 6, §3.4.3 · p. 7

3.4.4. ABLATION ON LLM BACKBONE AND COST ANALYSIS

  • Captain Agent with GPT-4-0125-preview has the best performance but highest cost; with GPT-4o-mini, it outperforms the other baselines at significantly lower cost.The cost accounting covers Captain Agent output, agent and tool selection, agent generation, and nested group conversation.Source: §3.4.4 · p. 7, §3.4.2 · p. 7, Figure 4 · p. 7

3.4.5. AGENT SELECTED IN EACH SCENARIO

  • The 541-agent library yields scenario-related selections, including frequent generic domain experts and an especially high selection rate for WebResearch Expert in information retrieval.The authors describe WebResearch Expert as a handcrafted agent that searches the internet and summarizes pages in Markdown.Source: §3.4.5 · p. 7

4. Related Work

Related work spans single-agent systems with reasoning and tool use, multi-agent collaboration, and task decomposition; several team-building approaches rely on static or manually assembled teams.

Related work

  • Selected agents are highly related to their scenario, and Verification Expert has a high selection rate.Source: Figure 5 · p. 8
  • ReAct interleaves reasoning and task-specific actions, while LLM agents can use external tools to combine internal capabilities with outside resources.The passage also situates these agents among work on reasoning, planning, and adapting to novel observations.Source: §4 · pp. 7–8
  • Static multi-agent methods use group-chat protocols and builders that produce agent lists, sometimes with a human and sometimes an LLM as builder.The passage also describes task decomposition into specialized agents as another approach.Source: §4 · p. 8
  • Task-decomposition methods assign specialized agents subtasks, but the cited approaches lack backtracking once agents are generated.Meta-prompting uses a meta-model to decompose tasks, assign subtasks, and aggregate completions.Source: §4 · p. 8

5. Conclusion and Discussion

The paper presents adaptive build through Captain Agent, which manages agent teams using adaptive building, nested group conversation, and reflection. It reports results across six real-world scenarios and ablations of the approach.

Conclusion and Discussion

  • Adaptive build is proposed as a multi-agent team-building paradigm intended to ensure diversity and prevent limited knowledge extraction.Source: §5 · p. 8
  • Captain Agent manages teams for problem-solving steps through adaptive multi-agent team building and nested group conversation with reflection.Source: §5 · p. 8
  • Across six real-world scenarios, Captain Agent achieves superior results to existing methods without prompt engineering; ablations report contributions from each component.Source: §5 · p. 8

Impact Statement

The impact statement describes a system that dynamically ensembles LLM agents and equips them with tools, while limiting real-world interaction experiments to GAIA web-search and browsing tasks. Agents cannot access publicly available information or publish content on the web.

Impact Statement

  • The method dynamically ensembles LLM agents and equips them with versatile tools for complex tasks.Source: Impact Statement · p. 9
  • Real-world interaction experiments are limited to GAIA tasks requiring web search and browsing.Source: Impact Statement · p. 9
  • Agents are restricted from accessing publicly available information and cannot publish content on the web, which the authors say keeps experiments non-invasive and safe.Source: Impact Statement · p. 9

A. Limitations

The limitations include cost, model-selection and evaluation concerns, and difficulty generalizing beyond the customized AutoGen communication framework. The comparison with other team-building frameworks highlights differences in how teams are assembled and used.

A. Limitations

  • GPT-4 conversations cost more than a single-agent system, and large conversation and profile contexts remain despite reducing nested-chat participants.The authors identify performance-cost trade-offs as future work.Source: A. Limitations · p. 13
  • The evaluation may not reflect real-world performance because data leakage during pretraining can misalign test and real-world results.Source: A. Limitations · p. 13
  • Captain Agent’s customized AutoGen communication framework limits generalization to other hand-crafted agents; the authors identify a universal communication protocol as important.Source: A. Limitations · p. 13

B. Difference Between Existing Team-building Frameworks

  • On the reduced ablation dataset, Captain Agent achieves the best performance with gpt-4-0125-preview; its gpt-4o-mini version is competitive with higher-cost baselines.Source: Table 5 · p. 13
  • Unlike AgentVerse’s fixed recruitment sequence and handcrafted teams, Captain Agent can join nested chats, cache teams, and recall them later.Source: B. Difference Between Existing… · p. 13
  • DyLAN begins with a fixed, manually predefined expert group and requires manual selection from a handcrafted pool of experts, prompts, and capabilities.Source: B. Difference Between Existing… · pp. 13–14

C. Case Study Comparing Captain Agent and Others

The case studies show Captain Agent tailoring expert teams to complex queries and reproducing a manually designed system’s performance in a supply-chain scenario.

C. Case Study Comparing Captain Agent and Others

  • Captain Agent matches the handcrafted three-agent OptiGuide system on a specific supply-chain optimization task while replacing it without human intervention.The case also reports that Captain Agent uses only a few lines of code, compared with OptiGuide’s fixed structure of over 100 lines.Source: C. Case Study Comparing Captain Agent… · p. 14
  • The two scenarios examine automatic tailoring of specialized agent systems and replacing a manually designed system without human intervention.The supply-chain case reports similar performance to OptiGuide.Source: C. Case Study Comparing Captain Agent… · p. 14
  • In the Figure 6 case, Captain Agent assembles relevant experts for a complex query, whereas the AutoGen two-agent system cannot complete it.The caption does not specify the query or experts; those are described in the accompanying case account.Source: Figure 6 · p. 14

C.1. Case Study 1: Arbitrary User Question Answering

The arbitrary-question case contrasts a specialized team assembled by Captain Agent with a simpler AutoGen two-agent system that fails on the query.

C.1. Case Study 1: Arbitrary User Question Answering

  • Captain Agent coordinates WebBrowsing, Myth, and Verification Experts to answer a question about an article by Emily Midkiff from June 2014.The agents address different domains and collaborate under Captain Agent’s coordination.Source: §C.1 · p. 14
  • The AutoGen two-agent system fails to resolve the article query because of limitations in the Assistant agent’s capabilities.The passage contrasts this failure with Captain Agent’s expert-team collaboration.Source: §C.1 · p. 14
  • Figure 7 presents a separate comparison with OptiGuide, a handcrafted multi-agent system for supply-chain-related queries.The figure caption identifies the comparison and application area but does not state its outcome.Source: Figure 7 · p. 15

C.2. Case Study 2: Domain Specific Question Answering

The supply-chain case reports similar performance to handcrafted OptiGuide with a much shorter implementation, while the appendix describes Captain Agent’s task instructions, expert conversations, reflection, and role matching.

C.2. Case Study 2: Domain Specific Question Answering

  • Captain Agent reproduces OptiGuide’s group structure and solves the supply-chain task with a few lines of code, compared with OptiGuide’s fixed structure of over 100 lines.The passage presents this as easing domain-specific development.Source: §C.2 · p. 15

D. Instruction of Captain Agent

D.1. System Message

  • The seek_experts_help tool builds a group for the supplied task, summarizes its conclusions, and is limited to one group initiation per response.The prompt also instructs Captain Agent not to alter task information from the user proxy.Source: §D.1 · p. 16
  • The system message specifies the execution task and its Markdown format, including task description, plan, output format, and completion constraints.It also permits delegating to experts or solving with coding and language skills.Source: §D.1 · p. 16

D.2. Reflector LLM

  • A reflector summarizes group-chat history, flags disputes, and reports whether results need checking; Captain Agent uses backward reasoning when disputes arise or results change.The reflector’s answer format includes task, reasons, contradictions, double-check status, and additional information.Source: §D.1 · p. 16, §D.2 · p. 17

D.3. Agent Selector LLM

  • The Agent Selector matches described roles to experts in the available expert pool, returning an expert or None for each role.Its answer format is JSON.Source: §D.3 · p. 17

E. Task Instructions

The appendix provides scenario-specific task instructions, with shared instructions for fair comparisons and distinct input, tool, and output requirements across the evaluated task types.

E. Task Instructions

  • Scenario instructions are manually designed and given equally to Captain Agent and baselines, except GAIA information retrieval uses leaderboard results.Instructions may suggest libraries such as pandas, numpy, scipy, and sympy.Source: E. Task Instructions · p. 18, §E.4 · p. 18

E.1. Instruction for Mathematics

  • Mathematics instructions allow Python approximation for difficult calculations and require verification before returning the final answer in box format.The listed preinstalled packages are sympy, numpy, and scipy; plotting is prohibited.Source: §E.1 · p. 18

E.2. Instruction for Programming

  • Programming tasks require completing a partially implemented function and returning standalone Python code that runs the provided unit tests.The instructions require importing run_tests and prohibit modifying the test-running line.Source: §E.2 · p. 18

E.3. Instruction for Data Analysis

  • Data-analysis tasks provide a CSV path and problem constraint, prohibit plotting, and require a specified final-answer format after verification.The passage leaves the requested format as a task-specific placeholder.Source: §E.3 · p. 18

E.4. Instruction for Science (Chemistry and Physics)

  • Science instructions cover chemistry and physics problems, permit Python approximation, require a specified answer unit, and request a verified boxed answer.The instructions list sympy, numpy, and scipy and prohibit plotting.Source: §E.4 · pp. 18–19

E.5. Instruction for World-information Retreival

  • World-information retrieval instructions tell the task builder to consider explicitly whether a task benefits from web navigation.Source: §E.5 · p. 19

F. Agent Library

The agent library includes a recorded example of a PythonProgramming_Expert. The tool library organizes manually crafted task-support functions into three categories.

F. Agent Library

  • The PythonProgramming_Expert example combines rocket physics and classical mechanics expertise with Python programming and scientific-computing libraries.The description names sympy, numpy, and scipy.Source: F. Agent Library · p. 19
  • The example’s instructions emphasize verifying physical and mathematical results for accuracy.They describe verification as important for reliable solutions.Source: F. Agent Library · p. 19

G. Tool Library

  • The tool library groups manually crafted functions into Information Retrieval, Data Analysis, and Math Problem Solving, using dataset patterns to identify tools that could enhance task resolution.Tables 6–8 correspond to those three categories.Source: G. Tool Library · p. 20, Table 6 · p. 20, Table 7 · p. 21

Related papers

Loading 2405.19425v3…