PaperlayerView exact revision on arXiv

Source-linked AI summary

Internet of Agents: Weaving a Web of Heterogeneous Agents for Collaborative Intelligence

Weize Chen, Ziming You, Ran Li, Yitong Guan, Chen Qian, Chenyang Zhao, Cheng Yang, Ruobing Xie, Zhiyuan Liu, Maosong Sun

arXiv:2407.07061v2cs.CL

TL;DR

Existing multi-agent frameworks have limited support for heterogeneous third-party agents, distributed deployment, and dynamically adapting coordination. IoA addresses these gaps with an integration protocol, instant-messaging-like architecture, and autonomous teaming and conversation-flow mechanisms. Across general assistant, embodied-agent, GAIA, and retrieval-augmented generation evaluations, it consistently outperforms state-of-the-art baselines, including a 66 to 76% win rate against AutoGPT and Open Interpreter individually.

  • Problem

    Existing multi-agent frameworks isolate agents within their own ecosystems, simulate collaboration on single devices, and rely on hard-coded communication and coordination.

    Source: §1 · pp. 1–2

  • Method

    IoA integrates third-party agents across devices and uses agent discovery, dynamic team formation, group chats, and autonomous conversation-flow control.

    Source: §1 · p. 2, §2.3.1 · p. 5, §2.3.3 · p. 6

  • Results

    IoA consistently outperforms state-of-the-art baselines across evaluated domains, achieving a 66 to 76% win rate against AutoGPT and Open Interpreter individually.

    Source: §1 · p. 2, §6 · p. 15

  • Takeaways & Limitations

    IoA provides a flexible and scalable foundation for collaboration among heterogeneous agents in distributed, Internet-like environments.

    Source: §1 · p. 2, §6 · p. 15

    LLM clients sometimes repeat information and fail to switch the group chat to pause & trigger, producing suboptimal communication patterns and high communication cost.

    Source: §4.2 · pp. 13–14

Abstract

from arXiv · show

The rapid advancement of large language models (LLMs) has paved the way for the development of highly capable autonomous agents. However, existing multi-agent frameworks often struggle with integrating diverse capable third-party agents due to reliance on agents defined within their own ecosystems. They also face challenges in simulating distributed environments, as most frameworks are limited to single-device setups. Furthermore, these frameworks often rely on hard-coded communication pipelines, limiting their adaptability to dynamic task requirements. Inspired by the concept of the Internet, we propose the Internet of Agents (IoA), a novel framework that addresses these limitations by providing a flexible and scalable platform for LLM-based multi-agent collaboration. IoA introduces an agent integration protocol, an instant-messaging-like architecture design, and dynamic mechanisms for agent teaming and conversation flow control. Through extensive experiments on general assistant tasks, embodied AI tasks, and retrieval-augmented generation benchmarks, we demonstrate that IoA consistently outperforms state-of-the-art baselines, showcasing its ability to facilitate effective collaboration among heterogeneous agents. IoA represents a step towards linking diverse agents in an Internet-like environment, where agents can seamlessly collaborate to achieve greater intelligence and capabilities. Our codebase has been released at \url{https://github.com/OpenBMB/IoA}.

1 INTRODUCTION

IoA is proposed as an Internet-inspired platform for heterogeneous autonomous agents, addressing ecosystem isolation, single-device simulation, and rigid coordination. It integrates third-party agents, supports dynamic teaming and conversation control, and outperforms state-of-the-art baselines across several task domains.

Motivation

  • IoA addresses ecosystem isolation, single-device simulation, and rigid communication and coordination in existing multi-agent frameworks.These limitations restrict third-party integration, distributed deployment, and adaptation to task-dependent collaboration needs.Source: §1 · p. 2

IoA Framework

  • The framework enables third-party agents on different devices to collaborate through an integration protocol and Internet-like architecture.Its design includes agent discovery, dynamic team formation, group chats, and flexible conversation-state control.Source: §1 · p. 2

Experimental Results

  • 66 to 76% win rate in open-domain evaluations shows IoA outperforming AutoGPT and Open Interpreter when compared with them individually.The framework also outperforms prior work on GAIA and substantially surpasses existing methods in retrieval-augmented generation.Source: §1 · p. 2

Implications

  • IoA’s results across domains suggest that Internet-like collaboration could support distributed multi-agent systems as capable agents become feasible on personal computers and mobile devices.The authors identify this as an opportunity for real-world multi-agent systems that collaborate across devices on complex problems.Source: §1 · p. 2

2 FRAMEWORK DESIGN AND KEY MECHANISMS OF IOA

IoA is an Internet-inspired platform for distributed collaboration among heterogeneous agents, combining layered server-client architecture with dynamic discovery, teaming, task execution, conversation control, and messaging. Its mechanisms support adaptive nested collaboration and coordinate a worked multi-agent research-paper task.

2.1 OVERVIEW OF IOA

  • IoA addresses distributed collaboration, adaptive communication, and heterogeneous-agent integration through an instant-messaging-like platform.Agents can operate across devices and locations, form teams based on task requirements, and integrate third-party capabilities through a flexible protocol.Source: §2.1 · pp. 2–3

2.2 ARCHITECTURE OF IOA

  • The server manages registration, discovery, group formation, and message routing, while clients wrap individual agents through matching layered architectures.Both components use interaction, data, and foundation layers to separate collaboration, information management, integration, networking, and security functions.Source: §2.1 · p. 3, §2.2.1 · pp. 3–4, §2.2.2 · p. 4

2.3.1 AGENT REGISTRATION AND DISCOVERY

  • Agent registration stores each client’s capabilities, and semantic search returns suitable collaborators from the server registry for task-specific collaboration.Agents register descriptions of their skills, while search client maps desired characteristics to a subset of registered clients.Source: §2.3.1 · pp. 4–5

2.3.2 AUTONOMOUS NESTED TEAM FORMATION

  • IoA forms teams dynamically and recursively, allowing agents to create nested sub-groups when subtasks require additional expertise.The resulting tree-like group structure decomposes complex work across specialized chats and can reduce communication complexity relative to fully connected single groups.Source: §2.3.2 · p. 5

2.3.3 AUTONOMOUS CONVERSATION FLOW CONTROL

  • Conversation flow is managed as a finite state machine whose LLM selects both the next state and next speaker from exchanged messages.States cover discussion, synchronous and asynchronous task assignment, pause and trigger, and conclusion, enabling structured communication and dynamic coordination.Source: §2.3.3 · p. 6

2.3.4 TASK ASSIGNMENT AND EXECUTION

  • Tasks are represented with descriptions and dynamically identified subtasks, then allocated synchronously or asynchronously to responsible clients.Synchronous allocation pauses the group chat, whereas asynchronous allocation permits parallel execution; integrated agents execute through a standardized run interface.Source: §2.3.4 · p. 7

2.4 COMPREHENSIVE MESSAGE PROTOCOL DESIGN

  • IoA’s extensible message protocol separates metadata headers from payloads and is implemented consistently by clients and the server for routing and processing.Headers support addressing and group identification, while payloads encode message purpose and expected responders.Source: §2.4 · pp. 7–8

2.5 PUTTING IT ALL TOGETHER: A WALKTHROUGH OF IOA IN ACTION

  • In the walkthrough, agents discover collaborators, form a group, create a specialized PDF-expert sub-group, execute parallel tasks, and conclude with synchronous editing and final review.The example shows conversation states, nested teams, and protocol-mediated exchanges coordinating a research-paper task through submission preparation.Source: §2.5 · p. 8

3 EXPERIMENTS

IoA is evaluated across heterogeneous tools, architectures, observation and action spaces, and knowledge bases using consistent prompts across task categories. Across these settings, IoA achieves strong comparative performance through dynamic collaboration among diverse agents.

3 EXPERIMENTS

  • Across experiments, IoA evaluates heterogeneous-agent collaboration in tools, architectures, embodied interaction, and knowledge while keeping prompts unchanged across tasks.The experiments target both collaboration among heterogeneous agents and adaptability across problem domains.Source: §3 · p. 9, §3.2 · p. 10, §3.3 · p. 11

3.1 HETEROGENEOUS TOOLS: GAIA BENCHMARK

  • IoA achieves the highest overall performance on GAIA, with particularly strong results on the more challenging Level 2 and Level 3 tasks.The evaluation uses four basic ReAct agents equipped with a web browser, code interpreter, Wikidata searcher, and YouTube transcript downloader.Source: §3.1 · p. 9

3.2 HETEROGENEOUS ARCHITECTURE: OPEN-ENDED INSTRUCTION BENCHMARK

  • IoA consistently outperforms AutoGPT and Open Interpreter across four open-ended task categories, with win rates of 76.5% and 63.4%, respectively.The benchmark covers search and report, coding, mathematics, and life assistance, and uses GPT-4-1106-preview as an impartial judge with alternating response order.Source: §3.2 · p. 10

3.3 HETEROGENEOUS OBSERVATION AND ACTION SPACE: EMBODIED AGENT TASKS

  • IoA outperforms Roco Dialog on four of five RoCoBench tasks and achieves success rates superior or comparable to Central Plan despite using partial observations.IoA obtains perfect scores on Cabinet, Sandwich, and Sort, while generally requiring slightly more decision steps than Central Plan.Source: §3.3 · p. 11

3.4 HETEROGENEOUS KNOWLEDGE: RETRIEVAL-AUGMENTED GENERATION

  • On RAG benchmarks, IoA often surpasses or matches GPT-4 despite using GPT-3.5, and heterogeneous knowledge outperforms homogeneous Apollo’s Oracle on two of four tasks.The evaluation uses TriviaQA, Natural Questions, HotpotQA, and 2WikiMultiHopQA with separate evidence pools for heterogeneous agents.Source: §3.4 · p. 12

4 ANALYSIS

IoA’s team formation mechanisms achieve substantial recall in both regular and nested settings, while cost analysis exposes communication overhead and repetition as important optimization targets.

4.1 TEAM FORMATION PRECISION

  • 41.4% Top@1 and 64.9% Top@10 recall demonstrate effective regular team formation, with relevant agents averaging rank 27.4 and reaching 50.1% MRR.These metrics evaluate initial team formation for assigned tasks.Source: §4.1 · p. 13, Table 4 · p. 13
  • 59.7% Top@1 and 81.8% Top@10 recall show strong nested team formation, with relevant agents typically ranked within the top 11 results.The nested setting addresses subtasks that may require additional expertise.Source: §4.1 · p. 13, Table 4 · p. 13

4.2 COST AND SUB-OPTIMAL COMMUNICATION PATTERN ANALYSIS

  • IoA-integrated agents incur $0.53 of communication cost per task, producing an overall cost of $0.99 despite lower individual agent costs.Task decomposition decreases the costs of both integrated agents, but communication adds overhead.Source: §4.2 · p. 13, Table 5 · p. 13
  • Repeated or rephrased information after asynchronous assignments can stagnate progress and contribute to high communication cost.Clients sometimes fail to switch the group chat into pause & trigger, as illustrated in Fig. 6.Source: §4.2 · pp. 13–14, Figure 6 · p. 14
  • Manually removing repetitions reduces communication costs by nearly 50%, highlighting communication efficiency and conversation-state handling as open alignment challenges.The analysis links this pattern to broader observations that LLMs may be better aligned for chatbot assistance than efficient agent communication.Source: §4.2 · p. 14

5 RELATED WORK

Prior work has advanced individual and multi-agent LLM systems across many domains, but existing frameworks remain limited in heterogeneous integration, distributed operation, and adaptive communication.

LLM-based Agents

  • LLM-based agents increasingly combine language interaction with tools and external knowledge for autonomous tasks in software, operating systems, and embodied environments.Examples include OS-Copilot, OpenDevin, XAgent, and Voyager.Source: §5 · p. 14

LLM-based Multi-Agent Systems

  • Multi-agent frameworks such as AgentVerse, AutoGen, ChatDev, and MetaGPT support collaboration, but often lack third-party integration, distributed execution, and flexible communication protocols.IoA is positioned as addressing these limitations for more scalable collaboration.Source: §5 · p. 14

6 CONCLUSION

IoA is presented as an Internet-inspired framework for heterogeneous multi-agent collaboration that integrates diverse agents, supports distributed collaboration, and adapts teaming and conversation flow.

6 CONCLUSION

  • IoA provides a flexible platform for integrating diverse third-party agents, enabling distributed collaboration and dynamic mechanisms for agent teaming and conversation flow control.The conclusion reports effectiveness across benchmarks and consistent outperformance of state-of-the-art baselines.Source: §6 · p. 15

A IMPLEMENTATION DETAILS OF IOA

IoA implements collaboration through a protocol-driven, layered client architecture connected by persistent messaging infrastructure and a lightweight third-party agent interface.

A.1 MESSAGE PROTOCOL

  • The message protocol uses headers and payload-related fields to support addressing, team formation, discussion, task assignment, and pause & trigger control.Header fields include sender, group-chat state, and communication ID; other fields encode goals, members, dialogue state, speakers, and task results.Source: §A.1 · pp. 20–21
  • IoA’s protocol fields encode nested team depth and turn limits, allowing group chats to manage recursive teaming and force conclusion after maximum turns.The team-up depth tracks nested formation, while max turns bounds discussion.Source: §A.1 · p. 20
  • The protocol’s triggers field lists task IDs requiring a trigger, enabling the pause & trigger mechanism used for conversation-flow control.Clients rely on the protocol to communicate the information needed for autonomous team formation and flow control.Source: §A.1 · p. 21

A.2 CLIENT

  • The client architecture comprises Foundation, Data, and Interaction layers that jointly provide communication, data management, and agent coordination.The implementation overview presents these layers as the main organization of the client component.Source: §A.2 · p. 21

A.2.1 FOUNDATION LAYER

  • Persistent WebSocket connections route messages through the server using comm id, while SQLite stores agent, group-chat, and task information.This foundation supports prompt message delivery and persistent collaboration state.Source: §A.2.1 · p. 21
  • Third-party agents integrate by implementing run(task desc: str) -> str, which accepts a task description and returns a task-completion summary.The protocol can be extended for more advanced functionalities and interaction patterns.Source: §A.2.1 · p. 21

A.2.2 DATA LAYER

IoA organizes collaboration through persistent agent, group, and task information, alongside interaction modules that support discovery, teaming, messaging, and task execution. Its server architecture combines foundation, data, and interaction layers, while security remains underdeveloped in the current implementation.

A.2.2 DATA LAYER

  • Agent Contact, Group Info, and Task Management modules retain collaboration histories, group states, and assigned-task progress for future decisions and coordination.Contact records inform partner selection; group records track goals, members, state, conclusions, nesting depth, and turn limits; task records track descriptions, assignees, status, and outcomes.Source: §A.2.2 · p. 22

A.2.3 INTERACTION LAYER

  • Clients can search for suitable agents, launch group chats, and coordinate discussion, assignment, synchronization, triggers, and conclusions through a structured message protocol.IoA limits team-formation tool calls to 10 by default and routes task results through integrated agents.Source: §A.2.3 · pp. 22–23

A.3 SERVER

  • The server provides layered infrastructure for agent registration, discovery, communication, and management across the Foundation, Data, and Interaction layers.Its data infrastructure uses Milvus for similarity search, while the server coordinates agent integration and message routing.Source: §A.3 · p. 23, §A.3.1 · p. 23

A.3.1 FOUNDATION LAYER

  • The current Security Module is not extensively used, although it is intended to verify and control third-party agent integration against malicious-agent threats.The paper identifies authentication, authorization, and monitoring as future enhancements for real-world deployment.Source: §A.3.1 · p. 23

A.3.2 DATA LAYER

IoA’s server data and interaction layers maintain agent connectivity, discover compatible agents, establish group chats, and route protocol messages. These modules support coordinated collaboration while leaving scalability and security enhancements for future development.

A.3.2 DATA LAYER

  • The Agent Registry stores names and capability descriptions for integrated clients, enabling similarity-based agent discovery and team formation.Agent queries convert requested characteristics into vectors and match them against registry entries, with BM25 or other retrieval methods also possible.Source: §A.3.2 · p. 23, §A.3.3 · p. 24
  • Session Management tracks WebSocket connections and group-chat memberships, ensuring messages reach the clients participating in each collaboration.It maintains mappings between agents, connections, and group chats for reliable routing.Source: §A.3.2 · p. 23

A.3.3 INTERACTION LAYER

  • Group Setup creates uniquely identified chats, notifies participating clients, and works with Session Management to establish communication mappings.The module initializes each requested group-chat instance with specified team members.Source: §A.3.3 · p. 24
  • Message Routing parses protocol messages by communication identifier and forwards them to all clients associated with the corresponding group chat.This routing depends on the connection and membership information maintained by Session Management.Source: §A.3.2 · p. 23, §A.3.3 · p. 24
  • The server’s layered modules enable agent discovery, team formation, and message exchange, but planned enhancements include load balancing, fault tolerance, scalability, and stronger security.These additions are presented as future work for handling growing real-world multi-agent demands.Source: §A.3.3 · p. 24

A.4 IMPLEMENTATION DETAILS OF DIFFERENT EXPERIMENTS

The experiments implement IoA across benchmark settings by adapting heterogeneous tools and agents, including a four-agent ReAct configuration for GAIA. GAIA evaluation uses the validation set because of budget constraints.

A.4 IMPLEMENTATION DETAILS OF DIFFERENT EXPERIMENTS

  • IoA’s experiment implementation covers multiple evaluations, with the appendix describing the implementation details for each conducted experiment.The supplied implementation subsection introduces the experimental details before describing the GAIA setup.Source: §A.4 · p. 24

A.4.1 GAIA

  • For GAIA, IoA integrates Web Browser, Code Executor, YouTube Transcript Downloader, and Wikidata Searcher ReAct agents.The Web Browser and Code Executor tools are adapted from AutoGen, while the YouTube and Wikidata tools are separately implemented or adapted for IoA.Source: §A.4.1 · p. 24
  • Tool-specific implementations and agent adaptations allow capabilities required by a task to be integrated into IoA and participate in task completion.For videos without available transcripts, the YouTube tool uses Whisper to transcribe spoken language.Source: §A.4.1 · p. 24
  • GAIA performance testing uses the validation set because of budget constraints, limiting the reported evaluation scope.The authors nevertheless describe the results as providing insight into IoA’s handling of complex, multi-step tasks.Source: §A.4.1 · p. 24

A.4.2 OPEN-ENDED INSTRUCTION BENCHMARK

The open-ended instruction benchmark evaluates IoA on 153 diverse tasks spanning search and reporting, coding, mathematics, and life assistance. Responses are compared using GPT-4 with order alternation and a strict consistency rule for wins.

A.4.2 OPEN-ENDED INSTRUCTION BENCHMARK

  • The benchmark contains 153 tasks across search & report, coding, math, and life assistance categories.Its composition is 52 search & report, 30 coding, 30 math, and 41 life assistance tasks.Source: §A.4.2 · p. 25
  • The benchmark combines real-world complex instructions, 10 manually created tasks, and 94 Self-Instruct-generated tasks after screening and modification.The resulting category diversity is intended to evaluate performance and versatility across real-world scenarios.Source: §A.4.2 · p. 25
  • IoA’s generated conclusion is treated as its final answer, while AutoGPT and Open Interpreter are prompted to provide conclusions after completing their multi-step tasks.This procedure aligns the compared systems around a final-answer format.Source: §A.4.2 · p. 25
  • GPT-4 performs pairwise comparisons with alternating response order, counting a win only when superiority is consistent in both orderings.Inconsistent judgments are classified as draws.Source: §A.4.2 · p. 25

A.4.3 EMBODIED AGENT TASKS

The experiments configure IoA for embodied collaboration and retrieval-augmented question answering, using task-specific communication settings and external evidence sources. RocoBench visualization illustrates coordination requirements across several embodied tasks.

A.4.3 EMBODIED AGENT TASKS

  • IoA validates its client implementation and communication mechanism on RocoBench by parsing formatted discussion outputs into environment actions.Two clients communicate without integrated agents, and RocoBench’s predefined parsers execute the resulting actions.Source: §A.4.3 · p. 26
  • For interdependent Sort, Sandwich, and Sweep tasks, IoA retains chat history, whereas Cabinet and Rope start a new group chat for each action to reduce costs.Other experimental settings remain consistent with the Roco Dialog baseline.Source: §A.4.3 · p. 26

A.4.4 RETRIEVAL-AUGMENTED GENERATION

  • RAG agents access Wikipedia’s top 10 retrieved results and Google’s top 5 search results as evidence pools during discussions.The Wikipedia index covers content up to January 20, 2021, and the tools are exposed to client-side LLMs.Source: §A.4.4 · p. 26
  • RAG evaluation samples 500 entries from four datasets and uses GPT-4 to compare dataset answers with model answers for correctness judgments.GPT-4 is instructed to provide chain-of-thought reasoning before its final judgment.Source: §A.4.4 · p. 26

B VISUALIZATION OF ROCOBENCH

  • RocoBench requires agents to coordinate constrained movements for cabinet, sweep, sandwich, sort, and rope-manipulation tasks.The tasks involve shared object manipulation, role coordination, matching placements, or selecting a correct path.Source: B VISUALIZATION OF ROCOBENCH · p. 26

C SIMULATED ENVIRONMENT FOR TEAM FORMATION EVALUATION

This section describes a simulated environment for evaluating IoA’s regular and nested team formation mechanisms. It covers the generation of structured tasks, heterogeneous agent profiles, and sub-tasks handled by existing or newly generated specialist agents.

C.1 REGULAR TEAM FORMATION SIMULATED ENVIRONMENT CONSTRUCTION

  • GPT-4-1106-preview generates diverse tasks and agents for simulated evaluation of regular team formation.The construction uses structured task and agent representations to support evaluation.Source: §C.1 · p. 27
  • The regular dataset samples 25 themes from 399 keyword categories and creates tasks requiring agents with capabilities spanning at least four themes.Task descriptions are generated through the GPT-4 API in JSON format.Source: §C.1 · p. 27
  • For each regular task, GPT-4 constructs at least two agents with varying capabilities, names, types, and descriptions aligned with IoA’s server-side registry.The profile format facilitates agent integration and interaction within IoA.Source: §C.1 · p. 27

C.2 NESTED TEAM FORMATION SIMULATED ENVIRONMENT CONSTRUCTION

  • GPT-4-1106-preview constructs the nested-team environment from two diverse task-and-agent sets derived from the regular-team dataset.The process distinguishes sub-tasks completed by existing agents from sub-tasks requiring additional agents.Source: §C.2 · p. 28
  • For existing-agent sub-tasks, 300 randomly selected regular-task sets are used to prompt GPT-4 to generate structured sub-tasks assigned to existing agents.Sub-task descriptions are produced in JSON format through the GPT-4 API.Source: §C.2 · p. 28
  • For additional-agent sub-tasks, GPT-4 generates requirements demanding specific expertise and creates agents with capabilities distinct from existing agents.The sub-task and additional-agent descriptions include names, types, and profiles in structured JSON format.Source: §C.2 · pp. 28–29
  • Diverse sets of tasks and agents form a simulated environment for evaluating regular and nested team formation.The environment is designed to support comprehensive assessment of both mechanisms.Source: §C.2 · p. 31
  • The evaluation tests whether IoA assembles appropriate teams to satisfy task requirements.Source: §C.2 · p. 31
  • This environment addresses existing benchmarks’ limited provision of suitable large-scale scenarios for agent evaluation.Source: §C.2 · p. 31

Related papers

Loading 2407.07061v2…