PaperlayerView exact revision on arXiv

Source-linked AI summary

Silo-Bench: A Scalable Environment for Evaluating Distributed Coordination in Multi-Agent LLM Systems

Yuzhe Zhang, Feiran Liu, Yi Shan, Xinyi Huang, Xin Yang, Yueqi Zhu, Xuxin Cheng, Cao Liu, Ke Zeng, Terry Jingchen Zhang, Wenyuan Jiang

arXiv:2603.01045v2cs.MAcs.AI

TL;DR

Limited single-model context leaves open whether agents with partial information can coordinate to compute globally correct answers. SILO-BENCH evaluates distributed coordination across information silos and finds that suitable communication does not ensure successful integration of distributed information. Its results identify coordination overhead as a limit on parallelization gains at high complexity.

  • Problem

    It remains underexplored whether LLM agents with partial information can coordinate to compute globally correct answers, a question made important by single-model context limits.

    Source: §1 · pp. 1–2

  • Method

    SILO-BENCH is a role-agnostic configurable environment that evaluates agents’ coordination over partitioned information using task-structural guidance.

    Source: §1 · p. 2

  • Results

    Agents spontaneously form task-appropriate topologies and exchange information, yet fail to integrate what they gather, and coordination overhead eliminates parallelization gains at high complexity.

    Source: §6 · p. 9

  • Takeaways & Limitations

    SILO-BENCH provides evaluation infrastructure for tracking progress in information sufficiency detection, consensus protocols, and adaptive coordination.

    Source: §6 · p. 9

    The evaluation covers three communication protocols, uniform underlying models, and three frontier LLMs, and excludes closed-source models due to cost and unverifiable, incomparable token usage.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models are increasingly deployed in multi-agent systems to overcome context limitations by distributing information across agents. Yet whether agents can reliably compute with distributed information, rather than merely exchange it, remains an open question. We introduce SILO-BENCH, a role-agnostic benchmark of 30 algorithmic tasks across three communication complexity levels, evaluating 54 configurations over 1,620 experiments. Our experiments expose a fundamental Communication-Reasoning Gap: agents spontaneously form task-appropriate coordination topologies and exchange information actively, yet systematically fail to synthesize distributed state into correct answers. The failure is localized to the reasoning-integration stage where agents often acquire sufficient information but cannot integrate it. This coordination overhead compounds with scale, eventually eliminating parallelization gains entirely. These findings demonstrate that naively scaling agent count cannot circumvent context limitations, and SILO-BENCH provides a foundation for tracking progress toward genuinely collaborative multi-agent systems. The code is available at https://github.com/jwyjohn/acl26-silo-bench .

1 Introduction

Single-model context limits motivate asking whether agents with partial information can coordinate to compute globally correct answers. SILO-BENCH evaluates that capability and reveals a gap between communication and reasoning integration.

1 Introduction

  • Limited context windows constrain centralized processing, while distributed agents can theoretically overcome single-model token limits by collaborating over partitioned information.The introduction also notes that quadratic attention costs make centralized processing increasingly impractical for large-scale tasks.Source: §1 · p. 1
  • Whether LLM agents can coordinate across information silos to compute globally correct answers remains underexplored.Existing benchmarks often prescribe communication structures, focus on social simulation, or target specific tasks.Source: §1 · p. 2
  • SILO-BENCH is a role-agnostic, configurable environment that generates evaluation instances and lets agents use their own coordination protocols.High-level task hints make it possible to observe whether agents translate structural understanding into effective coordination.Source: §1 · p. 2
  • The study evaluates 54 configurations and measures task success, token consumption, and communication density.The configurations span diverse protocols and computing paradigms.Source: §1 · p. 2
  • Agents form task-appropriate communication topologies but fail to turn effective coordination into correct distributed computation.Inefficient information synthesis compounds the failure as task complexity and agent scale increase.Source: §1 · p. 2

2 Related Work

Prior work addresses context limits and multi-agent problem solving, but leaves distributed collaboration and flexible coordination undermeasured. SILO-BENCH aims to evaluate that capability without fixed roles or communication scripts.

2 Related Work

  • Long-context benchmarks assess single-agent retrieval, while distributed collaboration across partitioned information remains unmeasured by standard evaluations.The section also notes that retrieval-augmented generation can fracture global context on tasks requiring reasoning across disjoint segments.Source: §2 · p. 2
  • The benchmark distinguishes aggregation, neighbor-to-neighbor mesh exchange, and all-to-all global-shuffle communication patterns.These levels characterize different coordination demands.Source: Figure 2 · p. 3
  • Role-specialized agents and fixed workflows can entangle reasoning with role priors or constrain information flow.SILO-BENCH instead uses role-agnostic task guidance and dynamic instances to assess whether agents develop effective coordination protocols.Source: §2 · pp. 2–3

3 SILO-BENCH

SILO-BENCH evaluates multi-agent collaboration under information silos through configurable tasks, metrics, and an execution pipeline. Its tasks span three levels of communication complexity, and its evaluation distinguishes answer correctness, partial quality, computational cost, and coordination.

3.2 Task Construction Pipeline.

  • Each configuration combines agent scale, communication protocol, and model; SILO-BENCH generates reproducible task instances from independently implemented random-input generators.The pipeline scales global input with agent count to keep per-agent workload constant and supplies exact ground-truth answers.Source: §3 · p. 3, §3.2 · p. 4

3.1 Task Space

  • Level I aggregation uses O(N) communication, Level II mesh tasks use O(N) communication, and Level III global-shuffle tasks span O(N log N) to O(N^2).Their representative structures are respectively reduction, neighbor-to-neighbor propagation, and potentially all-to-all exchange.Source: §3.1 · pp. 3–4

3.3 Evaluation Metrics

  • The metric suite also tracks token cost and communication density, distinguishing computational cost from the intensity of interagent interaction.Together, the metrics describe outcomes and how agents coordinate.Source: §3.3 · pp. 4–5
  • Success Rate measures correct convergence, while Partial Correctness Score captures task-specific partial answer quality and its gap from success isolates reasoning-integration loss.The score uses tolerance for Level I, correctly computed elements for Level II, and the longest correctly ordered subsequence for Level III.Source: §3.3 · p. 4

3.4 Execution Pipeline

  • The execution pipeline partitions global data without giving any agent privileged information, initializes agents with task guidance and protocol constraints, and runs parallel iterative collaboration before scoring answers and communication logs.Agents act on messages from the previous round; messages or files written in a round become visible in the next.Source: §3.4 · p. 5

4 Experiments

The experiments vary agent scale, communication protocol, and language model, then compare distributed performance with a single-agent baseline. Across configurations, agents often fail to turn coordination and acquired information into correct answers, with failures worsening as scale and task complexity increase.

4.1 Experimental Setup

  • The factorial evaluation spans six agent scales, three protocols, and three models, yielding 54 configurations and 1,620 experiments.A single-agent N=1 baseline provides the upper-bound comparison used to quantify coordination cost.Source: §4 · p. 5, §4.1 · pp. 5–6

4.2 Overall Performance

  • 36.9% average success rate for DeepSeek-V3.1 leads GPT-OSS-120B at 16.9% and Qwen3 at 8.2%, a 4.5× spread across models.Even the strongest model fails nearly two-thirds of the time under information silos.Source: §4.2 · p. 6
  • At k=2, multi-agent systems lose 15–49% of single-agent performance, while RCC reaches 80–100% at k=50 for Level-II and Level-III tasks.The Level-I to Level-III single-agent success difference is only about 15 percentage points, versus over 18 points for multi-agent systems.Source: §4.2 · p. 6, §4.1 · p. 6
  • At N≥50 on Level-III tasks, Success Rate falls to 0% while Partial Correctness Score remains at 8–16%, locating a bottleneck in integrating acquired information.On Level-I tasks, DeepSeek-V3.1 averages 88.0% PCS but 62.0% SR, a 26-point gap.Source: §4.2 · p. 6
  • DeepSeek-V3.1 drops from 62% on Level-I to 12% on Level-III, and Level-III success reaches zero at N≥50 while Level-I remains above 40% at 100 agents.All models degrade as agent count increases; communication density also decreases at larger scales.Source: §4.2 · p. 6

4.3 Protocol Suitability

  • Protocol performance varies by model: DeepSeek-V3.1 scores 40% with BP versus 32% with SFS, while GPT-OSS-120B scores 20% with P2P versus 14% with BP.SFS generally underperforms BP despite comparable information-transfer volume.Source: §4.1 · p. 5, Figure 3 · p. 5, §4.3 · p. 6

5 Analysis and Discussion

The analyses show that agents often form broadly task-appropriate communication patterns, but execution and reasoning failures prevent reliable distributed computation. At high complexity, coordination overhead can eliminate parallelization gains, motivating mechanisms for information-sufficiency checks, consensus, and adaptive protocols.

5.1 Case Study: Emergent Coordination Patterns

  • Agents form broadly appropriate topologies for Global Max, Prefix Sum, and Distributed Sort, but still over-communicate, distribute loads unevenly, or depart from the optimal pattern.Global Max develops a near-star with Agent 0 as aggregator; Prefix Sum shows neighbor communication plus redundant broadcasts; Distributed Sort is dense but uneven.Source: §5.1 · pp. 6–7

5.2 The Communication-Reasoning Gap

  • Premature Submission (37.2%), Consensus Failure (29.9%), and Computation Error (28.6%) capture failures to gather sufficient information, converge, or compute correctly.The modes can co-occur; 67 runs showed both Premature Submission and Consensus Failure.Source: §5.2 · p. 8
  • The Communication-Reasoning Gap lies in reasoning at the integration stage: agents manage coordination mechanics but fail to judge information sufficiency and synthesize distributed state.The analysis identifies reasoning quality, rather than communication intent, as the bottleneck.Source: §5.2 · p. 8

5.3 Implications and Future Directions

  • For Level-III tasks at N≥50, coordination overhead eliminates parallelization gains, while emergent leaders can hurt performance by becoming overloaded with global data.The aggregator must process a volume of global data it cannot handle.Source: §5.3 · pp. 8–9
  • The paper proposes information-sufficiency detection, formalized consensus protocols, and task-structure-based adaptive protocol selection as directions for future work.SILO-BENCH is presented as an evaluation foundation for tracking progress on these directions.Source: §5.3 · p. 9

6 Conclusion

The conclusion identifies a gap between agents’ communication competence and their ability to reason over distributed information. It argues that coordination alone does not reliably overcome information silos and points to mechanisms for improving information checks, consensus, and coordination strategies.

6 Conclusion

  • Agents form task-appropriate topologies and exchange information actively, yet fail to integrate what they gather, exposing a Communication-Reasoning Gap.The conclusion points to PCS–SR divergence and RCC analysis as evidence of this dissociation.Source: §6 · p. 9
  • At high complexity, coordination overhead eliminates parallelization gains, and spontaneous leader emergence can hurt performance by creating data-processing bottlenecks.The aggregator can become overwhelmed by global data volume.Source: §6 · p. 9
  • The conclusion calls for information-sufficiency detection, explicit consensus protocols, and adaptive coordination strategies, with SILO-BENCH tracking progress.Source: §6 · p. 9

Limitations

SILO-BENCH’s appendix specifies the scope and limits of its evaluation, three communication protocols, their round-based execution, and prompts that provide structural guidance without assigning roles or prescribing steps.

Limitations

  • The evaluation covers only three communication protocols, uniform-model configurations, and three frontier LLMs; it excludes closed-source models and may miss other models’ failure modes.The protocols omit hierarchical, gossip-based, and hybrid approaches; closed-source models were excluded because of cost and unverifiable, incomparable token-usage reports.Source: Limitations · p. 9

A.1 Protocol Overview

  • The protocols constrain how information is exchanged while leaving agents autonomous, spanning targeted peer messages, broadcast messages, and shared-state coordination.P2P requires named recipients, BP broadcasts without addressing, and SFS routes coordination through a shared file system; their default communication density also differs.Source: A Communication Protocols · p. 13, §A.1 · p. 13, Table 5 · p. 14

A.2 Peer-to-Peer Protocol (P2P)

  • P2P uses private SQLite-backed inboxes and asynchronous directed messages, requiring agents to choose both message content and recipients.Delivery is guaranteed within each sender-recipient pair, but there is no global ordering.Source: §A.2 · p. 13

A.3 Broadcast Protocol (BP) / A.4 Shared File System Protocol (SFS)

  • BP delivers each broadcast to all other agents through a compiled history, whereas SFS enables indirect coordination through a shared key-value store.SFS writes become visible to subsequent reads, and concurrent writes to the same path use last-writer-wins semantics.Source: §A.3 · p. 13, §A.4 · p. 13

A.5 The wait() Action: Formal Specification

  • Across protocols, wait() marks completion of an agent’s round; at the next round, agents see the prior round’s messages or committed shared-file state.The synchronous scheduler provides a consistent communication-state snapshot at round boundaries; submitted agents are not activated again.Source: §A.5 · pp. 13–14

B Prompt Design: Structural Guidance without Role Prescription / B.3 Design Principles

  • Prompts convey task structure and possible coordination patterns while leaving concrete decisions to agents, assigning no semantic roles or mandatory sequence.Agents must discover timing, recipients, consensus, and verification mechanisms themselves; all receive structurally identical prompts apart from ID and data shard.Source: B Prompt Design: Structural Guidance… · p. 14, §B.3 · pp. 14–15

B.1 Base Prompt Template

  • The base initialization prompt gives each agent its ID, team size, task description, local data, protocol, and available actions, and asks it to submit a globally correct answer.It explicitly says no single agent has enough information to solve the task independently.Source: §B.1 · p. 14

B.2 Task Description Examples

  • Task-description examples span global maximum, prefix sum, and distributed sort, illustrating aggregation, sequential dependencies, and global data reorganization.The examples describe the task structure and potential coordination patterns without prescribing a fixed execution sequence.Source: §B.2 · p. 14

C Experimental Details

The experiments probe coordination from pairwise teams to large groups, use a bounded synchronous execution setup, and run open-source models locally on a GH200 cluster.

C.1 Agent Scale Rationale

  • The six team sizes—N = 2, 5, 10, 20, 50, and 100—probe pairwise, peer-trackable, selective-communication, and large-scale coordination regimes.The authors describe hierarchical or highly selective coordination as effectively necessary at N = 50 and 100, and largely beyond current LLMs’ reach.Source: §C.1 · p. 15

C.2 Execution Parameters

  • Each configuration allows up to 100 synchronous rounds, with prior-round communications visible at the next round; configurations run once with fixed data-generation seeds.Agents that do not submit within the round limit receive a null answer counted as incorrect.Source: §C.2 · p. 15

C.3 Infrastructure

  • Experiments ran on a GH200 cluster, with up to 50 concurrent configurations and approximately 500+ GPU-hours equivalent for each experiment.Complete conversation histories, token counts, and timing information were recorded for all runs.Source: §C.3 · p. 15

C.4 Model Licenses and Intended Use

  • The study uses locally deployed open-source models: DeepSeek-V3.1, GPT-OSS-120B, and Qwen3-Next-80B-A3B, all described as permitting research and commercial use.The passage lists MIT for DeepSeek-V3.1 and Apache 2.0 for the other two models.Source: §C.4 · p. 15

D Token Budget Feasibility Analysis

The token analysis profiles prompt, data, and communication costs, then assesses context-window use and the model and task scales that fit different context budgets.

Token-cost profiling

Context-window utilization

  • DeepSeek-V3.1 averages 7.5% of its 128K context budget, while GPT-OSS-120B and Qwen3 each remain below 3%.The 95th-percentile cases are associated with redundant broadcasting, extended verbose rounds after failed convergence, or agents copying full message histories.Source: D Token Budget Feasibility Analysis · pp. 15–16

Feasibility by context size

  • Frontier models with 128K–200K context can run all configurations comfortably; 32K models handle over 90%, while 8K models suit N≤10 and Level I–II tasks.Level-III configurations at N≥50 may require truncation on 32K models.Source: D Token Budget Feasibility Analysis · p. 16

E Complete Task Specifications

The task-specification appendix maps SILO-BENCH tasks to their algorithmic foundations and identifies each task’s distributed adaptation approach.

E Complete Task Specifications

F.1 Task-Level Breakdown for DeepSeek-V3.1

DeepSeek-V3.1’s task-level results range from near-perfect Level-I aggregation to zero success on several Level-III tasks; broader results show performance falls with complexity and agent count.

F.1 Task-Level Breakdown for DeepSeek-V3.1

  • Level-I Distributed Vote and Any Match achieve near-perfect success across protocols, while four listed Level-III tasks achieve zero success across all protocols.The zero-success tasks are K-Means Iteration, Collaborative Filtering, PageRank Step, and Matrix Multiply.Source: §F.1 · p. 16, Table 10 · p. 18

F.2 Results by Model, Protocol, and Difficulty

  • Across all three models, P2P generally matches or outperforms BP for GPT-OSS-120B and Qwen3, SFS underperforms, and success declines with complexity and agent count.Source: §F.2 · p. 16

F.3 Communication Density Analysis

  • P2P produces higher communication density than BP across models, while SFS is sparser than P2P and systematically underperforms on success rate.BP densities cluster below 1.0; the passage attributes this to one-to-all single broadcasts and counts SFS transfer through successful file reads.Source: §F.3 · p. 16

G.1 Representative Failure Cases

  • Observed failures include submitting after contacting only 28 of 100 peers, consensus breakdown despite near-unanimous agreement, and off-by-one arithmetic after gathering all required data.Source: §G.1 · p. 16

G.2 Failure Mode Definitions and Co-occurrence

  • Premature submission and consensus failure co-occurred in 67 cases, suggesting early submissions can leave remaining agents with incomplete information and widen the convergence gap.The defined failure modes distinguish insufficient information at submission, unsynchronized answers, and incorrect computation despite sufficient information.Source: §G.2 · p. 17

G.3 Behavioral Patterns and Leader Emergence

Successful runs use fewer rounds than failed runs, while spontaneous leadership does not consistently improve outcomes. Illustrative successes use explicit synchronization, and prompt scaffolding does not remove the Communication-Reasoning Gap.

G.4 Successful Coordination Examples

  • Two successful patterns—centralized coordination and neighbor verification—both use explicit synchronization checkpoints; the 10-agent verification case succeeds despite higher message overhead.The five-agent coordinator case reaches 100% success within four rounds.Source: §G.4 · p. 18

H Prompt Scaffold Ablation

  • The planning round yields the most consistent gains (∼5–8 on Level-II/III), but the Communication-Reasoning Gap persists across scaffolds and scratchpad hints cannot prevent collapse at N≥50.Protocol reminders primarily help Level-I, while scratchpad hints benefit intermediate scales (N=10–20).Source: H Prompt Scaffold Ablation · p. 19, Table 18 · p. 20

G.3 Behavioral Patterns and Leader Emergence

  • Successful runs take 8.3 versus 12.7 rounds, verification appears in over 95% of runs regardless of outcome, and Level-III leader runs achieve 0% success versus 33.3% without a leader.Source: §G.3 · p. 18, Table 16 · p. 20, Table 17 · p. 20
Loading 2603.01045v2…