Source-linked AI summary
SwarmBench: Can Large Language Models Act as Agent Swarm Orchestrators?
Jinshan Gao, Zhuoran Jin, Tianyi Men, Kang Liu, Jun Zhao
TL;DR
Existing benchmarks do not systematically evaluate the orchestration capabilities required by dynamically organized Agent Swarms. The paper introduces SwarmBench, which measures these capabilities across multiple performance and process dimensions, and proposes SwarmExp based on extracted and replayed experience. Experiments show substantial differences among models and consistent improvements from SwarmExp.
Problem
Existing evaluations rely heavily on single-agent or general-purpose-agent benchmarks and final-task accuracy, limiting systematic assessment of Agent Swarm orchestration and collaboration-process quality.
Method
SwarmBench evaluates task decomposition, dynamic subagent creation and delegation, result aggregation, accuracy, efficiency, cost, and process quality across 8 tasks and 400 samples.
Results
Current models show uneven orchestration capabilities across accuracy, efficiency, cost, and process quality, with orchestration structure more important than simply increasing cost or parallelism.
Takeaways & Limitations
SwarmBench identifies orchestration bottlenecks and provides an open benchmark, while SwarmExp consistently improves orchestration performance across representative tasks.
Takeaways & Limitations
The benchmark covers only eight tasks, uses a fixed lightweight scaffold and subagent model pool, and partly relies on LLM-based judging that may introduce bias.
Abstract
from arXiv · showhide
Large language model-based multi-agent systems are evolving from fixed interaction topologies toward dynamically orchestrated Agent Swarms. However, existing benchmarks are still largely based on single-agent or general-purpose agent tasks, making it difficult to systematically evaluate key orchestration capabilities. We propose SwarmBench, a benchmark that evaluates model performance from multiple perspectives, including accuracy, efficiency, cost, and process quality. Experimental results show that current models exhibit substantial differences in orchestration capability. These differences are reflected not only in final accuracy, efficiency, and cost, but also in the overall quality of the orchestration process itself. Based on these findings, we further propose SwarmExp, a simple yet effective method based on experience extraction and experience replay, which consistently improves the orchestration performance of large language models.
1 Introduction
LLM-based multi-agent systems are shifting from fixed collaboration structures toward dynamic Agent Swarm orchestration, but existing benchmarks do not adequately evaluate this capability. SwarmBench addresses this gap by assessing orchestration across task capabilities, performance metrics, and process quality, while experiments reveal uneven model abilities and motivate SwarmExp.
- Motivation: Dynamic Agent Swarm systems shift orchestration from predefined roles and communication structures toward runtime organization of parallel subagents.The central challenge is whether LLMs can efficiently organize dynamic swarms for complex tasks.
- Benchmark gap: Existing benchmarks largely reuse single-agent or general-purpose-agent tasks and emphasize final answer quality over orchestration-process quality.They provide limited systematic evaluation of decomposition, subagent creation, delegation, and result aggregation.
- SwarmBench: SwarmBench organizes evaluation into task decomposition, subagent creation and delegation, and subagent result aggregation across 8 tasks and 400 samples.The benchmark covers tasks including root cause analysis, treasure hunt, and wide search.
- Evaluation: The benchmark evaluates accuracy, efficiency, cost, and process quality across task decomposition, subagent delegation, and result aggregation.Process quality provides fine-grained analysis of orchestration quality beyond final outcomes.
- Findings and method: Current models show uneven orchestration capabilities, while SwarmExp extracts experience from trajectories and reinjects it into the main agent context to improve performance consistently.The findings identify aggregation as a frequent bottleneck and report improvements from the experience-driven method.
2 SwarmBench
SwarmBench defines Agent Swarm as a hierarchical workflow in which an orchestrator dynamically creates, configures, delegates to, and aggregates outputs from heterogeneous subagents, often in parallel. Its benchmark spans three orchestration capability categories and evaluates them under a controlled lightweight scaffold.
- Definition of Agent Swarm: Agent Swarm is a hierarchical multi-agent system where an orchestrator coordinates a dynamically created set of subagents to complete a task.The paper distinguishes this from static workflows by emphasizing runtime orchestration.
- Definition of Agent Swarm: Agent Swarm combines hierarchical orchestration, dynamic heterogeneous subagent instantiation, and parallel subagent execution.Subagents may differ in roles, backbone models, tools, and local contexts, then execute subtasks concurrently.
- Formalization: Given an input task, the formal process includes an orchestrator, subagent configurations, orchestration and execution policies, tools or environments, and an execution budget.The budget can include step limits, agent count, latency, or monetary cost.
- Runtime orchestration: At each global step, the orchestrator observes the global state and outputs actions that can dynamically create subagents with specified roles, models, tools, and local contexts.This runtime action space distinguishes Agent Swarm from a fixed subagent set.
- Runtime orchestration: Parallelism is represented by the number of currently running agents, and stages with multiple concurrent agents constitute swarm behavior when subtasks are parallelizable.The paper defines system parallelism as ρ_t = |H_t|.
- Dataset construction: SwarmBench evaluates decomposition, dynamic subagent creation and delegation, and result aggregation through three task categories covering eight task types.The categories include decomposition-oriented, delegation-oriented, and aggregation-oriented tasks.
- Evaluation framework: The lightweight framework assigns the main agent responsibility for decomposition, creation, assignment, and aggregation, while subagents execute delegated subtasks in local contexts.The main agent cannot call tools or answer independently, and some task context is hidden from it.
3 Experiments
Experiments evaluate Agent Swarm orchestrators across accuracy, cost, efficiency, parallel execution, and process quality. Results show that orchestration quality—not simply model size or concurrency—shapes performance, with aggregation a recurring bottleneck.
- Evaluation setup: Models are evaluated using task accuracy, execution efficiency, monetary cost, and process quality across decomposition, delegation, and aggregation.Efficiency includes parallelism-based time gain, while process quality uses LLM-based scores for three orchestration dimensions.
- Accuracy results: GPT-5.4 achieves the best performance on six of eight tasks and is described as the strongest and most balanced orchestrator.Claude-Sonnet-4-6 is the closest proprietary competitor, while open-source models show more fragmented task-specific strengths.
- Cost-accuracy trade-off: Cost is not a linear function of performance: some tasks reach near-optimal results at lower cost, while larger models often offer better cost-performance trade-offs.Qwen3.5-397b-a17b more frequently lies on the Pareto frontier, whereas Qwen3-30b-a3b often costs more while performing worse.
- Parallelism and time efficiency: Parallelism rate does not determine parallelism gain; Gemini-3-flash achieves the highest gain with a moderate rate, while Kimi-k2.5 has the highest rate.Effective parallel structures reduce redundant waiting and repeated execution while organizing beneficial collaborative workflows.
- Orchestration quality: Task-centric scores identify decomposition difficulty for BD, MTU, and MATH, delegation difficulty for MPA, RCA, and TH, and aggregation difficulty for LTG and WS.MATH is hardest for decomposition, while LTG and WS show the largest aggregation drop.
- Orchestration quality: GPT-5.4 has the strongest and most stable orchestration quality, while aggregation scores are consistently lower than decomposition and delegation.Claude-Sonnet-4-6 is more balanced, whereas Qwen3.5-397b-a17b is stronger in decomposition and delegation.
4 SwarmExp: Experience-Driven Swarm Orchestration
SwarmExp improves Agent Swarm orchestration by extracting reusable experience from trajectories and replaying it during planning, delegation, and aggregation. It achieves the best results across four evaluated tasks, with transferable gains across models.
- Method: SwarmExp extracts skills, tricks, and model cards from Agent Swarm trajectories, then injects them into the main agent during inference.Skills encode workflow patterns, tricks capture fine-grained strategies, and model cards summarize subagent capabilities.
- Main results: SwarmExp achieves the best performance on all four evaluated tasks, improving over the initial Agent Swarm by +0.25 on MTU, +6.42 on RCA, +6.46 on LTG, and +2.92 on WS.The evaluation compares SwarmExp with single-agent, Best-of-N, Multi-Model Voting, and AOrchestra baselines.
- Transferability: Experience transfers across models, with 14 out of 16 settings improving after GPT-5.4-derived experience is injected.Gains are most obvious on LTG, while RCA and WS show more stable improvements; effects vary by target model.
- Ablation: Removing any experience component reduces performance, with model-card removal causing the largest average decline.Skill and trick removal also hurts performance, indicating contributions from both workflow templates and fine-grained execution guidance.
5 Related Work
Related work has moved from fixed multi-agent collaboration structures toward dynamic coordination, but evaluation remains insufficiently specific and fine-grained for Agent Swarm orchestration.
- Evolution of multi-agent systems: Earlier multi-agent systems commonly used predefined roles and interaction patterns such as debate, pipelines, and hierarchical manager-worker structures.Recent work shifts toward workflows that can adapt during execution.
- Evaluation gap: Many studies evaluate multi-agent systems with benchmarks designed for single-agent or general-purpose agents.These benchmarks do not systematically assess Agent Swarm orchestration capabilities.
- Evaluation gap: Existing evaluations often underemphasize whether systems decompose tasks appropriately, assign subagents effectively, and convert parallelism into meaningful gains.MultiAgentBench and related work share similarities, but SwarmBench targets the more specific Agent Swarm paradigm with finer-grained analysis.
6 Conclusion
SwarmBench evaluates Agent Swarm orchestration across tasks and multiple performance dimensions, finds clear limitations in current models, and motivates SwarmExp as an experience-driven improvement method.
- SwarmBench: SwarmBench contains 8 tasks and 400 samples covering task decomposition, subagent delegation, and result aggregation.It evaluates accuracy, efficiency, cost, and process quality.
- Conclusion: Current models still have clear limitations as swarm orchestrators, and performance depends more on effective orchestration structure than simply increasing cost or parallelism.The benchmark is intended to support future research on large language models as Agent Swarm orchestrators.
- Conclusion: SwarmExp is proposed as a simple experience-driven method for improving orchestration capability.Its design is based on extracting and reusing experience from task execution trajectories.
Limitations
SwarmBench’s conclusions are bounded by its task coverage, fixed experimental scaffold, and partial reliance on LLM-based judging.
- Scope: SwarmBench includes only eight tasks and may not capture the full range of Agent Swarm scenarios.This limits how broadly the benchmark can represent orchestration settings.
- Experimental setting: Evaluation uses a unified lightweight swarm scaffold with a fixed subagent model pool, which improves comparability but may limit conclusion generality.The scope boundary concerns both the scaffold and the available subagent models.
- Evaluation: Process analysis relies partly on LLM-based judging, which may introduce bias.The authors characterize this analysis as informative while acknowledging its potential limitation.
Ethical considerations
The work notes risks associated with multi-agent systems and states that its datasets are public and contain no personally identifiable information.
- Multi-agent systems may introduce error propagation, unintended coordination behaviors, and amplification of incorrect or misleading outputs.
A Data
SwarmBench comprises eight tasks spanning diverse orchestration requirements, including decomposition, delegation, and aggregation. The benchmark uses interactive environments and rule-based evaluation to measure both final outcomes and process behavior.
- Dataset composition: 8 tasks and 400 samples form SwarmBench, covering task decomposition, subagent creation and delegation, and subagent result aggregation.The dataset includes root cause analysis, treasure hunt, and wide search among other tasks.
- Batch Download: Batch Download tests decomposition of a large retrieval objective into parallel subtasks while maintaining consistency in the final file set.Its environment is a deterministic web-like universe containing synthetic pages, downloadable files, and distractors.
- Treasure Hunt: Treasure Hunt evaluates dynamic delegation in a partially observable grid world containing a base, keys, chests, walls, and swamp obstacles.Instances use 9x9 map templates with varied topologies and randomized object placement under reachability constraints.
- Treasure Hunt: Treasure Hunt assigns scouts and carriers distinct roles: scouts explore, collect keys, and open chests, while carriers retrieve opened chests to the base.The main agent remains stationary and dynamically creates and assigns these helper agents.
- Evaluation: Treasure Hunt scores the normalized value of chests delivered to the base and records auxiliary metrics such as exploration, actions, failures, and duplicate reveals.Opening a chest alone does not contribute to the primary score.
B Models
The experiments evaluate a diverse set of proprietary and open-source models, with ten candidates serving as orchestrators and a shared subagent model pool.
- All orchestrators share the same subagent model pool, enabling comparisons across orchestrator models under a common subagent configuration.
C Metrics
SwarmBench combines task-specific accuracy measures with efficiency, cost, process-quality, stability, and human-consistency analyses. Its experiments compare Agent Swarm against single-agent baselines under both original and matched-cost settings.
- Accuracy: Download F1 is the primary Batch Download accuracy metric, computed over required files using precision and recall.A file is correct only when it appears at the expected path and matches the gold checksum.
- Accuracy: Treasure Hunt reports task score as delivered chest value normalized by the total possible chest value.The evaluator uses the final environment state rather than the textual answer.
- Efficiency: Parallelism Rate measures the duration-weighted average number of concurrently active subagents, while Parallelism Gain measures latency reduction from overlapping execution.A higher Parallelism Rate does not by itself imply effective parallel execution; Parallelism Gain near 1 indicates nearly serial behavior.
- Cost: Total cost USD aggregates model-specific input and output token costs across calls from both the main agent and subagents.The paper reports monetary cost rather than raw token counts.
- Baseline comparison: Cost-matched single-agent baselines receive the average Agent Swarm cost as their budget ceiling and iteratively reason, reflect, and revise within that budget.
- Baseline comparison: Agent Swarm outperforms the cost-matched single agent on MTU, LTG, and WS, suggesting benefits beyond increased cost from its orchestration structure.The reported structure includes decomposition, delegation, complementary model capabilities, and localization of errors.
- Reliability: Repeated-run experiments execute selected model-task settings five times and report mean ± standard deviation to assess statistical stability.The representative subset includes MATH, BD, MTU, MPA, and LTG.
- Reliability: Human-consistency analysis compares human and LLM scores for task metrics and process quality using Pearson and Spearman correlations.It samples 100 outputs for each of LTG and MPA and 50 trajectories per task across all eight tasks.
D.4 Cross-Task Transfer of SwarmExp
SwarmExp’s cross-task transfer is task-dependent: experience from other tasks helps LTG, whereas WS benefits only from WS-specific experience. The results suggest that transferable orchestration patterns coexist with task-specific strategies.
- Transfer setup: Cross-task transfer evaluates whether experience extracted from MTU, RCA, LTG, and WS can improve performance on LTG and WS.Claude-Haiku-4-5 orchestrates the target tasks using experience extracted from GPT-5.4 trajectories.
- Transfer results: LTG improves with experience from other tasks, but LTG-specific experience performs best.The result suggests that high-level patterns such as long-horizon planning and result aggregation can transfer to open-ended generation tasks.
- Transfer results: Table 10 reports cross-task transfer results by source task for the LTG and WS target tasks.
- Transfer results: WS benefits only from WS-specific experience, while most cross-task experience hurts performance.This indicates that search-oriented tasks rely more heavily on task-specific retrieval and aggregation strategies.