PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards a Science of Scaling Agent Systems

Yubin Kim, Ken Gu, Chanwoo Park, Chunjong Park, Samuel Schmidgall, A. Ali Heydari, Yao Yan, Zhihan Zhang, Yuchen Zhuang, Yun Liu, Mark Malhotra, Paul Pu Liang, Hae Won Park, Yuzhe Yang, Xuhai Xu, Yilun Du, Shwetak Patel, Tim Althoff, Daniel McDuff, Xin Liu

arXiv:2512.08296v3cs.AI

TL;DR

How agent performance changes with coordination, model capability, and task factors remains underexplored. This paper evaluates controlled architectures across benchmarks and finds that architecture–task alignment, rather than agent count, determines collaborative success, with outcomes ranging from +80.8% to −70.0% versus single-agent baselines.

  • Problem

    The conditions under which multi-agent coordination benefits agentic tasks over single-agent systems remain insufficiently understood.

  • Method

    The study controls prompts, tools, and compute while varying five architectures and model capabilities across 260 configurations, six benchmarks, and three LLM families.

  • Results

    +80.8% to −70.0% relative to single-agent baselines, with architecture–task alignment determining collaborative success across evaluated tasks.

  • Takeaways & Limitations

    Agent effectiveness depends on matching coordination structure to task characteristics rather than increasing the number of agents.

  • Takeaways & Limitations

    The six evaluated benchmarks may not capture the full spectrum of agentic task characteristics, including embodied, multi-user, and long-horizon temporal environments.

Abstract

from arXiv · show

Agents, language model-based systems capable of reasoning, planning, and acting are widely adopted in real-world tasks, yet how their performance changes as these systems scale across key dimensions remains underexplored. We introduce quantitative scaling principles for agent systems as a predictive model, capturing how performance varies with coordination, model capability, and measurable system and task factors. Across 260 configurations spanning six agentic benchmarks, five canonical architectures (Single-Agent and four Multi-Agent: Independent, Centralized, Decentralized, Hybrid), and three LLM families, we perform controlled evaluations, standardizing tools, prompts, and compute to isolate architectural effects. The resulting model achieves a cross-validated R^2=0.373 across all six benchmarks (R^2=0.413 with a task-grounded capability metric). We identify a robust capability-saturation effect and additional patterns: (1) a coordination yields diminishing returns once single-agent baselines exceed certain performance; (2) tool-heavy tasks appear to incur multi-agent overhead; and (3) architectures without centralized verification tend to propagate errors more than those with centralized coordination. Relative performance change compared to single-agent baseline ranges from +80.8% on decomposable financial reasoning to -70.0% on sequential planning, demonstrating that architecture-task alignment determines collaborative success. The framework identifies the best-performing architecture for 87% of held-out configurations and shows consistent relative architecture preferences on unseen frontier models. Agent effectiveness depends on alignment between coordination and task structure, and that mismatched coordination degrades the performance.

1. Introduction

The paper develops a controlled framework for explaining when multi-agent coordination helps or harms agentic tasks, where architecture–task alignment determines success rather than agent count alone. Across 260 configurations, it quantifies scaling trade-offs involving capability, coordination overhead, and error propagation.

  • Agentic task definition: Agentic tasks require sustained multi-step environmental interaction, iterative information gathering under partial observability, and adaptive action, distinguishing them from static single-shot benchmarks.This distinction explains why monotonic team-size gains on static benchmarks do not generalize to tasks dominated by coordination overhead and error propagation.
  • Controlled evaluation: The study controls prompts, tools, computational budgets, and model capability while varying coordination structure across five architectures, three LLM families, six benchmarks, and 260 configurations.This design addresses prior evaluations that confounded architectural effects with implementation choices.
  • Contributions: The predictive framework achieves cross-validated R^2=0.373 across six benchmarks and R^2=0.413 with a task-grounded capability metric, identifying architecture–task alignment as the determinant of collaborative success.It uses empirical coordination metrics alongside reasoning capability and task properties to predict performance and identifies the optimal architecture with 87% accuracy.
  • Scaling patterns: Relative performance ranges from +80.8% for structured financial reasoning under centralized coordination to −70.0% for sequential planning under independent coordination.Decentralized coordination benefits dynamic web navigation by +9.2%, while architecture preferences vary systematically with task structure.
  • Scaling patterns: Multi-agent coordination incurs diminishing or negative returns on difficult agentic tasks: tool-heavy workflows suffer coordination overhead, while tasks exceeding 45% single-agent accuracy experience capability saturation.The tool-coordination trade-off has β=−0.096, p=0.002, and the capability ceiling has β=−0.236, p=0.004.

2. Related Work

Prior work distinguishes single-agent from topologically diverse multi-agent systems and separates interactive agentic tasks from static benchmarks. Findings suggest coordination benefits are domain- and architecture-dependent, motivating systematic evaluation of when collaboration outperforms strong single agents.

  • Multi-Agent Systems versus Single-Agent Systems: Single-agent systems use one reasoning locus, whereas multi-agent systems coordinate multiple LLM-backed agents through message passing, shared memory, or orchestrated protocols.MAS topologies include Independent, Decentralized, Centralized, and Hybrid architectures.
  • Scaling Laws and Coordination Mechanisms: The value of multi-agent coordination over strong single models with tools remains empirically open, as proposed scaling laws found no significant universal pattern [1].This motivates systematic evaluation of domain-specific collaborative scaling.
  • Agentic Tasks and Benchmarks: Agentic tasks require sustained multi-step interaction, information gathering under partial observability, and adaptive refinement from feedback, unlike single-shot non-agentic benchmarks.Examples span browsing [4] [46], trading [33], software engineering, and planning [21], while non-agentic benchmarks include GSM8K [48], MMLU [49], HumanEval [26], and SQuAD [50].
  • Agentic Tasks and Benchmarks: On non-agentic benchmarks, ensembles can improve monotonically—89% on HumanEval with five agents—whereas agentic coordination incurs interaction overhead and cascading errors.After 10 interactions, agents show 34% overlap, illustrating divergent world states [23].
  • Scaling Laws and Coordination Mechanisms: Coordination mechanisms and architecture-task alignment can determine whether collaboration amplifies or degrades performance, with query-dependent configurations achieving superior performance at 6-45% cost.Prior work includes workflows mitigating hallucination cascades [28] and structured interactions producing emergent behaviors [56].

3. Agent Systems and Tasks

The paper formalizes agent systems by their agents, environment, communication topology, and orchestration policy, distinguishing single-agent from multi-agent computation. It defines coordination architectures, agentic-task requirements, and error-amplification metrics to analyze how structure affects performance.

  • Agent-system formalism: An agent system comprises agents, a shared environment, communication topology, and orchestration policy; one agent defines SAS, while more than one defines MAS.Agents perceive, reason, and act through iterative feedback.
  • Architecture taxonomy: Communication topology determines information flow, while orchestration controls aggregation, overrides, persistent memory, and termination based on consensus or quality thresholds.The defined topologies are agent-to-aggregator Independent, orchestrator-to-agent Centralized, all-to-all Decentralized, and combined Hybrid.
  • Architecture taxonomy: The five architectures isolate coordination mechanisms from generic multi-agent effects, spanning sequential SAS and Independent, Centralized, Decentralized, and Hybrid MAS.Independent tests parallelism, Decentralized adds peer fusion, Centralized adds hierarchical verification, and Hybrid combines hierarchy with lateral flexibility.
  • Error metrics: Architectures with verification mechanisms contain errors more effectively than independent coordination, as measured by task-level and trace-level error amplification factors.The task-level factor compares MAS and SAS error rates, with values above one indicating net error amplification and below one indicating suppression.
  • Agentic tasks: Agentic benchmarks require sequential interdependence, partial observability, and adaptive strategy formation through interaction, information gathering, planning, and belief revision.Static benchmarks such as GSM8K and MMLU evaluate reasoning rather than currently necessary agentic capabilities.

4. Experiments & Results

Across 260 configurations, multi-agent systems show no universal advantage: mean improvement is −0.3%, with outcomes ranging from −70.0% to +80.8% depending on task structure and architecture. A regression scaling model predicts architecture effectiveness, achieving cross-validated R^2=0.373, or 0.413 with the task-grounded ACI metric.

  • Domain dependence: +80.8% is Finance Agent’s Centralized improvement, whereas PlanCraft degrades by −70.0% because coordination overhead exceeds task complexity in sequential work.Finance benefits from parallel information synthesis, while PlanCraft introduces redundant subtasks and coordination messages.
  • Capability saturation: −2.1% to −14.9% are the SWE-bench Verified degradations across MAS architectures, while Terminal-Bench shows only +1.7% for Independent coordination.The SWE-bench pattern is consistent with capability saturation above the single-agent threshold, whereas Terminal-Bench remains below it.
  • Overall results: −0.3% is the overall mean MAS improvement (95% CI: [−58.7%, +77.2%]), spanning −70.0% for PlanCraft Independent to +80.8% for Finance Centralized.The high variance (σ=37.5%) shows that multi-agent benefits are domain-specific rather than universal.
  • Architecture–LLM interactions: No architecture dominates across domains and vendors: Finance favors Centralized and Decentralized coordination, while vendor-specific preferences emerge within improvement-positive domains.On Finance Agent, Anthropic’s MAS-Centralized reaches +127.5%, demonstrating that coordination effects vary by LLM family.
  • Scaling model: Cross-validated scaling regression reaches R^2=0.373 across all 260 configurations and improves to R^2=0.413 with the task-grounded Agentic Capability Index.The model tests whether continuous empirical coordination metrics explain performance variance better than categorical architecture labels.

5. Limitations

The study’s scaling principles remain limited by coordination overhead, tool-heavy failure modes, restricted heterogeneity and prompting, benchmark coverage, economic costs, and statistical uncertainty. These constraints leave larger collectives, broader environments, and some regression patterns insufficiently validated.

  • Scaling collectives: Communication overhead grows superlinearly with agent count, while coordination efficiency degrades beyond moderate team sizes, limiting conclusions about larger collectives.The framework explores agent counts only up to nine, and beneficial emergent behavior in larger collectives remains unresolved.
  • Model heterogeneity: The heterogeneity analysis is narrow, and preliminary tests found no evidence that mixing models bypasses the capability-saturation threshold.Agents shared base architectures within model families, while the heterogeneous investigation covered 13 BrowseComp-Plus configurations.
  • Tool-intensive tasks: Tool-heavy environments are a primary multi-agent failure mode, with tool count negatively interacting with system efficiency.Potential remedies include explicit tool-access scheduling, capability-aware routing, and hierarchical tool delegation.
  • Prompting: Identical prompts were used across conditions rather than model-specific optimization, so prompt tuning could change the reported scaling characteristics.This limitation follows from known sensitivity of LLM behavior to prompt formulation.
  • External validity: Six diverse benchmarks may not capture the full spectrum of agentic tasks, especially long-horizon temporal dependencies, real-world feedback loops, embodied settings, and multimodal interaction.The reported principles therefore remain to be tested in domains such as robotic control, medical triage, and multi-user social interaction.
  • Cost and inference: Economic viability remains a barrier because token-centric coordination can increase total compute, while dataset-level clustering limits the certainty of regression inference.With G=6 clusters, some predictors significant under naive OLS lose significance under cluster-robust inference; results are framed as descriptive patterns supported by directional consistency.

6. Conclusion

Across 260 controlled configurations, the study characterizes how agent-system performance varies with coordination, model capability, and task properties. Capability saturation is the most robust scaling effect: coordination yields diminishing returns beyond approximately 45% single-agent baselines, confirmed by corrected statistical tests.

  • Conclusion: Coordination yields diminishing returns beyond approximately 45% single-agent baselines, establishing capability saturation as the most robust scaling effect.The finding remains significant under cluster-robust inference (p = 0.004) and Holm–Bonferroni multiple-comparison correction.
  • Conclusion: The study empirically characterizes agent-system performance across coordination structure, model capability, and task properties.The analysis covers 260 controlled configurations spanning three LLM families and six agentic benchmarks.
  • Conclusion: The conclusions are based on controlled evaluations spanning three LLM families and six agentic benchmarks.This scope supports comparing scaling behavior across diverse agent-system configurations.

Data Availability

All six benchmark datasets used in the study are publicly available, with documented instance counts and deterministic subset selection where applicable. Per-instance results for all 260 experimental configurations are provided in the code repository.

  • Data Availability: Six publicly available benchmarks comprise the evaluation: BrowseComp-Plus [32], Finance-Agent [33], PlanCraft [21], Workbench [34], SWE-bench Verified [20], and TerminalBench [25].The datasets contain 100, 50, 100, 100, 500, and 86 instances, respectively.
  • Data Availability: SWE-bench Verified uses a deterministic 20-instance subset selected with seed 42, while TerminalBench uses its first 20 instances.
  • Data Availability: Per-instance results for all 260 experimental configurations are available in the code repository at etc/analysis/.

Code Availability · Appendix

The study provides a public repository containing the evaluation framework and supporting artifacts needed to reproduce the reported analyses.

  • Code Availability: The repository includes evaluation code, configurations, prompt templates, analysis scripts, and sanitized execution traces, with additional reproduction artifacts documented online.Repository: https://github.com/ybkim95/agent-scaling

A. Model Intelligence Index

The study uses an extended Artificial Analysis Intelligence Index as a unified, non-agentic measure of LLM capability. For models lacking official scores, it reconstructs the index consistently while flagging estimates and methodological limitations.

  • Index construction: The Intelligence Index synthesizes performance across reasoning, knowledge, mathematics, coding, instruction following, long-context reasoning, and agentic workflow tasks.Its construction integrates eight evaluation suites, including MMLU-Pro [6.
  • Index construction: The index provides a model-capability baseline independent of agentic mechanisms and multi-agent collaboration through consistent zero-shot prompting, pass@1 scoring, and composite reasoning and problem-solving measurement.These design choices support a unified quantitative comparison across the models used in the study.
  • Reconstruction: For models without official platform results, the study independently reproduces selected evaluation suites and computes reconstructed Intelligence Index values using the equal-weighting formulation of Intelligence Index v3.0.Reconstructed values include constituent evaluation scores and model metadata in Table 6, while approximate estimates are marked with * when full reproduction is infeasible.
  • Limitations: Reconstructed scores require caution because some long-context and agentic evaluations are nondeterministic, while unavailable large-context support can make estimates upper-bound approximations.These values are methodologically consistent but not officially certified.

B. Out-of-Sample Validation … C.2. Domain Characterisation

Out-of-sample validation supports the scaling framework but exposes systematic extrapolation errors, especially for Hybrid architectures. The paper also operationalizes domain complexity using an ordinal score based on sequential interdependence, state-space complexity, and empirical difficulty.

  • B. Out-of-Sample Validation: MAS predictions achieve MAE = 0.061 versus 0.138 for SAS on held-out models, indicating better calibration than single-agent predictions beyond the training range.All three held-out models have Intelligence Index = 75, beyond the training range of 42–71; SAS systematically over-predicts because of linear extrapolation.
  • B. Out-of-Sample Validation: 73% of key finding–model pairs validate across three held-out models, with capability ceilings, Centralized or Decentralized optima, and Hybrid overhead generalizing universally.Per-model validation rates are 4/5, 4/5, and 3/5; BrowseComp’s Decentralized > Centralized pattern varies by model family.
  • B. Out-of-Sample Validation: At identical Intelligence Index = 75, GPT-5.2 reaches P_SA = 0.45 versus 0.36 and 0.34 for Gemini-3.0 Pro and Flash, while best MAS performance converges at 0.48–0.50.The convergence suggests that multi-agent architectures may compensate for single-agent limitations, but Intelligence Index may not be directly comparable across model families.
  • B. Out-of-Sample Validation: The scaling equation predicts Hybrid as optimal for all three held-out models, yet Centralized and Decentralized architectures perform better empirically.The equation over-predicts Hybrid by a mean error of +37.0%, compared with +3.0% for Centralized and +8.8% signed error for Decentralized, because extrapolation misses Hybrid’s high-capability overhead penalty.
  • C. Domain Complexity: Domain complexity D∈[0, 1] captures sequential interdependence and empirical difficulty to analyze when coordination benefits outweigh multi-agent overhead.The characterization is intended to distinguish performance gains from prohibitive coordination costs across evaluated benchmarks.
  • C.1. Complexity Score Assignment: The complexity score combines three empirical task properties, normalized to [0, 1], including sequential interdependence, state-space complexity, and overall empirical difficulty.Parallelizable tasks such as Finance Agent score low on sequential interdependence, whereas sequential constraint-satisfaction tasks such as PlanCraft score high; dynamic environments increase state-space complexity.
  • C.2. Domain Characterisation: Table 11 summarizes each benchmark’s domain complexity score and defining task characteristics.The table provides the benchmark-level domain characterization used for the complexity analysis.

C.3. Critical Threshold

The analysis identifies a critical complexity threshold at D≈0.40 governing when multi-agent architectures help or hurt. Below it, task decomposition and parallel reasoning produce positive returns; above it, coordination overhead degrades performance, constraining suitability by domain-intrinsic properties.

  • C.3. Critical Threshold: D≈0.40 marks the critical complexity threshold separating positive from negative multi-agent returns.Below this threshold, multi-agent architectures benefit from effective task decomposition and parallel reasoning.
  • C.3. Critical Threshold: Above D≈0.40, coordination overhead consumes computational resources otherwise allocated to reasoning, degrading performance.
  • C.3. Critical Threshold: Multi-agent suitability is fundamentally constrained by domain-intrinsic properties reflected in task complexity.

D. Datasets … E.3. Prompt Compilation System

The study evaluates agent architectures across six benchmarks spanning diverse reasoning, planning, tool-use, software, and terminal tasks. Its implementation standardizes model access, agent configurations, and prompt compilation while preserving dataset-specific task requirements.

  • D. Datasets: The datasets operationalize distinct evaluation settings, from expert-rubric finance analysis and multi-website synthesis to function-call workflows, sequential crafting, repository patching, and CLI execution.Evaluation uses structured rubrics, ground-truth comparison, exact function-call matching, environment success, test-suite passing, and objective terminal criteria.
  • E.1. Technical Infrastructure: The implementation uses unified API access and LangChain orchestration to standardize model switching, tool integration, conversation management, and structured prompting across providers.Provider-specific APIs support GPT, Gemini, and Claude models, while key rotation and context truncation address operational constraints.
  • E.1. Technical Infrastructure: Dataset-specific tool environments provide web search, code execution, mathematical operations, and completion markers through structured function-calling interfaces.Tools are dynamically bound to language models when function calling is available.
  • E. Implementation Details: Agent configurations fix comparable interaction budgets while supporting single-agent, independent, centralized, decentralized, and hybrid coordination patterns.Single agents use up to 10 iterations; multi-agent variants use specified agent counts, orchestration or debate rounds, and limited peer communication.
  • E.2. Agent Configuration: The framework supports heterogeneous role assignments, allowing high-capability orchestrators to coordinate efficient or diverse worker models.Model assignment is managed per agent role, including distinct worker models in decentralized systems.
  • E.3. Prompt Compilation System: Prompt compilation combines reusable YAML templates, role-specific extensions, conditional selection, and dataset variables to deliver consistent yet task-specific instructions across architectures.Variables include problem descriptions, context, and constraints, while shared dataset templates preserve task specificity.

E.4. Evaluation Methodology · E.5. Information Gain Computation

The evaluation balances computational cost and statistical significance across six benchmark subsets while controlling tools, context handling, resources, and model weights. Information gain measures uncertainty reduction from coordination using posterior variance reduction estimated from sampled reasoning traces.

  • E.4. Evaluation Methodology: Six benchmark subsets use 20–100 instances, with smaller SWE-bench Verified and Terminal-Bench samples reflecting Docker-based evaluation costs.Finance Agent uses 50 instances; BrowseComp Plus, WorkBench, and Plancraft use 100 each; SWE-bench Verified and Terminal-Bench use 20 each.
  • E.4. Evaluation Methodology: Identical tools, observation structures, truncation policies, rate limits, and retries control external feedback and resource-allocation differences across architectures.Frozen model weights and no fine-tuning isolate architectural effects from model optimization.
  • E.5. Information Gain Computation: Information gain ΔI quantifies the reduction in task uncertainty achieved through agent coordination via Bayesian posterior variance reduction.The method compares uncertainty before and after coordination rather than relying only on final task success.
  • E.5. Information Gain Computation: The uncertainty calculation uses binary task success Y and compares the agent’s pre-coordination state representation with its post-coordination aggregated output.spre denotes the initial reasoning trace, while spost denotes the final aggregated output.
  • E.5. Information Gain Computation: Monte Carlo estimation generates K=10 reasoning traces per state at temperature τ=0.7 and computes empirical variance of predicted success probabilities.For binary outcomes, the variance calculation reduces to a form based on sampled success-probability predictions.
  • E.5. Information Gain Computation: For binary outcomes, the estimator uses the mean predicted success probability across sampled traces, denoted p̂(s).This mean predicted probability supplies the state-level quantity used in the reduced variance expression.

F. SWE-bench Verified and Terminal-Bench Results

SWE-bench Verified and Terminal-Bench exhibit patterns consistent with a capability-saturation threshold, as shown alongside resolution rates reported with 95% bootstrap confidence intervals. These results extend the paper’s cross-benchmark scaling analysis to software-engineering and terminal-use tasks.

  • F. SWE-bench Verified and Terminal-Bench Results: Resolution rates for SWE-bench Verified and Terminal-Bench are reported with 95% bootstrap confidence intervals over 20 instances and 10,000 resamples.Table 16 presents each value as a point estimate with lower and upper confidence bounds.
  • F. SWE-bench Verified and Terminal-Bench Results: SWE-bench Verified and Terminal-Bench show scaling patterns consistent with the capability-saturation threshold.Figure 6 places both benchmarks within the cross-benchmark analysis of multi-agent performance versus single-agent baselines by Intelligence Index.
  • F. SWE-bench Verified and Terminal-Bench Results: Across six benchmarks, Figure 6 compares the best-performing multi-agent system with single-agent baselines by Intelligence Index, revealing benchmark-dependent scaling dynamics.OpenAI and Google show cooperative scaling in structured tasks, whereas Anthropic models show diminished or negative returns in open-ended environments.
Loading 2512.08296v3…