PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-agent Architecture Search via Agentic Supernet

Guibin Zhang, Luyang Niu, Junfeng Fang, Kun Wang, Lei Bai, Xiang Wang

arXiv:2502.04180v2cs.LGcs.CLcs.MA

TL;DR

Existing automated multi-agent design methods often seek static, complex systems that do not adapt inference resources to query difficulty and domain. MaAS optimizes a probabilistic, continuous agentic supernet and samples query-dependent systems from it. Across evaluated benchmarks, it reports higher performance, lower costs, and transferability across datasets and LLM backbones.

  • Problem

    Static, complex one-size-fits-all multi-agent systems fail to dynamically allocate inference resources according to query difficulty and domain.

  • Method

    MaAS optimizes a probabilistic, continuous distribution of agentic architectures and samples query-dependent multi-agent systems from the resulting agentic supernet.

  • Results

    MaAS surpasses existing handcrafted or automated multi-agent systems by 0.54% ∼16.89% while requiring lower training and inference costs and transferring across datasets and LLM backbones.

  • Takeaways & Limitations

    MaAS provides high-performing, resource-efficient multi-agent solutions tailored to queries across different domains and difficulty levels.

Abstract

from arXiv · show

Large Language Model (LLM)-empowered multi-agent systems extend the cognitive boundaries of individual agents through disciplined collaboration and interaction, while constructing these systems often requires labor-intensive manual designs. Despite the availability of methods to automate the design of agentic workflows, they typically seek to identify a static, complex, one-size-fits-all system, which, however, fails to dynamically allocate inference resources based on the difficulty and domain of each query. To address this challenge, we shift away from the pursuit of a monolithic agentic system, instead optimizing the \textbf{agentic supernet}, a probabilistic and continuous distribution of agentic architectures. We introduce MaAS, an automated framework that samples query-dependent agentic systems from the supernet, delivering high-quality solutions and tailored resource allocation (\textit{e.g.}, LLM calls, tool calls, token cost). Comprehensive evaluation across six benchmarks demonstrates that MaAS \textbf{(I)} requires only $6\sim45\%$ of the inference costs of existing handcrafted or automated multi-agent systems, \textbf{(II)} surpasses them by $0.54\%\sim11.82\%$, and \textbf{(III)} enjoys superior cross-dataset and cross-LLM-backbone transferability.

1. Introduction

Existing multi-agent systems often depend on manual design or optimize a static, resource-intensive architecture, limiting adaptation to query difficulty and domain. MaAS instead optimizes an agentic supernet that samples query-dependent systems with tailored resource allocation.

  • Motivation: Manual configurations in early multi-agent systems limit rapid adaptation across domains and application scenarios.These configurations include prompt engineering, agent profiling, and inter-agent communication pipelines.
  • Motivation: Static one-size-fits-all systems can optimize accuracy and pass@k but remain constrained on token cost, LLM calls, and inference latency.Complex systems may suit Ph.D.-level abstract algebra, whereas elementary arithmetic can work with a single zero-shot I/O.
  • MaAS: MaAS samples suitable multi-agent systems for different queries, providing satisfactory resolution and appropriate inference resources.The sampled systems are tailored to query difficulty, domain, and features.
  • Evaluation: 0.54% ∼16.89% higher performance is demonstrated across evaluations on seven benchmarks spanning code generation, mathematical reasoning, and tool usage.The evaluations also report significantly lower training and inference costs, transferability across benchmarks and LLMs, and strong inductive capacity.
  • MaAS: The agentic supernet reformulates architecture search as optimizing a probabilistic, continuous distribution of multiple agentic architectures.This shifts the objective away from seeking a single optimal multi-agent system.

2. Related Work

Research on automating agentic systems spans prompt optimization, inter-agent communication, and broader architecture search. MaAS follows the trajectory of neural architecture search by introducing supernet-based search for multi-agent systems.

  • LLM-Agents and Agentic Systems: Early multi-agent systems such as AutoGen and AgentVerse enhance capabilities through grouped agents but rely heavily on manually crafted designs.Manual design remains a recurring limitation in multi-agent system construction.
  • Automating Agentic Systems: Automated agentic-system research includes prompt optimization methods such as PromptBreeder, DsPy, and EvoPrompt.These methods automate optimization of prompts rather than necessarily searching the full agentic architecture.
  • Automating Agentic Systems: Inter-agent communication methods automate orchestration of interactions between agents, including GPTSwarm, DyLAN, EvoMAC, AgentPrune, and G-Designer.This category focuses on how agents communicate and are coordinated.
  • AutoML: Agentic automation has adopted reinforcement learning, evolutionary algorithms, Bayesian optimization, and MCTS approaches inspired by AutoML and neural architecture search.The paper situates its approach within this progression from black-box search toward efficient supernet training.
  • AutoML: MaAS introduces the first multi-agent-system search framework leveraging an agentic supernet.The approach extends the supernet paradigm associated with methods such as DARTS and SNAS to agentic automation.

3. Methodology

MaAS models agentic architectures as a query-conditioned probabilistic supernet, then samples and optimizes task-specific multi-agent systems under performance and cost considerations.

  • 3.2. Agentic Architecture Sampling: For each query, a controller samples a customized architecture from the supernet, executes it, and uses environment feedback to update the distribution and operators.The sampled architecture is generated from the query, parameterized distribution, and available operator set.
  • 3.1. Preliminary: MaAS defines an agentic operator as a composite LLM-agent invocation process involving multiple LLM calls and tool usage.Available operators include CoT, Self-RAG, and multi-agent debate, with different agent and tool allocations.
  • 3.1. Preliminary: The search space represents multi-agent systems as directed acyclic graphs whose nodes are selected agentic operators and whose edges encode connectivity.The agentic supernet contains a probabilistic distribution over possible multilayer operator configurations.
  • 3.1. Preliminary: MaAS optimizes a query-dependent architecture distribution rather than identifying one globally optimal multi-agent system.The objective combines system utility or performance with cost through a trade-off parameter.
  • 3.2. Agentic Architecture Sampling: An early-exit operator allows sampling to stop before the maximum depth when a query does not require additional layers.This addresses the redundancy of executing all layers for queries solvable with simple zero-shot input-output.
  • 3.2. Agentic Architecture Sampling: Query-dependent activation scores select operators sequentially until their cumulative score exceeds a threshold, varying the number of operators per layer with task complexity.The scores incorporate the query and preceding-layer operator representations, enabling dynamic resource allocation.
  • 3.3. Cost-constrained Supernet Optimization: MaAS estimates distribution gradients with empirical Bayes Monte Carlo because executing architectures through tools or API-based LLM calls is nondifferentiable.Cost-aware importance weights favor architectures that produce high-quality solutions with minimal token cost.
  • 3.3. Cost-constrained Supernet Optimization: Agentic operators are updated through textual gradients targeting prompts, model temperature, and operator-node structure.The textual-gradient approach approximates backpropagation for operators involving black-box tools and natural-language prompts.

4. Experiments

Across diverse benchmarks, MaAS is evaluated against single-agent, handcrafted, and automated multi-agent baselines, showing strong performance, efficiency, query-aware sampling, and transferability. Its analyses examine cost, operator selection, sensitivity, ablations, and cross-dataset or cross-LLM generalization.

  • Experiment Setup: MaAS is compared with single-agent, handcrafted multi-agent, and autonomous multi-agent baselines across math, code-generation, and tool-use benchmarks.The evaluation uses six public benchmarks and reports separate baseline comparisons for GAIA and the other task domains.
  • Performance Analysis: MaAS outperforms manually designed methods by 3.90–6.40% on average and automated methods by 2.07–8.26%.Across mathematical reasoning and code generation, it achieves an average best score of 83.59%.
  • Performance Analysis: 18.38% and 17.61% improvements are achieved on GAIA Level 1 and Level 2 tasks, respectively, through adaptive sampling of systems for different domains.The comparison includes automated systems and additional baselines such as AutoGPT, TapeAgent, and Sibyl.
  • Cost Analysis: 3.38$ is MaAS’s training cost versus 22.50$ for AFlow, while MaAS achieves comparable accuracy with 6.8× lower training cost.On MATH, MaAS also requires only 53 minutes for optimization, compared with 508 minutes for DyLAN and 129 minutes for GPTSwarm.
  • Cost Analysis: MaAS achieves the highest accuracy with a 0.42$ inference API cost and the lowest prompt/completion token consumption, API cost, and wall-clock time.The reported efficiency is attributed to dynamically allocating resources according to query difficulty.
  • Case Study: MaAS samples deeper architectures for challenging queries, while easy queries can exit at the second layer with probabilities of 0.37 and 0.47.It selects different operators at each layer, including direct I/O and ReAct, and increasingly favors early exit as supernet depth grows.
  • Framework Analysis: 89.5% →92.8% performance is observed as supernet depth increases from 2 to 4, after which gains become marginal while inference costs rise.L = 4 is selected; larger λ favors cost-efficient solutions with some performance degradation, and K = 4 provides satisfactory low-variance estimation.
  • Framework Analysis: Removing the textual gradient causes the largest performance drop, whereas removing early exit or the cost constraint has little performance impact but weakens query dependence or increases inference cost.The agentic supernet also transfers to Qwen-2.5-70b with 4.98% ∼5.50% performance gains and can activate a held-out Debate operator.

5. Conclusion

The paper reframes automated multi-agent design around an agentic supernet rather than a single optimal system. MaAS dynamically samples query-specific systems to balance performance and token efficiency across domains and difficulty levels.

  • The agentic supernet replaces searching for one optimal multi-agent system with optimizing a probabilistic, continuous distribution of agentic architectures.
  • MaAS dynamically samples multi-agent systems that provide satisfactory performance and token efficiency for queries across different domains and difficulty levels.

Impact Statement

MaAS presents a dynamic alternative to static, one-size-fits-all multi-agent architectures. The stated societal implication is more fine-grained resource allocation, improved efficiency, and greater workflow flexibility.

  • MaAS replaces static, one-size-fits-all architectures with a dynamic and adaptive agentic supernet.
  • The approach enables resource allocation tailored to query difficulty and domain while maintaining high-quality outputs.
  • MaAS is described as reducing inference costs and increasing the flexibility of multi-agent workflows.

B. Technical Details

The technical design uses a customizable repository of agentic operators and prompts, embedded operator profiles, and reusable operator implementations. Operators cover reasoning, debate, aggregation, refinement, testing, tool use, and query-dependent early exit.

  • Operator initialization: The operator repository includes CoT, LLM-Debate, Self-Consistency, Self-Refine, Ensemble, Testing, ReAct, and Early exit operators.These operators support reasoning, debate, answer aggregation, refinement, code testing, tool use, and variable-depth sampling.
  • Operator initialization: Early exit interrupts architecture sampling, making the depth of the agentic supernet query-dependent.
  • Operator initialization: Users can customize the operator repository by incorporating desired operators into MaAS.
  • Embedding Function: Each operator receives an LLM-generated profile, which MiniLM encodes into a fixed-dimensional embedding.
  • Operator implementation: The implementation defines asynchronous operator execution through prompts, typed operation classes, and an LLM-backed fill routine.
  • Operator implementation: Generated operators must follow the established input/output interface and provide complete, reliable, reusable code.

C.1. Dataset Statistics

The experiments split each dataset into training and test sets at a 1:4 ratio. For MATH, evaluation uses 617 difficulty-level-5 problems from four specified categories.

  • Each dataset is divided into training and test sets using a TRAIN:TEST ratio of 1:4.
  • The MATH subset contains 617 harder problems spanning Combinatorics & Probability, Number Theory, Pre-algebra, and Pre-calculus.All selected MATH problems are at difficulty level 5.
  • Dataset statistics are reported in Table 6.

C.2. Baseline Setups

The baseline setups combine standard single-agent reasoning methods with diverse multi-agent architectures, using published implementations or specified configurations for fair comparison.

  • Reasoning baselines: Chain-of-Thought prompts agents to reason through sequential steps rather than produce direct answers.The implementation follows Zhang et al. (2022).
  • Reasoning baselines: Self-consistency aggregates five CoT-generated solutions to enhance robustness.
  • Multi-agent baselines: LLM-Debate uses five role-specialized agents for up to two debate rounds, with majority voting determining the final decision.
  • Multi-agent baselines: LLM-Blender combines two gpt-4o-mini models, one Qwen-2.5-72b model, and one llama-3.1-70b model.
  • Multi-agent baselines: MacNet uses the MacNet-MESH variant, corresponding to a fully connected network topology.
  • Automated architecture baselines: GPTSwarm, AutoAgents, ADAS, AgentSquare, and AFlow follow published or official configurations, with AgentSquare and AFlow constrained to gpt-4o-mini under specified stopping limits.AgentSquare uses patience 5 early stopping; AFlow uses MAX ITERATION=20.

D. Supplementary Results

Supplementary results examine MaAS’s operator-sampling behavior and transferability. MaAS increasingly favors testing and self-refinement in deeper layers and can select an inference-only Debate operator despite not seeing it during training.

  • Operator sampling: MaAS avoids overly confident early stopping as sampling increases, prioritizing testing and self-refinement in deeper layers.
  • Transferability: Table 7 reports cross-model transferability by optimizing the agentic supernet with gpt-4o-mini and comparing backbone performance before and after equipping the optimized supernet.
  • Transferability: Table 8 reports cross-dataset transferability, with notation such as MATH→GSM8K indicating optimization on MATH and evaluation on GSM8K.
  • Layer-wise distributions: Figure 8 visualizes MaAS’s layer-wise distribution on HumanEval without the Debate operator.
  • Operator sampling: With Debate introduced only during inference, MaAS can still reasonably select the operator despite not being exposed to it during training.
Loading 2502.04180v2…