PaperlayerView exact revision on arXiv

Source-linked AI summary

MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems

Rui Ye, Shuo Tang, Rui Ge, Yaxin Du, Zhenfei Yin, Siheng Chen, Jing Shao

arXiv:2503.03686v1cs.CLcs.MA

TL;DR

Existing MAS approaches rely on manual configurations or multiple LLM calls, making query-specific construction difficult or costly. MAS-GPT trains an LLM to generate executable MAS from user queries in one inference using consistency-oriented query-MAS data. It consistently outperforms baseline MAS methods across diverse benchmarks and models.

  • Problem

    Existing MAS approaches rely on manual configurations or multiple advanced-LLM calls, creating challenges of inadaptability and high inference costs.

    Source: §1 · p. 1

  • Method

    MAS-GPT trains an LLM on consistency-oriented query-MAS pairs, representing each MAS as executable code generated from a user query.

    Source: §1 · p. 2

  • Results

    MAS-GPT consistently outperforms baseline methods across 9 benchmarks and 5 LLMs, with reported gains of 13.3% for o1 and 10.0% for DeepSeek-R1 on AIME-2024.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    A query-specific executable MAS can be generated in one inference and applied to process the query, simplifying MAS construction.

    Source: §3.3 · p. 5

    The potential negative impacts are similar to those of LLMs generally, including ethical concerns and misuse.

    Source: Impact Statements · p. 9

Abstract

from arXiv · show

LLM-based multi-agent systems (MAS) have shown significant potential in tackling diverse tasks. However, to design effective MAS, existing approaches heavily rely on manual configurations or multiple calls of advanced LLMs, resulting in inadaptability and high inference costs. In this paper, we simplify the process of building an MAS by reframing it as a generative language task, where the input is a user query and the output is a corresponding MAS. To address this novel task, we unify the representation of MAS as executable code and propose a consistency-oriented data construction pipeline to create a high-quality dataset comprising coherent and consistent query-MAS pairs. Using this dataset, we train MAS-GPT, an open-source medium-sized LLM that is capable of generating query-adaptive MAS within a single LLM inference. The generated MAS can be seamlessly applied to process user queries and deliver high-quality responses. Extensive experiments on 9 benchmarks and 5 LLMs show that the proposed MAS-GPT consistently outperforms 10+ baseline MAS methods on diverse settings, indicating MAS-GPT's high effectiveness, efficiency and strong generalization ability. Code will be available at https://github.com/rui-ye/MAS-GPT.

1. Introduction

Existing MAS approaches are difficult to adapt and can require multiple LLM calls, motivating a query-specific system generated in one inference. MAS-GPT represents MAS as executable code, trains on consistency-oriented query-MAS pairs, and consistently outperforms baselines across diverse settings.

1. Introduction

  • Existing MAS can be static and manually crafted, while adaptive alternatives may require multiple LLM inferences, limiting adaptability and increasing costs.The paper identifies these as barriers to broader application of MAS.Source: §1 · p. 1
  • MAS-GPT reframes building a MAS for each query as a generative task, producing an executable MAS from the user query in one inference.The representation is a Python forward function, with agent prompts as variables, LLM calls as functions, and agent relationships as string concatenation.Source: §1 · pp. 1–2
  • A consistency-oriented data pipeline constructs, evaluates, selects, and refines query-MAS pairs to support learning generalizable patterns and reasonable query-MAS correlations.Selection targets inter-consistency between similar queries and high-performing MAS; refinement targets intra-consistency between each query and its MAS.Source: §1 · p. 2
  • Users can apply the generated query-specific MAS directly to process the query, avoiding manual MAS design or multiple LLM calls to obtain one.The generated agents collaborate using a generated structure to produce the final solution.Source: §1 · p. 2
  • Across 9 benchmarks and 5 LLMs, MAS-GPT consistently outperforms 10+ baselines; on AIME-2024 it brings 13.3% and 10.0% gains for o1 and DeepSeek-R1, respectively.The AIME-2024 gains are reported as potential improvements to the reasoning performance of these models.Source: §1 · p. 2

2. Related Work

Prior MAS work includes fixed, manually designed teams and adaptive approaches that use repeated LLM calls. MAS-GPT instead trains an LLM to generate executable MAS from queries using constructed query-MAS data.

2. Related Work

  • MetaGPT, ChatDev, MedAgents, AgentVerse, and Multi-Agent Debate use fixed MAS designs that do not adapt to the given query.The cited systems include manually designed teams for coding or medical tasks and an iterative collaboration structure.Source: §2 · p. 2
  • DyLAN, GPTSwarm, ADAS, and AFlow use LLMs to select or generate task-oriented MAS, but require multiple LLM calls for each query.ADAS and AFlow iteratively generate MAS using strong LLMs; the passage notes that some approaches use over 10 API calls with lengthy context.Source: §2 · p. 2
  • MAS-GPT generates query-specific executable MAS in one LLM inference using query-MAS pairs produced by a data-construction pipeline.The pipeline is based on open-source LLMs.Source: §2 · pp. 2–3
  • Unlike prior SFT datasets for direct query answering, MAS-GPT's task requires teaching LLMs to generate MAS, and the paper constructs a query-MAS-paired dataset for this purpose.The paper states that LLMs have limited, if any, prior knowledge of MAS generation.Source: §2 · p. 3

3. Methodology

MAS-GPT streamlines query-specific multi-agent system construction to a single LLM inference. Its methodology represents systems as executable code and builds training data through steps designed to make query–MAS pairs effective and consistent.

3.1. Overall System Integrated with MAS-GPT

  • MAS-GPT generates a query-tailored MAS in one LLM inference, which can then process the query and deliver the final answer.This replaces manual or fixed system designs and multiple inference calls to obtain an MAS.Source: §3.1 · p. 3

3.2. MAS-GPT: Dataset Construction and Training

  • The dataset pipeline evaluates query–MAS pairs, selects effective pairs, and refines them for consistency between similar queries and within each pair.Refinement adjusts agent definitions to the query and adds a reasoning statement; refined MAS are accepted only if their score is not worse.Source: §3.2 · pp. 3–5
  • MAS are represented as executable Python forward functions, with agent prompts as variables, inferences as function calls, and agent interactions as string concatenations.The function takes a user query and returns the MAS-generated final answer.Source: §3.2 · pp. 3–4

3.3. Discussions

  • The authors describe the system as simple, cost-efficient, and adaptable, noting that training has a one-time cost while inference may be repeated in practical applications.A query-specific executable MAS is generated for processing the query.Source: §3.3 · p. 5

4. Experiments

Across diverse benchmarks, MAS-GPT outperforms baseline methods, including out-of-domain tests, and works with different MAS-driving LLMs. Experiments also examine reasoning performance, inference cost, data-construction choices, scaling, and generated-system examples.

4.1. Experimental Setups

  • The evaluation uses approximately 11k training query-MAS pairs and tests across math, coding, general QA, and science benchmarks with five LLMs.Baselines include single-agent and multi-agent methods; MAS-GPT is trained on Qwen2.5-Coder-32B-Instruct.Source: §4.1 · pp. 5–6

4.2. Main Results

  • MAS-GPT beats the second-best baseline by 3.89% on average across eight benchmarks, including out-of-domain tests.The comparison uses Llama-3-70B-Instruct to drive the MAS.Source: Table 2 · p. 6, §4.2 · p. 6
  • MAS-GPT consistently achieves superior performance with different MAS-driving LLMs, indicating compatibility across backbones.The evaluation includes Qwen2.5-72B-Instruct and GPT-4o-mini-2024-07-18.Source: Table 3 · p. 6, §4.2 · pp. 6–7
  • 13.34%: MAS-GPT improves over a single LLM on AIME-2024 with strong reasoning models including o1-preview.The experiments also use DeepSeek-R1; Figure 4(a) compares methods using o1-preview.Source: §4.2 · p. 7, Figure 4 · p. 7
  • 3.53%: MAS-GPT outperforms MATH-optimized AFlow on MATH and performs best across the compared benchmarks with fewer inference calls.AFlow uses ten proprietary-LLM API calls per query and depends on a hold-out validation set; MAS-GPT builds an MAS with one inference of a 32B model.Source: §4.2 · p. 7, Figure 4 · p. 7

4.3. Analysis of MAS-GPT

  • Ablations report that inter-consistency pair selection improves MATH performance by 8.39%, while MAS adjustment and added reasoning also improve performance.Table 4 summarizes the pair-selection, Refine-A, and Refine-R designs.Source: Table 4 · p. 7, §4.3 · p. 8
  • More training data generally improves code extractability, executability, and task performance, while larger models improve performance and the untrained base model fails to generate an effective MAS.Data size ranges from zero to 11k; the model-size comparison uses 7B, 14B, and 32B models.Source: §4.3 · p. 8, Figure 5 · p. 8
  • Case studies show query-specific MAS generation, generalization to unseen queries, and generation of novel MAS.Examples are presented in the appendix.Source: §4.3 · p. 8

5. Conclusion

MAS-GPT aims to make MAS creation less time-consuming by generating executable systems from user queries in one inference. Across benchmarks and LLM drivers, it outperforms baselines and shows potential to enhance reasoning and scale with further improvements.

  • MAS-GPT is trained to generate executable MAS from arbitrary user queries, using a consistency-oriented data-construction pipeline.The pipeline is intended to improve coherence and consistency of query-MAS pairs.Source: §5 · p. 8
  • Across nine benchmarks and five LLM drivers, MAS-GPT consistently outperforms more than ten baseline methods.Source: §5 · p. 8
  • The experiments indicate potential to enhance strong reasoning models and to improve further through continued scaling.Source: §5 · p. 8

Impact Statements

The impact statement describes MAS-GPT as simplifying MAS application and notes that potential negative impacts resemble those associated with LLMs generally.

  • MAS-GPT is presented as reducing the complexity of designing and deploying MAS for a wide range of tasks.Source: Impact Statements · p. 9
  • The authors describe the approach as improving accessibility, efficiency, and scalability for broader real-world application.Source: Impact Statements · p. 9
  • Potential negative impacts are described as LLM-associated ethical concerns and misuse.The statement characterizes these concerns as inherent to LLM use generally.Source: Impact Statements · p. 9

A.1. MAS-GPT Can Generate Query-Specific MAS

MAS-GPT analyzes a query’s domain, difficulty, and required knowledge, then selects and generates a query-specific multi-agent system within one inference.

A.1. MAS-GPT Can Generate Query-Specific MAS

  • MAS-GPT analyzes the query’s domain, difficulty, and required knowledge before choosing a multi-agent system.The analysis informs which agents and structure are appropriate for the query.Source: §A.1 · p. 12
  • The generated query-specific system uses five specialized agents and a final decision-making agent to answer the query.Source: §A.1 · p. 12
  • MAS-GPT generates the query-specific system within one LLM inference.Source: §A.1 · p. 12

A.2. MAS-GPT Can Generalize to Unseen Queries

For queries from domains unseen during training, MAS-GPT analyzes the query and generates a suitable multi-agent system, supporting generalization to unseen queries.

A.2. MAS-GPT Can Generalize to Unseen Queries

  • For an unseen-domain query, MAS-GPT analyzes the domain, required understanding, and difficulty before selecting suitable agents and a system structure.Source: §A.2 · p. 12
  • One generated system uses four parallel pipelines—two coding-based and two direct-answer pipelines—whose solutions are combined by an aggregation agent.Each coding pipeline has an agent for writing and executing code and another for organizing the final answer.Source: §A.2 · p. 13
  • MAS-GPT generalizes to unseen queries and generates appropriate multi-agent systems.Figure 6(b) visualizes an unseen-query case.Source: §A.2 · p. 13, Figure 6 · p. 13

A.3. MAS-GPT Can Generate Novel MAS

This section illustrates MAS-GPT’s generation of query-specific multi-agent systems through examples involving an unseen SciBench query and a Bertrand price-competition question. It also shows the systems’ parallel solution pipelines, final answer-selection agent, and a visualization of MASs in the system’s pool.

A.3. MAS-GPT Can Generate Novel MAS

  • For a query, MAS-GPT generated a system unseen in its base pool, with five parallel answer-and-refinement pipelines and a final decision-making agent.The paper presents this as evidence that MAS-GPT learns patterns for designing systems rather than merely memorizing its training data.Source: §A.3 · p. 13
  • For an unseen SciBench query, MAS-GPT generated a math-solving system with two code-execution pipelines, two direct-solution pipelines, and a final agent to evaluate the solutions.The code pipelines generate and execute Python, then organize solutions using the execution results.Source: §A.3 · pp. 14–15
  • For the Bertrand-model question, MAS-GPT generated a system that analyzes the domain, difficulty, and required knowledge, then uses independent solutions, refinement, and a decision agent to produce a final response.The example identifies the task as game theory and microeconomics, requiring reasoning about firms’ price-setting and Nash equilibrium.Source: §A.3 · p. 18

B. Visualization of Our MAS pool

C.1. Descriptions of Datasets

Table 9 provides an overview of the training and testing datasets.

C.1. Descriptions of Datasets

  • Table 9 gives an overall description of the training and testing datasets.Source: §C.1 · p. 20

C.2. Evaluations

The section evaluates MAS outputs by extracting answers or code from responses, then checking them against ground-truth answers or executable test cases.

C.2. Evaluations

  • For ground-truth tasks, LLMs extract the answer from each MAS response and compare it with the reference answer.The evaluation prompts use predefined extraction rules before comparison.Source: §C.2 · pp. 20–21
  • Because MAS responses are often unstructured, the authors use a two-step LLM process—answer extraction followed by evaluation—instead of direct ground-truth comparison.The process is based on prompts used in AutoGen (Wu et al., 2023).Source: §C.2 · p. 20
  • For coding tasks, performance is measured by test-case pass rate after an LLM extracts the code and a coding environment executes it.The section names HumanEval and MBPP as code benchmarks evaluated with test cases.Source: §C.2 · pp. 20–21

Related papers

Loading 2503.03686v1…