PaperlayerView exact revision on arXiv

Source-linked AI summary

X-MAS: Towards Building Multi-Agent Systems with Heterogeneous LLMs

Rui Ye, Xiangrui Liu, Qimin Wu, Xianghe Pang, Zhenfei Yin, Lei Bai, Siheng Chen

arXiv:2505.16997v1cs.AIcs.CLcs.MA

TL;DR

Existing MAS frameworks commonly drive all agents with one LLM, limiting the system to that model's capabilities. This paper introduces X-MAS-Bench and uses its observations to transition MAS to heterogeneous LLMs, finding consistent performance gains without structural redesign. The study positions heterogeneous model selection as a promising direction for improving collaborative AI systems.

  • Problem

    Existing MAS frameworks mostly use one LLM for all agents, limiting the system's intelligence to that model.

  • Method

    The paper benchmarks 27 LLMs across five MAS functions and five domains, then uses the findings to assign suitable models within existing MAS.

  • Results

    Heterogeneous MAS consistently outperforms homogeneous MAS without structural redesign across experiments on four MAS methods.

  • Takeaways & Limitations

    Heterogeneous LLM selection can harness collective intelligence without additional training.

  • Takeaways & Limitations

    The evaluation omits some LLMs, and the proof-of-concept currently relies on manual modification when converting homogeneous MAS.

Abstract

from arXiv · show

LLM-based multi-agent systems (MAS) extend the capabilities of single LLMs by enabling cooperation among multiple specialized agents. However, most existing MAS frameworks rely on a single LLM to drive all agents, constraining the system's intelligence to the limit of that model. This paper explores the paradigm of heterogeneous LLM-driven MAS (X-MAS), where agents are powered by diverse LLMs, elevating the system's potential to the collective intelligence of diverse LLMs. We introduce X-MAS-Bench, a comprehensive testbed designed to evaluate the performance of various LLMs across different domains and MAS-related functions. As an extensive empirical study, we assess 27 LLMs across 5 domains (encompassing 21 test sets) and 5 functions, conducting over 1.7 million evaluations to identify optimal model selections for each domain-function combination. Building on these findings, we demonstrate that transitioning from homogeneous to heterogeneous LLM-driven MAS can significantly enhance system performance without requiring structural redesign. Specifically, in a chatbot-only MAS scenario, the heterogeneous configuration yields up to 8.4\% performance improvement on the MATH dataset. In a mixed chatbot-reasoner scenario, the heterogeneous MAS could achieve a remarkable 47\% performance boost on the AIME dataset. Our results underscore the transformative potential of heterogeneous LLMs in MAS, highlighting a promising avenue for advancing scalable, collaborative AI systems.

1 Introduction

The paper argues that heterogeneous LLMs can overcome the limits of homogeneous multi-agent systems by assigning diverse models to specialized MAS functions and domains. It introduces X-MAS-Bench to guide this transition and reports benefits across chatbot-only and mixed chatbot-reasoner settings.

  • Motivation: Heterogeneous LLMs address the limitation that same-model agents may reproduce the underlying model's errors.The paper motivates diversity as a way to harness complementary capabilities from models trained on different corpora or by different teams.
  • X-MAS-Bench: X-MAS-Bench evaluates LLMs across five MAS functions and five domains spanning 21 test sets under controlled conditions.The functions are question-answering, revise, aggregation, planning, and evaluation; the domains include mathematics, coding, science, medicine, and finance.
  • X-MAS-Design: X-MAS-Design transitions existing MAS methods from homogeneous to heterogeneous LLMs by assigning agents appropriate models without structural redesign.The proof-of-concept evaluates three existing frameworks and a prototype MAS across five domains.
  • X-MAS-Bench: 27 LLMs and over 1.7 million evaluations identify diverse optimal model selections for each domain-function combination.These observations are intended to support researchers and practitioners building MAS.
  • Results: Heterogeneous MAS consistently outperforms homogeneous MAS across the reported experiments.The experiments cover chatbot-only and mixed chatbot-reasoner scenarios, and the paper releases its data, code, and evaluation results.

2 Related Work

The related work spans multi-agent systems, heterogeneous-LLM approaches, and general LLM benchmarking. The paper distinguishes itself by systematically benchmarking LLM capabilities specifically for MAS across multiple functions and domains.

  • LLM-based MAS: Existing MAS frameworks use multiple specialized agents for tasks including software development, mathematics, debate, planning, and evaluation.Examples include ChatDev, MetaGPT, EvoMAC, MACM, AgentVerse, DyLAN, and MAS-GPT [7] [9] [10] [15] [16] [6].
  • Heterogeneous LLMs: Prior heterogeneous-LLM methods ensemble or discuss candidate models, while MASRouter manually selects models for a specific framework.The paper instead aims to assess model capabilities across several MAS functions and domains for broader applicability.
  • Benchmarking LLMs: Unlike general domain or function benchmarks, this work benchmarks LLMs for MAS across 25 function-domain perspectives.The benchmark covers five domains and five MAS-related functions.

3 X-MAS-Bench: Evaluating LLMs for MAS

X-MAS-Bench evaluates LLMs across five MAS-related functions and five domains under controlled conditions, revealing substantial specialization rather than a universally strongest model. These differences support heterogeneous MAS configurations that assign suitable models to different scenarios.

  • Benchmark scope: 27 LLMs are evaluated across five MAS-related functions and five domains spanning 21 test sets under controlled experimental conditions.The benchmark includes 20 chatbots and 7 reasoners, with each assessment designed to vary only the LLM under evaluation.
  • Revise: Revise measures whether an LLM can reason over a query and an answer from a predefined model to produce a corrected final answer.All examined models receive the same query-answer prompt, and revised outputs are compared with ground-truth answers using accuracy.
  • Aggregation: Aggregation combines three fixed candidate responses with a query and scores the synthesized answer against ground truth using accuracy.Candidate responses and their order remain identical across examined models to ensure fair comparisons.
  • Results: No single LLM excels universally across the 25 function-domain scenarios, supporting heterogeneous model assignment in MAS.The benchmark reports substantial performance variation both across domains and functions, with Qwen2.5-7B shifting from exceptional revise-coding performance to mid-tier performance in revise-medicine and planning-coding.
  • Results: Smaller models can outperform larger models in niche function-domain pairs, including Qwen2.5-7B scoring 79.2 versus Qwen2.5-72B-Instruct at 77.3 for revise-coding.Qwen2.5-14B achieves the best performance among all models for aggregation-finance and evaluation-finance, indicating opportunities to improve performance and computational efficiency.
  • Results: Consistent high performers can anchor heterogeneous configurations while specialized models such as Llama3-OpenBioLLM-70B serve niche domains like medicine.Qwen-2.5-32B-Instruct, Qwen-2.5-72B-Instruct, and Llama-3.1-70B-Instruct frequently rank among the top across domains and functions.

4 X-MAS-Design: Leveraging Diversity for MAS

X-MAS-Design converts existing homogeneous MAS into heterogeneous systems by assigning appropriate LLMs to unchanged agent roles and interaction topologies. Across chatbot-only and mixed chatbot-reasoner settings, heterogeneous configurations generally outperform homogeneous alternatives, with gains linked to domain- and function-appropriate model diversity.

  • Chatbot-only experiments: X-MAS-Design consistently outperforms homogeneous configurations across four MAS methods, with DyLAN averaging 65.90 versus 62.73 for its best homogeneous model.The comparison uses four candidate LLMs across five domains; two outliers were attributed to missing specialized models.
  • Chatbot-only experiments: Heterogeneity improves performance across four differently designed MAS methods, indicating that the benefit is method-agnostic rather than tied to one architecture.The authors retain original agent roles and interaction topology while substituting heterogeneous LLM configurations.
  • Mixed chatbot-reasoner experiments: Mixing chatbot and reasoner agents outperforms standalone chatbot and reasoner configurations across all five domains and MAS methods.The reported gains are attributed to complementary strengths between the two model types, guided by X-MAS-Bench evaluations.
  • Mixed chatbot-reasoner experiments: 70% on AIME-2024 represents a 46.67%-point gain over the second-best homogeneous MAS for X-MAS-Proto.X-MAS-Design also exceeds the second-best chatbot-based homogeneous MAS by 33% on AIME-2025 and 34% on MATH-MAS.
  • Ablation study: Increasing candidate-model count generally improves X-MAS-Proto performance across math, coding, and science, except when an added model is poorly matched to science.The trend supports including diverse LLMs, while also showing that candidate relevance matters.
  • Ablation study: X-MAS-Design guided by X-MAS-Bench performs best against arbitrary LLM selection, supporting benchmark-based model assignment.The comparison evaluates X-MAS-Design against both homogeneous and arbitrary-selection alternatives.

5 Conclusions

The paper advocates heterogeneous LLM-driven MAS and supports this direction with broad evaluations and experiments across existing and newly designed systems. It presents heterogeneity as a route to collective intelligence without additional training, while identifying automated selection and scaling as future directions.

  • 5 Conclusions: The paper evaluates 27 LLMs across five MAS functions and five domains, then shows heterogeneous MAS can improve four methods without structural redesign.The study frames these results as support for building MAS with heterogeneous LLMs.
  • 5 Conclusions: Heterogeneous LLMs offer a direction for leveraging collective intelligence without additional training.Future work includes automated or dynamic model selection, scaling candidate models, and optimizing LLM-MAS synergy.

A Limitations

The evaluation does not include every available LLM, and the heterogeneous conversion currently relies on manual configuration. The paper also notes that its potential misuse risks are those generally associated with LLM use.

  • A Limitations: The benchmark omits some LLMs, limiting the coverage of its evaluation.The authors describe the study as comprehensive while acknowledging that inclusion is not exhaustive.
  • A Limitations: The current transition from homogeneous to heterogeneous MAS requires manual modification, although the modification is simple.The authors identify automated solutions as worthwhile future work.
  • A Limitations: Potential misuse risks mirror those associated with LLMs generally.The paper characterizes these risks as intrinsic to LLM usage rather than unique to the proposed approach.

C Experimental Setups

The experiments cover 27 LLMs, including general and domain-specific chatbots as well as reasoning models, under standardized token and temperature settings. The model pool spans multiple providers and domains relevant to MAS evaluation.

  • C Experimental Setups: The setup examines 27 LLMs comprising 20 chatbots and 7 reasoners.The chatbots include general models from Llama, Qwen, and Mistral, plus domain-specific mathematics, coding, science, medicine, and finance models.
  • C Experimental Setups: The model pool includes both generalist and specialist LLMs from multiple providers.This design supports comparisons across domain-specific capabilities within MAS functions.

D.1 Experiments setups of X-MAS-Design in Chatbot-Only Scenarios

The chatbot-only X-MAS-Design assigns different available LLMs to role functions across mathematics, coding, science, medicine, and finance. Assignments vary by domain and function, combining general-purpose, coder, math, and Mistral models.

  • Mathematics: Mathematics assigns Qwen-2.5-32B-Instruct to role assignment and evaluation, while coder and math models share solver duties and the coder handles criticism.
  • Coding: Coding uses Qwen-2.5-32B-Instruct for assignment and solving, and Qwen-2.5-Coder-32B-Instruct for criticism and evaluation.
  • Science: Science uses Qwen-2.5-32B-Instruct for assignment, solving, and criticism, with Mistral-Small-3.1-24B-Instruct-2503 evaluating.
  • Medicine: Medicine assigns Qwen-2.5-32B-Instruct to assignment and solving, while Mistral-Small-3.1-24B-Instruct-2503 handles criticism and evaluation.
  • Finance: Finance distributes roles across Mistral-Small-3.1-24B-Instruct-2503, Qwen-2.5-Coder-32B-Instruct, and Qwen-2.5-32B-Instruct, with Mistral handling assignment and evaluation.

D.2 Experimental Setups of X-MAS with Non-X-MAS-Bench-Guided Model Selections

This experiment compares X-MAS-Bench-guided chatbot-only model selection with five manually designed alternatives. All configurations are evaluated on MATH-500 to assess assignment sensitivity and the value of systematic guidance.

  • Compared configurations: The study compares the X-MAS-Bench-guided X-MAS-Design configuration with five manually designed configurations, X-MAS1 through X-MAS5.
  • X-MAS-Design: X-MAS-Design assigns Qwen-2.5-32B-Instruct to planning and evaluation, Qwen-2.5-Coder-32B-Instruct to solving and revision, and Mistral-Small-3.1-24B-Instruct-2503 to aggregation.
  • X-MAS-Design: The guided configuration combines domain-specialized solvers with stronger general-purpose planning and evaluation agents.
  • Manual alternatives: The five alternatives use general instruction-tuned LLMs without prior empirical optimization, providing a comparison for variance among manually configured pipelines.
  • Experimental evaluation: All configurations are evaluated on MATH-500 under the X-MAS-Proto scheme to examine heterogeneous assignments and X-MAS-Bench-guided selection.

E Experiments on X-MAS in Mixed Chatbot-Reasoner Scenarios

The mixed chatbot-reasoner experiments use Qwen-2.5-72B-Instruct and DeepSeek-R1-Distill-Qwen-32B across five domains and multiple MAS functions. The setup assigns roles by domain and evaluates model choices through benchmarked configurations.

  • Available models: Mixed chatbot-reasoner scenarios use Qwen-2.5-72B-Instruct and DeepSeek-R1-Distill-Qwen-32B as the available models.
  • Role assignment: Across mathematics, coding, science, and finance, DeepSeek-R1-Distill-Qwen-32B fills most solver, critic, and evaluator roles, while Qwen-2.5-72B-Instruct performs role assignment.
  • Role assignment: Medicine differs by assigning Qwen-2.5-72B-Instruct to role assignment, solving, and criticism, with DeepSeek-R1-Distill-Qwen-32B evaluating.
  • Debate and aggregation: Debate and aggregation configurations generally use DeepSeek-R1-Distill-Qwen-32B for debate, while aggregation varies between the two models by domain.
  • Node and ranking: Node-agent and ranker assignments likewise vary by domain, with DeepSeek-R1-Distill-Qwen-32B commonly serving as the node agent and Qwen-2.5-72B-Instruct commonly serving as ranker.
  • Five-role configurations: The five-role configurations combine the two models differently across planner, solver, reviser, evaluator, and aggregator roles, and Figure 5 benchmarks LLMs across five MAS-related functions and five domains.
Loading 2505.16997v1…