Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systems

1 Introduction

• Multi-LLM collaboration uses diverse models with different roles and adapted weights, while fixed-role or fixed-weight approaches limit flexible adaptation across tasks and contexts.Fixed roles can require substantial prompt engineering based on prior task knowledge, hindering automation and scaling to unseen tasks and contexts.Source: §1 · pp. 1–2
• Role-step learns task-specific LLM DAGs by decoding continuous adjacency matrices into discrete graphs, evaluating topological message passing, and updating matrices with particle swarm optimization.G-DECODE performs the continuous-to-discrete DAG decoding.Source: §1 · p. 2
• Weight-step uses JFK-score, aggregated model performance across DAG-position assignments weighted by assignment frequency, to optimize model weights with particle swarm optimization.The assignments are evaluated in the best-found network from the role-step.Source: §1 · p. 2
• Role-step and weight-step alternate until utility stops improving or a maximum iteration limit is reached, yielding a network with optimized graph structure and model weights.The framework also finds systems with heterogeneous roles and collaborative gains, benefits from model diversity, and can speed optimization through increased sparsity and “dropout”.Source: §1 · pp. 2–3
• 18.5% average improvement over 15 role- and/or weight-based approaches across 12 tasks spanning knowledge, reasoning, and agent contexts.Scaling collaboration from two to ten LLMs also yielded a 27.1% average improvement.Source: §1 · p. 2
• 27.1% average improvement came from scaling collaboration from two to ten LLMs through topological collaborative generation of smaller language models.The paper reports that roles and weights vary in importance across tasks.Source: §1 · pp. 2–3

2 Preliminary

• A multi-LLM system is a set of models and a binary adjacency matrix defining a DAG; models pass outputs as inputs and are called in topological order.The paper adapts PSO, an optimization method based on collective behavior, to optimize the adjacency structures and model weights.Source: §2 · p. 3

3 Methodology

• Heterogeneous Swarms jointly optimizes model roles and weights by alternating role-step and weight-step to adapt a multi-LLM system to a task utility function.The method starts from a pool of LLM experts and uses PSO to search diverse collaboration patterns and specialized model checkpoints.Source: §3 · p. 4, §3.2 · p. 5

3.1 Role-step

• Unlike prompt-engineered role assignments, role-step learns a DAG of LLM input-output relationships for collaborative generation.Multiple candidate DAGs explore the graph space to adapt to a task and its utility function.Source: §3.1 · p. 4
• G-decode converts continuous adjacency matrices into directed acyclic graphs by sampling nodes and edges, then calls models topologically and returns the end node’s output.Continuous adjacency entries represent edge likelihoods; adding edges only to existing nodes ensures the decoded graph is directed and acyclic.Source: §3.1 · p. 4

3.2 Weight-step

• JFK-score estimates each LLM’s individual contribution in the best-found role-step DAG, addressing the challenge of measuring a model’s utility within a multi-LLM system.The method samples model assignments to positions in that DAG and evaluates their system utility.Source: §3.2 · pp. 4–5
• JFK-score aggregates system utility weighted by each model’s assignment frequency, quantifying its contribution across roles and collaboration partners to guide weight optimization.Heterogeneous Swarms uses swarm intelligence to optimize model weights based on these contribution scores.Source: §3.2 · p. 5

4 Experiment Settings

• The prototype uses GEMMA-7B and a pool of 10 domain-fine-tuned LLM experts, with top-p 0.8 and up to 20 search iterations.The experts are fine-tuned from GEMMA-7B using 10 Tulu-v2 [44] domains; the implementation uses N = 10, M = 10, search patience 6, and grid search over other hyperparameters, reporting the best-found systems. MISTRAL-7B is also used in Table 5.Source: §4 · p. 5
• The comparison includes 17 baselines across five categories, including trivial and static-weight methods.Trivial baselines are Best Single expert and Prediction Merge; static-weight methods are Data Merge, Uniform Soup [97], and Dare-Ties [108] [103].Source: §4 · pp. 5–6
• 18.5% average improvement across tasks over TRIVIAL, STATIC WEIGHT, DYNAMIC WEIGHT, STATIC ROLE, and DYNAMIC ROLE approaches.Table 1 reports performance on 12 datasets.Source: Table 1 · p. 6
• Collaborative gains are positive on all four reported datasets, with C-Gain values of 0.143, 0.184, 0.101, and 0.426.Figure 3 evaluates gains by grouping problems according to how many of the 10 individual LLMs solve them correctly.Source: Figure 3 · p. 6
• Dynamic-weight baselines optimize weights using the utility function, while role baselines include fixed chain/star structures and dynamically optimized systems.Dynamic-weight methods are Greedy Soup [97], Pack of LLMs [66], LoraHub [40], and Model Swarms [26]; dynamic-role methods include GPT-Swarm, Meta-Agent [37], Agent-Prune [109], GNNs [110], AgentVerse [11], and MACNet [73].Source: §4 · p. 6
• Evaluation covers 12 datasets spanning knowledge, reasoning, agent, and miscellaneous tasks.Knowledge: MMLU-pro [94], Knowledge Crosswords [19], and COM2 [23]; reasoning: GSM8k [13], NLGraph, and Normad [76]; agent: GAIA-text [67] and the knowledge graph and lateral thinking puzzle subtasks of AgentBench [61]; miscellaneous: Qasper [16], AbstainQA [25], and WoW [106].Source: §4 · p. 6

5 Results

• For knowledge tasks, STATIC WEIGHT and DYNAMIC WEIGHT outperform role baselines by 4.3% on average, while role baselines are 9.2% better on agent tasks.The differing results indicate that jointly optimizing roles and weights enables adaptation to both scenarios.Source: §5 · p. 7
• DYNAMIC WEIGHT approaches outperform STATIC WEIGHT by 30.1% across tasks, while DYNAMIC ROLE outperforms STATIC ROLE by up to 8.2%.The paper attributes this advantage to dynamic adaptation guided by the utility function f rather than hand-crafted role design or weight merging.Source: §5 · p. 7

Collaborative Gains

• 0.213 average collaborative gain, and 18.1% of B0 problems were solved despite no component LLM solving them individually.HETEROGENEOUS SWARMS also achieves greater C-Gain than the baseline. C-Gain is positive when Acc(Bn) exceeds expected accuracy n/N.Source: §6 · pp. 7–8

Importance of Role and Weight through Ablation Study

• Role and weight importance varied by task and was consistent in 10 of 12 datasets, with weight more important for knowledge tasks and role for agent tasks.The ablation compares performance with the role-step or weight-step disabled against the corresponding role- and weight-based baselines.Source: Table 2 · p. 7, §6 · p. 8

Role Statistics

• Gemini-as-a-judge classified four roles—divide, refine, feedback, and irrelevant—and role distributions varied across tasks and DAG positions.NLGraph showed more divide-and-conquer, while K-Crosswords showed more feedback; branching nodes more often divided, while converging nodes more often refined or provided feedback.Source: §6 · p. 8, Figure 4 · p. 8

Diversity Matters

• 89% average relative improvement resulted from using the most diverse rather than the least diverse initial LLM pool.The controlled experiment fixed pool size at 10 and compared 1 × 10, 2 × 5, 5 × 2, and 10 × 1 model configurations.Source: Figure 5 · p. 9, §6 · p. 9

Role Statistics

• During NLGraph optimization, irrelevant roles decreased while divide, refine, and feedback roles increased.The authors attribute this trend to weight optimization improving the LLMs’ adaptation to the task.Source: Figure 4 · p. 8, §6 · p. 9

Encouraging Sparsity

• L1 normalization retained performance better, while threshold pruning produced a larger reduction in inference cost.The study encourages sparsity either by pruning adjacency values below threshold τ or by adding a λ∥A∥1 penalty to the utility function.Source: §6 · p. 9, Table 3 · p. 9

• Prior role-based approaches assign prompt-induced roles and exchange generated text [21] [25], but depend on prompt engineering and model steerability [83] [77].Heterogeneous Swarms instead treats roles as input-output relationships and optimizes DAGs to learn contextual roles and specialization.Source: §7 · p. 9
• Weight-based approaches use mixture-of-experts [102], route or aggregate hidden states and logits [54] [30], or merge models statically [108] [103] [45] and dynamically [66] [1] [40].Heterogeneous Swarms uses swarm intelligence to optimize model weights according to each LLM’s individual contribution, building on evolutionary algorithms for weight-based collaboration [26].Source: §7 · p. 10
• Heterogeneous Swarms jointly optimizes the roles and weights of diverse LLMs, discovering and adapting novel multi-LLM systems.Source: §7 · p. 10

8 Conclusion

• Heterogeneous Swarms outperforms 17 role- and weight-based baselines on 12 tasks, demonstrating collaborative gains and heterogeneous roles.The conclusion attributes these gains to multi-LLM collaboration.Source: §8 · p. 10
• The method alternates role-step and weight-step to optimize LLM network structure and model weights, yielding DAGs called in topological order for a given task.The role-step optimizes connections between LLMs, while the weight-step optimizes model weights.Source: §8 · p. 10

1. Claims

• The checklist’s justification states that claims in the abstract and introduction are supported by the experiments and results in Sections 5 and 6.Source: §1 · p. 19

2. Limitations

• The checklist’s limitations entry points to Appendix A for its justification.The supplied passage does not include the answer to whether limitations are discussed.Source: §2 · p. 19

3. Theory assumptions and proofs

• The paper reports no theoretical results.This is given as the justification for the theory-assumptions-and-proofs checklist item.Source: §3 · p. 20

4. Experimental result reproducibility

• The paper answers Yes that it fully discloses information needed to reproduce its main experimental results, citing Section 4 and Appendix D.The checklist’s reproducibility question concerns information needed to reproduce results insofar as they affect the paper’s claims or conclusions.Source: §4 · p. 20

5. Open access to data and code

• Code will be made publicly available upon acceptance, while the paper reports using open-access, publicly available data.The response to the open-access question is No.Source: §5 · p. 20

6. Experimental setting/details

• The authors mark the training and test details as specified in Section 4 and Appendix D.Source: §6 · p. 21

7. Experiment statistical significance

• The authors mark statistical significance reporting as addressed and point to Table 8.Source: §7 · p. 21

8. Experiments compute resources

• The authors mark information on experiment compute resources as provided in Appendix D.Source: §8 · p. 22

10. Broader impacts

• The authors report that Appendix B discusses both potential positive and negative societal impacts.Source: §10 · p. 22

11. Safeguards

• The safeguards declaration is NA because the study uses open data and models that are already publicly available.Source: §11 · p. 23

12. Licenses for existing assets

• The authors answer yes on existing-asset credit and terms, stating that the open data and models are allowed for academic research.Source: §12 · p. 23

13. New assets

• The paper marks new-asset documentation NA because it introduces no new assets.Source: §13 · p. 23

14. Crowdsourcing and research with human subjects

• The paper reports no crowdsourcing experiments.Source: §14 · p. 24
• The paper reports no human subjects, so its human-subjects risk and IRB declaration is NA.Source: §14 · p. 24

16. Declaration of LLM usage

• The LLM-usage declaration is NA, and the authors state that LLMs were not used in writing the paper.Source: §16 · p. 24

A Limitations

• HETEROGENEOUS SWARMS assumes a shared model architecture for weight-step optimization; heterogeneous architectures require role-step-only optimization or token-level collaboration [58] and [26].The alternatives are to optimize the network with only the role-step or to use token-level collaboration for the weight-step.Source: A Limitations · p. 25
• Calling multiple LLMs during optimization and inference increases potential cost.The paper proposes combining sparsity, time-complexity analysis, and Dropout-W/R strategies to speed up the system.Source: A Limitations · p. 25
• The authors report that sparsity, time-complexity analysis, and Dropout-W/R can speed up HETEROGENEOUS SWARMS, separately or together.The strategies are discussed in Table 3, Section C, and Figure 9, respectively.Source: A Limitations · p. 25
• Experiments use neural language models as nodes, but black-box models can be included if skipped during weight-step, and tools or APIs can serve as input-output nodes [59, 28, 39, 78, 73, 113, 69, 74, 20, 105, 90, 112, 64, 22, 8, 111, 85, 62, 15, 41].These possibilities were not evaluated in the paper.Source: A Limitations · p. 25
• Future work could allow cycles with an exit function or learnable instance-level edits to expand the network’s expressive power.Cycles would allow a model to be called multiple times before exiting.Source: A Limitations · p. 25

B Ethics Statement

• A compromised or biased component could pass harmful behavior to later models, so the authors urge careful selection of specialized LLMs and future methods to identify harmful components [51] [117].They also note risks in agentic versions [34, 104, 96, 72, 18, 31, 70, 115, 4, 10, 12, 14, 38, 42, 63, 81, 93, 101, 107, 114, 84, 2].Source: B Ethics Statement · p. 25

Time Complexity

• Optimization costs O(n(N + M)|f|) model inferences per iteration, while inference costs O(n); optimization is more expensive, but inference is on par with most baselines.The authors argue that performance improvements and collaborative gains justify the additional optimization cost.Source: C Analysis (cont.) · pp. 25–26, Table 4 · p. 26

Scaling Multi-LLM Collaboration

• Adding LLM experts from 2 to 10 yields consistent improvements across four datasets.The scaling study uses 2, 4, 6, 8, and 10 initial checkpoints, with 10 as the default.Source: Figure 6 · p. 25, C Analysis (cont.) · p. 25

Collaborative Strength

• Collaborations of weaker models can beat the top individual LLM, and smaller-model ensembles can outperform single larger models.The comparisons include systems built without the task’s top LLM or top half of LLMs, and ensembles of three Gemma-2-9B or six Gemma-3-4B models against single 27B models.Source: C Analysis (cont.) · pp. 26–27, Figure 10 · p. 27

Accelerating with Dropout-R/W

• Randomly skipping role-steps or weight-steps accelerates optimization with only a minor performance drop.The study evaluates role-step and weight-step skip likelihoods of 0.2, 0.5, and 0.8.Source: C Analysis (cont.) · p. 26, Figure 9 · p. 27

Another LLM

• Heterogeneous Swarms also works with Mistral-7B and outperforms the best single Mistral-based LLM.The paper reports performance using 10 Mistral language models.Source: C Analysis (cont.) · p. 26, Table 5 · p. 27

Generalization to Unseen Tasks

• Systems optimized on Knowledge Crosswords generalize better to the held-out WikiDYK task.Both datasets are in the encyclopedic knowledge domain.Source: C Analysis (cont.) · p. 27, Table 6 · p. 27

Collaborative Gains and Reasoning Comparisons

• Heterogeneous Swarms achieves higher collaborative gains than model-merging baselines and outperforms single-model reasoning enhancements.The reasoning comparison uses chain-of-thought and graph-of-thought with the best initial LLM expert.Source: C Analysis (cont.) · p. 27, Figure 11 · p. 27, Figure 12 · p. 28

Collaboration of Different Model Sizes

• A mixed system of Gemma-7B and Gemma-2B models outperforms both the best single Gemma-7B and the full Gemma-2B system.The mixed-size setting uses half 7B and half 2B models.Source: C Analysis (cont.) · pp. 27–28, Table 7 · p. 28

Dataset Details

• Evaluation spans 12 datasets; most use multiple-choice or exact-match scoring, while AgentBench-LTP and Qasper use Gemini answer-similarity ratings normalized to 0–1.Optimization typically samples 200 examples and evaluation 1,000, with downsampling when data are insufficient; one-tailed z-tests assess statistical significance, and Table 8 presents dataset statistics.Source: D Experiment Details · p. 28, Table 8 · p. 28

Implementation Details

• Experiments use ten LLM experts from Feng et al. [26], Gemma-based versions in the main paper, and Mistral-based versions for Table 5.The authors grid-search other hyperparameters, run up to 50–200 randomly selected settings, and use a cluster of 16 A100 GPUs with 40 GB memory each.Source: D Experiment Details · p. 29

Baseline Details

• Baseline comparisons use Feng et al.’s [26] settings for nine methods and specified structure-search procedures for chain, star, gpt-swarm, meta-agent, agent-prune, and GNN baselines.For example, chain and star assignments are randomly permuted 20 times, and the best-found structure on utility f is reported.Source: D Experiment Details · p. 29

Prompts

• The appendix provides default multi-LLM instructions, a Gemini-as-a-judge prompt, and a prompt for evaluating intermediate-output roles.Dataset-specific changes are made to the general prompt format when necessary; the prompts appear in Tables 9–11.Source: D Experiment Details · p. 29, Table 9 · p. 30, Table 10 · p. 30

Qualitative Examples

• Examples on Knowledge Crosswords and NLGraph show HETEROGENEOUS SWARMS adapting multi-LLM systems to tasks through heterogeneous roles and collaborative generation.The examples are shown in Figures 13 and 14.Source: D Experiment Details · p. 29, Figure 13 · p. 31, Figure 14 · p. 32

Illustrating the Optimization

• The optimization traces show co-improvement in graph utility and model weights across Knowledge Crosswords, NLGraph, and AB-kg.Figure 15 tracks role-step and weight-step utility over time on the three datasets.Source: D Experiment Details · p. 29, Figure 15 · p. 33