Source-linked AI summary
RouteMoA: Dynamic Routing without Pre-Inference Boosts Efficient Mixture-of-Agents
Jize Wang, Han Wu, Zhiyuan You, Yiming Song, Yijun Wang, Zifei Shan, Yining Li, Songyang Zhang, Xinyi Le, Cailian Chen, Xinping Guan, Dacheng Tao
TL;DR
MoA-based collaboration is costly and difficult to scale because existing filtering methods still infer all candidate models and lack principled selection. RouteMoA uses query-based pre-inference screening, posterior score refinement, and cost-aware ranking to route only promising models. It outperforms MoA across model-pool sizes while reducing large-pool cost by 89.8% and latency by 63.6%.
Problem
Existing MoA methods require all models to infer before judging, lack principled model selection, and become costly or exceed context limits in large pools.
Method
RouteMoA uses a lightweight query-aware scorer for initial screening, self- and cross-assessment for posterior correction without extra inference, and ranking that balances performance, cost, and latency.
Results
89.8% lower cost and 63.6% lower latency than MoA are reported for the large-scale model pool, while RouteMoA maintains strong accuracy and outperforms MoA and SMoA in reported evaluations.
Takeaways & Limitations
Dynamic prior-posterior routing provides a scalable and practical approach to efficient multi-LLM collaboration in heterogeneous model pools.
Takeaways & Limitations
The scorer requires retraining when new LLMs are added, although training on a small curated query set takes about 25 minutes.
Abstract
from arXiv · showhide
Mixture-of-Agents (MoA) improves LLM performance through layered collaboration, but its dense topology raises costs and latency. Existing methods employ LLM judges to filter responses, yet still require all models to perform inference before judging, failing to cut costs effectively. They also lack model selection criteria and struggle with large model pools, where full inference is costly and can exceed context limits. To address this, we propose RouteMoA, an efficient mixture-of-agents framework with dynamic routing. It employs a lightweight scorer to perform initial screening by predicting coarse-grained performance from the query, narrowing candidates to a high-potential subset without inference. A mixture of judges then refines these scores through lightweight self- and cross-assessment based on existing model outputs, providing posterior correction without additional inference. Finally, a model ranking mechanism selects models by balancing performance, cost, and latency. RouteMoA outperforms MoA across varying tasks and model pool sizes, reducing cost by 89.8% and latency by 63.6% in the large-scale model pool.
1 Introduction
RouteMoA addresses the cost, latency, and scalability limits of dense Mixture-of-Agents by dynamically selecting models before inference and refining those selections without additional inference. Its query-aware routing targets complementary model capabilities while maintaining strong accuracy and efficiency.
- Motivation: Specialized LLMs exhibit distinct task capabilities, making query-based prediction of model performance feasible.Qwen2.5-Coder leads coding, Qwen2.5-Math leads mathematics, Bio-Medical-Llama leads biomedical knowledge, and Gemma stands out in reasoning and reading.
- Motivation: Classical and sparse MoA incur high cost and latency because they forward all models, while sparse MoA additionally invokes a judge.These methods also lack principled model selection and become difficult to scale to large model pools because inference can exceed context limits.
- Method: RouteMoA uses a lightweight scorer to estimate model suitability from the query and narrow the initial candidate pool without executing inference.This enables only a subset of high-potential models to be activated, reducing inference overhead.
- Method: A mixture of judges combines scoring with self- and cross-assessment of existing responses, correcting scores without additional inference.Model ranking then balances performance, cost, and latency when selecting models.
2 Related Work
Related work establishes MoA as an effective layered collaboration strategy and contrasts it with routing approaches. RouteMoA combines dynamic model selection with subsequent multi-agent collaboration to retain complementary-model benefits while improving efficiency.
- General and task-specific LLMs: General-purpose and specialized LLMs provide broad and domain-specific capabilities, respectively.Examples include Qwen2.5-Math, Qwen2.5-Coder, and Bio-Medical-Llama as domain-specific variants.
- LLM routing: Sparse MoA reduces forwarded input tokens with a judge but still requires inference from all models, whereas RouteMoA selects suitable models dynamically without pre-inference.The router is designed to reduce both cost and latency.
- LLM routing: RouteMoA extends single-model routing by using subsequent multi-agent collaboration to enhance robustness and overall performance.Its routing process includes dynamic selection across layers rather than choosing only one model per query.
3 Methodology
RouteMoA dynamically routes a mixture-of-agents system by screening models before inference, refining scores from existing outputs, and ranking candidates by performance and efficiency. Its pipeline combines a scorer, mixture of judges, model ranking, and early stopping across layers.
- Routing Overview: RouteMoA dynamically selects a subset of top-performing LLMs for each layer without pre-inference to reduce cost and latency while maintaining performance.The framework retains the layered Mixture-of-Agents structure while routing only selected models.
- Score Acquisition: The first-layer scorer predicts coarse-grained performance for every model and narrows the pool to a small group of high-potential candidates.Precise score estimation is unnecessary because the scorer is intended for efficient candidate screening.
- Model Ranking and Selection: Model ranking selects top-k models by prioritizing performance, output-token cost, input-token cost, and latency, in that order.Model pricing and latency are sourced from OpenRouter.
- Model Ranking and Selection: Early stopping enters aggregation when the maximum adjusted score exceeds a threshold or the maximum layer number is reached.The final aggregation stage then produces the system output.
- SLM-based Scorer: The scorer uses query embeddings and learnable model embeddings to compute sigmoid-based performance scores, trained with contrastive objectives.Training data spans mathematics, reasoning, coding, reading comprehension, and biomedical domains, while the score combines ground-truth accuracy with a reward model.
- Mixture of Judges: For later layers, mixture-of-judges refines initial scores using previous-layer responses through self-assessment and cross-assessment.Cross-assessment selectively uses the highest-scoring model from the previous layer and applies only from the second layer onward.
4 Experiments
RouteMoA is evaluated across large- and small-scale model pools, diverse capability categories, and out-of-distribution tasks, with analyses of scorer quality and judge components. It improves accuracy while reducing cost and latency relative to MoA and SMoA.
- Experimental Setup: The evaluation compares RouteMoA with MoA and SMoA on large- and small-scale model pools, including 15-model and 5-model settings.The benchmarks cover five capability categories, multiple task datasets, and out-of-distribution generalization.
- Large-Scale Model Pool: 78.6 average accuracy surpasses MoA (71.3) and SMoA (69.7) on the large-scale model pool.RouteMoA also reports especially large gains in Math Reasoning (+47.5%) and Language Generation (+23.9%).
- Large-Scale Model Pool: 89.8% lower total cost and 63.6% lower latency than MoA are achieved while RouteMoA also outperforms SMoA in efficiency and accuracy.RouteMoA remains practical for large pools where full inference becomes costly and can exceed context limits.
- Small-Scale Model Pool: 81.4% lower inference cost and 38.7% lower average latency than MoA accompany RouteMoA’s highest small-pool average score of 83.1.The improvement over SMoA is statistically significant (t = 2.296, p = 0.0217 < 0.05).
- Out-of-Distribution Generalization: RouteMoA exceeds SMoA on out-of-distribution accuracy (54.62 vs. 52.92) while reducing cost by 11.5% and latency by 24.7%.Accuracy gains are reported for Geography (+7.04), History (+5.10), Physics (+4.50), and Biology (+4.77).
- Scorer and Judge Analysis: Top-3-Agree reaches 96.2%, indicating that scorer-selected candidates usually contain high-performing models for later refinement.Top-3-Hit is 97.9%, and the judge ablation reports average scores of 82.6 without self-assessment, 82.7 without cross-assessment, and 83.1 for RouteMoA.
5 Conclusion
RouteMoA addresses the resource limitations of classical MoA through dynamic routing that combines query-based screening with posterior score refinement. The framework reduces cost and latency while maintaining strong performance, with reported OOD generalization and large-pool scalability.
- 5 Conclusion: RouteMoA uses a lightweight scorer for prior-knowledge candidate screening, then mixture-of-judges refinement based on model outputs.The framework dynamically routes models without pre-inference and balances performance, cost, and latency.
- 5 Conclusion: RouteMoA significantly reduces cost and latency while maintaining strong performance across evaluated settings.The conclusion also reports strong out-of-distribution generalization and scalability for large model pools.
6 Limitation
The scorer requires retraining to support new LLMs, while RouteMoA’s inference-stage prompts coordinate aggregation, self-assessment, and cross-assessment across layers.
- Limitation: Retraining the scorer is required when supporting new LLMs.The paper notes that future work will explore retrain-free routing.
- Limitation: 25 minutes of lightweight-scorer training on a small curated query set is reported for integrating a new LLM.
- Inference prompts: Layer-specific prompts aggregate previous-layer responses and assess answers through self-assessment and cross-assessment.Layer 1 combines aggregation and self-assessment, while later layers use the corresponding assessment prompts.
- Training-data generation: Training data generation evaluates model answers and aggregated responses against ground-truth answers, using InternLM2-1.8B-Reward as a judge model.
C Clustering Details for Sample-Sample Loss
The scorer is trained with sample-sample contrastive clustering, and its selection quality is evaluated using Top-1-Hit, Top-3-Hit, and Top-3-Agree metrics.
- Clustering details: Sample-sample contrastive loss clusters prompt embeddings into Q groups and samples in-group and out-group queries for training.t-SNE and k-means produce the low-dimensional clustered representations.
- Evaluation metrics: Top-1-Hit measures whether the scorer’s top prediction is among models capable of answering correctly.The metric averages binary scores over answerable test cases.
- Evaluation metrics: Top-3-Hit measures whether at least one correct-answering model appears among the scorer’s top-three predictions.The metric is averaged over test cases answerable by at least one model.
- Evaluation metrics: Top-3-Agree compares the scorer’s top-three models with the three best models according to ground-truth performance.It assigns scores of 1, 0.6, 0.3, or 0 when three, two, one, or zero selections overlap.
- Evaluation metrics: The Top-3-Agree score is averaged over test cases that can be answered correctly by at least one model.
Scorer evaluation under different α and λ
Scorer evaluation varies the training hyperparameters α and λ and reports Top-1-Hit under those combinations.
- Evaluation: Three scorer metrics are calculated under different combinations of training hyperparameters α and λ.The metrics are Top-1-Hit, Top-3-Hit, and Top-3-Agree.
- Evaluation: Figure 10 reports Top-1-Hit under different scorer training hyperparameters λ and α.
E Dataset Statistics for Scorer Training of Small-Scale Model Pool
Scorer-training dataset statistics are reported in Table 6, with train, development, and test partitions generally following each dataset’s original split.
- Dataset statistics: Table 6 presents the dataset statistics used for scorer training.
- Dataset splits: Train, development, and test splits generally follow the original partitioning of each dataset.
- Related evaluation: Figure 11 reports Top-3-Hit under different scorer training hyperparameters.
- Related evaluation: Figure 12 reports Top-3-Agree under different scorer training hyperparameters.
- Dataset splits: Datasets without a development split are divided into training and development subsets from the original training portion.
F Evaluation Details on Large-Scale Model Pool
RouteMoA is evaluated on a 15-model large-scale agent pool and is designed to select query-appropriate subsets while controlling cost and performance. The evaluation uses 30 test sets, including three out-of-distribution sets.
- The large-scale agent pool contains 15 recent LLMs spanning 4B to 235B parameters, capabilities, and think modes.
- MoA and SMoA are infeasible for large pools because they require every LLM to infer before aggregating responses.This creates substantial inference cost as the pool grows.
- Large pools can exceed context limits, while manually choosing powerful models raises cost and choosing smaller models may reduce performance.
- RouteMoA selects agent subsets using criteria based on user-query categories and complexity to lower cost while maintaining competitive performance.Its scorer is trained with a compact query pool assembled from diverse datasets.
- Evaluation covers 30 test sets with 15 non-overlapping samples each, including lcqmc, mrpc, and cluewsc2020 as out-of-distribution sets.
G Model & Data License and Intended Use Statement
The paper states that its publicly available models and benchmark datasets are used for their intended research purposes under the creators’ terms of use. Tables document the large agent pool, test categories, and scorer-training query pool.
- The models and datasets are publicly available, cited, and used consistently with their creators’ intended research purposes.The authors state that they follow each model’s and dataset’s stipulated terms of use.
- Table 7 documents the models forming the larger agent pool.
- Table 8 documents the test dataset categories for the large agent pool.
- Table 9 documents the query pool used to train the scorer for the larger agent pool.