PaperlayerView exact revision on arXiv

Source-linked AI summary

EvoRouter: Cost-Efficient Multi-Agent Routing via Insight-Guided On-Policy Distillation

Yitao Xiao, Xu Zhang, Ye Cheng, Shaoyong Guo, Yan Sun, Ziyue Zhu, Sitao Xie, Guo Tong, Haoran Luo

arXiv:2610.35690v1cs.AIcs.MA

TL;DR

Jointly configuring multi-agent systems is difficult because the configuration space is combinatorial and execution rewards provide sparse supervision for interdependent decisions. EvoRouter distills cross-task configuration insights into on-policy teacher supervision and combines it with cost-aware reinforcement learning. Across in-domain and out-of-domain settings, it achieves higher average accuracy at lower computational cost than evaluated routing baselines.

  • Problem

    Joint MAS configuration creates a combinatorial planning space, while sparse execution rewards provide limited guidance for interdependent routing decisions.

    Source: §1 · pp. 1–2

  • Method

    EvoRouter combines cost-aware reinforcement learning with on-policy distillation conditioned on cross-task configuration insights and execution feedback.

    Source: §1 · pp. 2–3, §4.4 · p. 5

  • Results

    Across in-domain and out-of-domain settings, EvoRouter achieves higher average accuracy at lower computational cost than evaluated routing baselines.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The resulting routing policy operates without privileged feedback at inference.

    Source: §6 · p. 8

    The current framework is designed for cloud-based agent services; extending it to resource-constrained edge environments is future work.

    Source: §6 · p. 8

Abstract

from arXiv · show

Serving web queries at scale with multi-agent systems requires jointly configuring large language models, collaboration modes, agent counts, interaction rounds, and skills. However, the resulting configuration space grows combinatorially, while sparse execution rewards provide limited fine-grained supervision for interdependent planning decisions. We propose EvoRouter, a self-evolving multi-agent routing framework driven by insight-guided on-policy distillation. A dedicated expert periodically synthesizes performance-cost feedback from configuration trajectories across tasks into reusable conditional routing insights about capability requirements, returns from resource scaling, and interactions among configuration choices. A privileged teacher combines these evolving insights with current execution feedback to provide dense token-level supervision for the router's own planning trajectories through on-policy distillation. Cost-aware reinforcement learning jointly optimizes task performance and computational cost. Together, these training signals internalize execution insights into the routing policy. The updated policy generates new trajectories that support subsequent insight extraction, forming an execution-driven self-improvement loop. We evaluate EvoRouter on 19 in-domain datasets spanning five domains and two out-of-domain benchmarks. EvoRouter achieves an average in-domain accuracy of 68.69%, outperforming the strongest routing baseline by 7.23 percentage points while reducing inference compute cost by 50.95%. Our code is open-sourced at https://github.com/CenYangtze/EvoRouter.

1 Introduction

Joint MAS routing must choose models, collaboration modes, agent counts, rounds, and skills to balance performance and cost, but conditional configuration benefits and sparse feedback make learning difficult. EvoRouter distills cross-task execution insights into routing and combines on-policy distillation with cost-aware reinforcement learning.

1 Introduction

  • Joint MAS routing extends model selection to models, collaboration modes, agent counts, interaction rounds, and agent skills and subtasks.The goal is to adapt configurations across Web-query domains and difficulty levels while optimizing reasoning performance and compute cost.Source: §1 · p. 1
  • Configuration benefits depend on both task requirements and surrounding choices, so isolated execution trajectories reveal outcomes only for their specific configurations.Sparse terminal feedback also gives limited guidance about which coupled decisions are effective or waste resources, or how to adjust them.Source: §1 · p. 2
  • EvoRouter combines conditional insights distilled from cross-task performance–cost feedback with on-policy distillation and cost-aware reinforcement learning.The expert extracts task-specific capability bottlenecks, resource-scaling returns, and configuration interactions; the training signals provide token-level supervision and optimize the performance–cost trade-off.Source: §1 · pp. 2–3
  • The introduction reports validation across 21 scenarios and seven domains, with better performance, lower cost, and greater robustness than representative baselines.The supplied passages do not provide the underlying measurements.Source: §1 · p. 3

2 Related Work

Prior MAS routing work either learns policies from execution feedback or evolves retrievable experience; EvoRouter combines policy learning with experience-derived insights for joint configuration.

2 Related Work

  • EvoRouter targets pre-execution joint configuration over fixed heterogeneous workers, guided by analyses of task-dependent scaling and configuration interactions.Related approaches also address per-query model selection, budget allocation, sequential invocation, delegation, agent or context selection, and execution state.Source: §2 · p. 3
  • EvoRouter applies group-relative reinforcement learning and on-policy distillation to student-generated plans, with the teacher conditioned on evolving cross-task insights and execution feedback.The resulting policy operates without privileged feedback at inference.Source: §2 · p. 3

3 Routing Insights and Problem Definition

MAS configuration outcomes vary with task demands, resource scale, and interactions among configuration choices. The reported examples show why routing must condition both resource selection and combination on those factors.

3.1 Routing Insights In Multi-Class Tasks

  • Switching from m_β+IO to m_α+FullConnected improves Multimodal success rate by 13.1 points but reduces QA success rate by 5.0 points.The passage uses this contrast to support task-aligned resource selection.Source: §3.1 · p. 3
  • Different models and collaboration strategies peak at different interaction rounds or agent counts, and further scaling may raise costs while degrading success rates.Resource allocation therefore need not produce proportional performance gains.Source: §3.1 · p. 3

EvoRouter RSI Loop

The supplied fragments label parts of an RSI loop as task analysis, subtask decomposition, and LLM, skills, and rounds configuration.

EvoRouter RSI Loop

Training Stage 2

Training Stage 2 distills configuration-dependent routing knowledge from tasks, configurations, outcomes, and costs. The problem formulation treats routing as selecting a detailed MAS configuration and optimizing verified task success against budget-normalized execution cost.

Training Stage 2

  • Resource scaling has configuration-dependent saturation points, so allocation should be tailored to the underlying model and collaboration strategy.Source: Training Stage 2 · p. 4
  • Under FullConnected, all-mβ beats all-mα; under Debate, all-mα wins and uses 79.8% fewer MAS FLOPs, so effective combinations depend on collaboration mode.Source: Training Stage 2 · p. 4
  • The framework distills condition-aware routing knowledge by jointly analyzing tasks, complete configurations, execution outcomes, and costs.The insights address task alignment, scaling saturation, and configuration interactions.Source: Training Stage 2 · p. 4

3.2 Problem Definition

  • For each task, the router constructs a MAS configuration from available models, collaboration modes, skills, subtasks, and agent assignments.Source: §3.2 · p. 4
  • The environment executes the configuration to produce an outcome; a task-specific verifier then evaluates the final answer and marks trajectory success.Source: §3.2 · p. 4
  • The routing policy jointly optimizes task success and resource efficiency through utility that balances success against budget-normalized execution cost.The cost penalty is controlled by λ_C.Source: §3.2 · p. 4

4 Method

EvoRouter initializes a routing policy through supervised fine-tuning, then iteratively combines execution-derived insights, on-policy distillation, and reinforcement learning. Its routing trajectories provide the observations used to refine insights and update the policy.

4.1 Evolution Loop

  • The router analyzes task context, produces a structured multi-agent plan for a deterministic executor, then uses execution observations to generate its final answer.Each trajectory records the task, configuration, execution outcomes, and verifier-evaluated answer.Source: §4.1 · p. 5

4.2 Supervised Initialization

  • Supervised fine-tuning on valid, correct bootstrap-router demonstrations initializes task analysis, planning, and result synthesis, and supplies a frozen reference policy.Prompt contexts and environment feedback are masked during loss computation.Source: §4.2 · p. 5

4.3 Cost-Aware Policy Optimization

  • The policy reward combines verifier-based task success with a budget-normalized compute-cost penalty, and group-relative updates are regularized against a frozen reference policy.The cost penalty is capped so successful trajectories have positive rewards while unsuccessful ones have nonpositive rewards; the KL term is token-level.Source: §4.3 · p. 5

4.4 Insight-Guided On-Policy Distillation

  • The expert updates routing insights from recent rollouts by inducing conditional rules, refining preconditions, and pruning redundant entries under a capacity ceiling.The rules address capability alignment, scaling saturation, and configuration interactions; distillation uses the latest insights.Source: §4.4 · p. 5
  • Evolving insights and execution feedback condition teacher supervision of student routing plans, while subsequent execution trajectories feed further insight refinement.Distillation targets student-generated analysis and plan content tokens, closing a loop among execution, insight synthesis, and policy internalization.Source: §4.4 · pp. 5–6

5 Experiments

EvoRouter outperforms routing baselines across several in-domain capability categories and unseen domains, with lower inference cost; its analyses examine ablations, iterative refinement, sensitivity to cost penalties, and generalization to unseen resources.

5.2 Performance Analysis

  • 7.23 percentage points higher accuracy than RCR-Router and 50.95% lower mean MAS cost (15.28k FLOPs units); total inference cost is 18.94k units.Total cost is 76.5% below MasRouter and 39.2% below RCR-Router, including router overhead.Source: Figure 5 · p. 7, §5.2 · p. 7
  • EvoRouter variants outperform baselines across four of five in-domain capability categories, with average accuracies of 68.69% for EvoRouter-4B and 71.23% for EvoRouter-8B.On unseen Code and Reasoning domains, sample-weighted accuracies range from 74.67% to 78.93%, exceeding the strongest baseline by 7.74 to 12.00 points.Source: §5.2 · p. 7

5.3 Framework Analysis

  • Full EvoRouter-4B improves accuracy over SFT alone by 8.92 in-domain and 11.47 out-of-domain points while reducing MAS costs by 9.0% and 13.9%, respectively.With matched iterations, insight-guided OPD gains 5.46 and 4.80 points over RL, compared with 1.31 and 1.06 points for OPD without insights.Source: §5.3 · p. 7
  • Iterative refinement first raises success by 2.62 points while lowering compute by 16.4%, then later gains 1.35 points with 7.4% lower cost.Between these stages, subsequent updates use additional compute for further accuracy gains.Source: §5.3 · p. 7
  • Higher λ_C steers policies toward lower-cost configurations, and increasing penalties shifts per-question compute toward lower budgets while suppressing the expensive upper tail.Accuracy sensitivity to the penalty varies across domains.Source: §5.3 · pp. 7–8

5.4 Generalization Analysis

  • With LLM, collaboration-mode, and skill pools replaced, EvoRouter-4B beats MasRouter by 3.85 in-domain and 7.47 out-of-domain points while reducing compute by 6.7% and 18.5%.It also exceeds the strongest out-of-domain baseline by 5.9 points using less than half its compute.Source: §5.4 · p. 8, Figure 9 · p. 8

5.5 Case Study

  • A mathematical-reasoning case illustrates accurate, cost-effective MAS routing; Figure 11 provides an example of distilled routing insights.The passage does not report quantitative results for this illustration.Source: §5.5 · p. 8

6 Conclusion

EvoRouter jointly configures heterogeneous LLM-based multi-agent systems from execution experience, combining cost-aware reinforcement learning with on-policy distillation. The framework is evaluated across in- and out-of-domain settings, while its current design targets cloud-based agent services.

6 Conclusion

  • EvoRouter internalizes cross-task configuration insights and trajectory feedback through cost-aware reinforcement learning and on-policy distillation, then deploys without privileged feedback.Across 21 datasets in seven domains, it achieves higher average accuracy at lower computational cost than evaluated routing baselines in both in- and out-of-domain settings.Source: §6 · p. 8
  • The current framework targets cloud-based agent services; future work proposes coordinated cloud–edge routing that accounts for device heterogeneity and inter-agent communication overhead.Source: §6 · p. 8

A.1 Resource

  • The available collaboration modes include single-agent IO, CoT, and Reflection; multi-agent Chain, FullConnected, and Debate; and Self-Consistency and Hierarchical approaches.Some modes constrain configurations: for example, CoT supports at most one agent, so a two-agent CoT plan is rejected.Source: §A.1 · p. 11

A.2 Dataset

The evaluation uses a partitioned question dataset, task-specific cost budgets, three router backbones, and a set of routing baselines. Accuracy is averaged across runs, with a fixed strong backbone for baselines lacking native model selection.

A.2 Dataset

  • The dataset contains 12,800 questions across five domains, split into 10,500 training, 1,000 development, and 1,300 test questions; out-of-domain evaluation uses 175 LiveCodeBench and 200 BBH-LD questions.Source: §A.2 · p. 11
  • Task-domain budgets use the median across questions of each question’s 25th-percentile successful-trace cost, plus a 15% margin and a 512-unit minimum.Example budgets are 12,374.00 for math and 5,722.02 for commonsense tasks.Source: §A.2 · p. 11

A.3 Training

  • Three router backbones are trained on eight A100 80GB GPUs, with mutually disjoint task sets for SFT, RL, and evaluation.DeepSeek-V4-Pro supplies bootstrap trajectories for supervised initialization and serves as the insight-distillation expert, but does not directly supervise the student.Source: §A.3 · p. 11

A.4 Evaluation

  • Accuracy is averaged over three runs, and baselines without native model selection use GLM-5, the pool’s largest-active-parameter model.The appendix describes vanilla LLM, single-agent, multi-turn, and multi-agent routing baselines.Source: §A.4 · pp. 11–12

B Derivation of Inference Computational Cost

The appendix estimates inference cost with a FLOPs-based proxy rather than API prices, which can vary across providers and time. It derives the proxy from active model parameters and token lengths, then aggregates estimated worker-call costs along a trajectory.

B Derivation of Inference Computational Cost

  • API pricing can fluctuate across providers and over time, so the paper models computation for a decoder-only Mixture-of-Experts model instead.Its active matrix parameter count includes always-active backbone matrices and the routed experts activated across layers.Source: B Derivation of Inference… · p. 12
  • An invocation’s estimated FLOPs combine dominant parameter-matrix operations, causal-attention interactions, and remaining operations under KV caching.The matrix term is 2A_m FLOPs per token, while attending to t positions adds 4LDt FLOPs.Source: B Derivation of Inference… · p. 12
  • In experiments, parameter-matrix operations exceed 95% of total FLOPs, so the proxy omits attention interactions and R_m and is 2A_m(n_in+n_out); 2A_0 FLOPs define one cost unit, with A_0=3×10^9 for Qwen3.5-35B-A3B.Source: B Derivation of Inference… · p. 12

Related papers

Loading 2610.35690v1…