Source-linked AI summary
EvoRoute: Experience-Driven Self-Routing LLM Agent Systems
Guibin Zhang, Haiyang Yu, Kaiming Yang, Bingli Wu, Fei Huang, Yongbin Li, Shuicheng Yan
TL;DR
Complex agentic systems deliver strong results but face an intrinsic trade-off among performance, monetary cost, and completion speed. EvoRoute addresses this challenge with experience-driven, per-step routing that selects Pareto-optimal LLMs, achieving comparable or improved performance while reducing cost and latency on challenging benchmarks.
Problem
Complex agentic systems face an intrinsic tension among performance, monetary cost, and rapid task completion, while existing systems can incur substantial costs and delays.
Method
EvoRoute uses an evolving experience base, multi-faceted retrieval, and Pareto-optimal selection to dynamically choose LLM backbones for individual workflow sub-tasks.
Results
Up to 10.3% higher performance, approximately 20% of baseline cost, and nearly 3× faster execution were achieved across five challenging benchmarks.
Takeaways & Limitations
EvoRoute empirically addresses the agent system trilemma by reducing monetary cost and latency without compromising task success on benchmarks including GAIA and BrowseComp+.
Takeaways & Limitations
The evaluation focuses on CK-Pro and Smolagent rather than other deep-research frameworks, with exhaustive broader benchmarking limited by substantial token costs.
Abstract
from arXiv · showhide
Complex agentic AI systems, powered by a coordinated ensemble of Large Language Models (LLMs), tool and memory modules, have demonstrated remarkable capabilities on intricate, multi-turn tasks. However, this success is shadowed by prohibitive economic costs and severe latency, exposing a critical, yet underexplored, trade-off. We formalize this challenge as the \textbf{Agent System Trilemma}: the inherent tension among achieving state-of-the-art performance, minimizing monetary cost, and ensuring rapid task completion. To dismantle this trilemma, we introduce EvoRoute, a self-evolving model routing paradigm that transcends static, pre-defined model assignments. Leveraging an ever-expanding knowledge base of prior experience, EvoRoute dynamically selects Pareto-optimal LLM backbones at each step, balancing accuracy, efficiency, and resource use, while continually refining its own selection policy through environment feedback. Experiments on challenging agentic benchmarks such as GAIA and BrowseComp+ demonstrate that EvoRoute, when integrated into off-the-shelf agentic systems, not only sustains or enhances system performance but also reduces execution cost by up to $80\%$ and latency by over $70\%$.
1 Introduction
Complex agentic systems achieve strong performance on demanding tasks but face an intrinsic trade-off among performance, cost, and efficiency. EvoRoute addresses this Agent System Trilemma with experience-driven, fine-grained routing that selects models per sub-task and improves performance-cost-latency trade-offs.
- Motivation: Complex agentic systems coordinate heterogeneous models, tool workflows, and memory modules to solve multi-turn tasks.These systems have demonstrated strong results across machine learning engineering, information search, report generation, and issue repair.
- Motivation: $3 per task, up to 40 minutes, and failed 24-hour completion illustrate the cost and latency burden of high-performing agentic systems.The cited examples include substantial execution costs, long delays for simple queries, and failure to complete all MLE-Bench tasks within a day.
- Motivation: The Agent System Trilemma is the intrinsic tension among task performance, execution efficiency, and computational or monetary cost.Existing systems can excel in individual aspects but do not spontaneously achieve all three objectives.
- Approach: EvoRoute dynamically selects an LLM for each sub-task using historical retrieval and Pareto-optimal filtering across cost, efficiency, and performance.It avoids both a single model for the full task and a fixed multi-agent configuration.
- Approach: A dual-phase design explores multiple trajectories to populate its knowledge base, then uses one optimized execution path for faster and cheaper inference.The knowledge base and routing policy are refined through accumulated experience and environmental feedback.
- Empirical validation: Up to 10.3% higher performance, approximately 20% of baseline cost, and nearly 3× faster execution were achieved across five challenging benchmarks.The experiments include GAIA and BrowseComp+ and compare EvoRoute with vanilla agent systems.
2 Related Work
Related work places EvoRoute within two streams: increasingly automated multi-agent systems and model-routing methods that assign queries to suitable models. These approaches motivate dynamic routing but leave room for more resource-aware selection in complex workflows.
- Agentic AI systems: Multi-agent systems range from handcrafted configurations to partially automated systems that automate selected components.The taxonomy organizes systems by their level of automation.
- LLM and agent routing: Model routing assigns queries to appropriate models to exploit complementary capabilities while potentially reducing computational costs.Classical approaches include neural-network routers trained on performance or cost signals.
- LLM and agent routing: EvoRoute extends this routing direction to complex agentic systems whose workflows combine multiple specialized agents and tools.Its motivation is consistent with the need to select models according to varying task requirements.
3 Preliminary
The paper models an agentic system as a sequential, tool-using workflow whose agents operate over shared state and interaction history. It then replaces static agent-to-model assignments with a dynamic policy optimized over performance, cost, and execution delay.
- System definition: An agentic system resolves a query through sequential specialized roles, shared state, tool invocations, and scheduler-selected actions.The workflow can include task decomposition and sub-task execution.
- System definition: The execution trajectory is a sequence of states and actions ending at a terminal step from which the final answer is synthesized.The trajectory records the system’s interaction over the workflow.
- Static assignment: Existing systems commonly use a static, human-predefined mapping from agent roles to LLM backbones, limiting adaptation to sub-task difficulty.Examples include hard-coded model choices for web browsing and planning agents.
- Dynamic routing: EvoRoute learns a dynamic routing policy that selects an LLM for the active agent at each workflow step.This policy is intended to overcome the limitations of fixed agent-to-model assignments.
- Dynamic routing: The routing objective seeks a Pareto-optimal policy that maximizes task success while minimizing cumulative cost and wall-clock delay.Performance, cost, and efficiency are measured over execution trajectories.
4 Methodology
EvoRoute builds a self-evolving experience base from step-level execution records, then retrieves analogous precedents and selects models through Pareto filtration and uncertainty-aware sampling. Feedback from each completed action is logged back into the knowledge base to refine future routing decisions.
- Self-Evolving Experience Base: The experience base K stores step-level records containing task context, selected models, tools, cost, duration, execution success, and task-level success.Each record captures the interaction between sub-task characteristics, model choices, and resulting performance, cost, and efficiency.
- Self-Evolving Experience Base: After each workflow, EvoRoute appends newly generated records to K, creating an empirical repository for later retrieval and selection.The repository is updated with all records produced during the completed task.
- Multi-Faceted Retrieval: For a new sub-task, EvoRoute retrieves precedents matching agent role, semantic similarity, or anticipated tool use, combining these sets into K_cand.Semantic retrieval uses instruction embeddings and a similarity threshold, while tool retrieval targets overlapping operational profiles.
- Pareto-Optimal Filtration and Selection: For each candidate LLM, EvoRoute averages performance, cost, and duration over retrieved records, then removes models dominated across all three axes.A model is dominated when another is at least as strong on performance and no worse on cost and duration, with strict improvement on one axis.
- Pareto-Optimal Filtration and Selection: Thompson sampling draws from Normal-Inverse-Gamma posteriors for Pareto-optimal models and selects the highest sampled utility under weighted trilemma preferences.The weights reflect the desired trade-off among performance, cost, and duration.
- Self-Evolving Experience Base: After the selected model completes its action, EvoRoute logs the observed outcome into K, closing a feedback loop that continually updates routing knowledge.This feedback mechanism makes each decision and outcome available to subsequent routing steps.
5 Experiment
The experiments compare agent frameworks and routing methods across performance, cost, and latency, showing EvoRoute’s gains on challenging benchmarks and the value of its core components.
- Experiment Setup: The evaluation uses ReAct, Smolagents, and Cognitive Kernel-Pro, representing increasing architectural complexity.Baselines include predefined configurations and routing methods such as PromptLLM, MasRouter, and GraphRouter.
- Experiment Setup: EvoRoute is compared across performance, cost, and latency using GAIA, BrowseComp+, HotpotQA, DS-1000, and DDXPlus.Figure 3 visualizes normalized performance, inverted cost, and delay across GAIA subsets.
- Main Results: 74.60% average performance exceeds Gemini-2.5-Pro’s 74.27%, while EvoRoute uses 26% of its cost ($138.40 vs. $521.61) and reduces latency by 25% (92.23h vs. 123.00h).On GAIA, EvoRoute also reaches 63.18% versus Claude-4’s 58.28% while cutting cost by over 76%.
- Main Results: On BrowseComp+, EvoRoute outperforms Claude-4 at 38.72% versus 33.50%, with cost below 36% ($79.30 vs. $220.50) and roughly half the latency.The comparison reports 10.33h versus 20.15h latency.
- Main Results: EvoRoute allocates stronger models to cognitively demanding Plan Agent tasks and cost-efficient models to simpler File Agent operations.Gemini-2.5-Pro accounts for 37.4% of Plan Agent selections, while Qwen3-14B accounts for 25.79% of File Agent selections.
- Framework Analysis: Removing the knowledge base decreases performance by 13.21%, while disabling multi-faceted retrieval decreases performance by 11.22% and produces the highest cost and delay.Replacing Thompson sampling causes a 6.52% drop, whereas omitting Pareto filtration causes a 1.52% drop.
6 Conclusion
The paper presents EvoRoute as a dynamic routing framework for the agent system trilemma. Experiments on GAIA and BrowseComp+ report large reductions in cost and latency without compromising task success, supporting more practical deployment.
- EvoRoute is introduced as a dynamic model routing framework designed to address the agent system trilemma.
- The experiments report monetary-cost reductions of up to 80% and latency reductions exceeding 70% without compromising task success.The evaluation covers challenging benchmarks including GAIA and BrowseComp+.
- The authors characterize EvoRoute as a step toward making powerful agentic AI systems more practical, scalable, and economically viable for real-world deployment.
Limitation & Ethical Concerns
The evaluation focuses on CK-Pro and Smolagent rather than several other deep-research frameworks, partly because exhaustive benchmarking would be costly.
- The evaluation covers CK-Pro and Smolagent but excludes OWL, Agent-Orchestra, and AIME.
- The authors state that substantial token costs make exhaustive large-scale benchmarking impractical.
- The paper reports no immediate ethical risks because it uses standard public benchmarks and widely used open frameworks.
Contributions
The listed contributors divide responsibilities across implementation, manuscript preparation, visualization, experimentation, and project supervision.
- G. Zhang led method implementation, manuscript preparation, visualization, and experimental analysis.
- K. Yang conducted a substantial portion of the experimental work and served as a core contributor.
- H. Yu and B. Wu provided extensive guidance and project discussions, while F. Huang, Y. Li, and S. Yan offered senior-level supervision.
A Cold Start Issue
EvoRoute addresses the cold-start problem by populating its initially empty experience base through diverse exploration before deployment. This initialization required modest resources while producing an informative prior.
- The cold-start problem arises because an initially empty experience base makes retrieval and routing ineffective.
- EvoRoute uses a dedicated exploration strategy to populate the experience base with diverse initial experiences before operational deployment.
- 50 TaskCraft tasks were executed with both Smolagent and CK-Pro frameworks to collect initial experience data.
- At each execution step, the active agent’s LLM backbone was selected uniformly at random from the available model pool.
- $28.8 and approximately 3.6 hours produced around 480 step-level records for the initial experience base.
B Tool Prediction Function
PredictTools begins with a near-instantaneous heuristic lookup that maps explicit trigger keywords to corresponding tools, handling clear-cut cases without latency or API cost.
- PredictTools uses a two-stage hybrid strategy intended to balance predictive accuracy with minimal computational overhead.
- A predefined dictionary maps explicit keywords such as “search,” “run,” and “plot” to corresponding tools.
- Clear-cut tool predictions receive zero latency or API cost through the initial heuristic check.
D Dataset Details
The evaluation covers multiple agentic and task-specific benchmarks, while the manuscript also reports AI-based tools supporting preparation activities.
- GAIA contains 165 tasks across three difficulty tiers: 53 basic, 86 intermediate, and 26 advanced.
- BrowseComp+ includes 830 evaluation instances, while DS-1000 consists of 1,000 tasks.
- HotpotQA contains 7,405 queries, and DDXPlus was subsampled to 1,000 instances from approximately 130K samples.
- AI-based tools, including large language models, supported manuscript language polishing, clarity, reference formatting, and visualization generation.