Source-linked AI summary
ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestration
Hongjin Su, Shizhe Diao, Ximing Lu, Mingjie Liu, Jiacheng Xu, Xin Dong, Yonggan Fu, Peter Belcak, Hanrong Ye, Hongxu Yin, Yi Dong, Evelina Bakhturina, Tao Yu, Yejin Choi, Jan Kautz, Pavlo Molchanov
TL;DR
Complex agentic tasks remain difficult and costly for general-purpose language models, motivating systems that coordinate diverse tools and models. ToolOrchestra trains an 8B orchestrator with reinforcement learning using outcome, efficiency, and preference rewards, and reports stronger benchmark performance at lower cost than prior agents.
Problem
Large language models remain of limited utility on complex agentic tasks such as HLE, where tool use is needed to extend their capabilities.
Method
ToolOrchestra trains a small orchestrator end-to-end with reinforcement learning to coordinate diverse tools and models using outcome, efficiency, and user-preference rewards.
Results
Orchestrator-8B achieves state-of-the-art performance across challenging benchmarks while operating at significantly lower cost than larger models.
Takeaways & Limitations
The results support composing diverse tools with a lightweight orchestration model as an efficient approach to tool-augmented reasoning.
Takeaways & Limitations
Prompted off-the-shelf orchestrators can exhibit self-enhancement bias and default to the strongest available tool regardless of cost or relative utility.
Abstract
from arXiv · showhide
Large language models are powerful generalists, yet solving deep and complex problems such as those of the Humanity's Last Exam (HLE) remains both conceptually challenging and computationally expensive. We show that small orchestrators managing other models and a variety of tools can both push the upper bound of intelligence and improve efficiency in solving difficult agentic tasks. We introduce ToolOrchestra, a method for training small orchestrators that coordinate intelligent tools. ToolOrchestra explicitly uses reinforcement learning with outcome-, efficiency-, and user-preference-aware rewards. Using ToolOrchestra, we produce Orchestrator, an 8B model that achieves higher accuracy at lower cost than previous tool-use agents while aligning with user preferences on which tools are to be used for a given query. On HLE, Orchestrator achieves a score of 37.1%, outperforming GPT-5 (35.1%) while being 2.5x more efficient. On tau2-Bench and FRAMES, Orchestrator surpasses GPT-5 by a wide margin while using only about 30% of the cost. Extensive analysis shows that Orchestrator achieves the best trade-off between performance and cost under multiple metrics, and generalizes robustly to unseen tools. These results demonstrate that composing diverse tools with a lightweight orchestration model is both more efficient and more effective than existing methods, paving the way for practical and scalable tool-augmented reasoning systems.
1. Introduction
ToolOrchestra addresses the limits of monolithic tool-use agents by training a small orchestrator to coordinate heterogeneous tools and models. Its reward design targets outcome quality, efficiency, and user preferences, with evaluations reporting strong performance at lower cost.
- Motivation: LLMs remain limited on complex agentic tasks such as HLE, although external tools such as search engines and code interpreters can improve accuracy and reduce hallucinations.This motivates combining models and tools rather than treating tool use as a single-model add-on.
- Motivation: The orchestration paradigm delegates subproblems to tools with different capabilities and costs instead of relying on one powerful model for the entire task.The orchestrator selects tools and their order according to the task.
- Motivation: Prompting off-the-shelf models produces brittle delegation patterns, including self-enhancement bias and defaulting to the strongest available tool regardless of cost or utility.GPT-5 disproportionately calls GPT-5-mini, while Qwen3-8B relies heavily on GPT-5.
- Approach: ToolOrchestra trains an 8B orchestrator with reinforcement learning to decide when and how to invoke models, web search, code interpreters, and other tools.The training objective balances correctness, resource efficiency, and alignment with user cost and tool preferences.
- Contributions: ToolOrchestra includes ToolScale, a publicly intended dataset contribution for training tool-use agents.The dataset is designed to facilitate further research on tool-use agent training.
2. Agentic Problem Formulation
The paper formulates multi-turn tool use as a Markov Decision Process in which an orchestrator selects actions from interaction history and receives correctness, cost, latency, and preference signals. Rollouts alternate reasoning, tool calls, and observations to produce task solutions.
- 2.1. Task Formulation: The task is represented as an MDP with instruction, state, action, observation, transition, emission, reward, initialization, and discount components.This formalization follows conventions similar to prior work.
- 2.1. Task Formulation: The MDP objective maximizes trajectory correctness and user-preference alignment while minimizing cumulative action cost and latency.Each action has cost, latency, and preference-alignment terms, while the completed trajectory receives a correctness reward.
- 2.1. Task Formulation: The orchestrator chooses each action from the full interaction history, and the environment returns the next state and observation after that action.The policy conditions on the instruction, initial observation, prior actions, and subsequent observations.
- 2.1. Task Formulation: After N interaction steps, the trajectory is the accumulated history and the environment supplies a correctness reward between 0 and 1.The objective also aggregates preference alignment, monetary costs, and operational latencies across the trajectory.
- 2.2. Multi-Turn Rollout: A rollout interleaves chain-of-thought reasoning, tool actions, and environment observations across multiple turns.The rollout begins from a system prompt and question, and each generated step ends with an EOS token.
- 2.2. Multi-Turn Rollout: At each turn, reasoning analyzes the current state and plans the next action before the orchestrator selects an available tool.The supplied passage identifies APIs and specialized tools as examples of the available action choices.
3. ToolOrchestra
ToolOrchestra trains a small model to coordinate heterogeneous tools through multi-step reasoning, using reinforcement learning that balances task outcomes, efficiency, and user preferences. Its training data and configurations are synthesized to support diverse environments, tool availability, and cost structures.
- Core approach: ToolOrchestra trains an 8B model to generate multi-step reasoning and tool-use trajectories while dynamically selecting external tools.The toolset includes domain-specialized models and exposes tools through a unified interface.
- Reward design: The reward design combines binary task success with penalties for monetary cost and wall-clock latency.Compute and latency rewards are defined as negative trajectory cost and elapsed time.
- Reward design: User preference vectors specify how strongly the orchestrator should optimize tool usage, outcome, compute, and latency.For example, a preference can prioritize accuracy while assigning no weight to computational cost.
- Training procedure: GRPO generates trajectory batches, assigns scalar rewards, normalizes rewards within each group, and updates the policy with a clipped surrogate objective.Training also filters weakly informative, format-inconsistent, or invalid rollouts to stabilize reinforcement learning.
- Data synthesis: ToolScale synthesizes training data by generating simulated databases and tool APIs, then producing diverse tasks with golden function calls and required information.Generated tasks are checked for solvability using the supplied databases and APIs.
- Data synthesis: Training varies available tool subsets and pricing schedules across instances to encourage adaptation to heterogeneous tool configurations and costs.This construction is intended to prevent overfitting to fixed usage patterns.
4. Experimental Setting
The evaluation compares Orchestrator-8B with prompted orchestrators and monolithic language models across HLE, FRAMES, and τ2-Bench. The setup reports performance alongside cost and latency while accounting for benchmark and implementation differences.
- Baselines: Orchestrator-8B is compared with prompted LLM orchestrators and monolithic systems without tools, with basic tools, or with expanded specialized-model toolsets.The off-the-shelf comparison includes GPT-5, Claude Opus 4.1, and several open-source models.
- Benchmarks and metrics: The evaluation covers HLE, FRAMES, and τ2-Bench, using official proprietary-model prices and pricing systems for open-source models.Inference uses temperature 0, and Orchestrator may take up to 50 turns to solve a task.
- Metrics and reporting: Table 1 reports accuracy-oriented results alongside cost in US cents and latency in minutes averaged across HLE and FRAMES.Basic tools include domain functions, search, and code interpretation; additional τ2-Bench efficiency statistics appear in Table 16.
- Evaluation caveats: The reported HLE comparison uses the text-only subset for the baselines and Orchestrator, whereas existing state-of-the-art results use the full set.GPT-5 reached 77.7 in these experiments rather than its reported 84.2 because of implementation differences.
5. Experimental Results
Across HLE, FRAMES, and τ2-Bench, Orchestrator-8B outperforms strong baselines while using substantially lower cost and avoiding strong tool-selection biases.
- Claude Opus 4.1 with tools improves from 11.7 to 19.8 on HLE and from 58.2 to 63.5 on FRAMES, but costs 2.8x more and has 4x higher latency.
- Qwen3-235B-A22B with tools and models rises from 14.0 to 32.8 on HLE and from 39.5 to 74.2 on FRAMES, while consuming more than 2 times the cost and latency.
- Orchestrator-8B makes more balanced tool calls than other strong foundation models without strong bias toward a particular tool or model.The call proportions are averaged across HLE, FRAMES, and τ2-Bench.
- 37.1 on HLE and 76.3 on FRAMES, Orchestrator-8B surpasses all baselines by a large margin.On τ2-Bench, it also outperforms GPT-5 using basic tools by 2.5%.
6. Analysis
Analysis shows that Orchestrator-8B strategically allocates tool calls, retains the best performance-cost trade-off, generalizes to unseen models, and better follows user preferences.
- 6.1. Tool Use Analysis: Orchestrator-8B strategically selects among tools and models, unlike Claude Opus 4.1’s frequent GPT-5 calls and GPT-5’s preference for GPT-5-mini.
- 6.2. Cost Analysis: Orchestrator-8B consistently outperforms GPT-5, Claude Opus 4.1, and Qwen3-235B-A22B at a given HLE budget and reaches similar results at substantially lower cost.The comparison varies maximum allowed turns among 10, 20, 50, and 100.
- 6.3. Generalization: With models unseen during training, Orchestrator-8B consistently achieves the best performance at the lowest cost across HLE, FRAMES, and τ2-Bench.It adapts by using model descriptions to understand the unseen models’ strengths and weaknesses.
- 6.4. User Preferences: Orchestrator-8B exhibits remarkably better adherence to heterogeneous user preferences at test time than strong monolithic systems such as GPT-5.
7. Related Work
Related work frames tool learning as sequential decision-making and increasingly targets efficiency and controllability alongside correctness.
- Table 3 compares preference performance and reports that Orchestrator-8B best adapts to user preference during test time.
- Tool-learning research has progressed from supervised fine-tuning on tool-labeled dialogues toward reinforcement-learning formulations of sequential tool use.
- Recent methods emphasize reducing computational costs and improving controllability, while balancing accuracy, efficiency, and reliability through reward shaping.
8. Conclusion
The paper presents ToolOrchestra as a reinforcement-learning approach for coordinating diverse tools and models, with Orchestrator-8B achieving strong performance at lower cost.
- ToolOrchestra trains a small orchestration model end-to-end with reinforcement learning to unify diverse tools and specialized models.Its strategy is guided by outcome quality, efficiency, and human-preference rewards.
- Orchestrator-8B attains state-of-the-art performance while operating at significantly lower cost than larger models.
- These biased invocation patterns motivate training a dedicated small orchestrator to decide when and how to invoke more intelligent models.
B. Evaluation Benchmarks
The evaluation uses HLE and FRAMES benchmarks to assess difficult reasoning and retrieval-augmented generation capabilities.
- HLE evaluates iterative search and intensive reasoning on PhD-level questions spanning mathematics, humanities, and natural sciences.
- FRAMES evaluates end-to-end retrieval-augmented generation.
- The HLE text-only subset excludes questions solvable by simple web search and includes 10–14% requiring images.
- The evaluated model shows strong mathematical, quantitative, scientific, logical, coding, and factual reasoning, with weaknesses on specialized, computationally heavy, and niche humanities items.
D. Tools in training
The training toolkit combines diverse model-based and classical tools, accessed through third-party APIs and configured to support user preferences about tool use.
- Training samples subsets from a larger heterogeneous tool set for each example, while evaluation fixes the tool set for fair comparison.
- The training toolset includes query writers, web and local search, code writers and interpreters, math models, and generalist models.
- Web search uses the Tavily API, while local search uses a Faiss index with Qwen/Qwen3-Embedding-8B.
- Code tools combine expert models with a Python sandbox for code execution.
- The evaluation applies Together AI pricing, and the preference example favors local or open-source tools while avoiding API calls.
H. Generalization of pricing configurations
The method evaluates pricing generalization by testing Orchestrator-8B under costs not seen during training, while ToolScale supplies synthesized tool-use tasks and validation procedures.
- Under an unseen DeepInfra pricing configuration, Orchestrator-8B adapts its tool-calling strategy across outcomes, efficiency, and user preferences.
- Orchestrator-8B consistently achieves the best performance, cost, and latency under varied pricing configurations, demonstrating robust generalization.
- ToolScale generates end-to-end interaction data by simulating tool environments and producing tasks with corresponding ground-truth solutions.
- Synthesized tasks are filtered when golden execution errors occur, language models cannot solve them in pass@8, or no actions are required.
- Task success requires execution correctness, process fidelity, and operation completeness.
L. Calculation of rewards for preference-aware benchmark
Preference-aware benchmark rewards are computed from tool-use trajectories using baseline-normalized tool invocation statistics and aggregated across benchmark examples.
- The evaluation records each trajectory’s tool-invocation vector and compares it with a baseline vector obtained from the starting checkpoint.
- The final preference-aware reward is calculated for each benchmark example after constructing the baseline and trajectory statistics.
- The benchmark result sums the example-level rewards across all examples.