PaperlayerView exact revision on arXiv

Source-linked AI summary

AOrchestra: Automating Sub-Agent Creation for Agentic Orchestration

Jianhao Ruan, Zhihao Xu, Yiran Peng, Fashen Ren, Zhaoyang Yu, Xinbing Liang, Jinyu Xiang, Yongru Chen, Bang Liu, Chenglin Wu, Yuyu Luo, Jiayi Zhang

arXiv:2602.03786v2cs.AIcs.CL

TL;DR

Existing sub-agent designs lack a dynamic abstraction view, limiting adaptability for complex, long-horizon tasks. AORCHESTRA models agents as a unified Instruction, Context, Tools, Model tuple and creates tailored executors on demand through an orchestrator. Across three challenging benchmarks, it achieves a 16.28% average gain in pass@1 over established baselines with Gemini-3-Flash.

  • Problem

    Existing sub-agent designs lack a dynamic abstraction view, limiting adaptability for complex, long-horizon tasks.

  • Method

    AORCHESTRA uses a unified four-tuple interface to compose specialized working memory and capabilities, while an orchestrator creates and delegates to task-tailored executors on demand.

  • Results

    16.28% average gain in pass@1 across GAIA, Terminal-Bench, and SWE-Bench-Verified is achieved over established baseline frameworks with Gemini-3-Flash.

  • Takeaways & Limitations

    AORCHESTRA supports plug-and-play executors and learnable orchestration, including cost-aware routing that approaches favorable performance–cost trade-offs.

Abstract

from arXiv · show

Language agents have shown strong promise for task automation. Realizing this promise for increasingly complex, long-horizon tasks has driven the rise of a sub-agent-as-tools paradigm for multi-turn task solving. However, existing designs still lack a dynamic abstraction view of sub-agents, thereby hurting adaptability. We address this challenge with a unified, framework-agnostic agent abstraction that models any agent as a tuple Instruction, Context, Tools, Model. This tuple acts as a compositional recipe for capabilities, enabling the system to spawn specialized executors for each task on demand. Building on this abstraction, we introduce an agentic system AOrchestra, where the central orchestrator concretizes the tuple at each step: it curates task-relevant context, selects tools and models, and delegates execution via on-the-fly automatic agent creation. Such designs enable reducing human engineering efforts, and remain framework-agnostic with plug-and-play support for diverse agents as task executors. It also enables a controllable performance-cost trade-off, allowing the system to approach Pareto-efficient. Across three challenging benchmarks (GAIA, SWE-Bench, Terminal-Bench), AOrchestra achieves 16.28% relative improvement against the strongest baseline when paired with Gemini-3-Flash. The code is available at: https://github.com/FoundationAgents/AOrchestra

1. Introduction

AORCHESTRA addresses the limited adaptability of existing sub-agent designs by treating agents as dynamically composable units and creating task-specific executors on demand. Its orchestrator decomposes objectives, curates context, selects capabilities, and delegates execution, with strong results across three benchmarks.

  • Existing multi-agent systems incur coordination overhead and provide limited control over context routing, complicating robust long-horizon execution.
  • Context-isolated sub-agents mitigate context rot but lack on-demand specialization, while static roles are inflexible, leave coverage gaps, and require heavy human engineering.
  • AORCHESTRA models each agent as a unified four-tuple—INSTRUCTION, CONTEXT, TOOLS, MODEL—for runtime composition of task-specific working memory and capabilities.Relevant context is filtered for the current subtask, while tools and models define the agent’s task-specific functionality.
  • Its dedicated orchestrator dynamically decomposes the objective, creates tailored sub-agents through the four-tuple interface, and delegates execution via explicit tool calls.The orchestrator focuses on orchestration rather than directly executing tasks.
  • 16.28% relative improvement against the strongest baseline is achieved across GAIA, Terminal-Bench 2.0, and SWE-Bench-Verified with Gemini-3-Flash.The framework consistently outperforms representative orchestration approaches and popular agent frameworks in the training-free setting.
  • +11.51% pass@1 on GAIA follows supervised fine-tuning, while cost-aware routing reduces average cost by 18.5%.These results support learning the orchestration policy through task orchestration and model-routing strategies.

2. Related Work

Prior work uses multi-agent collaboration and sub-agents invoked as tools to decompose tasks and manage context. However, existing sub-agent systems do not fully specialize agents and commonly rely on fixed specialists or manual configuration.

  • Multi-agent systems organize specialized roles through predefined workflows and communication protocols to enhance language-model task solving.
  • Sub-agent-as-tools systems invoke sub-agents for decomposed problems, including recursive spawning and context folding to manage intermediate steps.
  • Existing approaches do not treat sub-agents as fully specialized agents, while practical systems often configure fixed specialists with manual design.

3. Methodology

AOrchestra formulates orchestration as learning over structured actions and dynamically instantiated agents, each specified by instruction, context, tools, and model. The orchestrator delegates execution, routes context and capabilities, and can optimize task performance against execution cost.

  • A unified four-tuple agent abstraction: An agent is represented as Φ = (I, C, T, M), combining task instruction, curated context, tools, and an underlying model.The tuple separates working memory from capabilities and supports runtime creation of parametrized sub-agents.
  • Action Space of Orchestrator: AOrchestra treats orchestration as a policy over delegation and termination actions, while separating orchestration from environment execution.The orchestrator delegates subtasks through structured actions or finishes with a final answer; executors perform the delegated work.
  • Implementation of Delegate and Finish: Delegation instantiates an executor with a specified instruction, context, tool set, and model, returning a structured observation to the orchestrator.The executor conditions only on the delegated instruction and context and is restricted to the selected tools.
  • Advantages of AORCHESTRA: The orchestrator dynamically decomposes objectives and creates tailored sub-agents, while remaining independent of their internal implementations.This design supports task-specific capabilities, structured context, pluggable executors, and learning from interactive experience.
  • Learnable Orchestrator: Learning targets task orchestration and model routing, including subtask decomposition, context and tool selection, and performance–cost balancing.The paper uses supervised fine-tuning for orchestration and iterative in-context instruction optimization for cost-aware behavior and Pareto-efficient trade-offs.

4. Experiments

AORCHESTRA is evaluated across three interactive benchmarks and multiple orchestration analyses. It consistently outperforms baselines, while context control, learnable orchestration, cost-aware routing, and pluggable sub-agents provide additional benefits.

  • Experiment Setup: AORCHESTRA is evaluated on Terminal-Bench 2.0, SWE-Bench-Verified, and GAIA, using pass@1/pass@3 comparisons against representative agentic systems.The benchmarks cover terminal operations, software engineering, and general-purpose real-world tasks.
  • Main Results: 22.13% average pass@1 improvement over the best baselines is achieved with Gemini-3-Flash across the three benchmarks.AORCHESTRA outperforms the baselines on all evaluated environments.
  • Main Results: 80.00 pass@1 and 86.06 pass@3 are achieved on GAIA with Gemini-3-Flash as both orchestrator and sub-agent, including a 13.94-point pass@1 gain over OpenHands.With Claude-4.5-haiku as sub-agent, AORCHESTRA still attains 60.61 pass@1.
  • Main Results: 52.86 pass@1 and 57.14 pass@3 are achieved on Terminal-Bench 2.0 with Gemini-3-Flash, including a 64.29-point pass@1 improvement over Mini-SWE.Performance remains comparable to or better than specialized coding systems under other backbones.
  • Main Results: 82.00 pass@1 and 86.00 pass@3 are achieved on SWE-Bench-Verified with Gemini-3-Flash, outperforming ReAct and OpenHands under the same model setting.The benchmark evaluates patches for real open-source repository issues against provided tests.
  • Advantage 1: Context Control: Task-relevant compressed context outperforms both No-Context and Full-Context by retaining execution traces while filtering irrelevant information.The ablation changes only the Context field, keeping the sub-agent model, tools, and system prompt fixed.
  • Advantage 2: A Learnable Orchestrator: 68.48% accuracy after SFT improves over Qwen3-8B’s 56.97% baseline, while increasing average cost by $0.32 and attempts by 56%.The result supports orchestration as a learnable skill, though the gain has an associated cost increase.
  • Advantage 2: A Learnable Orchestrator: 75.15% accuracy at $0.57 average cost improves over 72.12% at $0.70 under mixed-model routing after Pareto-oriented in-context learning.Across model choices, AORCHESTRA configurations form a Pareto frontier with improved cost–performance trade-offs.

5. Conclusion

AOrchestra automates sub-agent creation through a unified four-tuple interface, enabling task-tailored executors with specialized working memory and capabilities. Across three benchmarks, it shows consistent improvements, including a 16.28% average pass@1 gain.

  • AOrchestra represents each sub-agent with the unified tuple (Instruction, Context, Tools, Model).This interface supports automated sub-agent creation within an orchestration-centric system.
  • The orchestrator dynamically spawns task-tailored executors with specialized working memory and capabilities.Working memory comprises instruction and context, while capabilities comprise model and tools.
  • On-demand specialization uses task-sufficient context while keeping sub-agents plug-and-play across implementations.The abstraction is decoupled from execution and can be optimized through supervised fine-tuning and context learning.
  • 16.28% average gain in pass@1 is reported across GAIA, Terminal-Bench, and SWE-Bench-Verified when paired with Gemini-3-Flash.The paper describes these results as strong and consistent improvements over established baseline frameworks.

A.1. Datasets

The evaluation covers three agentic benchmarks spanning digital-world question answering, sandboxed command-line workflows, and software engineering. Each benchmark has a specified test or validation split, with cost-driven sampling for Terminal-Bench and SWE-Bench.

  • GAIA evaluates general AI assistants on realistic tool-augmented questions involving web browsing and multi-step reasoning.The validation split contains 165 tasks.
  • Terminal-Bench 2.0 evaluates end-to-end real-world workflows in a sandboxed command-line environment using executable tests.Its test split contains 89 tasks, with 70 randomly sampled for the main experiments due to cost.
  • SWE-Bench-Verified evaluates software agents by requiring patches for real GitHub issues, verified through tests.The human-screened test split contains 500 tasks, with 100 randomly sampled for evaluation due to cost.

A.2. SFT Hyper-parameters.

The experiments use specified hyperparameters, with the SFT configuration summarized in Table 5.

  • The experiments use the hyperparameters specified for the reported evaluations.
  • The supplied passages identify the configuration source but do not report individual hyperparameter values.
  • Table 5 presents the SFT hyperparameters used in the study.

A.3. Baseline Implementations

Baseline implementations use diverse agentic frameworks, with benchmark-specific orchestration prompts guiding delegation, verification, model selection, and final submission. Some baseline results are omitted because of architecture or compatibility constraints.

  • Baseline implementations: Claude Code results are omitted from the main experiments because its architecture and intended usage are considered unsuitable for GAIA open-world multi-hop question answering.
  • Baseline implementations: DeepSeek-V3.2 is excluded from Table 1 because the authors found poor native compatibility with Claude Code.
  • GAIA: The GAIA prompt supports decomposing questions into subtasks, delegating specific work, reviewing subtask history, and completing once results are sufficient.Model selection favors cheaper models for simple tasks and more capable models for complex reasoning or critical attempts.
  • Execution controls: The prompts track progress, remaining attempts or steps, subtask history, and required output formats during execution.GAIA also requires precise, concise answers in its answer field.
  • Terminal-Bench: The Terminal-Bench prompt directs the main agent to verify requirements and choose submit or delegate_task based on status and test coverage.It also accounts for limited attempts and model-selection costs.
  • SWE-Bench: The SWE-Bench prompt requires checking the issue, prior progress, code changes, and tests before submitting or delegating further work.A submit action triggers the official FAIL_TO_PASS and PASS_TO_PASS evaluation suite.

C.1.1. CASE OVERVIEW

AOrchestra demonstrates strong GAIA performance on complex tasks, including long interaction chains and hard-task recovery. A representative case shows iterative correction, evidence propagation, and verification leading to a correct final answer.

  • 80.0% overall success was achieved across 165 GAIA tasks, with 88.7% at Level 1, 80.2% at Level 2, and 61.5% at Level 3.
  • AOrchestra completed representative long-horizon tasks after 10 attempts while maintaining context integrity, at costs of $5.93 and $3.06.
  • Hard tasks sometimes succeeded after repeated failures, including cases requiring 8–10 attempts through reflection, replanning, and iterative sub-agent execution.
  • The representative GAIA question asked for the number of figures with visible hands in a 2015 Metropolitan Museum of Art zodiac exhibition, whose expected answer was 11.
  • The case study task was solved in 10 attempts through error correction, intermediate-finding propagation, and hypothesis verification.

Attempt 10 - Main Decision (Final)

The final decision completes the GAIA case after identifying the relevant exhibition and zodiac set. The appendix also frames the system around orchestrator planning and specialized task delegation.

  • The final decision identifies “Celebrating the Year of the Ram” and the “Twelve animals of the Chinese zodiac” jade set as the basis for the correct answer.
  • The appendix introduces the framework’s available actions, tools, and execution constraints for each benchmark.
  • The appendix describes the Main Agent as responsible for global task planning and subtask delegation.
  • The delegate task action assigns a well-scoped subtask to a specialized Sub-Agent.

Action Schema: delegate task

The delegate-task schema packages a sub-agent request with task instructions, context, model selection, and optional tools, while completion submits the final answer and terminates execution.

  • The delegate-task request contains task instructions, context, model, and tools fields.
  • The task_instruction field specifies detailed, actionable instructions for the Sub-Agent.
  • The context field supplies additional information distilled from prior attempts.
  • The model field selects a model alias, while tools optionally restrict the tools exposed to the Sub-Agent.
  • The complete action submits a final answer string and terminates the task.

D.2. Sub-Agent Tools For each Benchmark

Table 6 summarizes the tools available to Sub-Agents for each benchmark together with their corresponding constraints.

  • Table 6 organizes Sub-Agent tools by benchmark and reports the constraints associated with each tool set.

D.2.1. GAIA TOOLS

The GAIA tool inventory equips sub-agents with web retrieval, code execution, and multimodal analysis, while the broader system defines benchmark-specific execution constraints and sandboxed environments.

  • GAIA tools: GAIA sub-agents use web search, webpage-content extraction, code execution, image analysis, and audio processing tools.Web search uses the Serper API, webpage extraction uses the Jina API, and multimodal actions call vision- or audio-capable LLM backends.
  • GAIA tools: GoogleSearchAction performs web search through the Serper API with query, result-count, and locale parameters.
  • GAIA tools: ExtractUrlContentAction retrieves webpage content through the Jina API using a URL and browsing query.
  • Execution constraints: ExecuteCodeAction runs Python or Bash code in an isolated workspace with a default 10-second timeout and restrictions on potentially destructive operations.Disallowed operations include file deletion, privilege escalation, permission changes, root-level redirection, and system-level commands.
  • Execution constraints: Web access is mediated through tool APIs rather than direct external-service access, with a 30-second search timeout and 50-second URL-extraction timeout.Retrieved web content and API responses or metadata may be cached and logged for reproducible evaluation.
  • Benchmark environments: Terminal-Bench and SWE-Bench use separate sandboxed execution setups, with 600-second and 300-second default test-related timeouts, respectively.Terminal-Bench supports Docker, E2B, and Daytona; SWE-Bench runs tasks in isolated Docker containers and executes pytest.
Loading 2602.03786v2…