Source-linked AI summary
Orchestra-o1: Omnimodal Agent Orchestration
Fan Zhang, Vireo Zhang, Shengju Qian, Haoxuan Li, Hao Wu, Jinyang Wu, Donghao Zhou, Zhihong Zhu, Zheng Lian, Xin Wang, Pheng-Ann Heng
TL;DR
Existing agent systems have limited support for heterogeneous modalities, motivating Orchestra-o1, which coordinates modality-aware decomposition, specialized sub-agents, and parallel execution. It establishes a new state of the art on OmniGAIA, improving over Gemini-3-Pro by 10.3%.
Problem
Existing LLM agent systems support limited modalities, leaving a gap for unified understanding and decision-making across interacting text, image, audio, and video inputs.
Method
Orchestra-o1 separates high-level orchestration from tool-augmented execution through dynamic task decomposition, online specialization, parallel sub-tasks, and DA-GRPO training.
Results
10.3% improvement over Gemini-3-Pro establishes Orchestra-o1 as the state-of-the-art system on the OmniGAIA benchmark.
Takeaways & Limitations
Orchestra-o1 achieves strong gains over native omnimodal agents and orchestration baselines, while its open-source Orchestra-o1-8B improves over strong open-source omnimodal baselines.
Takeaways & Limitations
Orchestration adds system complexity through sub-agent histories, tool schemas, backend configurations, cost accounting, and asynchronous execution.
Abstract
from arXiv · showhide
The recent success of agent swarms has shifted the paradigm of large language model (LLM)-based agents from single-agent workflows to multi-agent systems, highlighting the importance of agent orchestration for task decomposition and collaboration. However, existing orchestration frameworks are limited to a narrow set of modalities and struggle to generalize to more complex settings where heterogeneous modalities coexist and interact. This limitation becomes particularly pronounced in omnimodal scenarios, where tasks require the unified understanding and coordination of diverse inputs such as text, image, audio, and video. In this work, we propose Orchestra-o1, an omnimodal agent orchestration framework designed to support efficient agent collaboration across multiple modalities. Orchestra-o1 introduces a unified orchestration mechanism that enables modality-aware task decomposition, online sub-agent specialization, and parallel sub-task execution. This scalable design allows agent systems to effectively tackle complex real-world tasks involving heterogeneous information sources, surpassing the second-best approach by 10.3% accuracy on the OmniGAIA benchmark. Furthermore, we introduce decision-aligned group relative policy optimization (DA-GRPO), an efficient agentic reinforcement learning approach for training Orchestra-o1-8B, which also achieves state-of-the-art performance against all existing open-source omnimodal agents.
1. Introduction
Existing LLM agent systems generally support limited modalities, leaving a gap for real-world tasks involving interacting text, image, audio, and video. Orchestra-o1 addresses this gap with modality-aware orchestration and DA-GRPO training, achieving strong OmniGAIA improvements.
- Motivation: Existing agent systems typically focus on text or vision-language tasks, limiting their ability to handle real-world interactions among text, image, audio, and video.This modality gap motivates omnimodal agent orchestration.
- Results: 32.8% improvement over AOrchestra and 10.3% improvement over Gemini-3-Pro establish Orchestra-o1 as a new state-of-the-art on OmniGAIA.Both improvements are reported for Orchestra-o1 with 2025] as the main agent.
- Training: DA-GRPO is an efficient offline agentic reinforcement learning algorithm designed to enhance orchestration decision-making in an open-source main agent.It extends GRPO with a design tailored specifically for agent orchestration.
- Orchestra-o1: Orchestra-o1 provides modality-aware task decomposition, online sub-agent specialization, and parallel sub-task execution for complex omnimodal agent tasks.The framework decouples high-level orchestration from specialized perception and action execution.
2. Related Work
Related work has progressed from single-agent tool use and planning toward multi-agent orchestration, while omnimodal intelligence expands multimodal modeling beyond bimodal settings. Orchestra-o1 addresses this gap with unified, modality-aware coordination for scalable collaboration across heterogeneous inputs and tools.
- Agent orchestration: LLM-agent research has shifted from single-agent reasoning with iterative tool interaction toward multi-agent systems coordinated by a central orchestrator.Early efforts emphasized tool use and planning within one agent, whereas newer systems coordinate multiple specialized agents.
- Research gap and contribution: Orchestra-o1 differs from prior studies by combining modality-aware task decomposition with dynamically adaptive coordination, parallel execution, heterogeneous tool integration, and scalable multi-agent collaboration.The framework is designed for real-world omnimodal environments requiring flexible coordination strategies.
- Omnimodal intelligence: Omnimodal intelligence unifies text, image, audio, and video processing, extending earlier multimodal work that primarily focused on bimodal vision-language or audio-language settings.Early vision-language models demonstrated alignment between visual and textual representations.
3. Methodology
Orchestra-o1 formulates omnimodal orchestration as a closed-loop, dependency-aware decision process that decomposes heterogeneous inputs into specialized sub-tasks, assigns suitable models and tools, executes independent work in parallel, and aggregates evidence. Its training recipe introduces DA-GRPO to align main-agent decisions with successful orchestration behavior while enabling offline decision scoring.
- Problem Definition: Omnimodal orchestration is defined as multi-round decision-making over a question and heterogeneous image, audio, or video inputs, maximizing reward against the ground-truth answer.The task instance is x = (q, M), with M containing auxiliary modality inputs, and the system produces a concise final answer.
- System Formulation: The main agent acts as an orchestrator, choosing either completion or delegation; delegation generates sub-tasks with contexts, selected backends, and assigned tool subsets whose results update structured history.Each independent sub-agent returns status, an answer-like result, a summary, and an execution trace before the main agent continues or answers.
- Modality-aware Task Decomposition: The main agent induces a dependency graph with modality and tool masks, selects executable ready sub-goals, and runs independent sub-tasks in parallel while delaying dependent work.The framework explicitly models collaboration as dependency-aware parallel scheduling with learnable decomposition, model and tool selection, evidence aggregation, and stopping decisions.
- Backend and Tool Selection: Orchestra-o1 coordinates perception tools for image, audio, and video analysis with action tools for web search, page visits, and code execution, routing sub-tasks to backends according to capability and cost-latency profiles.The framework sends easier extraction and search tasks to cheaper models and difficult omnimodal reasoning tasks to stronger backends.
- Framework Summary: Orchestra-o1 uses a closed-loop process that maintains structured memory, refines the dependency graph from new evidence, and compresses specialized results until the final answer is sufficiently supported.Its formulation separates high-level planning from specialized perception and action execution.
- Training Recipe: DA-GRPO trains the open-source main agent to favor strategically aligned orchestration decisions and scores decisions offline from reconstructed states, avoiding repeated execution of expensive sub-agent trajectories.The method is presented as a GRPO-style training approach and is contrasted with outcome-only reinforcement learning.
4. Experiments
Experiments evaluate Orchestra-o1 on the heterogeneous OmniGAIA benchmark using accuracy, category and difficulty breakdowns, and comparisons against native, orchestration-based, and proprietary baselines. Orchestra-o1 achieves strong accuracy and efficiency, while DA-GRPO provides the best reported post-training result.
- Benchmark and Baselines: Orchestra-o1 is evaluated on OmniGAIA, which contains heterogeneous text, image, audio, and video inputs across Easy, Medium, and Hard tasks.Accuracy is the primary metric, with additional category- and difficulty-level analyses.
- Category-wise Comparison: 72.8% overall accuracy is achieved by Orchestra-o1-GPT-5, exceeding Gemini-3-Pro by 10.3% absolute accuracy and AOrchestra-GPT-5 by 32.8% absolute accuracy.Improvements are reported across most categories, including geography, technology, history, sport, art, movie, and science.
- Difficulty-level Comparison: 80.3%, 75.0%, and 56.4% accuracy are reached by Orchestra-o1-GPT-5 on easy, medium, and hard tasks, respectively.Its gains over AOrchestra-GPT-5 are 35.2%, 35.0%, and 24.3% absolute accuracy across those difficulty levels.
- Efficiency Analysis: 72.8% accuracy at a cost of 341.6 is achieved by Orchestra-o1 overall, compared with AOrchestra’s 40.0% accuracy at a cost of 565.7.The comparison covers easy, medium, hard, and overall splits, with the advantage attributed to two design choices.
- Post-training Ablation: 30.0% accuracy is achieved by DA-GRPO, compared with 27.7% for Vanilla GRPO and 28.6% for SFT.The result supports rewarding decision-level alignment, tool reasonableness, and strategic orchestration quality.
5. Conclusion
Orchestra-o1 is an omnimodal agent orchestration framework that separates high-level orchestration from low-level tool-augmented action execution. Its open-source Orchestra-o1-8B instantiation uses DA-GRPO to optimize strategic orchestration behaviors.
- Framework: Orchestra-o1 separates high-level orchestration from low-level tool-augmented action execution.The framework is introduced as an omnimodal agent orchestration system.
- Framework: The main agent dynamically decomposes complex tasks into dependency-aware sub-tasks and dispatches independent sub-tasks to specialized sub-agents in parallel.This enables coordinated execution across specialized sub-agents.
- Framework: Orchestra-o1 maintains compact context memory and decides when accumulated evidence is sufficient to produce the final answer.These mechanisms support evidence-based completion of orchestrated tasks.
- Training: Orchestra-o1-8B is an open-source instantiation of the main agent trained with DA-GRPO.DA-GRPO rewards format correctness, action validity, tool reasonableness, and decision quality.
A. Proof of Theorems · A.1. Proof of Proposition 1
The proof establishes that linear orchestration sums sub-task runtimes, whereas Orchestra-o1 executes conditionally independent ready sub-tasks in parallel and adds synchronization overhead. Parallel execution is no slower exactly when its resulting latency does not exceed the linear schedule, with equality requiring equal sub-task runtimes.
- A.1. Proof of Proposition 1: The proof begins by modeling linear orchestration latency as the sum of all sub-task runtimes.For a linear orchestrator, sub-tasks execute sequentially, so round latency is additive.
- A.1. Proof of Proposition 1: Orchestra-o1 launches all ready sub-tasks simultaneously when they are conditionally independent and share no mutable state.This parallel launch is followed by aggregation of the returned results by the main agent.
- A.1. Proof of Proposition 1: The parallel round latency is the maximum ready sub-task runtime plus synchronization time.The round completes after the slowest sub-task finishes and the main agent aggregates the results.
- A.1. Proof of Proposition 1: Parallel execution is no slower than linear execution exactly when its latency condition holds.Under this condition, the denominator of S_t is at most the numerator, implying S_t≥1.
- A.1. Proof of Proposition 1: The proof uses the fact that every sub-task runtime is no greater than the maximum runtime among the ready sub-tasks.This inequality supports the comparison between summed sequential latency and maximum-based parallel latency.
- A.1. Proof of Proposition 1: Equality requires the synchronization term to meet the corresponding equality condition in the latency comparison.The supplied proof states that equality additionally requires the runtime-sum condition described next.
- A.1. Proof of Proposition 1: The runtime-sum equality holds only when all sub-task runtimes are equal, completing the proposition’s proof.The proof identifies this as the necessary equality case for the comparison.
A.2. Proof of Proposition 3.2
The proof uses the chain rule of mutual information and the specialization assumption to establish strict information gain for orchestration over the native agent. Under log loss, this gain implies lower Bayes-optimal risk when specialized sub-agents provide additional task-relevant evidence and the orchestrator preserves it.
- Information gain: The chain rule decomposes I(Y; E_orch | q) across the modality-wise components E_1, ..., E_R.The proof begins with I(Y; E_orch | q) = I(Y; E_1, ..., E_R | q).
- Information gain: The component tuple is assumed to contain all task-relevant information retained by E_0, while specialization makes every conditional information term no smaller and at least one strictly larger.These assumptions yield a strict information advantage for the orchestrated representation over the native representation.
- Information gain: The resulting inequality establishes strict information gain: I(Y; E_orch | q) > I(Y; E_0 | q).Equation (36) provides the comparison, followed by the proof’s explicit conclusion of strict information gain.
- Log-loss consequence: Under log loss, the Bayes-optimal predictor is p(Y|q, E), and the minimum achievable expected loss equals conditional entropy.This connects the information comparison to the achievable log-loss risk.
- Log-loss consequence: Because H(Y|q) is fixed, strict information gain implies R_log(E_orch) < R_log(E_0), making orchestration theoretically preferable when specialized sub-agents provide more evidence and preserve it.The conclusion applies when the orchestrator preserves the additional task-relevant evidence supplied by specialized sub-agents.
B. More Experimental Details · B.1. Tool Configurations
Orchestra-o1 uses a unified tool ecosystem in which the main agent assigns tool subsets to specialized sub-agents rather than calling tools directly. The toolkit supports web research, page verification, computation, image, audio, and video analysis, with modality-specific delegation conditions.
- B.1. Tool Configurations: The main agent assigns each sub-task a subset of shared tools, leaving specialized sub-agents to perform perception or action execution.This keeps the main agent focused on high-level orchestration.
- B.1. Tool Configurations: Web Search retrieves external information for public knowledge, recent facts, entity disambiguation, and missing contextual information using the Serper API.The tool is used when the answer is not fully contained in the provided modality inputs.
- B.1. Tool Configurations: Page Visit extracts readable content from candidate web pages through the Jina Reader API when search snippets are insufficient for fact verification.It complements web search by enabling detailed source inspection.
- B.1. Tool Configurations: Code Execution runs Python in a controlled workspace for numerical computation, table processing, date conversion, unit conversion, and other deterministic operations.Computation can be delegated only after prerequisite values are extracted from media or retrieved from the web.
- B.1. Tool Configurations: Image Analysis handles visual recognition, scene understanding, chart interpretation, OCR-like inspection, and extraction of image-grounded evidence.The main agent is instructed to process relevant images before external search because images may contain task-specific information unavailable in text.
- B.1. Tool Configurations: Audio Analysis transcribes and analyzes speech clips, sound events, and other audio-grounded clues, writing its transcription and summary into the sub-task history.Later rounds can use the returned textual evidence.
- B.1. Tool Configurations: Video Analysis combines visual frames and, when appropriate, audio for temporal reasoning, event recognition, spoken-video understanding, and multimodal evidence extraction.Because video analysis can be expensive, the main agent should provide specific instructions instead of requesting an overly broad description.
B.2. System Prompt for Main Agent · B.3. System Prompt for Sub-agent · B.4. Prompt for Rubric Rewards
The prompts define phased, modality-aware orchestration through parallel independent subtasks, dependency-aware sequencing, adaptive model and tool selection, and budget-conscious retries. They also specify focused sub-agent behavior and a four-dimensional rubric for evaluating orchestration decisions.
- B.2. System Prompt for Main Agent: The Main Agent analyzes each question, plans multi-phase execution, and delegates subtasks while maximizing parallelism subject to dependencies.It identifies independent subtasks that can run immediately and dependent subtasks that must wait for later results.
- B.2. System Prompt for Main Agent: Each delegation round submits all currently runnable independent subtasks together, while later rounds use results from previous phases to unblock dependent work.Subtasks in the same round must not depend on one another, but may use prior findings through the context field.
- B.2. System Prompt for Main Agent: The Main Agent reviews subtask history, completes when results suffice, and retries failed or incomplete subtasks with adjusted instructions, context, or models when necessary.It selects the next parallel batch from subtasks newly unblocked by prior results.
- B.2. System Prompt for Main Agent: Limited attempts require maximizing parallelism per delegation, trusting verified results, and assigning cheaper models to simple tasks while reserving stronger models for complex reasoning or video analysis.Different models may be assigned to parallel subtasks according to complexity.
- B.3. System Prompt for Sub-agent: The Sub-agent prompt requires focused task completion, step-by-step reasoning, memory of key observations, immediate finishing when done, and adherence to image-analysis and repeated-search limits.When remaining_steps <= 5 or the same behavior pattern has been repeated five times, the sub-agent should finish with its best available partial results.
- B.3. System Prompt for Sub-agent: Sub-agent outputs must be only valid JSON using an exact available tool name or finish, with tool-specific parameters in the params object.The Main Agent’s final answer format is likewise constrained to precise, concise answers without explanations in the answer field.
- B.4. Prompt for Rubric Rewards: The rubric scores Main Agent decisions on FORMAT_CORRECT and ACTION_VALID from 0-1, and TOOL_REASONABLE and DECISION_QUALITY from 0-3.The evaluated actions are delegate_task and complete, with required JSON structure and parameters.
- B.4. Prompt for Rubric Rewards: DECISION_QUALITY prioritizes progress toward solving the task and accepts strategies that differ from the expert when they are reasonable, creative, or directly correct.A correct complete answer receives 3 regardless of the expert action, while reasonable delegation when completion was expected receives 1-2.
C. Limitations
Orchestra-o1’s orchestration improves modularity and efficiency but increases system complexity and engineering demands. Its training recipe optimizes the main agent while leaving sub-agent backends fixed, limiting end-to-end adaptation.
- System complexity: Orchestration requires maintaining sub-agent histories, tool schemas, backend configurations, cost accounting, and asynchronous execution.These additional components must be carefully engineered and monitored.
- Training limitations: The training recipe focuses on the main agent rather than jointly optimizing all sub-agents and tools.DA-GRPO improves decision-level orchestration, but sub-agent backends remain fixed during training.
- Training limitations: A more complete learning system could jointly adapt the main agent, sub-agent policies, and tool-selection behavior from end-to-end task outcomes.