PaperlayerView exact revision on arXiv

Source-linked AI summary

Maestro: Reinforcement Learning to Orchestrate Hierarchical Model-Skill Ensembles

Jinyang Wu, Guocheng Zhai, Ruihan Jin, Yuhao Shen, Zhengxi Lu, Fan Zhang, Haoran Luo, Zheng Lian, Zhengqi Wen, Jianhua Tao

arXiv:2605.22177v1cs.LGcs.CL

TL;DR

Existing multimodal-agent frameworks struggle to coordinate heterogeneous models and skills across tasks. Maestro trains an RL orchestrator to compose model-skill ensembles, achieving 70.1% average accuracy while generalizing to unseen models and skills without retraining.

  • Problem

    Existing frameworks inadequately coordinate heterogeneous models and skills, leaving their synergistic interdependencies insufficiently understood across multimodal tasks.

  • Method

    Maestro formulates orchestration as sequential decision-making over a hierarchical model-skill registry, using compositional actions to select experts, invoke skills, and terminate.

  • Results

    70.1% average accuracy across ten multimodal benchmarks exceeds frontier models, while the learned routing policy generalizes to unseen models and skills without retraining.

  • Takeaways & Limitations

    The results suggest intelligent model-skill orchestration is a high-leverage alternative to scaling model size while preserving low inference latency.

  • Takeaways & Limitations

    Routing can fail on tasks spanning skill categories because the policy may commit to one skill without reconsidering within the allotted turns.

Abstract

from arXiv · show

The proliferation of large language models (LLMs) and modular skills has endowed autonomous agents with increasingly powerful capabilities. Existing frameworks typically rely on monolithic LLMs and fixed logic to interface with these skills. This gives rise to a critical bottleneck: different LLMs offer distinct advantages across diverse domains, yet current frameworks fail to exploit the complementary strengths of models and skills, thereby limiting their performance on downstream tasks. In this paper, we present Maestro (Multimodal Agent for Expert-Skill Targeted Reinforced Orchestration), a Reinforcement Learning (RL)-driven orchestration framework that reframes heterogeneous multimodal tasks as a sequential decision-making process over a hierarchical model-skill registry. Rather than consolidating all knowledge into a single model, Maestro trains a lightweight policy to dynamically compose ensembles of frozen expert models and a two-tier skill library, deciding at each step whether to invoke an external expert, which model-skill pair to select, and when to terminate. The policy is optimized via outcome-based RL, requiring no step-level supervision. We evaluate Maestro across ten representative multimodal benchmarks spanning mathematical reasoning, chart understanding, high-resolution perception, and domain-specific analysis. With only a 4B orchestrator, Maestro achieves an average accuracy of 70.1%, surpassing both GPT-5 (69.3%) and Gemini-2.5-Pro (68.7%). Crucially, the learned coordination policy generalizes to unseen models and skills without retraining: augmenting the registry with out-of-domain experts yields a 59.5% average on four challenging benchmarks, outperforming all closed-source baselines. Maestro further maintains high computational efficiency with low latency. The source code is available at https://github.com/jinyangwu/Maestro.

1 Introduction

MAESTRO addresses the heterogeneity of multimodal tasks by replacing static or monolithic dispatch with an RL-trained orchestrator that coordinates external expert models and hierarchical skills. It frames coordination as sequential decision-making and reports strong performance with a 4B orchestrator, including generalization to unseen models and skills without retraining.

  • 1 Introduction: Multimodal tasks are inherently heterogeneous, requiring different inductive biases and expertise for geometric proofs, medical reports, and high-resolution satellite imagery.
  • 1 Introduction: MAESTRO trains a high-level orchestrator to coordinate heterogeneous external capabilities through sequential decision-making over a hierarchical model-skill registry.
  • 1 Introduction: MAESTRO is evaluated on 10+ multimodal benchmarks covering mathematical reasoning, chart understanding, medical analysis, high-resolution perception, and embodied question answering.
  • 1 Introduction: MAESTRO formalizes model-skill coordination as a finite-horizon POMDP and optimizes routing with outcome-based reinforcement learning without step-level supervision.
  • 1 Introduction: 70.1% performance is achieved by the 4B orchestrator, exceeding frontier models such as GPT-5 while supporting plug-and-play generalization to unseen models and skills without retraining.

2 Related Works

Prior work has progressed toward modular LLM agents with reusable skills, reinforcement-learning-based optimization, and multimodal collaboration mechanisms. These strands motivate systems that combine structured skills, learned policies, and visual-language tools for complex tasks.

  • LLM Agent and Skills: LLM agents have evolved from prompt-based interaction to modular systems that support autonomous reasoning and tool invocation.Early frameworks used fixed reasoning traces or predefined action spaces, while newer work represents task-specific procedures as reusable skills.
  • Reinforcement Learning for Agent Optimization: Reinforcement learning aligns LLM agents with complex objectives by enabling trial-and-error exploration beyond static supervised demonstrations.Recent work also investigates recursive RL for co-evolving agent policies and skill banks.
  • Multimodal LLM Collaboration: Multimodal agents integrate visual perception with linguistic reasoning through specialized vision-language models, executable vision tools, structured action spaces, and modular tools.Other approaches address high-density visual-task challenges using optical self-compression and hierarchical memory.

3 Method

MAESTRO formulates multimodal reasoning as RL-driven orchestration over heterogeneous models and hierarchical skills. Its policy composes model-skill actions, integrates observations iteratively, and optimizes task success with structurally constrained rewards.

  • Framework overview: MAESTRO dynamically composes model and skill ensembles through an RL-driven policy for adaptive, multi-step reasoning in complex multimodal environments.The framework is described as non-invasive and jointly orchestrates models and skills rather than relying on static execution.
  • Heterogeneous registries: The framework maintains a dual registry of frozen expert LLMs and hierarchical skills, learning which model-skill ensemble best fits each reasoning step.Candidate models provide distinct inductive biases, including visual perception, mathematical reasoning, and code generation.
  • Action space: 55?
  • Action space: The compositional search action is a triplet containing a selected expert model, functional skill, and semantic query, while termination returns the final answer.Search actions are serialized as <search> Model@@Skill: Query </search>, and final responses use <answer> tags.
  • Context transition: Observations from external invocations are injected into the context, recursively refining the orchestrator’s belief state for subsequent decisions.The transition updates context by concatenating prior context, action, and standardized observation feedback.
  • Policy optimization: The policy interleaves latent reasoning with dynamic external invocations and is optimized with GRPO using group-relative advantages over sampled trajectories.The objective maximizes expected total trajectory reward while addressing sparse rewards in long-horizon reasoning.

4 Experiments

Across ten multimodal benchmarks, MAESTRO’s lightweight orchestrator achieves strong in-domain and out-of-domain performance, generalizes to augmented model-skill registries without retraining, and maintains low latency and token usage. Ablations further show that hierarchical skills and complementary reward components materially improve routing and task performance.

  • In-Domain Performance: 70.1% average accuracy with a 4B orchestrator surpasses GPT-5 (69.3%) and Gemini-2.5-Pro (68.7%) in-domain.On Geometry3K, MAESTRO reaches 77.4%, compared with GPT-4o (34.1%) and GLM-4.6V (60.4%).
  • Out-of-Domain Generalization: 88.0% on VStar and 79.6% on HRBench-4K outperform DeepEyes (85.6% on VStar) and Thyme (77.0% on HRB-4K).These results cover unseen high-resolution benchmarks and support generalization beyond task-specific memorization.
  • Plug-and-Play Generalization: 52.7% average accuracy from the unaugmented default registry outperforms all “Think with Images” baselines while remaining comparable to strong closed-source models.The default configuration uses five expert models and five Level-1 skills; MAESTRO* adds two experts and four Level-1 skills without retraining.
  • Efficiency: 2.88s average latency and 648.20 tokens are the lowest reported by MAESTRO across the ten benchmarks.Hierarchical routing selects a skill-expert pair directly, avoiding redundant image zooming and repetitive prompting used by iterative methods.
  • Scaling with Skill Pool Size: 60.7% to 66.5% accuracy (+5.8%) results from expanding the skill pool from N=2 to N=8.VStar improves by 7.4% (80.6%→88.0%) and Slake by 7.8% (57.9%→65.7%).
  • Reward Design Ablation: −13.1% without rfmt and −8.8% without rans show that structural constraints and outcome supervision provide complementary benefits.rfmt preserves reliable multi-turn communication, while rans primarily drives routing quality and task performance.

5 Conclusion

MAESTRO frames heterogeneous model-skill orchestration as sequential decision-making while decoupling coordination logic from model parameters. Across ten multimodal benchmarks, it outperforms leading closed-source models, discovers model-skill synergies, generalizes to out-of-domain settings, and maintains low inference latency.

  • Conclusion: MAESTRO reframes heterogeneous model-skill orchestration as a sequential decision-making problem, decoupling coordination logic from underlying model parameters.This separation places orchestration decisions outside the underlying model weights.
  • Conclusion: Across ten multimodal benchmarks, MAESTRO outperforms leading closed-source models while maintaining low inference latency.The conclusion links benchmark performance with computational efficiency.
  • Conclusion: MAESTRO uncovers non-trivial model-skill synergies and generalizes its routing logic to out-of-domain settings.These findings indicate that the learned orchestration behavior extends beyond the evaluated settings.

A Theoretical Analysis … D Detailed Experimental Details

MAESTRO’s analysis attributes its performance and efficiency to hierarchical action-space compression, joint model-skill compatibility, and extensibility constrained by routing regret. Algorithmic and taxonomy details implement this design through RL orchestration, two-tier skills, and non-invasive routing across default and OOD domains.

  • A. Theoretical Analysis: MAESTRO compresses routing from all model-skill pairs to coarse Level-1 skills, reducing redundant calls and explaining lower latency, token consumption, and ablation-supported performance gains.Level-2 routing remains skill-local, where domain-specific sub-routines provide execution depth.
  • B Algorithmic Details: At each step, the orchestrator conditions on context to think, invoke a selected model-skill pair, or answer, while training samples trajectory groups and updates the policy with GRPO.Rewards combine answer and formatting terms, with group-relative advantages used for optimization.
  • A.3 Model-Skill Compatibility: Joint routing captures positive model-skill compatibility that independent model or skill retrieval would miss, matching ablations showing both pools matter and model removal hurts more on reasoning tasks.The compatibility term contributes value beyond selecting a strong model and relevant skill separately.
  • A.4 Extensibility: Registry expansion cannot reduce the oracle upper bound, but practical gains require oracle improvements to exceed additional routing regret.Semantic descriptions help identify new entries, yet extensibility is not guaranteed unconditionally.
  • C Detailed Hierarchical Skill Taxonomy: The library contains 9 Level-1 and 24 Level-2 skills, with S1–S5 used by default and S6–S9 added only for extended OOD evaluation.The hierarchy minimizes the 4B orchestrator’s action space while preserving domain-specific precision.
  • C.1 Default Skill Configuration (S1–S5): Default skills cover geometry, charts, counting, fine-grained perception, and scientific imagery through specialized OCR, detection, zooming, captioning, and reasoning routines.Examples include geometric structure extraction, chart-type routing, spatially indexed counting, micro-structure inspection, and multimodal diagram interpretation.
  • C.2 Extended Skill Configuration (S6–S9): Extended skills address embodied reasoning, OCR, diagram reasoning, and Python generation, including trajectory analysis, text extraction, topology judgments, and executable-code repair.These capabilities are plugged into the existing registry without orchestrator retraining.
  • C.3 Hierarchical Execution Protocol: Routing is non-invasive and two-stage: the 4B policy selects a Level-1 skill and expert model, then keyword activation or expert classification invokes the Level-2 sub-routine.This separates strategic resource allocation from domain-specific execution.

D.1 Training Data Statistics

The MAESTRO orchestrator is trained on a 9,200-sample mixture spanning seven datasets and five task domains, excluding extended out-of-domain benchmarks. Training and evaluation are separated by using official training splits for optimization and held-out test splits for evaluation, with no sample-level overlap.

  • Training mixture: 9,200 samples span seven datasets covering five task domains in the default skill configuration.Table 4 summarizes the training-mixture composition used to optimize the MAESTRO orchestrator.
  • Training mixture: Extended OOD benchmarks ERQA, OCRBench, VlmsAreBlind, and Humaneval_V are excluded from training.This preserves a clean separation between the training distribution and out-of-domain evaluation.
  • Data separation: Training uses official dataset training splits, while evaluation uses corresponding held-out test splits with no sample-level overlap.This separation applies even when benchmark names appear in both training and evaluation splits, including ChartQA, Geometry3K, TallyQA, Slake, MicroVQA, and MSEarthMCQ.

D.2 Evaluation Benchmark Statistics … E.3 Test-Time Scaling

MAESTRO is evaluated across diverse in-domain and out-of-domain multimodal benchmarks against broad model, workflow, and tool-use baselines. Its analysis shows that expanding skills and test-time computation improves accuracy, while focused expert invocation supports efficiency.

  • D.2 Evaluation Benchmark Statistics: The evaluation spans chart, geometric and mathematical reasoning, scientific and medical VQA, counting, high-resolution perception, embodied reasoning, OCR, synthetic diagrams, and agentic tool use.Table 5 distinguishes ID benchmarks, OOD benchmarks for zero-shot generalization, and OOD∗ benchmarks evaluated with an augmented registry.
  • D.3 Baselines: The baselines include proprietary and open-source multimodal models, direct answering, an untrained workflow-enabled model, and specialized visual reasoning or tool-use systems.These comparisons isolate raw backbone ability, workflow effects, model-selection gains, and performance against reinforcement-learning and tool-oriented approaches.
  • D.4 Implementation Details: MAESTRO is trained with GRPO from Qwen3-VL-4B-Thinking on four 80GB A100 GPUs for 3 days and 11 hours, using 380 update steps and eight trajectories per prompt.The implementation uses FSDP, AdamW with learning rate 1 × 10−6, and asynchronous vLLM rollouts.
  • E.1 Efficiency and Scalability Analysis: Reported token consumption and end-to-end latency include the 4B orchestrator, every 4–9B expert invocation, and associated skills.The orchestrator decomposes requests itself, triggering narrowly scoped experts only when specific capabilities are needed, which explains lower cost than single-model alternatives despite multiple experts.
  • E.2 Scaling with Skill Pool Size: Expanding the skill pool from N=2 to N=8 raises average accuracy from 60.7% to 66.5% (+5.8%).The configurations add counting, science, perception, embodied scene, OCR, and Python code-generation skills.
  • E.2 Scaling with Skill Pool Size: Specialized benchmarks gain 9.3% on ERQA and 10.1% on OCRBench when the skill pool expands from N=2 to N=8.ERQA increases from 43.0% to 52.3%, while OCRBench increases from 74.9% to 85.0%.
  • E More Results and Analysis: Average latency increases from 3.27s to 4.03s as skills expand from N=2 to N=8, remaining sub-linear relative to the performance gains.The result suggests the RL-driven orchestrator learns efficient coordination as the skill pool grows.
  • E.3 Test-Time Scaling: Moving from pass@1 to sc@16 steadily improves average accuracy, including gains of 2.7% on MathVision and 4.4% on TallyQA.Self-Consistency samples multiple reasoning trajectories and selects answers by majority vote, exploiting the orchestrator’s distribution of coordination paths.

E.4 Additional Discussion

MAESTRO’s RL policy converges toward stable, confident orchestration and delivers strong performance and efficiency across in-domain and out-of-domain evaluations. RL substantially improves routing quality, while format reward is more important than outcome reward for effective tool use.

  • Overall Performance and Efficiency: MAESTRO achieves the best overall performance and efficiency across systematically evaluated in-domain and out-of-domain datasets.The reported comparisons use absolute differences versus the strongest baseline for each metric.
  • Performance Upper Bound: +17.4% gain in pass@1 average accuracy, from 52.7% to 70.1%, shows RL training is the primary driver of routing performance.Pass@k evaluation also reveals meaningful headroom at higher sampling budgets.
  • Detailed Ablation Results: Removing format reward causes invalid action outputs that block tool execution and largely reduce the system to direct answering by the 4B backbone.The failure mode includes sequentially emitting both <search>...</search> and <answer>...</answer> when only one action should be selected.
  • Detailed Ablation Results: Removing outcome reward preserves well-formatted tool calls, but selected calls become less consistently optimal; its accuracy degradation is smaller than removing format reward.Even imperfect tool use generally provides stronger support than the 4B backbone alone.
  • RL Training Convergence: Mean reward rises steadily to a stable plateau while policy entropy declines smoothly, indicating increasingly confident orchestration.The format-reward variant converges to a higher reward level.

E.5 Statistical Significance Analysis

A Wilcoxon signed-rank analysis finds that MAESTRO significantly outperforms VTOOL-R1 across all ten benchmarks and on out-of-domain benchmarks. These results support MAESTRO’s robustness and generalization capability.

  • Statistical significance across benchmarks: p = 9.7 × 10−4 (p < 0.05) rejects H0 of no significant difference between MAESTRO and VTOOL-R1 across all ten benchmarks.The comparison uses the Wilcoxon signed-rank test and pairs MAESTRO with the strongest “Think with Images” baseline, VTOOL-R1.
  • Out-of-domain significance: p = 6.1 × 10−3 (p < 0.05) indicates a significant difference on out-of-domain benchmarks.The authors interpret this result as validating MAESTRO’s robustness and generalization capability.

F Case Study · G Additional Discussion

MAESTRO’s case studies show task-aware selection of model–skill pairs for diverse multimodal questions, including fine-grained perception and medical imaging. They also illustrate zero-shot extensibility to newly added out-of-domain experts and skills.

  • F Case Study: MAESTRO orchestrates expert models and hierarchical skills across diverse multimodal tasks in both in-domain and out-of-domain settings.Representative examples are presented in Figures 10–14.
  • F Case Study: Task semantics guide MAESTRO’s selection of the appropriate model–skill combination.Figures 10 and 12 demonstrate this task-aware orchestration behavior.
  • F Case Study: In VStar, MAESTRO combines GLM-4.6V-Flash with Perception_Problem_Solver to zoom into the relevant image region and identify the scarf color as red.The system returns the correct answer (B) for the fine-grained color perception question.
  • F Case Study: Figure 12 shows MAESTRO applying task-aware orchestration to a chest X-ray question in Slake.The supplied passage introduces this example but does not include its completed model–skill sequence or answer.
  • F Case Study: Table 9 compares performance upper bounds at pass@16 versus pass@1 with and without RL training.The supplied passage provides the table’s comparison scope but no numerical entries.
  • F Case Study: Table 10 reports that the model pool and skill library each contribute independently, while their combination is essential for peak performance.This is presented as an ablation study of MAESTRO components.
  • F Case Study: MAESTRO extends zero-shot to newly added experts and skills on the ERQA benchmark.Figure 13 presents an embodied-scene question involving robot arm actions.
  • F Case Study: On ERQA, MAESTRO jointly invokes Qwen3.5-9B and Embodied_Scene_Problem_Solver, neither of which was present during training.Together, they analyze the gripper, open jar, and nearby lid to conclude that the robot is closing.

G.1 Limitation and Failure Case Analysis … G.8 Future Work

MAESTRO’s limitations center on manual skill construction, boundary-case failures, and limited adaptation, while its open-source, lightweight ensemble remains computationally efficient. The discussion positions joint model-skill RL routing as distinct from retrieval and outlines future work on self-evolving registries, online adaptation, richer correction, modalities, and theory.

  • G.1 Limitation and Failure Case Analysis: Boundary tasks can trigger commitment to one Level-1 skill without reconsideration, while the skill library still depends on human-authored descriptions that require manual scaling effort.A representative failure involves chart questions requiring domain-specific scientific knowledge.
  • G.2 Clarification on Model Scale and Computational Cost: MAESTRO uses five open-source expert models and a 4B orchestrator, with all registry components below 10B parameters and expert models invoked selectively.The framework is described as computationally efficient and accessible compared with frontier closed-source models.
  • G.4 Sensitivity to Skill Descriptions: Discriminative descriptions based on input type and expected output format produce more consistent orchestration decisions than descriptions framed as abstract capabilities.The orchestrator remains robust to minor paraphrasing.
  • G.5 Detailed Comparison with Concurrent Works: MAESTRO differs from concurrent methods by learning outcome-based RL policies over joint model-skill assignments, rather than routing skills alone or using independent retrieval.The model pool ablation drops 12.1%, compared with 2.7% when removing the skill pool.
  • G.6 Why Reinforcement Learning over Retrieval-Based Routing: RL captures cross-modal model-skill synergies and supports multi-turn revision, whereas embedding retrieval independently selects components in a single step.The comparison identifies these as two key advantages of RL-based dispatching.
  • G.7 Emergent Behavior During Training: After approximately 50 steps, the policy shifts from multiple searches and malformed sequences toward well-formed, single-call trajectories for straightforward tasks.Early training is characterized by high entropy and volatile rewards.
  • G.8 Future Work: Future work includes self-evolving skill registries, online policy adaptation, explicit multi-turn self-correction, broader modalities and action types, and theoretical analysis of orchestration learning.These directions address fixed manual libraries, offline training, limited revision, modality coverage, and missing formal guarantees.

H Broader Impact

Maestro’s collaborative orchestration paradigm coordinates heterogeneous expert models and modular skills through a lightweight policy rather than consolidating capabilities in one large model. This design may broaden access to specialized AI while introducing risks from scalable misuse and inherited harmful behaviors.

  • Positive societal implications: Maestro coordinates heterogeneous expert models and modular skills with a lightweight policy instead of consolidating all capabilities into a single large model.This decouples task routing from model parameters.
  • Positive societal implications: Decoupling task routing from model parameters lowers the barrier for domain experts in medicine, science, and engineering to integrate purpose-built models.
  • Risks: Coordinating specialized models could be misused to build automated pipelines for disinformation generation, targeted content manipulation, or privacy-violating information aggregation at scale.
  • Risks: Because Maestro leaves all expert models frozen, harmful behaviors present in underlying models remain a risk.
Loading 2605.22177v1…