PaperlayerView exact revision on arXiv

Source-linked AI summary

SciOrch: Learning to Orchestrate Expert LLMs for Solving Frontier Multimodal Scientific Reasoning Tasks

Jingru Guo, Xiangyuan Xue, Lian Zhang, Wanghan Xu, Siki Chen, Philip Torr, Wanli Ouyang, Lei Bai, Zhenfei Yin

arXiv:2606.15872v1cs.CL

TL;DR

Frontier multimodal scientific reasoning remains difficult for LLMs, while complementary strengths across models are underused by single-model approaches. SciOrch trains a lightweight orchestrator to delegate sub-problems among frontier models, achieving 56.66% average accuracy and outperforming both the strongest standalone model and multi-agent baseline.

  • Problem

    Frontier multimodal scientific reasoning remains unreliable for LLMs, while complementary strengths across models are not captured by single-model evaluation.

  • Method

    SciOrch trains a lightweight 8B VLM to orchestrate frontier models through sub-problem delegation, using step-level MCTS trajectories and offline policy optimization.

  • Results

    56.66% average accuracy on 240 questions is the best result on both evaluation slices, exceeding the strongest standalone model by 3.74% and the strongest multi-agent baseline by 3.33%.

  • Takeaways & Limitations

    SciOrch converts cross-model complementarity into higher scientific-reasoning accuracy while using substantially lower inference cost than typical multi-agent methods.

  • Takeaways & Limitations

    The modest 240-question evaluation cannot conclusively establish generalization beyond the evaluated distribution, especially for individual disciplines.

Abstract

from arXiv · show

Frontier scientific reasoning remains a major challenge for large language models (LLMs), where even the strongest commercial systems fall short of expert-level performance. A closer look at model behavior reveals substantial complementarity that single-model evaluation hides: different frontier models excel on different question types, and no single model captures the full picture. We present SciOrch, a framework that trains a lightweight 8B model to orchestrate frontier LLMs for scientific reasoning. The orchestrator decomposes each question, delegates sub-problems to selected commercial models through API calls, and synthesizes a final answer. Training such an orchestrator is fundamentally harder than conventional agentic RL: each action triggers an API call that is expensive in both dollar cost and latency, making standard online rollouts infeasible. We address this with MCTS-based approach, producing diverse orchestration trajectories, extracting per-node single-turn samples, and optimizing the orchestrator with GRPO-style training. On a 240-question test set spanning SGI-Reasoning and Scientists' First Exam, SciOrch reaches 56.66% average accuracy, outperforming the strongest single commercial model by 3.74% and the strongest multi-agent baseline by 3.33%. It also attains the best accuracy on both SGI and SFE with less than half the API cost of typical multi-agent methods.

1. Introduction

SciOrch addresses the limits of frontier multimodal scientific reasoning by exploiting complementary model and sub-task strengths through adaptive orchestration. It trains a lightweight 8B model to delegate sub-problems to commercial frontier models, achieving higher accuracy at lower API cost.

  • Motivation: 89% oracle coverage across all evaluated models exposes substantial complementarity that single-model accuracy hides.Three models already cover 74% of the test set, while pooling all evaluated models reaches an 89% oracle upper bound.
  • Motivation: Different models excel across disciplines and sub-tasks, motivating orchestration rather than reliance on a stronger individual model.Models may differ in figure parsing, domain-specific recall, and quantitative reasoning.
  • Method: SciOrch trains a lightweight 8B vision-language model to choose between delegating a sub-problem to a specified model and submitting a final answer.This frames long-context multimodal reasoning as a multi-step decision problem using commercial model APIs as tools.
  • Results: 56.66% average accuracy establishes SciOrch as the strongest system across the 240-question test set.It achieves 49.30% on SGI-Reasoning and 68.10% on Scientists’ First Exam.
  • Results: 3.74% improvement over Gemini-3-Pro and 3.33% over the strongest prompt-based multi-agent baseline demonstrate gains over both standalone and multi-agent systems.SciOrch costs only $10.42 over the full test set, lower than the best self-consistency baselines.
  • Method: MCTS-based training makes orchestration reinforcement learning tractable by generating diverse trajectories and decoupling offline updates from API latency.The approach addresses the expense and latency of API-triggering actions during training.

2. Related Work

Related work progresses from saturated broad scientific benchmarks to harder graduate-level and multimodal evaluations, while also exploring multi-model reasoning and domain-specialized scientific agents. Graduate-level benchmarks remain difficult for frontier reasoning models, which achieve only 50 to 60 percent accuracy.

  • Scientific reasoning benchmarks: Broad early scientific-reasoning benchmarks using multiple-choice questions are now largely saturated by frontier models.Later evaluations increased difficulty through graduate-level questions designed to resist shallow retrieval.
  • Multimodal scientific benchmarks: Multimodal scientific benchmarks test interpretation of figures, diagrams, and experimental data as vision-language models mature.The emerging frontier setting requires interpreting real experimental figures.
  • Multi-model reasoning: Multi-model and multi-path methods seek to surpass single-model limits through voting, debate, or structured collaboration.Prompt-based protocols are fixed, inference cost grows linearly with the number of paths, and accuracy gains are limited.
  • AI for Science: AI for Science systems build domain-specialized scientific agents, including predictive models for protein structure, materials, and chemistry.This research line also pairs LLMs with search or evolutionary procedures.

3. Method

SciOrch formulates multimodal scientific reasoning as selective, multi-step delegation by a lightweight vision-language orchestrator, and trains it with a three-phase MCTS procedure that generates diverse, informative trajectories without standard online-RL costs. It converts MCTS trajectories into per-node relative-return samples and optimizes the orchestrator with REINFORCE++ using token-level stabilization.

  • Orchestration: The lightweight vision-language orchestrator decomposes each question, delegates sub-problems to selected commercial models, and synthesizes their responses into a final answer.It routes each sub-question to one model rather than querying the entire pool in parallel.
  • Orchestration: Each JSON-formatted action either delegates a tagged sub-question to a model or submits the final answer, with malformed actions rejected during sampling.Delegation records the target model, instruction, and returned response in the interaction history.
  • MCTS training: MCTS uses diverse rollout, signal-driven expansion, and pruning to share common-prefix API costs while producing many state-action samples.Expansion prioritizes nodes with large sibling return gaps while penalizing depth; pruning retains at most K children per parent.
  • MCTS training: Phase 1 selects two children using different model-pool prompts and the pair with the largest sentence-embedding cosine distance among candidate actions.This targets diversity during both candidate generation and child selection.
  • Policy optimization: Per-node training replaces unstable trajectory-level gradients with samples whose return compares the selected child against the average explored siblings, then applies REINFORCE++.Token-level KL penalties, a fixed initial-checkpoint reference policy, batch advantage normalization, and a clipped PPO-style objective stabilize optimization.

4. Experiments

Experiments use a 176-question training set and 240-question test set combining SGI-Reasoning with SFE, and compare SciOrch against frontier, multi-agent, routing, and training-based baselines. SciOrch achieves the highest accuracy on both test slices while using substantially lower API cost than typical multi-agent methods.

  • Dataset: 176 training and 240 test questions combine SGI-Reasoning with SFE, with the test set spanning all nine SGI disciplines.The training set contains 145 SGI and 31 SFE questions; the test set contains 146 SGI and 94 SFE questions.
  • Baselines and evaluation: SciOrch is evaluated against frontier models, self-consistency, multi-agent debate, P2L routing, SFT, and GRPO on the 240-question test set.Evaluation reports exact-match accuracy on SGI, SFE, and the combined test set, together with API cost.
  • Baseline results: 27.52% average accuracy is achieved by vanilla Qwen3-VL-8B, while SFT and GRPO improve over it by only 3.72% and 1.58%, respectively.Post-trained Qwen3-VL-8B reaches at most 26.00% on SGI and 44.68% on SFE.
  • Baseline results: 67.02% SFE accuracy makes GPT-5.4 the leading frontier model on SFE, but it trails Gemini-3-Pro on SGI, showing complementary model strengths.Self-consistency improves GPT-5.4 by 0.42%, whereas multi-agent debate degrades it by 2.08% and increases API spend by 5 times.
  • SciOrch results: 56.66% average accuracy makes SciOrch the strongest method, with 49.30% on SGI and 68.10% on SFE, while spending $10.42 across the test set.SciOrch uses an 8B orchestrator to dynamically dispatch among frontier models and spends less than half the cost of self-consistency.

5. Conclusion

SciOrch trains a lightweight model to orchestrate expert models for challenging multimodal scientific reasoning. It frames reasoning as multi-step API-call routing to exploit cross-model complementarity and uses step-level MCTS to make RL-based learning tractable under API costs.

  • Framework: SciOrch trains a lightweight model to orchestrate expert models for challenging multimodal scientific reasoning.The framework targets scientific reasoning tasks that require coordinating multiple expert models.
  • Problem formulation: Each action is an API call, casting frontier scientific reasoning as a multi-step routing problem.This formulation exposes the task as a sequence of model-selection decisions rather than single-model evaluation.
  • Model complementarity: SciOrch exploits cross-model complementarity that single-model evaluation hides.Different expert models can contribute complementary capabilities to the overall reasoning process.
  • Training: Step-level MCTS-based learning makes the RL process tractable under per-action API costs.The method addresses the expense of API calls during multi-step reinforcement-learning training.

6. Limitations

SciOrch’s evaluation is limited by a modest 240-question test set across nine scientific disciplines and by restriction to multiple-choice questions. These constraints prevent conclusive claims about generalization and leave a gap from open-ended scientific practice.

  • Evaluation scale: 240 questions across nine scientific disciplines make the evaluation modest, so generalization beyond the evaluated distribution remains inconclusive.Individual per-discipline cells are small, limiting the conclusiveness of the findings.
  • Multiple-choice format: Multiple-choice training and evaluation differ from open-ended scientific practice, which requires hypotheses, free-form derivations, and arguments judged for partial correctness or semantic equivalence.Exact-match correctness supplies a clean verifiable reward in the evaluated setting.

7. Ethical Considerations

SciOrch makes competitive scientific reasoning more accessible by lowering API costs, but this accessibility may enable misuse. Because it routes and aggregates potentially hallucinated commercial-model responses without fact-checking, its coherent answers can still be factually wrong and require verification.

  • Potential misuse: Lowering the cost of competitive scientific answers improves accessibility for under-resourced research groups but could also be exploited by bad actors.SciOrch uses a lightweight 8B orchestrator to route across frontier commercial APIs.
  • Model hallucination: SciOrch does not verify individual API responses, instead routing and aggregating outputs that may hallucinate on frontier scientific questions.Ground truth is scarce in pretraining data for these questions.
  • Model hallucination: The system can confidently produce well-formatted, internally coherent answers that are factually wrong and harder to detect than errors from a weaker single model.SciOrch’s outputs should be treated as candidate reasoning requiring verification, not substitutes for primary literature.

A. The Use of LLMs

LLMs were used exclusively to refine the manuscript, while the authors reviewed and validated all LLM-generated or refined content. They were not used for conceptual development, data collection, code implementation, experimental design, or results interpretation.

  • A. The Use of LLMs: LLMs were used exclusively for manuscript refinement, including correcting typographical and grammatical errors and improving linguistic expression.The authors used LLMs to correct typographical errors, fix grammatical problems, and improve linguistic expression.
  • A. The Use of LLMs: The authors carefully reviewed and validated all content generated or refined with LLM assistance.
  • A. The Use of LLMs: LLMs were not used for conceptual development, data collection, code implementation, experimental design, or interpretation of results.

B. Model Documentation

SciOrch uses Qwen3-VL-8B-Instruct to make structured delegation or final-answer decisions while conditioning on the multimodal question, interaction history, and commercial model pool. Its expert pool comprises GPT-5.4, Gemini-3-Pro, and Claude-Sonnet-4.5, with P2L-1.5B providing a one-shot routing baseline.

  • Orchestrator backbone: SciOrch’s policy backbone is Qwen3-VL-8B-Instruct, which either delegates a sub-problem to a commercial model or submits a final answer.At each decision step, the policy observes the original multimodal question, accumulated interaction history, and available commercial model pool.
  • Expert models: The three commercial expert models are GPT-5.4, Gemini-3-Pro, and Claude-Sonnet-4.5.SciOrch delegates selected sub-problems to these models through API calls, appending each response to the interaction history before the next decision step.
  • Routing baseline: P2L-1.5B is the routing baseline, selecting one model from SciOrch’s three-model commercial pool using the question stem and OCR-extracted figure text.This baseline tests whether one-shot text-based routing can match SciOrch’s multi-step delegation.

C. Dataset Documentation

The paper documents SGI-Reasoning as its primary benchmark and SFE as an augmentation source, with both providing multimodal scientific multiple-choice questions for training and held-out evaluation.

  • SGI-Reasoning: SGI-Reasoning contains 291 multimodal scientific reasoning questions across nine disciplines, with 145 training and 146 held-out test questions.Each example requires reasoning over scientific visual content and selecting a multiple-choice answer.
  • SGI-Reasoning: SGI-Reasoning receives no additional quality gating because it is the target evaluation benchmark.This gating decision applies specifically to SGI-Reasoning.
  • Scientists’ First Exam: SFE contains 830 bilingual visual question-answering items across five scientific disciplines, yielding 281 English multiple-choice candidates before quality gating.The disciplines are Astronomy, Chemistry, Earth Science, Life Science, and Materials Science.
  • Scientists’ First Exam: 31 training and 94 test questions remain in the SFE subset after retaining examples that at least one of GPT-5.4, GPT-4o, Gemini-3-Pro, Gemini-2.5-Pro, and Claude-Sonnet-4.5 answers correctly.Quality gating uses these five frontier models.

D. Prompt Templates

The prompt templates define a strict orchestrator–specialist workflow: the Orchestrator delegates focused uncertainties, routes them by task–pool fit, and submits one final option only after evidence-based gating. They also constrain outputs to concise JSON with explicit reasoning, cross-checks, and specialist confidence.

  • Orchestrator workflow: The Orchestrator routes focused sub-questions to specialists, then decides and submits one final option rather than solving the multimodal question itself.Each turn requires exactly one JSON object containing either a focused delegation or a boxed answer choice.
  • Routing: Routing matches task type to pool: vision reasoning uses Vision, hard calculation or science uses Frontier, medium tasks use Strong, and easy lookup or one-step arithmetic uses Lightweight.The template explicitly prioritizes task–pool fit over model reputation.
  • Delegation rules: Each sub-question isolates one missing fact, maps its answer to the options, avoids leaking guesses, and must not repeat an already answered sub-question.These rules keep delegation focused and prevent bundled unknowns or contamination from prior conclusions.
  • Submit gating: Submit gating requires delegate-supported decisive evidence; high-confidence delegate conclusions additionally need an independent cross-check or a justification for omitting one.The reasoning template also requires evidence, a potential belief-flipping counter-check, and further delegation when gating fails.
  • Specialist responses: Specialists answer only the focused question, reason independently from the original context and images, cite observed evidence when relevant, and provide confidence in [0, 1].Their output is structured as a direct answer, brief evidence-grounded explanation, and numeric confidence.
Loading 2606.15872v1…