PaperlayerView exact revision on arXiv

Source-linked AI summary

MedRouter: Demystifying Knowledge Differences Across Medical LLMs for Routing-Based Reasoning

Lang Cao, Binghang Lu, Yuhao Shen, Yue Guo

arXiv:2609.33119v1cs.CLcs.AI

TL;DR

Medical LLMs can have complementary question-level strengths, while existing routing often prioritizes answer quality and inference cost rather than exploiting that complementarity. MedRouter routes questions to specialists and uses SCALE to train the Router with correctness supervision followed by generator-level reinforcement learning. Across eight medical QA benchmarks, it achieves the highest mean accuracy among compared methods with both tested generators.

  • Problem

    Existing routing methods often balance answer quality and inference cost, while how to exploit differences in medical specialists’ competence for reasoning remains open.

    Source: §1 · p. 1

  • Method

    MedRouter uses an embedding-based multi-label Router to select specialists for a generator, and SCALE trains the Router with specialist correctness supervision followed by reinforcement learning from generator outcomes.

    Source: §1 · p. 2

  • Results

    Across eight benchmarks, MedRouter achieves the highest mean accuracy among compared methods with both generators; with Gemma-4-E2B it reaches 46.1%, an 8.2% relative improvement over Encoder Router (42.6%).

    Source: §1 · p. 3, §4.2 · p. 7

  • Takeaways & Limitations

    Specialists have complementary question-level strengths, and a per-question oracle reaches 64.3% accuracy versus 43.6% for best-per-dataset selection.

    Source: §1 · pp. 1–2

    MedRouter does not lead on every multimodal benchmark, and its three-specialist vision-capable pool and frozen generator may limit image-question accuracy.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

Medical question answering spans diverse specialties and modalities, and individual medical large language models (LLMs) exhibit distinct strengths across tasks and domains. This heterogeneity suggests that combining specialists may enable broader coverage of medical questions than relying on any single model. However, existing LLM routing methods primarily seek to balance answer quality and inference cost, leaving open how to exploit differences in specialist competence to improve medical reasoning. In this paper, we introduce MedRouter, an agentic system that uses an embedding-based multi-label router to select and query specialist LLMs, then passes their responses to a generator to produce the final answer. We further propose SCALE (Specialist Competence-Aware Learning), a two-stage training framework that first trains the Router with specialist correctness supervision and then optimizes its selections through reinforcement learning. The second stage uses a Performance Gain Reward (PGR) that measures how specialist information affects the generator's answer correctness relative to answering without that information. Experiments on eight text-based and multimodal medical QA benchmarks show that MedRouter outperforms the strongest routing baseline by 8% in average accuracy. Our analysis of specialist outputs further reveals distinct strengths and complementary question-level coverage, motivating learned routing to combine these capabilities for more comprehensive medical reasoning.

1 INTRODUCTION

Medical specialists differ in which questions they answer correctly, so selecting by overall or dataset-level strength can miss useful capabilities. MedRouter routes questions to specialists and trains selections using specialist correctness and generator-level feedback.

1 INTRODUCTION

  • A per-question oracle reaches 64.3% accuracy, 20.7 percentage points above the 43.6% best-per-dataset selection, revealing question-level routing headroom.The analysis spans six specialists across four text-based and four multimodal QA benchmarks; multimodal evaluation uses three vision-capable specialists.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Existing routing work often balances answer quality and inference cost, while the paper focuses on using complementary specialist knowledge for medical reasoning.The introduction notes that systems using one LLM for both selection and generation can make it difficult to optimize routing to exploit specialist information.Source: §1 · p. 2
  • MedRouter separates multi-label specialist selection from answer generation, while SCALE trains the Router first with correctness supervision and then with reinforcement learning from generator-level feedback.The Performance Gain Reward compares generator answer correctness with and without selected specialist responses; only the Router is updated.Source: §1 · p. 2
  • 45.9% and 46.1% mean accuracy with Qwen3.5-2B and Gemma-4-E2B, respectively, were the highest among compared methods with both generators.Both exceed Best-in-Pool’s 43.6% benchmark average.Source: §1 · p. 3
  • The paper characterizes specialist complementarity, introduces MedRouter and SCALE, and evaluates routing across eight text-based and multimodal medical QA benchmarks.Its contributions also include out-of-distribution tests and ablations of the training stages.Source: §1 · p. 3

2 RELATED WORK

Prior work addresses model selection, medical reasoning, and model collaboration, but the paper identifies limited explicit learning of complementary medical knowledge and its integration for reasoning.

2 RELATED WORK

  • LLM selection methods commonly target response-quality and inference-efficiency trade-offs, including budget-aware routing, suitability prediction, cascades, and reinforcement-learning policies.The paper distinguishes its motivation as exploiting complementary medical knowledge rather than primarily balancing quality and cost.Source: §2 · p. 3
  • Medical reasoning research includes specialized corpora, ensembles, confidence-driven helper routing, multi-agent collaboration, and reinforcement-learning coordination of triage and specialist agents.The cited examples address diagnosis, clinical prediction, evidence synthesis, and medical QA.Source: §2 · pp. 3–4
  • Figure 2 depicts MedRouter’s embedding-based Router selecting from a frozen specialist pool, with a frozen generator producing the final answer from selected specialist information.Only the Router is trained, first with specialist correctness supervision and then with reinforcement learning using a reward comparing correctness with and without specialist information.Source: Figure 2 · p. 4
  • The paper identifies a gap in explicitly learning to exploit complementary knowledge across existing medical LLMs and integrate it effectively for medical reasoning.Prior approaches are described as often relying on one LLM, fixed or heuristic combinations, or costly multi-agent interactions.Source: §2 · p. 4

3 METHODOLOGY

MedRouter separates specialist selection from answer generation, using a multi-label Router and a generator to combine selected specialist responses. SCALE trains the Router first from specialist correctness and then from generator-level feedback.

3.1 MEDROUTER: SPECIALIST SELECTION AND ANSWER GENERATION

  • MedRouter uses an embedding-based Router to select specialists and a generator to produce the final answer from their responses and the question.Specialists are selected as a set rather than as a single training target.Source: §3 · p. 4, §3.1 · p. 4

3 METHODOLOGY

  • The methodology treats specialist selection as multi-label prediction, allowing the Router to select more than one specialist for a question.At inference, specialists meeting the 0.5 score threshold are selected; if none meet it, the highest-scoring specialist is chosen.Source: §3 · p. 4, §3.1 · p. 4

3.2 SPECIALIST COMPETENCE-AWARE LEARNING

  • SCALE first trains the Router on individual specialist correctness, then refines its selections using feedback from the frozen generator.The embedding encoder, specialists, and generator remain frozen during training; only Router parameters are updated.Source: §3.2 · p. 5

3.2.1 STAGE I: LEARNING SPECIALIST COMPETENCE

  • Stage I assigns each specialist a binary correctness label for each training question and trains the Router with binary cross-entropy.Questions may have multiple positive specialist labels, and specialists are evaluated independently of the generator.Source: §3.2.1 · p. 5

3.2.2 STAGE II: LEARNING FROM GENERATOR OUTCOMES

  • Stage II samples specialists independently, evaluates selected responses through the generator, and updates Router selections with REINFORCE and a learned value baseline.The Performance Gain Reward compares generator answers with and without selected specialist responses: it is positive when adding responses improves correctness and negative when they reduce it.Source: §3.2.2 · p. 5

4 EXPERIMENTS

Across eight text-based and multimodal medical QA benchmarks, MedRouter improves average accuracy over the strongest routing baseline by selecting and integrating specialist responses. The experiments also examine SCALE’s training stages and routing outcomes, using public benchmark datasets and frozen specialists and generator.

4.1 EXPERIMENTAL SETUPS

  • MedRouter uses a frozen embedding encoder and trainable multi-label routing head to select specialists, while a separate generator integrates their responses.The specialist pool contains three text-only and three multimodal models; the study evaluates Qwen3.5-2B and Gemma-4-E2B generators.Source: §4.1 · p. 6
  • Evaluation covers four text and four multimodal medical QA benchmarks, with MedXpertQA subsets reserved exclusively for out-of-distribution evaluation.The combined test sets contain 19,034 questions; training combines 30,000 text-based and 26,365 multimodal examples.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

  • 45.9% with Qwen3.5-2B and 46.1% with Gemma-4-E2B are MedRouter’s average accuracies, exceeding Encoder Router’s 42.6% and Best-in-Pool’s 43.6%.With Qwen3.5-2B, MedQA accuracy rises from the best routing baseline’s 62.5% to 71.8%.Source: §4.2 · p. 7, Table 1 · p. 7
  • MedRouter does not lead on every multimodal benchmark; its pool has only three vision-capable specialists, and the frozen generator is not trained to improve visual understanding.Specialist responses can omit visual evidence or introduce conflicting information.Source: §4.2 · p. 8

4.3 ABLATION STUDY OF SCALE

  • 45.9% average accuracy and the best result on seven of eight benchmarks come from full two-stage SCALE; Stage I alone reaches 43.6%, and Stage II alone reaches 44.8%.The results suggest correctness supervision provides a useful initialization and subsequent reinforcement learning further improves specialist selection for the generator.Source: §4.3 · p. 8, Table 2 · p. 8
  • Replacing PGR with correctness-only reward lowers average accuracy from 45.9% to 44.7%; PGR compares generator correctness with and without specialist information.The reward distinguishes specialist information that corrects an otherwise incorrect answer from information unnecessary for an already-correct answer.Source: §4.3 · pp. 8–9

4.4 ANALYSIS OF ROUTING

  • MedRouter produces more correct answers supported by specialist knowledge, and answers 332 questions correctly even when no specialist response is correct.Some questions remain incorrect despite routing to a knowledgeable specialist, with the generator frozen and not optimized in this framework.Source: §4.4 · p. 9
  • SCALE selects multiple specialists and uses 29 distinct routes, preserving more route diversity and entropy than RL-only training.Fine-tuning alone has more distinct routes and higher route entropy than SCALE.Source: §4.4 · p. 9, Figure 3 · p. 9

5 CONCLUSION

The paper finds that medical specialist LLMs have complementary question-level strengths and introduces MedRouter and SCALE to exploit them. Across eight text-based and multimodal benchmarks, MedRouter improves average accuracy over the strongest routing baselines while keeping specialists and generator frozen.

5 CONCLUSION

  • Specialist LLMs have complementary strengths at the question level, motivating routing that combines their capabilities.MedRouter separates specialist selection from answer generation; SCALE trains the router with correctness supervision followed by reinforcement learning based on downstream response value.Source: §5 · p. 9

Contents of Appendix

The authors used generative AI tools to assist with manuscript work and reviewed the assisted work, manually checking technical claims, citations, results, and content.

A AI USE STATEMENT

  • The authors report using generative AI for writing and research-support tasks, reviewing its work and manually checking technical claims, citations, results, and manuscript content.They state that AI did not generate datasets, conduct experiments without author verification, or make final research decisions.Source: A AI USE STATEMENT · p. 16

B ETHICS STATEMENT

MedRouter is presented as a research framework, not a clinical decision-making tool; real-world clinical use would require professional validation, privacy protections, and rigorous safety assessment.

B ETHICS STATEMENT

  • The authors caution that system outputs may be inaccurate, incomplete, or inconsistent and are not professional medical advice.Any real-world clinical use would require qualified medical validation, privacy protections, and rigorous assessment of safety and potential harm.Source: B ETHICS STATEMENT · p. 16

C LIMITATIONS

MedRouter’s evaluation and scope leave several limitations: it depends on specialist models, may vary across datasets and deployment settings, and is evaluated only on medical question answering.

C LIMITATIONS

  • MedRouter depends on the availability and reliability of its specialist models.The paper also notes that performance may vary across datasets and deployment settings.Source: C LIMITATIONS · p. 16
  • The evaluation is limited to medical question answering, while many real-world medical tasks involve more complex reasoning and information synthesis.Such tasks may require gathering, organizing, and integrating specialist input over multiple steps.Source: C LIMITATIONS · p. 16
  • The authors identify extending MedRouter to complex medical reasoning and improving specialist-information use as future work.Source: C LIMITATIONS · p. 16

D EXPERIMENTAL SETUP DETAILS

The setup caches specialist responses for consistent comparisons, trains the router in two stages, and evaluates against routing baselines using specified infrastructure.

D EXPERIMENTAL SETUP DETAILS

  • Specialist responses are generated once offline and cached so every routing method receives the same information.Generation uses temperature 0.0, a 1536-token limit, and no thinking mode; selected specialists contribute reasoning, not final answers.Source: D EXPERIMENTAL SETUP DETAILS · p. 16
  • Stage I trains the policy head to predict specialist correctness, and its checkpoint initializes Stage II reinforcement learning.Stage I uses binary cross-entropy; Stage II randomly initializes the value head and trains policy and value heads with separate learning rates.Source: D EXPERIMENTAL SETUP DETAILS · p. 16
  • The KNN baseline retrieves 50 nearest training questions and the MLP baseline uses the MedRouter policy-head architecture over the same embeddings.Table 3 reports individual-model accuracy across text-based and multimodal medical QA benchmarks, with Best-in-Pool selecting the top specialist per dataset; training and evaluation use one NVIDIA GH200 GPU.Source: D EXPERIMENTAL SETUP DETAILS · pp. 16–17, Table 3 · p. 17

E PERFORMANCE OF MEDICAL SPECIALIST LLMS

Specialist strengths differ by modality and dataset: one specialist leads across the text benchmarks, while multimodal leadership varies, making per-dataset selection stronger than the best single model there.

E PERFORMANCE OF MEDICAL SPECIALIST LLMS

  • 48.5% multimodal average accuracy for dataset-level Best-in-Pool exceeds the 46.8% average of the best single specialist.Text-based Best-in-Pool matches MediX-R1-8B’s average; its text average is 38.7%.Source: E PERFORMANCE OF MEDICAL SPECIALIST LLMS · p. 17
  • MediX-R1-8B performs best among specialists on all four text-based datasets, averaging 38.7% accuracy.Source: E PERFORMANCE OF MEDICAL SPECIALIST LLMS · p. 17
  • On multimodal QA, MediX-R1-8B leads on SLAKE and MedXpertQA-MM, while MedGemma-4B leads on PathVQA and VQA-RAD.MedGemma-4B has the highest multimodal average at 46.8%. Text-only specialists are not evaluated on multimodal datasets because they do not accept image inputs.Source: E PERFORMANCE OF MEDICAL SPECIALIST LLMS · p. 17

F CASE STUDY: INTEGRATING COMPLEMENTARY SPECIALIST INFORMATION

The case study shows how a generator can combine useful intermediate specialist reasoning while rejecting incorrect conclusions; the accompanying prompts present specialist responses as fallible evidence.

F CASE STUDY: INTEGRATING COMPLEMENTARY SPECIALIST INFORMATION

  • The generator answers the MedXpertQA-Text case correctly from routed reasoning traces, although all three specialists choose incorrect final options.The baseline generator also answers incorrectly without specialist information.Source: F CASE STUDY: INTEGRATING… · pp. 18–19
  • OpenBioLLM-8B contributes the correct organism hypothesis but wrong morphology, while MedGemma-4B offers useful evidence against Neisseria despite its incorrect option.MediX-R1-8B instead reinforces an incorrect competing diagnosis.Source: F CASE STUDY: INTEGRATING… · pp. 18–19
  • The generator retains useful organism-level cues, rejects contradictory specialist conclusions, and returns the correct coccobacillus answer.Its final answer is correct, though its mechanistic explanation is not entirely precise: the hemolysin reference is unsupported.Source: F CASE STUDY: INTEGRATING… · pp. 19–20

G PROMPTS USED IN THIS WORK

Related papers

Loading 2609.33119v1…