PaperlayerView exact revision on arXiv

Source-linked AI summary

MedOrch: Medical Diagnosis with Tool-Augmented Reasoning Agents for Flexible Extensibility

Yexiao He, Ang Li, Boyi Liu, Zhewei Yao, Yuxiong He

arXiv:2506.00235v1cs.CL

TL;DR

Medical AI systems can be rigid and narrowly specialized, while general language models may lack grounding in specialized knowledge and tools. MedOrch coordinates customizable medical agents and tools with transparent, multiple reasoning trajectories; across three medical domains, it reports competitive performance, including 93.26% accuracy on Alzheimer’s diagnosis versus 89.05% for the previous state-of-the-art baseline.

  • Problem

    Medical AI systems often lack adaptable multimodal integration and auditable reasoning, while specialized models and general language models have complementary limitations for clinical decision-making.

    Source: §1 · pp. 1, 3

  • Method

    MedOrch dynamically orchestrates specialized agents and tools through a modular framework that supports customizable workflows, transparent reasoning, and multiple diagnostic trajectories.

    Source: §1 · p. 3, §3.1 · p. 5

  • Results

    MedOrch showed competitive performance across Alzheimer’s assessment, chest X-ray diagnosis, and multimodal visual question answering; its three-class Alzheimer’s diagnosis accuracy was 93.26% versus 89.05% for ConvNext+GPT.

    Source: §1 · p. 3, §4.2.3 · p. 11, §5 · p. 14

  • Takeaways & Limitations

    The authors conclude that MedOrch’s transparent reasoning and multiple diagnostic trajectories support comprehensive clinical decision-making and adaptation to specific workflows.

    Source: §5 · p. 14

    The reported baseline comparisons include methods relying heavily on synthetic, template-generated training data and requiring schema access, image-label mappings, and fine-grained manual engineering.

    Source: §4.4.2 · p. 13

Abstract

from arXiv · show

Healthcare decision-making represents one of the most challenging domains for Artificial Intelligence (AI), requiring the integration of diverse knowledge sources, complex reasoning, and various external analytical tools. Current AI systems often rely on either task-specific models, which offer limited adaptability, or general language models without grounding with specialized external knowledge and tools. We introduce MedOrch, a novel framework that orchestrates multiple specialized tools and reasoning agents to provide comprehensive medical decision support. MedOrch employs a modular, agent-based architecture that facilitates the flexible integration of domain-specific tools without altering the core system. Furthermore, it ensures transparent and traceable reasoning processes, enabling clinicians to meticulously verify each intermediate step underlying the system's recommendations. We evaluate MedOrch across three distinct medical applications: Alzheimer's disease diagnosis, chest X-ray interpretation, and medical visual question answering, using authentic clinical datasets. The results demonstrate MedOrch's competitive performance across these diverse medical tasks. Notably, in Alzheimer's disease diagnosis, MedOrch achieves an accuracy of 93.26%, surpassing the state-of-the-art baseline by over four percentage points. For predicting Alzheimer's disease progression, it attains a 50.35% accuracy, marking a significant improvement. In chest X-ray analysis, MedOrch exhibits superior performance with a Macro AUC of 61.2% and a Macro F1-score of 25.5%. Moreover, in complex multimodal visual question answering (Image+Table), MedOrch achieves an accuracy of 54.47%. These findings underscore MedOrch's potential to advance healthcare AI by enabling reasoning-driven tool utilization for multimodal medical data processing and supporting intricate cognitive tasks in clinical decision-making.

1 Introduction

Medical AI systems are often rigid, narrowly focused, or opaque, limiting flexible multimodal reasoning and clinical review. MedOrch orchestrates customizable tools and agents with transparent, diverse reasoning trajectories, and reports competitive results across three medical domains.

1 Introduction

  • Clinical workflows require goal-driven integration of heterogeneous sources, but existing systems are rigid, narrowly focused, and often need substantial re-engineering to adapt.Standalone LLMs are less effective on specialized medical tasks, while specialized models cannot integrate broader clinical contexts or adapt their reasoning flow.Source: §1 · pp. 1–2
  • Many healthcare AI systems provide only a final diagnosis, leaving clinicians without intermediate reasoning steps to validate or audit.Transparent pathways can help professionals check reasoning against clinical expertise and identify potential biases or errors.Source: §1 · p. 2
  • Current systems typically offer a single diagnostic conclusion, limiting access to alternative hypotheses and reasoning strategies for ambiguous or complex cases.The passage identifies overlapping conditions, rare diseases, and multiple valid interpretations as settings where diverse diagnostic perspectives may be valuable.Source: §1 · p. 3
  • MedOrch uses a modular agent-based framework to coordinate specialized tools, expose its reasoning pathway, and generate multiple diagnostic trajectories.Clinicians can customize workflows and integrate new agents or tools without changing the core reasoning framework.Source: §1 · p. 3
  • On ADNI, MedOrch achieved 93.26% accuracy for Alzheimer’s diagnosis, outperforming the best baseline by over 4 percentage points, and improved disease-progression prediction by 19 percentage points.Source: §1 · p. 3

2 Related Work

Prior medical AI spans specialized language, vision-language, and decision-support systems, while tool-using and multi-agent work has advanced general capabilities. The related work identifies limited dynamic reasoning, multimodal clinical integration, and medical-domain adaptation as remaining gaps.

2.1 AI Systems for Medical Decision Support

  • Medical language, clinical decision-support, and vision-language systems offer specialized capabilities but are often limited to text, narrow domains, predefined pipelines, or standalone task performance.Med-PaLM, GatorTron [14], Clinical-BERT [15], and BioMedLM [16] focus on medical language; CLIP-based models [20], RadFM [21], MultiMedBench [22], and MedVInT [23] address vision or multimodal tasks with stated limits on broader integration or reasoning.Source: §2.1 · p. 4

2.2 Tool-Using and Agentic AI Systems

  • Tool-using and multi-agent systems advance tool invocation and reasoning, but cited approaches often lack dynamic compositional reasoning or the medical adaptations needed for multimodal clinical scenarios.Toolformer [24] and HuggingGPT [25] established early tool-use approaches; later systems include Gorilla [26], ToolACE [27], xLAM [28] [29], Search-o1 [30], WebThinker [31], Agentic Reasoning [32], AutoGen [33], and Reflexion [34].Source: §2.2 · pp. 4–5

3 Proposed Method

MedOrch integrates specialized tools into configurable clinical reasoning, incorporating their outputs into a trace that leads to a final answer. Its architecture supports autonomous tool use, workflow customization, and extension with additional agents.

3.1 System Overview

  • The reasoning model uses a registry of tool capabilities, contexts, and input/output specifications, allowing tools to be added without retraining or architectural changes.The workflow can also be customized through system-prompt and registry changes to match clinical protocols without retraining.Source: §3.1 · p. 5, §3.2 · p. 7

3.2 Core Reasoning Mechanism

  • In-context tool-calling tokens trigger query routing, tool execution, and reintegration of results until the model has enough information to answer.The model generates these tokens without fine-tuning or additional training.Source: §3.2 · pp. 6–7
  • The system can generate different reasoning pathways for the same case by varying analytical priorities or diagnostic hypotheses, offering complementary perspectives for complex cases.In the illustrated workflow, web search, patient-history retrieval, and chest X-ray analysis are integrated into the reasoning trace before a final diagnosis.Source: Figure 3 · p. 6, §3.2 · p. 7

3.3 General-Purpose Agents

  • General-purpose agents provide internet search, code execution, EHR query translation, and retrieval of clinical guidelines for broad reasoning tasks.Text2SQL can also retrieve similar cases from the clinical database.Source: §3.3 · p. 7

3.4 Task-Specific Medical Agents

  • Domain-specific agents analyze medical images, answer visual questions, track clinical-entity relationships, and summarize longitudinal trends; registry updates add agents without modifying the core engine.The image-analysis agent can convert neuroanatomical measurements into structured data for diagnostic reasoning.Source: §3.4 · p. 8

4 Experiments

The experiments evaluate MedOrch on ADNI diagnosis and progression prediction, MIMIC-CXR interpretation, and EHRXQA visual question answering. They report strong ADNI results and competitive performance across tasks using a fixed reasoning model with registered domain tools.

4.2.3 Results and Analysis

  • 93.26% accuracy on three-class AD vs. MCI vs. NC classification with o1-mini best@5 exceeds ConvNext+GPT’s 89.05%.This is a 4.21 percentage-point absolute improvement over the previous state-of-the-art baseline.Source: Table 1 · p. 9, §4.2.3 · p. 11

4.1 Experiment Overview

  • EHRXQA requires switching among symbolic querying, visual interpretation, and natural-language explanation within one dialogue, without changing the underlying reasoning architecture.For MIMIC-CXR, new image encoders can also be plugged in without retraining the core reasoning engine.Source: §4.1 · p. 9
  • Across tasks, MedOrch keeps a fixed reasoning model and registers appropriate tools, offering competitive performance with transparent, auditable, flexible reasoning.The experiments include ADNI, MIMIC-CXR, and EHRXQA.Source: §4.1 · pp. 8–9

4.2.1 Dataset and Task

  • ADNI tests diagnosis from partial patient information and prediction from historical visits, using 638 diagnostic questions and 158 progression questions.Diagnostic questions require the system to retrieve additional patient data as needed; prediction questions test temporal-pattern analysis.Source: §4.2.1 · pp. 10–11

4.2.2 Baselines

  • Baselines include CNN/U-Net, multimodal methods, and ConvNext+GPT; evaluation uses ACC, SPE, and SEN with GPT-4o or o1-mini under three trajectory strategies.The strategies are best@1, majority@5, and best@5.Source: §4.2.2 · p. 11, §4.2.3 · p. 11

4.2.3 Results and Analysis

  • 50.35% accuracy with best@5 and 38.61% with best@1 show MedOrch’s advantage on Alzheimer’s disease progression prediction.The task is more challenging for all compared methods.Source: §4.2.3 · p. 11
  • Three-class accuracy rises from 88.71% (best@1) to 93.26% (best@5), while multiple traces provide distinct perspectives with logged reasoning steps and tool outputs.The text also reports statistically significant gains from multiple diagnostic perspectives.Source: §4.1 · p. 9, §4.2.3 · pp. 11–12
  • On MIMIC-CXR, o1-mini achieves the highest scores in every AUC column and macro-F1, while GPT-4o is competitive with the best baseline.The comparison covers chest X-ray classification and detection performance.Source: Table 3 · p. 12

4.3 Chest X-ray Diagnosis

MedOrch is evaluated on chest X-ray diagnosis and multimodal EHRXQA, with results emphasizing macro-level chest X-ray performance and cross-modal question answering. The evaluation also highlights that MedOrch works without the manually engineered training corpus used by benchmark baselines.

4.3.1 Dataset and Task

  • The chest X-ray task uses MIMIC-CXR radiographs and asks the system to identify the dataset’s 14 labeled thoracic pathologies.MIMIC-CXR includes radiographs with associated radiology reports.Source: §4.3.1 · p. 12

4.3.2 Baselines

  • The chest X-ray comparison evaluates MedOrch against competitive multimodal baseline methods.The listed methods include IFCC, R2Gen, UniXGen, XrayGPT, and LLM-CXR.Source: §4.3.3 · p. 12, §4.3.2 · p. 12

4.3.3 Results and Analysis

  • MedOrch achieves substantially better Macro AUC and Macro F1 than the compared chest X-ray methods, while LLM-CXR slightly outperforms it on Micro F1.The analysis associates the macro results with performance across pathology classes, including less common conditions.Source: §4.3.3 · pp. 12–13

4.4.1 Dataset and Task

  • EHRXQA combines structured MIMIC-IV records with MIMIC-CXR images and includes table-, image-, and Image+Table-related questions.The evaluation samples 300 test questions: 69 image-related, 108 table-related, and 123 Image+Table-related.Source: §4.4.1 · p. 13

4.4.2 Baselines

  • MedOrch answers EHRXQA questions without handcrafted training corpora or program annotations, unlike baselines that rely on synthetic question-program-answer data.Those baselines require database schemas, image-label mappings, and fine-grained manual engineering.Source: §4.4.2 · p. 13

4.4.3 Results and Analysis

  • 54.47% accuracy is achieved by MedOrch on the most challenging EHRXQA Image+Table questions.Across modalities, it also reaches 78.70% execution accuracy on table-related questions and 41.67% accuracy on image-related questions.Source: §4.4.3 · p. 13
  • MedOrch operates in zero-shot or few-shot fashion without manual effort, unlike structured QA pipelines that require synthetic training corpora and executable programs.The authors connect this distinction to privacy, annotation cost, and schema variability across institutions, and argue the approach can adapt to unseen hospital systems, schemas, and patient cohorts.Source: §4.4.3 · p. 14

5 Conclusion

The conclusion presents MedOrch as a customizable medical decision-support framework that coordinates tools, records auditable reasoning, and generates multiple diagnostic pathways. Its evaluation across three medical applications demonstrates competitive performance and customizability.

5 Conclusion

  • MedOrch coordinates diverse medical tools across modalities and domains while adapting workflows to clinical goals and institutional requirements.Its recorded decision pathways allow clinicians to audit and verify recommendations, and its multiple reasoning pathways offer diagnostic perspectives beyond a single conclusion.Source: §5 · p. 14
  • MedOrch’s evaluation across Alzheimer’s assessment, chest X-ray diagnosis, and multimodal visual question answering demonstrates competitive performance and high customizability.The authors frame the framework as supporting comprehensive clinical decision-making through transparent reasoning and multiple diagnostic trajectories.Source: §5 · p. 14
  • The authors aim to extend MedOrch to broader clinical scenarios and additional modalities, and identify potential applications beyond healthcare.They point to its modular and customizable architecture as the basis for possible adaptation to other complex domains.Source: §5 · p. 14

A Implementation Details

All experiments use GPT-4o and o1-mini as the core reasoning models.

A Implementation Details

A.1 General-purpose Agents

The general-purpose agents provide web retrieval, sandboxed coding, clinical database querying, and guideline retrieval.

A.1 General-purpose Agents

  • The Web Search Agent fetches top-k Bing results and extracts nearby context, keeping the context small enough to keep query latency under two seconds.Source: §A.1 · p. 18
  • The Coding Agent runs short Python scripts in a sandbox to provide on-the-fly calculations, plots, and data reformatting within the audit trail.Source: §A.1 · p. 18
  • The Text2SQL Agent converts patient-information questions into SQL and checks generated queries before execution, including temporal-query and clinical-join handling.Source: §A.1 · p. 18
  • The RAG system retrieves relevant clinical-guideline sections from its structured knowledge base using semantic similarity and recency.Source: §A.1 · p. 18

A.2 Domain-specific Agents

MedOrch adds specialized agents for ADNI, MIMIC-CXR, and EHRXQA. The ADNI agents analyze baseline scans and longitudinal patient records, while the other agents support radiograph and multimodal-question analysis.

Domain-specific agents for ADNI

  • 84.34% test accuracy is reported for the ADNI Medical Image Analysis Agent’s fine-tuned 3D ResNet-50 on baseline T1 scans.The agent wraps FreeSurfer for basic preprocessing and uses an 80:15:5 patient-level split.Source: §A.2 · p. 19
  • The ADNI Longitudinal Data Analysis Agent aligns patient records to a baseline-anchored monthly timeline and derives moving averages, slopes, and rate-of-change ratios.Short gaps under three months are forward-filled; longer gaps are linearly interpolated before features are passed to the reasoning model.Source: §A.2 · p. 19

Domain-specific agents for MIMIC-CXR and EHRXQA

  • For MIMIC-CXR and EHRXQA, MedOrch combines an M3AE-and-GPT-4V Medical Imaging QA Agent with a UMLS-based Clinical Knowledge-Graph Agent.GPT-4o labels relations among lifted clinical concepts, and the resulting triples are indexed for local and global similarity search.Source: §A.2 · p. 19

B Example of Multiple Reasoning Trajectories

MedOrch can produce different diagnostic trajectories for the same patient by varying the sequence of tool use. The example contrasts a CSF-and-cognitive-score route ending in an AD diagnosis with an MRI-led route ending in an MCI assignment and follow-up recommendation.

B Example of Multiple Reasoning Trajectories

  • Trajectory 1 queries CSF measurements and cognitive scores, computes the Tau/ABETA ratio, and consults guidelines before diagnosing AD, whereas Trajectory 5 starts with MRI analysis and assigns MCI after cross-checking cognitive scales.Trajectory 5 interprets the hippocampal Z-score with targeted literature search and recommends follow-up.Source: B Example of Multiple Reasoning… · p. 19
  • The same patient can receive distinct diagnostic conclusions when the reasoning strategy changes.The example’s two trajectories reach AD and MCI conclusions, respectively.Source: B Example of Multiple Reasoning… · p. 19
  • MedOrch logs intermediate calculations and evidence sources for clinical reviewers to inspect.Its modular orchestration permits strategy variation without changing the core engine.Source: B Example of Multiple Reasoning… · p. 19

C Comparision of MedOrch and ChatGPT + M3AE (BM25)

The comparison contrasts a template-dependent ChatGPT-plus-M3AE baseline with MedOrch’s automatic tool invocation. The example traces MedOrch from database retrieval to image analysis and a confirming visual question-answering check.

C Comparision of MedOrch and ChatGPT + M3AE (BM25)

  • The ChatGPT-plus-M3AE baseline depends on hand-written SQL or NeuralSQL templates, with four annotators spending about two months authoring them.NeuralSQL templates embed FUNC_VQA(...) to pass study IDs to M3AE for visual sub-questions.Source: C Comparision of MedOrch and ChatGPT +… · p. 19
  • At inference, the baseline retrieves template instances with BM25 and asks ChatGPT to generate similar SQL or NeuralSQL; new question forms or schema changes require template edits.The fixed logical skeleton makes adaptation dependent on writing or revising templates.Source: C Comparision of MedOrch and ChatGPT +… · p. 19
  • MedOrch automatically invokes agents or tools from a question and necessary information, without manual work, and can be adapted to other problem types.The example asks whether an attribute appears in a specified patient’s chest X-ray.Source: C Comparision of MedOrch and ChatGPT +… · p. 20
  • In the MedOrch example, database retrieval supplies a study ID, an imaging agent checks the image, and a medical-imaging QA agent double-checks the answer.The example concludes that the queried attribute is present in the specified object.Source: C Comparision of MedOrch and ChatGPT +… · p. 21

Related papers

Loading 2506.00235v1…