Source-linked AI summary
MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflow
Ziyue Wang, Junde Wu, Linghan Cai, Chang Han Low, Xihong Yang, Qiaxuan Li, Yueming Jin
TL;DR
Existing medical VLMs and agentic systems do not adequately reflect clinical diagnostic processes or support sufficiently fine-grained quantitative analysis. MedAgent-Pro uses hierarchical, guideline-informed planning and patient-level evidence-based reasoning, outperforming GPT-4o by 34% and 22% on glaucoma and heart disease diagnosis.
Problem
Existing VLMs and medical agentic systems fall short of clinical diagnostic standards because they lack sufficient medical knowledge, fine-grained visual perception, and clinically oriented workflows.
Method
MedAgent-Pro uses hierarchical disease-level planning and patient-level reasoning with retrieved guidelines, quantitative tools, and stepwise reliability verification.
Results
Across 10+ imaging modalities, 20+ anatomies, and 50+ diseases, MedAgent-Pro surpasses mainstream VLMs, medical agentic systems, and task-specific models, outperforming GPT-4o by 34% and 22% on glaucoma and heart disease diagnosis.
Takeaways & Limitations
MedAgent-Pro advances evidence-based, clinically aligned multi-modal diagnosis by combining guideline-based planning, quantitative analysis, and reliability checks.
Takeaways & Limitations
The framework depends on visual tools that remain limited in some medical domains, while qualitative analysis remains vulnerable to VLM inconsistency and hallucination.
Abstract
from arXiv · showhide
In modern medicine, clinical diagnosis relies on the comprehensive analysis of primarily textual and visual data, drawing on medical expertise to ensure systematic and rigorous reasoning. Recent advances in large Vision-Language Models (VLMs) and agent-based methods hold great potential for medical diagnosis, thanks to the ability to effectively integrate multi-modal patient data. However, they often provide direct answers and draw empirical-driven conclusions without quantitative analysis, which reduces their reliability and clinical usability. We propose MedAgent-Pro, a new agentic reasoning paradigm that follows the diagnosis principle in modern medicine, to decouple the process into sequential components for step-by-step, evidence-based reasoning. Our MedAgent-Pro workflow presents a hierarchical diagnostic structure to mirror this principle, consisting of disease-level standardized plan generation and patient-level personalized step-by-step reasoning. To support disease-level planning, an RAG-based agent is designed to retrieve medical guidelines to ensure alignment with clinical standards. For patient-level reasoning, we propose to integrate professional tools such as visual models to enable quantitative assessments. Meanwhile, we propose to verify the reliability of each step to achieve evidence-based diagnosis, enforcing rigorous logical reasoning and a well-founded conclusion. Extensive experiments across a wide range of anatomical regions, imaging modalities, and diseases demonstrate the superiority of MedAgent-Pro to mainstream VLMs, agentic systems and state-of-the-art expert models. Ablation studies and human evaluation by clinical experts further validate its robustness and clinical relevance. Code is available at https://github.com/jinlab-imvr/MedAgent-Pro.
1 Introduction
Clinical diagnosis requires standardized, evidence-supported, step-by-step integration of visual and textual information, but existing VLMs and reasoning models remain inadequate for fine-grained quantitative analysis. MedAgent-Pro addresses this gap through hierarchical disease- and patient-level reasoning grounded in medical guidelines, quantitative assessment, and evidence verification.
- Introduction: Clinical diagnosis synthesizes medical images and patient records, requiring standardized, step-by-step evaluation rather than one-hop visual question answering.
- Introduction: Its hierarchical design separates disease-level planning from patient-level reasoning to mirror modern clinical procedures.
- Introduction: Existing VLMs provide strong task performance but lack sufficient medical knowledge and fine-grained visual perception for in-depth quantitative analysis.
- Introduction: MedAgent-Pro introduces a systematic, evidence-based agentic workflow for versatile multimodal medical diagnosis.
- Introduction: A retrieval-augmented disease-level planner uses medical guidelines, while patient-level reasoning combines quantitative analysis with evidence-based verification.
2 Related work
Related work spans multimodal medical diagnosis, VLM-based agents, and medical agentic systems. Existing approaches advance imaging assessment, medical VQA, and tool integration, but the supplied discussion identifies limitations in current diagnostic and clinical workflows.
- Multi-modal Medical Diagnosis: Multimodal medical diagnosis research covers imaging classification [24], detection, segmentation, end-to-end VQA [53], and medical VLMs [57].The field has become a primary research objective, while medical VQA [7] is characterized as overly simplified.
- VLM-based AI Agent: VLM-based agents have advanced across industrial engineering [59], scientific experimentation [64], embodied agents [67], gaming [69] [70], and societal simulation [73] [74].The passage presents agent-based methods as an increasingly important direction for developing autonomous intelligent systems.
- Medical Agentic System: Medical agentic systems comprise VLM-enhancement methods using debate or majority voting [31] and toolkits combining orchestrator agents with specialized models [33].The supplied passage criticizes these systems for gluing tools together rather than following a clinically oriented workflow.
3 Methods
MedAgent-Pro mirrors clinical diagnosis through disease-level guideline-based planning and patient-level personalized reasoning. It combines two-stage retrieval, executable specialist tools, and stepwise reliability checks to produce evidence-based diagnoses.
- Hierarchical Diagnostic Workflow: The workflow separates standardized disease-level planning from patient-level reasoning, adapting guideline-based diagnostic steps to each patient’s multimodal data.Disease plans are generated from medical guidelines, while patient-specific reasoning selects steps according to available inputs.
- Disease-Level Knowledge-Based Planning: The RAG agent filters guideline documents by keyword summaries, then performs vector retrieval to extract the five most relevant chunks for the disease query.Documents are chunked and indexed in a vector database, with pre-generated one-sentence summaries accelerating the initial filtering stage.
- Executable Diagnostic Planning: The VLM converts guideline-derived clinical indicators and tool-operation descriptions into an executable plan whose functions consume and produce typed data properties.Plans are stored as JSON entries linking predefined Python tools with expected input and output properties, enabling automatic function invocation when data properties match.
- Patient-Level Evidence-Based Reasoning: Patient-level orchestration selects only plan steps supported by available data, skipping unavailable modalities such as OCT or visual-field inputs.For glaucoma, the system can execute fundus-image steps while filtering out operations requiring missing data.
- Quantitative Analysis: Specialized tool agents provide quantitative assessments through segmentation, grounding, and LLM-based coding models, bridging multimodal AI outputs with clinical indicators.Examples include Medical SAM Adapter, MedSAM, Cellpose, Maira-2, and Copilot; segmentation tools can extract optic-cup and optic-disc masks.
- Evidence-Based Reasoning Paradigm: At each reasoning step, the VLM labels outputs Continue, Terminate, or Complete by assessing result reliability, using accepted evidence sequentially and halting when evidence is unreliable.The final diagnosis compares the accumulated reasoning state with a risk threshold θ, integrating reliable external evidence with expert knowledge.
4 Experiment
MedAgent-Pro is evaluated across multiple diagnostic datasets and modalities against general VLMs, medical agentic systems, and task-specific models. Results show broad performance gains, while ablations support the contributions of planning, quantitative analysis, evidence-based reasoning, and structured indicator fusion.
- Experimental Setup: Experiments span increasingly challenging datasets for glaucoma, heart disease, multi-disease chest X-ray diagnosis, and multiple-choice clinical questions, using bAcc, F1, or accuracy as appropriate.GPT-4o [12] serves as the baseline VLM, LangChain implements the RAG agent, and competing medical agentic systems also use GPT-4o.
- Comparison with Medical Agentic Systems: MedAgent-Pro consistently outperforms medical agentic systems across all diseases and domains, while improving NEJM performance by 7.9% overall and gaining most in visually supported tasks.The gains cover cell imaging, chest X-rays, CT/MRI, and ophthalmology, while remaining robust without visual tool support.
- Effectiveness of the Proposed Key Components: Planning significantly improves performance, and adding visual quantitative analysis and evidence-based reasoning produces further gains, including F1 increases of 34.5% and 20.7%.The ablation evaluates planning, quantitative analysis, and evidence-based reasoning sequentially on glaucoma and heart disease diagnosis.
- Analysis of the Impact of Indicator Accuracy: Indicator-analysis errors have only marginal effects on final diagnostic accuracy when qualitative analysis uses the ophthalmic-specific VisionUnite model.The study separately examines qualitative and quantitative analysis on glaucoma diagnosis because qualitative accuracy is difficult to quantify directly.
- Analysis of Decision-Making Strategies: Structured fusion consistently outperforms flat fusion across indicator counts by assigning explicit risk-based weights to clinical indicators.Flat fusion directly feeds raw indicators into the VLM, whereas structured fusion explicitly weights them for final decision-making.
5 Human Evaluation with Clinical Experts
Clinical-expert evaluation shows that MedAgent-Pro produces higher-quality diagnostic content than other VLMs across five dimensions, while its workflow step counts align with clinicians’ perceived task difficulty and time demands. Visual-tool integration reduces steps for quantitatively assessable conditions, whereas qualitative diagnoses remain step-intensive.
- Alignment with Real-World Clinical Workflow: Across 12 chest X-ray tasks, MedAgent-Pro’s workflow step counts broadly match clinicians’ rankings of diagnostic difficulty and time demand.Tasks were ranked from 1 to 12 by thoracic clinicians.
- Alignment with Real-World Clinical Workflow: Fracture requires the most steps and is ranked most complex, whereas Support Devices requires the fewest steps and is ranked least complex.These examples illustrate alignment between workflow effort and clinicians’ perceived task complexity.
- Alignment with Real-World Clinical Workflow: Visual-tool integration significantly reduces workflow steps for Pleural Effusion and Cardiomegaly, while Fracture and Edema remain step-intensive.Quantitative indicators such as the cardiothoracic ratio can be automated, whereas qualitative assessments still require sequential reasoning.
- Assessment of Generated Diagnostic Content Across Other Methods: MedAgent-Pro outperforms other VLMs across relevance, comprehensiveness, clinical reliability, reasoning coherence, and language clarity in expert evaluations of glaucoma and chest X-ray diagnosis.Clinicians scored each dimension on a 1–5 scale.
6 Conclusion
MedAgent-Pro advances evidence-based multi-modal medical diagnosis by combining guideline-based planning, quantitative analysis, and stepwise reliability verification. Its clinical impact remains limited by dependence on visual tools and inconsistent, hallucination-prone VLM qualitative analysis.
- Broader Impact: MedAgent-Pro integrates medical guidelines, quantitative analysis, and reasoning-step verification to bridge AI systems with clinical procedures and support evidence-based medicine.This design targets accurate, multi-modal diagnosis rather than treating diagnosis as an empirical task.
- Limitations: The framework depends on visual tools that remain unavailable or limited in certain medical domains.
- Limitations: Qualitative analysis still relies on VLMs, whose inherent inconsistency and hallucinations constrain diagnostic reliability and clinical impact.Addressing these limitations is necessary to further improve computer-aided diagnosis.