Source-linked AI summary
MedRAX: Medical Reasoning Agent for Chest X-ray
Adibvafa Fallahpour, Jun Ma, Alif Munim, Hongwei Lyu, Bo Wang
TL;DR
CXR interpretation is clinically important but existing task-specific tools are fragmented, while multimodal models can hallucinate and struggle with complex diagnostic reasoning. MedRAX unifies multimodal reasoning with dynamically orchestrated CXR tools without additional training, and achieves state-of-the-art performance on complex interpretation tasks. The paper also introduces ChestAgentBench to evaluate these capabilities across diverse medical queries.
Problem
Existing CXR models operate in fragmented task-specific workflows, while LMMs can hallucinate and struggle with systematic multi-step diagnostic reasoning.
Method
MedRAX uses an LLM-driven ReAct loop to dynamically orchestrate specialized CXR tools without additional training.
Results
MedRAX achieves state-of-the-art performance on complex CXR interpretation tasks and outperforms general-purpose and specialized medical models.
Takeaways & Limitations
The findings support combining large-scale reasoning with domain-specific tools as a structured alternative to purely end-to-end medical models.
Takeaways & Limitations
MedRAX can struggle with contradictory outputs from classification and segmentation tools, incurs computational overhead, and lacks robust uncertainty quantification.
Abstract
from arXiv · showhide
Chest X-rays (CXRs) play an integral role in driving critical decisions in disease management and patient care. While recent innovations have led to specialized models for various CXR interpretation tasks, these solutions often operate in isolation, limiting their practical utility in clinical practice. We present MedRAX, the first versatile AI agent that seamlessly integrates state-of-the-art CXR analysis tools and multimodal large language models into a unified framework. MedRAX dynamically leverages these models to address complex medical queries without requiring additional training. To rigorously evaluate its capabilities, we introduce ChestAgentBench, a comprehensive benchmark containing 2,500 complex medical queries across 7 diverse categories. Our experiments demonstrate that MedRAX achieves state-of-the-art performance compared to both open-source and proprietary models, representing a significant step toward the practical deployment of automated CXR interpretation systems. Data and code have been publicly available at https://github.com/bowang-lab/MedRAX
1. Introduction
CXR interpretation is clinically important but burdensome, while existing AI tools remain fragmented and foundation models can be unreliable for complex reasoning. MedRAX addresses this gap by orchestrating specialized tools and introducing a benchmark for complex CXR queries.
- CXR interpretation supports critical disease decisions but requires radiologists to spend hours on systematic anatomical analysis.
- Task-specific AI models automate classification, segmentation, and report generation, but their fragmented nature limits practical clinical utility.
- Foundation models offer unified image-text reasoning, yet LMMs can hallucinate, reason inconsistently, and lack specialized transparent workflows for diagnostic tasks.
- MedRAX is a specialized framework that dynamically orchestrates multiple CXR analysis tools without additional training.
- ChestAgentBench contains 2,500 complex medical queries across 7 categories, constructed from 675 expert-curated clinical cases.
- MedRAX outperforms general-purpose and biomedical specialist models on complex reasoning tasks while maintaining transparent workflows.
2. Related Work
Prior agentic and multimodal systems improve medical reasoning but remain constrained by domain robustness, computational cost, flexibility, or task coverage. These limitations motivate specialized clinical benchmarks and tool-integrated approaches such as MedRAX.
- 2.1. LLM-based Agent Architectures: LLM-based agents adapt across applications without additional training, but few are evaluated for robustness in high-stakes medical settings.
- 2.1. LLM-based Agent Architectures: MDAgents and MMedAgent use multi-agent interaction or external medical models to strengthen clinical reasoning and decision-making.
- 2.1. LLM-based Agent Architectures: MDAgents incurs coordination overhead, while MMedAgent’s broad multimodal scope may dilute specialization and requires retraining for new tools.
- 2.2. Medical Agents: o1-powered agents provide strong multi-step reasoning but face high computational demands, proprietary constraints, and inefficiencies from redundant reasoning.
- 2.2. Medical Agents: Radiology-focused systems such as RaDialog and M4CXR explore report generation, conversational assistance, and multitask CXR interpretation.
- 2.3. Evaluation Frameworks: Existing benchmarks assess agent reasoning, tool use, interactive healthcare, and software tasks, revealing persistent weaknesses in context retention, planning, and medical-task consistency.
3. MedRAX
MedRAX uses an LLM-driven ReAct loop with memory to decompose CXR queries, select and execute specialized tools, and integrate their outputs. Its modular architecture supports diverse interpretation tasks, parallel and configurable deployment, and extensible tool integration.
- Framework: MedRAX dynamically reasons, plans, and executes multi-step CXR workflows by combining multimodal reasoning with structured tool-based decisions.
- ReAct Loop: The agent selects tools, executes independent tools in parallel, updates memory with results, and returns either a response or timeout message.
- ReAct Loop: The ReAct framework iterates through observation, thought, and action, using prior tool findings and short-term memory to guide subsequent reasoning.
- Specialized Tools: MedRAX supports visual question answering through CheXagent and LLaVA-Med, segmentation through MedSAM and ChestX-Det, and grounding through Maira-2.
- Specialized Tools: Its tool suite also includes report generation, disease classification, CXR generation, DICOM processing, custom plots, and figure visualization.
- Memory and Control: Memory caches tool outputs to prevent redundant computations, while the agent monitors outputs and errors to inform later tool selection.
- Flexible Integration: The framework accepts text-only or multimodal LLMs, supports local or cloud deployment, and allows tools to be modified, replaced, or added without training.
- Deployment: A Gradio interface supports standard radiological formats including DICOM, multi-turn chat, and display of intermediate tool outputs.
4. ChestAgentBench
ChestAgentBench is designed to evaluate complex, multi-step chest X-ray interpretation rather than simple single-step visual question answering. It combines expert-curated Eurorad cases with six-choice questions spanning seven core competencies and five question types, with automated quality verification.
- Dataset diversity: The 675 cases span 53 anatomical areas and diverse clinical settings and pathologies, with lung, thorax, and mediastinum among the most frequent regions.The figure reports lung at 51.2%, thorax at 42.8%, and mediastinum at 15.8%.
- Evaluation format: Each question uses a structured six-choice format to evaluate complex CXR interpretation with straightforward, reproducible accuracy measurement.Accuracy is measured as the percentage of correct answers across questions.
- Core competencies: The benchmark covers seven CXR competencies: detection, classification, localization, comparison, relationship, diagnosis, and characterization.These competencies assess identifying, positioning, comparing, relating, interpreting, and describing findings.
- Question design: ChestAgentBench uses five question types that combine targeted competencies with medical reasoning: detailed finding analysis, pattern recognition and relations, spatial understanding, clinical decision making, and diagnostic characterization.Each question type evaluates a specific combination of analytical skills.
- Quality control: GPT-4o generated questions for each case and question type, while automated checks removed items lacking six-choice consistency, clinical grounding, or verifiability.Questions were required to have one correct answer explicitly supported by the original Eurorad case material.
5. Experiments
MedRAX is evaluated against multimodal and specialized CXR models across four benchmarks spanning complex reasoning, visual question answering, report generation, and medical VQA. It achieves state-of-the-art results on several evaluations and demonstrates tool-mediated resolution of complex cases.
- ChestAgentBench: MedRAX achieves 63% accuracy across all seven ChestAgentBench categories, exceeding GPT-4o at 56.4%, Llama-3.2-90B at 57.9%, CheXagent at 39.5%, and LLaVA-Med at 28.7%.The benchmark contains 2,500 six-choice questions covering detection, classification, localization, comparison, relationship, characterization, and diagnosis.
- CheXbench: MedRAX scores 68.7% on Rad-Restruct and 82.9% on SLAKE visual QA, surpassing CheXagent on both subsets and GPT-4o on Rad-Restruct.On OpenI image-text reasoning, performance drops across all models, with CheXagent reaching only 59.0% accuracy against a 50% random baseline.
- MIMIC-CXR Radiology Report Generation: MedRAX reaches mF1-14 of 79.1% and mF1-5 of 64.9% on MIMIC-CXR report generation, outperforming M4CXR at 60.6% and 61.8%, respectively.Its macro-averaged scores are lower than the leading baselines, with M4CXR highest on MF1-14 and Med-PaLM M 84B highest on MF1-5.
- SLAKE VQA: MedRAX achieves 90.35% accuracy and 91.23% recall on SLAKE VQA, exceeding M4CXR at 85.1% and 86.0% and CheXagent at 71.1% and 73.2%.The benchmark contains 114 close-ended English chest X-ray questions.
- Case Studies: In case studies, MedRAX combines sequential or conflicting tool outputs to identify a chest tube and left pneumothorax where GPT-4o makes incorrect diagnoses.The examples illustrate tool integration for device identification and comparative lung analysis.
6. Discussion
MedRAX combines structured tool orchestration with large-scale reasoning to improve complex CXR interpretation, while revealing benefits and limitations of this hybrid approach.
- MedRAX achieves state-of-the-art performance in complex CXR interpretation, outperforming both general-purpose and specialized medical models.
- Task Decomposition: Explicit task decomposition dynamically composes complex reasoning chains, maintains computational efficiency, and produces clear decision traces.The reported performance gap suggests that explicit decomposition provides advantages beyond scale alone.
- Generalists Versus Specialists: General-purpose models GPT-4o and Llama-3.2-90B outperform specialized models LLaVA-Med and CheXagent on the evaluated tasks.MedRAX bridges this gap by integrating domain-specific tools while maintaining generalist reasoning.
- Limitations: MedRAX sometimes struggles when classification and segmentation tools provide contradictory interpretations in fine-grained visual tasks.
- Limitations: Multiple specialized tools can increase response times compared with end-to-end models, and the framework lacks robust uncertainty quantification.
- Future Work: Future work should analyze tool-utilization trade-offs, prompting for critical tool evaluation, reinforcement learning, and comprehensive clinical validation.The stated trade-offs include accuracy, computational cost, speed, error propagation, and spurious correlations.
7. Conclusion
MedRAX integrates structured tool orchestration with large-scale reasoning for CXR interpretation and demonstrates superiority over general-purpose and domain-specific models.
- MedRAX establishes a benchmark in AI-driven CXR interpretation by integrating structured tool orchestration with large-scale reasoning.
- The evaluation demonstrates MedRAX’s superiority over both general-purpose and domain-specific models.
- The findings highlight combining foundation models with specialized tools as a principle applicable to broader healthcare domains.
Impact Statement
MedRAX is designed to augment clinical expertise in CXR interpretation, with privacy-conscious deployment options and acknowledged safety requirements.
- MedRAX aims to improve CXR interpretation, diagnostic accuracy, and efficiency as an AI co-pilot that augments rather than replaces clinical expertise.
- The paper acknowledges model bias, hallucinations, and data privacy as AI risks requiring robust validation before clinical deployment.
- Development used the de-identified Eurorad database under CC BY-NC-SA 4.0 licensing, with ChestAgentBench released under the same terms.
- Azure OpenAI Service was configured to opt out of data logging for privacy-sensitive datasets such as MIMIC-CXR.
- MedRAX supports local LLM deployment for greater data control, with planned guidelines for privacy-preserving configurations.
A. MedRAX Core Methodology
MedRAX uses a multimodal LLM-driven ReAct loop to reason, invoke specialized CXR tools, observe results, and update memory until producing an answer.
- The ReAct loop iteratively cycles through reasoning, acting, and observing to orchestrate tools for complex CXR interpretation.
- The iterative process supports multi-step analysis using various diagnostic tools sequentially or in parallel.
- A multimodal LLM analyzes the query, images, history, and prior results before answering directly or requesting tool execution.
- Specialized tools cover CXR classification, segmentation, visual question answering, grounding, and report generation through standardized wrappers.
- Tool wrappers specify names, descriptions, input schemas, and execution logic so the LLM can identify capabilities and invoke models.
- The Orchestrator parses LLM responses, selects and executes requested tools, updates memory, and decides when the cycle is complete.
- Memory stores user input, LLM responses, thoughts, tool identities, and results as structured messages that provide context for later reasoning.
- Independent tools may execute in parallel, while deployment supports quantization and distribution across CPU or GPU resources.
A.5. Implementation Architecture
MedRAX uses a state-machine workflow architecture in which processing nodes update shared agent state and edges control unconditional or conditional transitions. This design supports dynamic adaptation while preserving clear decision traces.
- Node and Edge Structure: Processing nodes are Python functions or class methods that consume the current Memory/State and return updates to it.
- Node and Edge Structure: Standard edges define unconditional transitions between workflow nodes.
- Node and Edge Structure: Conditional edges use routing functions to select the next node from the current Memory/State.
- The resulting framework is described as flexible, robust, and transparent for dynamic CXR interpretation across diverse clinical scenarios.
B. ChestAgentBench: Comprehensive Benchmark Statistics
ChestAgentBench includes 675 cases spanning emergency, intensive-care, and other hospital settings, with 20 pathological findings distributed across the benchmark. The findings range from common masses to less frequent cardiomegaly, reflecting varied clinical prevalence.
- Clinical Settings: 675 cases span emergency rooms, intensive care units, and other hospital settings.
- Clinical Settings: 19.7% (133 cases) come from emergency rooms, while 4.9% (33 cases) come from intensive care units.
- Clinical Settings: 75.4% (509 cases) come from other hospital settings.
- Pathological Findings: The benchmark covers 20 distinct pathological findings, ranging from 26.3% (Mass) to 1.5% (Cardiomegaly).Structural abnormalities and fluid collections are most common, whereas cardiac and infectious conditions appear less frequently.