Source-linked AI summary
RadAgent: A tool-using AI agent for stepwise interpretation of chest computed tomography
Mélanie Roschewitz, Kenneth Styppa, Yitian Tao, Jiwoong Sohn, Jean-Benoit Delbrouck, Benjamin Gundersen, Nicolas Deperrois, Christian Bluethgen, Julia E. Vogt, Bjoern Menze, Farhad Nooralahzadeh, Michael Krauthammer, Michael Moor
TL;DR
Existing 3D VLM approaches provide limited visibility into the intermediate reasoning behind chest CT reports. RadAgent addresses this gap with an RL-trained, tool-using workflow that produces traceable decisions and improves report generation over standalone 3D VLMs across internal and external datasets, while requiring substantial computing resources and depending on its trained tool set.
Problem
Conventional 3D VLMs largely generate reports in a single step without exposing the intermediate factors shaping their outputs, limiting inspectability.
Method
RadAgent is an RL-trained radiology agent for chest CT analysis that uses a diagnostic checklist, specialized tools, and an explicit trace of intermediate decisions.
Results
RadAgent improves diagnostic performance, robustness to misleading contextual cues, and faithfulness over standalone 3D VLMs and its training-free alternative across internal and external datasets.
Takeaways & Limitations
The explicit agent trace anchors decisions in provided evidence and supports more reliable, inspectable chest CT report generation.
Takeaways & Limitations
The system requires a multi-GPU setup and the trained agent is optimized for the specific tool set available during training.
Abstract
from arXiv · showhide
Vision-language models (VLM) have markedly advanced AI-driven interpretation and reporting of complex medical imaging, such as computed tomography (CT). Yet, existing methods largely relegate clinicians to passive observers of final outputs, offering no interpretable reasoning trace for them to inspect, validate, or refine. To address this, we introduce RadAgent, a tool-using AI agent that generates CT reports through a stepwise and interpretable process. Each resulting report is accompanied by a fully inspectable trace of intermediate decisions and tool interactions, allowing clinicians to examine how the reported findings are derived. In our experiments, we observe that RadAgent improves chest CT report generation over its 3D VLM counterpart, CT-Chat, across three dimensions. Clinical accuracy improves by 5.8 points (35.4% relative) in macro-F1 and 5.1 points (18.6% relative) in micro-F1. Robustness under adversarial conditions improves by 24.7 points (41.9% relative). Furthermore, RadAgent achieves 37.0% in faithfulness, a new capability entirely absent in its 3D VLM counterpart. By structuring the interpretation of chest CT as an explicit, tool-augmented and iterative reasoning trace, RadAgent brings us closer toward transparent and reliable AI for radiology.
EFFUSION
RadAgent verifies preliminary chest CT findings through sequential tool calls before synthesizing a final report. The trace includes intermediate findings such as pleural effusion and a small lung nodule.
- EFFUSION: RadAgent begins with preliminary findings and checks suspected pleural effusion using segmentation, slice extraction, and slice VQA tools.The process then continues with additional 3D visual question answering before report generation.
- EFFUSION: The final report records normal heart findings, left pleural effusion, no pericardial effusion, clear main airways, and a 2 mm lower-left-lobe nodule.The report synthesizes accumulated evidence after the agent determines that investigation is complete.
- EFFUSION: The agent iteratively plans diagnostic actions, selects tools, updates its memory, and refines findings until sufficient evidence is collected.This workflow is guided by a clinician-inspired checklist and maintains intermediate reasoning information.
- EFFUSION: RadAgent’s training and evaluation pipeline combines composite rewards, a clinician-reviewed checklist, GRPO-based training, and internal and external benchmarks.The supplied overview identifies chest CT benchmarks as part of evaluation.
Results
RadAgent combines specialized tools with an RL-trained agent that follows a diagnostic checklist and produces traceable intermediate reasoning. It outperforms CT-Chat across datasets and improves accuracy, robustness, and faithfulness, though reward design involves trade-offs.
- Results: RadAgent is an RL-trained chest CT agent that plans sequential diagnostic strategies and produces traceable intermediate outputs.Its workflow uses specialized tools and maintains a record connecting report decisions with supporting tool outcomes.
- Results: RadAgent significantly outperforms CT-Chat across CT-RATE validation, CT-RATE test, and external RadChestCT datasets.The comparison tests whether the agent can refine or correct the initial report produced by the baseline VLM.
- Results: Mixed curriculum rewards were selected because removing sequence rewards caused checklist violations, while using them from the start reduced exploratory tool-call diversity.The authors report a trade-off between report quality, checklist adherence, and tool-sequence coherence.
- Results: Reinforcement learning adds gains beyond the training-free tool-using variant and improves out-of-domain performance on RadChestCT.After reinforcement learning, trained RadAgent reverses the training-free variant’s disadvantage against CT-Chat on external micro-F1.
- Results: 83.7% robustness and 37.0% faithfulness versus 58.9% and 0.0% for CT-Chat show gains under misleading hints and in output faithfulness.RadAgent’s explicit trace helps distinguish evidence-supported decisions from factors that steer model outputs.
Discussion
RadAgent’s learned, iterative tool-use process improves report generation over standalone 3D VLMs while exposing intermediate reasoning and supporting clinician validation. The discussion highlights gains in accuracy, robustness, and faithfulness, alongside computational, toolbox-dependence, and remaining-faithfulness limitations.
- Performance and mechanism: RadAgent improves report-generation performance over standalone 3D VLMs and training-free alternatives across internal and external datasets.The authors attribute these improvements mainly to the learned, iterative agentic process.
- Clinical implications: RadAgent-based report generation improves diagnostic performance and produces outputs more resistant to misleading contextual cues.The discussion presents this robustness as making the system more reliable for clinical use.
- Performance and mechanism: RadAgent anchors individual decisions in provided evidence, supporting higher robustness and faithfulness than conventional 3D VLMs.The process refines an initial report step by step using specialized tools and a diagnostic checklist.
- Clinical implications: Inspectable intermediate reasoning and tool use may support human-AI collaboration by allowing clinicians to interrogate and validate underlying findings.The proposed workflow includes requesting segmentation of a pleural effusion to visually verify a positive finding.
- Broader significance: RadAgent learns a high-performing tool-calling strategy that dynamically routes subtasks to specialized tools, potentially combining general adaptability with expert precision.The learned policy can potentially be distilled into a fixed workflow, while expanding the tool set may improve breadth, coverage, and practical utility.
- Limitations: The system requires multiple GPUs, may remain too computationally demanding for resource-constrained settings, and is optimized for the tool set available during training.Substantially changing the toolbox may require rerunning the reinforcement-learning pipeline; faithfulness also remains limited at 37.0%.
Methods
RadAgent generates chest CT reports through iterative reasoning, clinician-inspired checklist guidance, and specialized tools spanning image analysis, pathology screening, localization, slice selection, and report refinement.
- Agent workflow: RadAgent is an agentic system for 3D chest CT analysis equipped with a diagnostic checklist for report generation.
- Agent workflow: Report generation follows a ReAct-style iterative process in which the agent decides whether to call additional tools or finalize the report.The agent investigates specific findings and stops when its investigation is sufficient.
- Agent workflow: The workflow begins with an initial report draft, which the agent verifies and refines using a user-specific diagnostic checklist.The checklist is intended to improve the preliminary report and identify omissions.
- Toolbox: The toolbox provides specialized capabilities for visual question answering, disease classification, report drafting, segmentation, slice selection, and CT windowing.Its tools support image understanding, pathology screening, anatomical and pathological localization, and evidence extraction across the radiological workflow.
- Toolbox: The toolbox combines volumetric and slice-level analysis, including CT-Chat VQA for volumes, a 2D VQA model for extracted slices, and CT-CLIP-based classification of eighteen thoracic pathologies.The slice-level VQA tool requires prior slice extraction and returns a free-text answer.
- Training and execution: Training uses GRPO with rewards for report quality and tool-use trajectory quality, including successful calls, tool diversity, and checklist adherence.The system also evaluates whether tool outputs are consumed coherently by later calls and discourages unnecessarily long or computationally heavy trajectories.
User Prompt
A sample chest CT interaction shows RadAgent repeatedly checking anatomical regions and abnormalities, recovering from a failed volume-level query by extracting axial slices and using slice-level VQA.
- Example interaction: The agent sequentially calls report_generation(), disease_classifier(), and ct_vqa() to examine airways, parenchyma, pleura, heart, and abdomen.
- Example interaction: The workflow identifies diffuse ground-glass opacities suggestive of Covid-19 viral pneumonia and verifies the absence of other major abnormalities.
- Error recovery: After a failed device query requiring the CT volume, the agent extracts five axial slices before retrying with slice_vqa().The recovery path uses evenly spaced axial slices from the CT volume.
- Example interaction: The device-check query asks whether catheters, tubes, lines, pacemakers, or surgical clips are present and whether their position causes complications.