Source-linked AI summary
MedOpenClaw and MedFlowBench: Auditing Medical Agents in Full-Study Workflows
Weixiang Shen, Chengzhi Shen, Yanzhu Hu, Che Liu, Junde Wu, Jiayuan Zhu, Xiao Han, Zongyue Li, Jingpei Wu, Min Xu, Daguang Xu, Yueming Jin, Benedikt Wiestler, Daniel Rueckert, Jiazhen Pan
TL;DR
Medical imaging benchmarks often use selected images, leaving incomplete evidence about agents’ ability to search complete studies and operate imaging software. This paper introduces a bounded runtime and full-study benchmark for auditable evidence production, finding that answer-only scoring overestimates competence and that advanced tools do not ensure reliable multi-step execution.
Problem
Existing medical imaging evaluation often uses pre-selected images, whereas clinically relevant findings may require complete-study search, viewer-state management, and auditable evidence.
Method
MEDOPENCLAW provides bounded interaction with 3D Slicer and QuPath, and MEDFLOW-BENCH evaluates full-study episodes using structured evidence and deterministic evidence gates.
Results
Answer-only scoring overestimates workflow competence, with marked drops on complex tasks when correct supporting evidence is required; advanced operations do not automatically improve reliable execution.
Takeaways & Limitations
Medical imaging agents should be evaluated on producing correct, checkable evidence while navigating complete volumetric studies or whole-slide images.
Takeaways & Limitations
The reported failure analysis identifies constrained workflow execution, including invalid quantitative evidence from weak screen grounding and unvalidated measurement procedures.
Abstract
from arXiv · showhide
Medical imaging benchmarks often evaluate VLMs on pre-selected 2D images, slices, crops, or patches, making evaluation closer to visual recognition. Real clinical workflows impose a different burden: readers must search through complete studies, operate imaging software, navigate across slices and magnifications, and document visual evidence that can be audited. We argue that this evidence-producing workflow is a critical missing evaluation axis for medical imaging agents. To study it, we introduce MedFlowBench, a full-study benchmark for VLM agents, together with MedOpenClaw, a controlled and replayable runtime in which agents operate medical imaging viewers such as 3D Slicer and QuPath. In each episode, an agent inspects a complete radiology study or whole-slide pathology image, returns a task answer, and submits structured evidence, including key slices, coordinates, regions of interest, or lesion-state fields. This evidence is automatically checked against withheld masks, annotations, and labels. Across evaluated models, final answer-only scoring gives an overly optimistic picture: when answers must also be supported by correct evidence, performance drops substantially on complex workflows. We further find that adding image-analysis tools does not by itself solve the problem. Tools help when they make a complex procedure simple and reliable, but agents still struggle when they must choose inputs, manage viewer state, and verify intermediate outputs over multiple steps. MedFlowBench exposes whether medical imaging agents can produce auditable evidence from complete studies, rather than plausible answers from selected images.
1 Introduction
Medical imaging benchmarks often simplify evaluation to selected images, while clinical workflows require searching complete studies and producing auditable evidence. MEDOPENCLAW and MEDFLOW-BENCH evaluate this broader interaction, showing that answer-only scores overestimate competence and that tools do not automatically resolve multi-step workflow failures.
- Motivation: Pre-selected 2D images and patches omit the study-level search required across volumetric radiology and whole-slide pathology.Findings may emerge across slices, sequences, timepoints, or after localization.
- Contribution: MEDOPENCLAW lets VLMs operate 3D Slicer and QuPath through bounded actions, while MEDFLOW-BENCH evaluates five full-study radiology and pathology task families.The tasks include MRI subtype and treatment-response assessment, lung PET/CT, breast histology, and lymph-node metastasis assessment.
- Evidence-grounded evaluation: Each episode requires a task answer plus structured evidence such as key slices, coordinates, regions of interest, or longitudinal lesion-status fields.Deterministic checks compare submitted evidence against withheld masks, annotations, or labels.
- Results: Answer-only scoring substantially overestimates workflow competence because scores drop markedly when complex-task answers must also be supported by correct evidence.The benchmark therefore evaluates both task accuracy and checkable evidence.
- Results: Specialized tools help when they collapse complex procedures into reliable calls, but dependent segmentation and registration workflows remain brittle.Agents may choose inappropriate parameters, lose image-space or viewer-state alignment, or fail to verify generated artifacts.
2 Related Work
Prior medical benchmarks and agent systems increasingly broaden multimodal reasoning and tool use, but much related work still assumes fixed images, isolated APIs, or simplified environments. MEDOPENCLAW instead situates bounded interaction within professional medical imaging software.
- Static medical VQA and medical VLM benchmarks: Static medical VQA and multimodal benchmarks mainly evaluate language-conditioned reasoning over fixed or pre-selected image inputs.Related report-generation and image-grounded QA systems likewise largely assume fixed images.
- Medical agents: Medical-agent research makes multi-step reasoning and evidence gathering more explicit, yet many systems still operate on fixed images, isolated APIs, or abbreviated workflows.Radiology-oriented agents and broader multimodal agents often select specialized tools rather than navigate complete studies.
- Full-study and interactive medical imaging systems: Interactive medical-imaging work addresses volumetric reasoning, simulated radiology environments, and copilot-style assistance inside software such as 3D Slicer.These directions are closer to study-level interaction but represent distinct partial approaches.
- General-purpose agent runtimes: MEDOPENCLAW relates to general-purpose software and desktop-agent runtimes while adapting the interaction contract to medical imaging.Its bounded control is implemented within the medical-imaging interface rather than as an open-ended runtime.
3 MEDOPENCLAW Runtime
MEDOPENCLAW provides a bounded, software-native contract between backbone VLM agents and medical imaging viewers. It exposes navigation, evidence capture, and analysis actions while logging state and artifacts for replayable audit.
- Runtime interface: The runtime connects backbone VLM agents to 3D Slicer for volumetric radiology and QuPath for whole-slide pathology.Viewer state naturally represents selected series, slice or viewport settings, and image coordinates.
- Runtime interface: Agents can select series, scroll slices, adjust display settings, bookmark views, measure, pan and zoom slides, convert viewer points to image coordinates, and export evidence.The runtime operates externally without modifying viewer source code.
- Action space: Viewer Control, Evidence Capture, and Analysis Operators organize the exposed action space.Evidence actions save checkable views, locations, regions or masks, and measurements from the current viewer state.
- Auditability: The bounded design makes auditability operational by logging action calls, arguments, viewer-state snapshots, and generated artifacts.These records support reconstruction of accessed views, executed operations, and available evidence.
- Scope: The same traceable interaction contract supports human-in-the-loop interfaces, but MEDCOPILOT is presented as a demonstration rather than evidence of clinical deployment or efficiency gains.Generated actions and evidence artifacts remain visible for clinician inspection.
4 MEDFLOW-BENCH: Study-Level Evaluation
MEDFLOW-BENCH evaluates study-level and whole-slide reasoning through standardized task families, tracks, schemas, and evidence-based metrics. Its episodes require agents to navigate complete cases, acquire evidence, and perform clinically meaningful multimodal or longitudinal operations.
- Benchmark design: MEDFLOW-BENCH evaluates full-study interaction rather than perception from pre-selected 2D images, including active evidence acquisition and professional-software execution.The benchmark emphasizes cross-modality, longitudinal reasoning, and differential diagnosis.
- Example episodes: Representative traces show longitudinal MRI registration and fused-view inspection alongside breast WSI screening, magnification changes, coordinate mapping, and diagnosis.The traces preserve operation calls, viewer states, visual outputs, locations, and final responses for auditability.
- Task families: Five task families span brain-tumor subtype assessment, longitudinal MRI treatment response, lung PET/CT, breast histology, and lymph-node metastasis assessment on whole-slide images.They require lesion localization, cross-sequence or PET/CT integration, longitudinal comparison, and gigapixel-slide analysis.
- The Three-Track Design: The three-track design uses identical cases, formulations, and metrics while separating viewer-native inspection, advanced operations, and diagnostic compatibility.Track A focuses on primitive navigation and evidence acquisition; Track B tests invoking, configuring, and verifying specialized operations.
- Scoring metrics: Task measures primary answer performance, Strict applies the evidence gate, and Loc. separately measures coordinate localization when spatial ground truth exists.Strict equals Task only when module-specific evidence fields pass deterministic checks.
5 Experiments
Experiments evaluate full-study radiology and pathology workflows under answer-plus-evidence scoring, including viewer-native access, runtime-agnostic alternatives, and advanced operations. Results show that evidence-grounded performance remains weak in difficult localization and multi-step software workflows, while automated operations can provide selective gains.
- Viewer-Native Radiology Baselines: Brain MRI is the most favorable radiology setting, with Gemini 3.1 Pro Preview reaching 0.73 Task, 0.73 Strict, and 0.96 Loc.The target tumor is relatively spatially salient, supporting stronger localization and evidence-grounded performance.
- Viewer-Native Radiology Baselines: Longitudinal MRI and Lung PET/CT remain difficult: the best Strict scores are 0.18 and 0.14, while the best Lung PET/CT localization score is 0.26.These workflows require longitudinal comparison or precise localization of smaller thoracic lesions and RAS evidence.
- Viewer-Native Pathology Baselines: Pathology results likewise show plausible case-level answers but weak evidence and coordinate localization in QuPath whole-slide workflows.Viewer-native pathology uses coordinate-grounded scoring under the same auditable-evidence requirement.
- Track C: Runtime-Agnostic: Runtime-agnostic alternatives preserve the evidence-grounding bottleneck, while frontier models using 3D Slicer outperform native medical 3D VLM baselines on aggregate Task, Strict, and localization metrics.The comparison includes static slice montages and native volumetric models, so the bottleneck is not tied to one runtime interface.
- Track B: Advanced Operations: Advanced operations produce local and inconsistent gains for segmentation and registration, whereas MONAI-based 3D VISTA3D yields consistent Lung PET/CT localization gains and model-dependent Strict improvements.Multi-step operations require source-volume selection, spatial input, parameterization, state tracking, and output verification; VISTA3D is more automated and easier to invoke.
6 Failure Case Analysis
Failure cases reveal that medical VLM agents struggle to preserve clinical intent, ground actions spatially, manage software state, calibrate operation outputs, and compose stable procedures. Auditable traces expose these failures beyond what final answers alone can show.
- Workflow intent management: Longitudinal registration calls can be locally reasonable yet drift away from the clinical question of cross-timepoint response assessment.One case mixed longitudinal CT1 registration, CT1-derived FLAIR resampling, and intra-timepoint alignment.
- Spatial and parametric action grounding: Weak 2D screen boxes were elevated into millimeter-scale lesion measurements without validated boundaries, registration, or a measurement procedure.The resulting RAS coordinates and reported lesion diameters and SPD change were treated as quantitative clinical evidence despite poor grounding.
- Stateful artifact management: A registered MRI fusion workflow became inconsistent when the requested CT1 transform resolved to a FLAIR transform and the displayed fusion did not match the cited CT1 evidence.The failure reflects software-state misbinding rather than only imperfect registration quality.
- Operation-result calibration and verification: In a lung PET/CT case, the agent discarded an initially close VISTA3D segmentation and used an incorrect manual point to prompt a self-confirming result.The failure was not uniformly poor segmentation; it was failure to arbitrate between operation outputs and anatomical plausibility.
- Procedural stability of advanced operations: Only 3 of 10 registration runs completed the expected BRAINSFit →resample/apply-transform →registered-fusion sequence, while segmentation outputs ranged from plausible masks to seed-only or no-mask results.Repeated runs exposed procedural instability in composing advanced operations into a stable workflow.
- Synthesis: Auditable traces expose failures in objective preservation, spatial grounding, state tracking, output calibration, and procedural control that final answers alone can miss.The traces record operation arguments, viewer states, generated artifacts, and evidence chains.
7 Discussion and Conclusion
The paper reframes medical-agent evaluation as dynamic, auditable interaction with professional software and complete studies. Its findings show that correct answers alone can conceal unsupported evidence and insufficient workflow competence.
- Discussion: Evidence-constrained evaluation distinguishes correct answers from evidence that is wrong, missing, or ungrounded.Strict metrics require checkable slices, coordinates, regions, or longitudinal evidence alongside the task answer.
- Discussion: Advanced segmentation, registration, and expert-model operations do not automatically improve diagnostic accuracy.Reliable execution still requires spatial grounding, parameterization, software-state tracking, and artifact verification.
- Discussion: Operating 3D Slicer or QuPath requires maintaining a clinical objective while manipulating persistent viewer state and selecting relevant operations.
- Conclusion: MEDOPENCLAW provides a secure, bounded, auditable runtime, while MEDFLOW-BENCH evaluates full-study interaction with checkable evidence.
A Limitations and Roadmap
The benchmark is intentionally a controlled evaluation environment rather than a clinical-deployment system. Its current scope is bounded by pathology operations, cohort coverage, trace comparability, and proxy reference standards.
- Limitations: The benchmark targets controlled evaluation rather than clinical deployment.
- Limitations: Current pathology experiments emphasize QuPath navigation and coordinate-grounded evidence, while richer operations and larger whole-slide cohorts remain extensions.
- Limitations: Runtime-agnostic methods enable pipeline comparison but do not necessarily produce the same auditable software trace.
- Limitations: Public labels and hidden annotations provide reproducible reference standards but remain proxies for full clinical adjudication.
- Runtime scope: The callable viewer interface remains strictly explicit, with REST-wrapped Slicer functions and named bridge handlers for operations outside that interface.
C Benchmark Modules, Dataset Sources, and Scoring
This appendix documents benchmark modules, eligible units, task sources, and scoring rules. MEDFLOW-BENCH combines task answers with checkable evidence under deterministic protocols.
- Benchmark overview: Table 4 summarizes benchmark modules, input units, case counts, primary tasks, and evidence requirements.
- Eligibility: NSCLC includes 162 eligible cases by default after AMC exclusion, or 211 cases when AMC cases are included.
- Eligibility: LUMIERE retains 139 longitudinal questions from 360 after evidence-availability and label-frequency filtering.
- Scoring: Radiology Strict scoring requires answer correctness under benchmark-specific evidence constraints, while pathology evidence and localization use deterministic point lookup.
C.1 Module construction and evaluation protocols
The benchmark combines radiology and pathology modules with deterministic answer-and-evidence protocols. It also includes a static slice-montage baseline that removes interactive viewer capabilities for runtime-agnostic comparison.
- Module construction: Radiology modules cover longitudinal MRI lesion-evolution reasoning, multi-sequence brain-tumor diagnosis, and paired NSCLC PET/CT analysis.
- Evaluation protocols: Radiology Strict scoring requires the final answer and task-specific evidence to be correct under module-specific rules.
- Evaluation protocols: UCSF-PDGM uses three key-slice offsets for Strict scoring, while its RAS point is scored separately for localization.
- Evaluation protocols: NSCLC Strict scoring requires all three key slices and an RAS point inside the hidden lesion mask, otherwise the case receives zero.
- Evaluation protocols: Pathology evidence and localization use deterministic coordinate hits against annotations, masks, or XML polygons rather than human or LLM judging.
- 2D slice-montage baseline: The 2D slice-montage baseline compresses each volume into a static grid, removing dynamic scrolling, cross-plane confirmation, fusion, measurements, and coordinate conversion.
D Runtime-Agnostic Track Results
The runtime-agnostic track compares alternatives with 3D Slicer viewer-native runs under the same answer-and-evidence schema, while diagnostic examples clarify auditable interaction regimes and workflow failures. Differences are task-dependent, and the cases show that locally plausible operations can still fail when workflow intent becomes unstable.
- Runtime-agnostic comparison: Runtime-agnostic baselines return canonical answers and evidence fields from rendered or volumetric inputs, but do not produce replayable viewer-state traces.They are scored with the same parsers as viewer-native runs, making the comparison task-dependent and primarily diagnostic.
- Interaction regimes: Figures 4–6 illustrate primitive viewer-native interaction, segmentation-assisted evidence capture, and MONAI VISTA3D advanced operations within auditable episodes.The examples show how agents inspect studies, localize lesions, invoke selected operations, and record structured evidence such as masks, key slices, and RAS coordinates.
- Operation budget: The diagnostic study varies the maximum number of explicit operation calls available to the agent as an operation-budget control.The supplied passages identify Table 7 as the budget-control study and note that it reports results for Qwen3.5-35B-A3B and Gemini 3.1 Pro Preview.
- Workflow failure analysis: Longitudinal response assessment failed when the agent mixed cross-timepoint CT1 registration, CT1-derived FLAIR resampling, and intra-timepoint FLAIR-to-CT1 alignment.The trace contained plausible intermediate artifacts and repeated registration calls, but the final workflow no longer supported the intended longitudinal comparison.
G.2 Case F2: Pseudo-quantification from ungrounded coordinates
Case F2 shows how weak screen-space localization can be converted into apparently precise quantitative evidence. The resulting measurements support an incorrect longitudinal response assessment because they lack validated spatial and measurement grounding.
- Observed outcome: The agent predicted progressive disease from an approximately 119% SPD increase derived from ungrounded lesion coordinates.The ground-truth response was partial response, whereas the prediction was progressive disease.
- Failure mechanism: The measurements came from weakly grounded screen-space boxes converted into RAS coordinates and interpreted as lesion diameters.The workflow did not validate lesion boundaries, slice planes, or the longitudinal frame before producing millimeter-scale evidence.
- Benchmark implication: This case exposes spatial grounding failure: uncertain localization was elevated into quantitative RANO-style reasoning without validated boundaries, segmentation, registration, or measurement procedures.The issue is therefore more fundamental than an inaccurate size estimate; the evidence itself was not clinically grounded.
G.3 Case F3: Transform-layer misbinding in registered MRI fusion
This case shows that registration failures arose from misbinding transforms, image nodes, viewer layers, and reported evidence, producing unstable or non-auditable fusion despite plausible operation objects.
- Transform-layer misbinding: The executed trace resolved a CT1 registration request to the FLAIR transform and transformed the wrong nodes.The requested CT1 transform was not the transform actually applied during execution.
- Evidence mismatch: The viewer fusion used baseline and follow-up FLAIR layers, conflicting with the final rationale describing registered CT1 views.The displayed background and foreground were both FLAIR volumes, whereas the rationale cited registered CT1 volumes.
- Transform-layer misbinding: The CT1 transform lacked its expected CT1 child node, while the FLAIR transform contained both follow-up CT1 and FLAIR nodes.This inconsistent artifact binding explains why nominal registration objects did not guarantee correct displayed layers.
- Operation-result calibration: The agent discarded an initially useful segmentation candidate, selected an incorrect point, and used the resulting localization as final coordinate evidence.This demonstrates failure to arbitrate between operation outputs or verify anatomical plausibility before reporting evidence.
- Repeated-workflow instability: Only 3 of 10 repeated registration runs completed the expected BRAINSFit, resampling, and fusion-verification sequence.Other runs omitted verification, fused unregistered volumes, or repeated intermediate calls without closing the workflow.
- Repeated-workflow instability: Identical inputs produced markedly different final registration views and segmentation outputs across repeated runs.The instability affected both displayed image evidence and segmentation masks, including broad over-segmentation, local masks, seed-only evidence, and failures.