Source-linked AI summary
A Modular Agent for Reliable and Auditable Spatial Relation Verification in CT Scans
Simon Vincent Abel, Heiko Hillenhagen, Michael Götz, Timo Ropinski, Ayhan Can Erdur, Daniel Santak Wolf
TL;DR
Current VLMs remain weak at controlled, image-grounded spatial reasoning needed for radiological understanding. The paper introduces a modular agent that parses queries, localizes anatomy, and verifies relations geometrically, with the best hybrid configuration achieving 94.1% accuracy and 94.2% F1. The study positions explicit spatial verification as a building block for future report-oriented imaging agents.
Problem
VLMs struggle to reliably ground anatomical spatial relations in image evidence, despite spatial understanding being important for radiological reporting.
Method
A modular agent parses natural-language queries, matches anatomical entities, localizes them with YOLO, and applies deterministic geometric verification.
Results
94.1% accuracy and 94.2% F1 were achieved by Qwen2-VL + hybrid agent, a 42.5-percentage-point accuracy gain over direct Qwen2-VL prompting.
Takeaways & Limitations
Explicit modular spatial verification provides interpretable intermediate representations and enables module-level attribution of errors.
Takeaways & Limitations
The study is limited to spatial verification in 2D axial CT slices and does not address full volumetric reasoning or several richer spatial relations.
Abstract
from arXiv · showhide
Reliable spatial understanding is an important prerequisite for future medical vision-language systems that aim to support radiological report generation and structured image understanding. While modern vision-language models (VLMs) show promising performance on many medical imaging tasks, recent evidence suggests they remain weak in controlled spatial reasoning and often fail to reliably ground spatial relations in image evidence. Given that radiological reasoning hinges on understanding the relative positions of anatomical structures and findings, this spatial weakness poses risks to diagnostic accuracy. We present a modular medical imaging agent for binary spatial relation verification in axial CT slices. Instead of directly predicting spatial answers end-to-end, the system decomposes the task into explicit stages: language parsing, anatomical localization, and deterministic geometric verification. Natural-language queries are converted into structured relation tuples, queried organs are localized with a YOLO-based detector, and the final spatial decision is computed from object centers using deterministic geometric rules. We evaluate the approach on the held-out MIRP spatial QA benchmark and compare it against representative end-to-end VLM baselines. The best-performing hybrid configuration reaches 94.1% accuracy and 94.2% F1, outperforming direct Qwen2-VL prompting by 42.5 percentage points in accuracy, while preserving interpretable intermediate representations and auditable reasoning stages. The results suggest that explicit modular spatial verification can serve as a promising building block for future report-oriented medical imaging agents.
1 Introduction
The paper targets reliable, auditable verification of anatomical spatial relations, addressing weaknesses in end-to-end VLM spatial reasoning. It proposes a modular agent that exposes intermediate representations and delegates verification to specialized components.
- Motivation: VLMs used in radiology must ground spatial statements in verifiable image evidence because anatomical localization is central to reporting.The introduction connects spatial language to radiological reporting and notes potential consequences of localization failures.
- Motivation: Agentic systems expose intermediate representations and delegate reasoning to specialized tools rather than relying on monolithic prediction.The proposed framework applies this approach specifically to controlled, image-grounded, auditable anatomical relation verification.
- Contribution: The architecture is intended as a foundation for adding tools for other medical reasoning tasks and structured report generation.The paper specifically mentions presence verification, measurement-oriented reasoning, and report generation as possible extensions.
- Contribution: The system decomposes spatial reasoning into language parsing, anatomical localization, and geometric verification, enabling explicit attribution of failures to pipeline stages.The architecture is presented as a controlled subproblem of report-oriented radiological image understanding.
- Contribution: The work evaluates a modular spatial-verification pathway against end-to-end VLM baselines on the MIRP spatial question-answering benchmark.The evaluation uses a unified protocol for comparing the proposed agent with representative VLM systems.
2 Method
The spatial pathway converts a natural-language query into structured entities, localizes those entities in an axial CT slice, and computes the relation deterministically from detected object geometry. Explicit intermediate states support structured outputs and fine-grained auditing.
- Architecture: The agent preserves a conventional VLM interface while routing image-grounded spatial questions through an explicit verification pathway.A VLM controller receives the image and prompt, then selects the specialized task pathway.
- Spatial verification pathway: The pathway extracts a relation tuple, matches entities to detector classes, localizes structures with YOLO, and verifies spatial ordering from object centroids.Unsupported ontology matches return an invalid result with an audit entry, while the detector passes the highest-confidence box per class onward.
- Spatial verification pathway: The final binary truth value is computed from horizontal or vertical coordinate comparisons rather than neural answer prediction.This design uses tool-provided coordinates to make the spatial decision explicit.
- Auditability: The agent records prompts, parsed queries, ontology matches, detections, binary results, and audit information as intermediate states.These records make the returned result structured and verifiable.
- Auditability: Explicit stages allow each incorrect or invalid case to be assigned to its earliest failing component.Possible categories include parsing, ontology matching, missing detections, localization, geometric ambiguity, and runtime errors.
3 Experimental Setup
The experiments compare direct VLM answers with internally modularized answers under the same external prompts and deterministic decoding. Evaluation uses axial CT spatial questions and reports accuracy, F1, invalid-output rate, and stage-wise failures.
- Evaluation design: The study evaluates MedGemma 4b and Qwen2-VL 7b both directly and within the hybrid agent on the MIRP RQ1 benchmark.The framework is model-agnostic, but these two publicly weighted VLMs instantiate the comparison.
- Evaluation design: Direct VLMs receive the CT slice and full prompt, whereas the hybrid agent extracts, routes, and verifies the same question modularly.All systems therefore share the same user input while differing in internal reasoning architecture.
- Evaluation protocol: Direct VLM decoding uses temperature 0, and the benchmark prompt requires exactly one binary character as the answer.The prompt defines 1 as true and 0 as false for each spatial statement.
- Evaluation protocol: Evaluation reports accuracy, F1 score, and invalid-output rate, counting invalid, failed, or non-binary outputs as incorrect.Hybrid-agent errors are additionally assigned to the earliest failing stage.
4 Results
The hybrid agent substantially outperformed direct VLM prompting on MIRP spatial verification, while exposing intermediate outputs for stage-wise failure analysis.
- 94.1% accuracy and 94.2% F1 were achieved by Qwen2-VL + hybrid agent, exceeding direct Qwen2-VL prompting by 42.5 percentage points in accuracy.MedGemma + hybrid agent reached 91.6% accuracy, while direct VLM baselines remained near 52% accuracy.
- Stage-wise failure attribution: The hybrid agent returns parsed questions, matched classes, detections, geometric comparisons, answers, and audit information for inspecting successes and failures.Each failure is assigned to the earliest pipeline stage preventing the correct answer.
- Stage-wise failure attribution: 50.9% of the 55 incorrect cases arose from imprecise YOLO localization, followed by parsing/query-extraction errors at 27.3% and missing detections at 21.8%.No errors were attributed to question extraction, routing, ontology matching, geometry ambiguity, or formatting/runtime failures under the attribution rule.
- Accuracy and F1 in Table 1 are percentages, but prior leaderboard baselines were not re-evaluated and therefore are not strictly paired comparisons.The evaluated results use a held-out test set that differs from the leaderboard data.
5 Discussion
With identical external prompts, explicit modular reasoning improved spatial verification by replacing implicit prediction with detections, deterministic comparisons, and auditable intermediate representations.
- The unified prompt protocol isolates internal reasoning as the difference between direct VLMs and the hybrid agent.Both receive the same CT slice and prompt, while the hybrid routes the question through spatial verification.
- The modular agent grounds spatial decisions in anatomical detections and deterministic coordinate comparisons rather than implicit multimodal prediction.Correct extraction, ontology mapping, and detections allow the final decision to be traced to a coordinate comparison.
- Table 2 attributes failures among 55 incorrect or invalid outputs of Qwen2-VL + hybrid agent to the earliest identifiable failing stage.Fallback cases without a separately stored raw query were conservatively attributed to parsing/query extraction.
- The study evaluates pairwise 2D spatial relations in axial CT slices, not volumetric reasoning, distance-sensitive relations, containment, overlap, or multi-structure consistency.The architecture is described as extensible to future 3D routing or volumetric measurement tools.
6 Conclusion
The paper presents explicit tool-based spatial verification in axial CT as a proof of concept that improves performance and makes errors attributable to specific modules.
- The modular agent decomposes language-grounded spatial verification into explicit stages and replaces end-to-end prediction with tool-based verification.The framework targets axial CT slices and exposes intermediate reasoning stages.
- The hybrid agent achieves markedly higher accuracy than standard end-to-end VLMs, which operate near chance level on the spatial reasoning task.
- Exposed intermediate stages enable errors to be attributed to specific modules rather than remaining hidden inside end-to-end multimodal prediction.
- The work establishes a foundation for extending the modular agentic framework to additional auditable clinical pathways.The paper describes spatial relation verification as an initial proof of concept for explicit tool use in medical image reasoning.