Source-linked AI summary
Do Medical Vision Models Reason About Anatomy? Probing the Spatial Inductive Biases of Learned Visual Representations
Naren Akash, Neeraja Ramanan
TL;DR
It remains unclear whether medical vision representations support spatial comparisons needed for radiological interpretation. Using SPAR-Bench, the paper finds that encoders retain canonical organ-position information but struggle with patient-specific relational computation, while pooled probing and system-level evaluation can obscure these differences.
Problem
Diagnostic accuracy does not reveal whether medical vision representations support localization, relational reasoning, and spatial queries needed for radiological interpretation.
Method
The authors construct SPAR-Bench, eight probes over multi-organ abdominal CT that separate coordinate localization, relational reasoning, and spatial queries across architectures and medical foundation models.
Results
Encoders perform well on canonical anatomy in-domain but relational and spatial probes often collapse under zero-shot transfer; reading identical frozen features with full tokens raises relational recovery from 0.7% to 67.8%.
Takeaways & Limitations
The evaluated encoders carry information about typical organ locations but little machinery for comparing structures within an individual scan, and pooled probing understates their representations.
Takeaways & Limitations
The evaluation does not isolate where encoder–MLLM differences arise, and the probes still require testing in 3D, other body regions, and other modalities.
Abstract
from arXiv · showhide
Interpreting a CT scan means comparing structures on either side, judging how far apart organs sit, and knowing where each one belongs. Medical vision encoders are evaluated on diagnostic accuracy, or through assembled multimodal systems where a failure is hard to attribute, so it remains unclear whether their representations support any of this. We construct SPAR-Bench, eight probes over multi-organ abdominal CT that separate coordinate localization, relational reasoning, and spatial queries, and apply them to five architectural configurations and three medical foundation models, frozen and finetuned. Probes that ask for a comparison within the slice stay at chance, and neither pretraining scale, finetuning, nor architecture closes the gap. Probes that appear solved in domain fall to chance under zero-shot transfer, indicating that their accuracy reflects recall of canonical anatomy rather than computation over the image. Reading the same frozen features with a pooled head rather than the full set of tokens moves relational recovery from 0.7% to 67.8%, so pooled probing understates what a representation holds. Questions the encoders answer well are answered at chance by four open-weight MLLMs. Our results suggest these encoders carry a map of where organs usually lie, and little of the machinery for comparing structures within a particular patient. Code and data will be available at https://spar-bench.github.io.
1 Introduction
Radiological interpretation depends on spatial relationships among anatomical structures, but diagnostic evaluations do not reveal whether models represent or compute those relationships. SPAR-Bench is introduced to probe localization, relational reasoning, and spatial queries directly.
- Spatial coordination matters clinically because interpretation uses canonical position, proximity to landmarks, and contralateral comparison.For example, staging a pancreatic mass requires quantifying its contact with the superior mesenteric artery.
- Diagnostic accuracy can hide whether a model represents the geometry underlying anatomical judgments.Correct findings may arise from texture or context rather than locating, measuring, and comparing structures.
- The study targets a gap left by prior work focused mainly on preserving spatial information or reconstructing disrupted inputs.Those analyses do not directly test whether vision encoders reason about anatomical space.
- SPAR-Bench separates spatial competency into coordinate localization, relational reasoning, and six spatial-query types.The benchmark evaluates organ centroids, shuffled anatomical arrangements, and questions about relative position, symmetry, size, distance, and counting.
- The benchmark evaluates five architectural configurations and three medical foundation models as frozen or finetuned encoders.
2 Related Works
Prior medical vision research uses spatial structure for representation learning, but existing benchmarks do not directly test whether vision encoders reason over anatomical space. Open questions remain about scaling and architectural inductive bias.
- Self-supervised medical methods use spatial disruption and restoration, including jigsaw solving, Rubik’s cube recovery, and patch rearrangement.
- Existing benchmarks cover natural vision and medical multimodal language models, but not encoder reasoning about anatomical space.
- Medical foundation models are often evaluated through classification transfer, which can succeed using local texture without spatial reasoning.
- The paper examines whether medical pretraining scale and CNN-versus-ViT spatial mechanisms affect functional spatial reasoning.Prior work raised doubts that scaling alone improves spatial understanding, while architectural differences in spatial information handling had not been functionally tested.
3 Spatial Anatomical Reasoning (SPAR) Benchmark
SPAR-Bench constructs mask-derived abdominal CT probes across three levels: localization, permutation-based relational recovery, and whole-slice spatial queries. Its design removes surrounding context for tile tasks and defines spatial reasoning as composing measurements within one image.
- Benchmark Construction: SPAR-Bench uses 150 WORD abdominal CT volumes with voxel-level masks for 16 organs, split into 100 training, 20 validation, and 30 test volumes.The dataset also includes 20 LiTS volumes with tumor annotations, totaling 170 volumes.
- Benchmark Construction: Cropping Levels 1 and 2 removes neighboring context, requiring correct tile placement to rely on position encoded in the organ’s own appearance.
- Benchmark Construction: The benchmark defines spatial reasoning as composing at least two spatial measurements within one image, excluding probes requiring unsupported 3D context.Containment and vessel adjacency are outside the support of 2D axial slices.
- Level 1: Localization: Level 1 predicts normalized organ-centroid coordinates from 16 tiles and reports Mean Euclidean Distance and Success Detection Rate at 10 pixels.
- Level 2: Relational Reasoning: Level 2 shuffles 16 organ tiles and evaluates recovery of the centroid-based spatial arrangement rather than a fixed class order.A Sinkhorn-normalized permutation matrix is converted to hard assignments with the Hungarian algorithm.
- Level 3: Spatial Queries: Level 3 pairs whole 224×224 windowed CT slices with six heads covering relative position, orientation, bilateral symmetry, organ size, distance, and counting.The probes use classification targets derived from slice geometry and tumor annotations.
- Architectural Design: The study factors CNN versus ViT backbones and MLP versus Transformer heads in a 2×2 design plus a unified configuration.ResNet-50 and ViT-B/32 are used to separate convolutional locality from global attention without confounding hybrid architectures.
- External Evaluation: External evaluation tests the trained configurations on TotalSegmentator without retraining and asks the same Level 3 questions to four open-weight MLLMs.
4 Results and Discussion
Across SPAR-Bench, spatial competence is uneven: some in-domain scores are high, but relational and patient-specific comparisons often fail under transfer or depend strongly on token access. Encoder and MLLM performance also diverge, while finetuning trades localization for relational accuracy.
- Architectural and model comparisons: Permutation accuracy ranges from 0.7% for frozen BiomedCLIP+MLP to 82.0% for finetuned RadDINO+Trans, confirming the task is learnable.A ViT trained from scratch reaches 76.2%, so low scores are not attributable solely to task difficulty.
- Architectural and model comparisons: Frozen features yield 0.7% permutation accuracy with an MLP head but 67.8% with a Transformer head, showing pooling can hide relational structure.The MLP reads a pooled vector, whereas the Transformer reads the full token set.
- Finetuning effects: Finetuning RadDINO improves permutation accuracy by 47 points with an MLP head and 18 with a Transformer head, while degrading localization from 9.88 to 11.46 px for the MLP.The observed shift favors relational accuracy over localization, although the study measures task accuracy rather than representation geometry.
- Zero-shot transfer: Relative position exceeds 98% in domain but falls to 50.03 ± 0.04 for frozen RadDINO+MLP on TotalSegmentator, consistent with canonical-anatomy recall rather than image-based comparison.Orientation and localization also fall under transfer, while distance and organ size hold, indicating a selective rather than uniform collapse.
- Zero-shot transfer: All 13 configurations score 25.6–36.4% on symmetry, and neither architecture, pretraining scale, nor finetuning moves performance beyond chance.Symmetry remains in the 27.3–34.9% band on TotalSegmentator and is also near chance for the MLLMs.
- Encoder and system competence: Relative position is near ceiling for encoders but 49.5–50.1% for all four MLLMs, while organ-size performance separates the MLLMs across a 75-point range.Because multiple components intervene between encoder and system measurements, competence at one level should not be assumed at the other.
5 Conclusion
Spatial scores must be tested across representations, probe heads, and cohorts because capabilities that appear established can disappear when any one changes. The benchmark also leaves open whether these findings generalize beyond 2D abdominal CT and the tested modalities and body region.
- 5 Conclusion: Spatial scores depend jointly on the representation, probe head, and cohort used for measurement.Changing each factor while holding the others fixed caused some apparently established capabilities to fail.
- 5 Conclusion: Probes with answers that are consistent across patients can reward recall of canonical anatomy rather than computation over an individual image.The authors argue that probes without canonical answers are valuable because the evaluated systems did not solve them.
- 5 Conclusion: The same probes should be tested in 3D, other body regions, and other modalities.These settings define the stated scope boundary for the reported conclusions.
Supplementary Material
Supplementary results provide per-organ and per-configuration detail for localization, permutation, and zero-shot spatial-query transfer. They show organ-dependent difficulty, recovery by Transformer heads on some variable organs, and persistent near-chance symmetry under transfer.
- Localization: Localization error is lowest for kidneys and highest for colon and intestine across every configuration.Table 3 reports MED in pixels, averaged over three seeds, with lower values indicating better localization.
- Relational Reasoning: Frozen MLP heads fall to near zero on esophagus permutation, whereas a Transformer head on the same features recovers it.This per-organ pattern supplies detail behind the broader pooling effect discussed in the main text.
- Zero-Shot Transfer: Symmetry stays near chance for all 13 configurations in zero-shot transfer, while relative position and orientation vary across configurations.Table 5 reports Level 3 transfer to TotalSegmentator without retraining.