Source-linked AI summary
Layer Selection in VLMs for Zero-Shot OOD Detection via Multi-Resolution Entropy Estimation
Shyam Nandan Rai, Francesco Di Salvo, Sebastian Doerrich, Christian Ledig
TL;DR
Medical zero-shot OOD detection with VLMs relies on image–text alignment, but existing methods largely use final-layer representations despite modality-dependent OOD signals. The paper selects intermediate layers using multi-resolution entropy estimation and reports consistent gains over state-of-the-art zero-shot baselines across medical benchmarks and VLM backbones.
Problem
Existing VLM-based zero-shot OOD methods primarily rely on final-layer representations, while whether aggregating layer-wise scores improves medical image detection remains unexplored.
Source: §1 · pp. 1–2Method
The method selects intermediate-layer subsets by aggregating histogram entropy estimates across multiple discretization scales.
Source: §1 · p. 2, §3.2 · p. 4Results
Across two medical OOD benchmarks and two VLM backbones, the method consistently improves over state-of-the-art zero-shot baselines; single-resolution binning sensitivity can degrade AUROC by up to 19.3%.
Source: §1 · p. 2Takeaways & Limitations
The results support multi-resolution entropy estimation as a simple, effective strategy for zero-shot OOD detection across datasets and backbones.
Source: §6 · p. 7The approach depends on the VLM backbone and has been evaluated in two medical modalities; further validation across imaging domains is needed.
Source: §6 · p. 7
Abstract
from arXiv · showhide
Out-of-distribution (OOD) detection is crucial for safe deployment of medical AI systems, where domain shifts arise across institutions, acquisition protocols, and patient populations. VLMs enable zero-shot OOD detection by embedding images into a language-aligned latent space, where cross-modal similarity serves as a non-parametric confidence signal for identifying in-distribution samples. Yet existing methods rely almost exclusively on final-layer embeddings, implicitly assuming that the deepest representations are universally optimal. We first show that this assumption does not hold in medical imaging: intermediate layers provide complementary OOD signals, and the optimal representational depth depends on the respective image modality. While prior work selects layer combinations via entropy minimization of normalized histograms, we demonstrate that single-resolution entropy estimation is highly sensitive to binning choices, leading to performance variations of up to 19.3% AUROC. To address this instability, we propose a multi-resolution entropy estimation strategy that aggregates histogram statistics across multiple discretization scales, enabling robust and stable intermediate-layer selection. Across two medical OOD benchmarks, namely MIDOG and OASIS, covering distinct imaging modalities, diverse shift types, and different VLM backbones, our method consistently outperforms state-of-the-art approaches, offering a lightweight and stable solution for zero-shot OOD detection.
1 Introduction
Zero-shot VLM-based OOD detection uses image–text alignment to assess samples in medical settings with shifts across institutions, devices, protocols, and patient populations. This paper examines modality-dependent layer signals and proposes multi-resolution entropy estimation for more stable intermediate-layer selection.
1 Introduction
- The paper investigates whether aggregating layer-wise OOD scores can improve zero-shot detection in medical image analysis, where existing VLM methods primarily use final-layer representations.VLMs use image–text alignment as a confidence signal, and prior work reports OOD-discriminative signals across multiple depths.Source: §1 · pp. 1–2
- Texture-driven histopathology shifts are primarily captured in earlier layers, whereas semantic and anatomical brain MRI shifts benefit from deeper representations.The best representational depth therefore depends on modality, rather than always being the final layer.Source: §1 · p. 2, Figure 1 · p. 2
- Across two medical OOD benchmarks and two VLM backbones, the proposed method consistently improves OOD detection over state-of-the-art zero-shot baselines.It also reduces bin-size sensitivity through multi-resolution entropy estimation.Source: §1 · p. 2
- Single-resolution histogram entropy can make layer selection unstable in small medical datasets, with discretization sensitivity degrading AUROC by up to 19.3%.The proposed multi-resolution strategy aggregates entropy estimates across discretization scales to support robust, stable selection.Source: §1 · p. 2
2 Intermediate layers for medical OOD detection
A layer-wise analysis with UniMedCLIP shows that medical OOD signals vary with imaging modality, motivating selection of intermediate representations rather than relying only on the final layer.
2 Intermediate layers for medical OOD detection
- Intermediate representations are projected with the final layer’s pretrained projection head into the shared image–text space, then scored using layer-wise MCM.The encoder has N sequential layers, with L_N(I) as its canonical final-layer embedding.Source: §2 · p. 3
- For MIDOG histopathology, early layers perform best; for OASIS brain MRI, mid-level layers outperform the final-layer representation.The study uses UniMedCLIP and Maximum Concept Matching (MCM), averaging performance across near- and far-OOD shifts.Source: §2 · p. 3, Figure 2 · p. 3
- The proposed selection framework evaluates candidate layer subsets that include the final layer, collecting in-distribution scores and selecting the subset with minimum multi-resolution entropy.The resulting selected subset is used for test-time inference.Source: Figure 2 · p. 3
3 Method
The method scores image–text alignment with VLMs, then selects intermediate-layer combinations using entropy estimated across multiple histogram resolutions. The selected subset is determined from in-distribution data and used at test time.
3 Method
- The method is proposed as a robust way to select intermediate layers because OOD signals emerge at modality-dependent representational depths.The method follows the observation that optimal depth varies with imaging modality.Source: §3 · p. 3
3.1 Preliminaries
- Maximum Concept Matching scores an image by its highest softmax probability over similarities to in-distribution text prompts.The score is computed over image–text similarities.Source: §3.1 · p. 4
3.2 Multi-resolution entropy estimation
- Candidate layer subsets always include the final layer, and their aggregated layer-wise MCM scores across in-distribution images form the distributions used for entropy estimation.The method evaluates all possible subsets of intermediate layers while enforcing final-layer inclusion.Source: §3.2 · p. 4
- The method averages entropy estimates across histogram resolutions and selects the subset with minimum average entropy.The histogram ensemble uses K binning scales; the selected subset is determined using in-distribution data and remains fixed for every test sample.Source: §3.2 · p. 4
4 Experimentation
Experiments compare zero-shot OOD detection methods across MIDOG and OASIS, two medical imaging benchmarks, using two VLM backbones and near- and far-OOD shifts. The proposed method performs strongly across most configurations, although BioMedCLIP results on MIDOG are mixed.
Datasets and metrics
- Evaluation covers MIDOG histopathology and OASIS brain MRI, with near- and far-OOD shifts, using AUROC and FPR95.PhaKIR was excluded because the VLM backbones were not pretrained on surgical data, making predictions on that dataset unreliable.Source: Datasets and metrics · p. 5
Implementation details
- Experiments use BioMedCLIP and UniMedCLIP in a training-free, inference-only setting, with the same flat prompt set across methods.The method and single-resolution baseline select subsets of at most four layers, including the final layer, using in-distribution data.Source: Implementation details · p. 5
Results
- On MIDOG with UniMedCLIP, near-OOD AUROC is 54.6% and FPR95 is 94.4%, while far-OOD AUROC is 97.4% and FPR95 is 13.4%.The paper reports the best near- and far-OOD results for this backbone and dataset.Source: Results · p. 5
- Across MIDOG and OASIS, the method achieves the best or competitive performance in nearly all configurations; BioMedCLIP results on MIDOG are mixed.MCM has the best near-OOD results and De la Jara et al. the best far-OOD results for BioMedCLIP on MIDOG.Source: Results · p. 5, Table 1 · p. 6
- Approximately 20% lower FPR95 on OASIS with BioMedCLIP versus the strongest baseline in both near- and far-OOD settings.Table 1 spans both backbones, datasets, and shift settings; the reported largest gains are in FPR95.Source: Results · p. 5, Table 1 · p. 6
5 Ablation studies
The ablations show that intermediate-layer aggregation can improve OOD detection, while its benefit depends on the backbone; multi-resolution entropy estimation avoids the bin-size sensitivity of single-resolution selection.
5 Ablation studies
- The ablations examine the number of aggregated intermediate layers and histogram-bin sensitivity, using UniMedCLIP unless otherwise stated.The bin-count sweep uses OASIS (near), a setting with fewer than 1,000 in-distribution samples.Source: §5 · p. 6, §5.2 · p. 6
5.1 Number of selected layers
- On MIDOG (near), adding selected intermediate layers consistently improves UniMedCLIP AUROC, while BioMedCLIP AUROC eventually decreases as more layers are aggregated.The analysis uses MIDOG because OASIS near AUROC is near-saturated; selecting layers benefits BioMedCLIP compared with using all layers.Source: §5.1 · p. 6, Figure 3 · p. 7
5.2 Sensitivity of histogram bins
- On OASIS (near), single-resolution selection varies by up to 19.3%, while multi-resolution entropy estimation achieves stable, near-optimal performance.The single-resolution analysis has standard deviation σ = 8.2%; the multi-resolution approach avoids manual bin-size selection.Source: §5.2 · p. 6, Figure 3 · p. 7
6 Discussion
The paper concludes that modality-dependent OOD signals support intermediate-layer aggregation, while noting backbone dependence and limited modality coverage. It also identifies uniform layer averaging as a constraint and prompt-wise score distributions as a future direction.
6 Discussion
- The approach remains dependent on the VLM backbone, and further validation is needed beyond the two evaluated medical imaging modalities.The paper also notes that uniform averaging may miss optimal weighted layer combinations.Source: §6 · p. 7
- The authors identify finer-grained scoring beyond MCM, such as using entire prompt-wise score distributions, as a future direction.The paper concludes that multi-resolution entropy estimation offers a simple strategy for enhancing zero-shot OOD detection across datasets and backbones.Source: §6 · p. 7