PaperlayerView exact revision on arXiv

Source-linked AI summary

What Do Medical Vision-Language Models Learn in Radiology? Transfer, Alignment, and Source-Proxy Leakage Under Distribution Shift

Ayoub Louaye Bouaziz, Lokmane Chebouba, Yassine Himeur

arXiv:2608.25251v1cs.CVcs.AI

TL;DR

Medical VLMs may look reliable in-domain while their representations fail under shifts in acquisition domain, paired supervision, or evaluation protocol. This paper uses leakage-aware cross-dataset transfer, multimodal retrieval, and source-proxy probes to stress-test that gap, finding improved matched transfer with BYOL, unstable stronger adversarial adaptation, weak external OpenI exact-pair retrieval, and recoverable source-proxy signals. The findings support explicit stress testing and precise interpretation of evaluation protocols.

  • Problem

    Medical VLMs can appear reliable in-domain while representation usefulness changes under acquisition, supervision, or evaluation shifts.

  • Method

    The study stress-tests representations using NIH-to-CheXpert visual transfer, PadChest-to-OpenI multimodal evaluation, strict pair-index retrieval, and metadata-derived source-proxy probes.

  • Results

    BYOL improves matched NIH-to-CheXpert transfer over ImageNet, stronger adversarial adaptation becomes unstable, OpenI exact-pair retrieval remains weak, and source-proxy signals remain recoverable.

  • Takeaways & Limitations

    Apparent competence under a single protocol can conceal transfer, alignment, and shortcut-related failure modes, motivating stress-tested evaluation under distribution shift.

  • Takeaways & Limitations

    The study uses one paired training corpus and one external stress-test dataset, exact-index retrieval lacks a validated equivalence map, and archived transfer values lack seed-level records for significance testing.

Abstract

from arXiv · show

Medical vision-language models (VLMs) can appear reliable in-domain while failing when acquisition domain, paired supervision, or evaluation protocol changes. We study this failure mode as a representation-level blind spot relevant to epistemic intelligence, without claiming a formal estimator of epistemic uncertainty. Using NIH ChestXray14 and CheXpert, we first isolate source-only cross-dataset visual transfer from unsupervised domain-adaptation diagnostics. Using PadChest and OpenI, we then evaluate multimodal alignment under strict pair-index retrieval and quantify metadata-derived source-proxy information retained in frozen embeddings. Self-supervised visual initialization improves NIH-to-CheXpert transfer over supervised ImageNet initialization in matched ResNet-18 comparisons, whereas adversarial adaptation is useful only in a narrow regime and becomes unstable as adversarial pressure increases. Multimodal exact-pair retrieval remains low under external OpenI stress testing, and source-proxy information remains recoverable from learned representations. Qualitative nearest-neighbor and Grad-CAM analyses show clinically plausible cross-dataset structure and thoracic attention patterns in many cases, while device-heavy and false-positive cases remain ambiguous. Auxiliary architecture checks are task-dependent and do not support a universal backbone ranking. Overall, the study shows that apparent competence under a single protocol can conceal transfer, alignment, and shortcut-related failure modes, motivating stress-tested evaluation of medical VLMs under distribution shift.

1. Introduction

The paper examines why medical VLM representations that appear competent in-domain may fail when datasets, supervision, or evaluation protocols change. It frames transfer, multimodal alignment, and source-proxy recoverability as distinct stress-test targets rather than formal epistemic-uncertainty estimation.

  • Motivation: Chest X-ray models can degrade across hospitals when acquisition pipelines, patient populations, and reporting conventions change.Such shifts can encourage reliance on scanner, view, or documentation regularities that do not transfer with pathology.
  • Motivation: Strong in-domain performance does not establish representation usefulness under changes in image domain, report style, or supervision regime.Robustness, grounding, and evaluation under dataset variation remain open problems for medical VLMs.
  • Research Question: The study stress-tests chest X-ray representations across datasets without estimating epistemic uncertainty or claiming a formal account of what models know.It operationalizes an EIML-relevant blind spot through controlled tests of transfer, alignment, and recoverable source-proxy information.
  • Study Design: NIH ChestXray14 and CheXpert support visual transfer, while PadChest and OpenI support paired multimodal evaluation.Matched ResNet-18 encoders isolate initialization effects, with architecture sensitivity examined separately as an auxiliary tier.
  • Contributions: The paper separates source-only transfer, paired-supervision effects, external paired retrieval, and metadata-derived source-proxy recoverability in multimodal representations.Its contributions include self-supervised initialization comparisons, dual-scale multimodal evaluation, leakage-precise terminology, and qualitative retrieval and Grad-CAM analyses.

2. Related Work

Prior work shows that cross-institution chest-radiography generalization is vulnerable to acquisition and cohort-specific shortcuts. Medical image–text systems likewise exhibit useful structure whose evaluation depends on dataset construction, prompts, candidate pools, and explicit nuisance controls.

  • Cross-Institution Generalization: Cross-institution chest-radiography generalization remains difficult because models can exploit acquisition and cohort-specific shortcuts.Self-supervised learning is presented as a transfer baseline when labeled data are limited or shifted.
  • Medical VLMs: Medical contrastive systems have progressed from paired pretraining to chest X-ray-specific language–image models.The paper uses this literature as context for diagnostic cross-dataset evaluation rather than proposing a new architecture.
  • Evaluation: Evaluation conclusions can depend strongly on dataset construction, prompt formulation, and candidate-pool definition.This motivates asking which findings survive conservative cross-dataset stress tests.
  • Shortcut Learning: Multimodal shortcut risks motivate patient-disjoint splits and explicit nuisance probes because image and report modalities may share source-specific regularities.The study pairs utility metrics with source-proxy probes without equating metadata-derived proxies with verified hospital-site labels.

3. Experimental Setup

The experiments assign fixed roles to four chest X-ray datasets, compare controlled visual initializations, and separately diagnose adaptation, retrieval, and source-proxy behavior. The framework emphasizes leakage-aware held-out evaluation, strict pair-index retrieval, and task-dependent architecture checks.

  • 3.1. Datasets and Roles: NIH ChestXray14 is the labeled visual-transfer source, CheXpert the external target, PadChest the paired training corpus, and OpenI the external paired stress-test set.Images share grayscale conversion, 224×224 resizing, and common normalization; patient-disjoint splits are used when identifiers are available.
  • 3.2. Visual Initialization: The primary controlled tier varies ImageNet, BYOL, and eligible OpenI CLIP initialization while keeping ResNet-18 fixed.OpenI-pretrained CLIP is excluded from held-out OpenI quantitative claims to avoid evaluation contamination.
  • 3.3. Source-Only Transfer: Source-only transfer uses NIH labels only, with frozen-backbone linear probing and partial fine-tuning of the final residual block plus classifier.CheXpert labels are not used for updates, model selection, or early stopping.
  • 3.3. Source-Only Transfer and Unsupervised Adaptation: DANN and CORAL are evaluated as unsupervised adaptation diagnostics because they use unlabeled target features for domain or covariance losses.Scheduled adversarial strength and full training trajectories expose instability rather than establishing pure domain-generalization benefits.
  • 3.4. Multimodal Construction and Retrieval: The multimodal model projects image and BioClinicalBERT text encodings into a normalized shared space and trains with symmetric contrastive loss on paired PadChest data.Text uses standardized label-derived phrases or automatically translated free-form reports.
  • 3.4. Multimodal Construction and Retrieval: Retrieval uses strict pair-index Recall@K for K ∈{1, 5, 10}, with OpenI evaluated over N = 6800 candidate entries.Because duplicate templates and report-level multiple-image associations exist, exact-index misses are not treated as clinical semantic mismatches.
  • 3.5. Source-Proxy Leakage: OpenI source-proxy probes predict site_parent and site_folder classes from frozen embeddings while keeping patient or study groups disjoint.These metadata fields are not treated as verified hospital identities, and patient identity is not a probe target.
  • 3.6. Auxiliary Checks: Adversarial proxy unlearning, CORAL alignment, and InstanceNorm are reported jointly with downstream utility and proxy predictability.Auxiliary architecture runs are task-dependent and are not combined into a universal backbone ranking.

4. Results

Results show that transfer, adaptation, retrieval, qualitative structure, and source-proxy behavior vary substantially across evaluation settings. Auxiliary architecture checks further indicate that backbone effects are task-dependent rather than universally rankable.

  • Matched NIH-to-CheXpert Transfer: BYOL initialization improves AUC over ImageNet initialization in the architecture-matched NIH-to-CheXpert ResNet-18 comparison.The result supports a limited initialization claim and does not establish that initialization dominates architecture generally.
  • Unsupervised Domain-Adaptation Behavior: Target AUC rises early under DANN before degrading as adversarial pressure increases, while CORAL is less erratic in the observed runs.Neither adaptation method is treated as a substitute for source representation quality.
  • Multimodal Pair-Index Retrieval: Exact pair-index retrieval on the external OpenI pool remains low and close to chance at larger K, limiting the conclusion to weak transfer of PadChest-learned pair identity.The diagnostic uses candidate entries rather than unique reports and does not model duplicate or report-level positives.
  • Qualitative Representation Structure: Nearest-neighbor and Grad-CAM analyses retain clinically plausible anatomy, view, and thoracic attention patterns, but harder, device-heavy, and false-positive cases remain ambiguous.These are qualitative diagnostics, not evidence of lesion localization accuracy or causal grounding.
  • Source-Proxy Leakage and Mitigation: Metadata-derived source proxies remain recoverable from frozen representations, while InstanceNorm reduces proxy accuracy most but incurs the largest utility loss.The trade-off is supported only for the evaluated proxy and task, and proxy recoverability does not prove hospital-site identification.
  • Architecture Sensitivity Is Task-Dependent: Swin-T reaches 0.5373 and is marked unstable in the auxiliary consolidation run, supporting a task-dependent rather than universal architecture comparison.The auxiliary task is not directly comparable with the primary NIH-to-CheXpert transfer table.

5. Limitations

The study is bounded by limited dataset coverage, exact-index retrieval constraints, incomplete statistical and run-level records, and an unrecoverable translation-model snapshot.

  • The multimodal study uses one large paired training corpus and one external paired stress-test dataset.

6. Conclusion

The study finds that medical VLMs can retain clinically plausible structure while still failing under cross-dataset transfer, exact-pair alignment, and source-proxy diagnostics. These results support stress-tested evaluation and precise interpretation of what evaluation protocols measure.

  • Self-supervised initialization improves matched NIH-to-CheXpert transfer, whereas stronger adversarial adaptation becomes unstable.
  • External OpenI evaluation shows weak strict exact-pair retrieval, while source-proxy signals remain recoverable from frozen embeddings.
  • Qualitative retrieval and Grad-CAM analyses reveal plausible cross-dataset structure and thoracic attention patterns, alongside ambiguity in difficult cases.
  • Auxiliary architecture checks are task-dependent and do not support a universal backbone ranking.
  • The paper argues for explicit stress tests, contamination-aware baselines, and precise claims about what evaluation protocols actually measure.

Supplementary Material

Supplementary zero-shot baselines provide contextual numerical reference under a unified protocol, but their provenance and architectural mismatch limit the conclusions they can support.

  • Zero-shot baselines are included as contextual reference points under unified preprocessing, label mapping, and AUROC evaluation.
  • These external checkpoints are not architecture-matched controls and are not used as primary evidence for architecture or transfer claims.
  • Public CXR-CLIP checkpoint compositions may include CheXpert, so held-out CheXpert generalization requires verified checkpoint provenance.

B. Extended Architecture Tables and Exploratory Results

The supplementary material separates exploratory architecture checks from the primary transfer experiment and documents the preprocessing, grouping, label mapping, and dataset roles used for controlled evaluation.

  • Extended Architecture Tables and Exploratory Results: Auxiliary architecture comparisons use different task definitions and cannot support a single universal architecture-ranking claim.
  • Extended Architecture Tables and Exploratory Results: The separate consolidation check reports Swin-T at 0.5373, but the run was unstable and is not primary evidence.
  • Leakage-Aware Evaluation Protocol: The protocol assigns NIH ChestXray14 to source transfer, CheXpert to external targeting, PadChest to paired training and in-domain evaluation, and OpenI to external retrieval stress testing.
  • Leakage-Aware Evaluation Protocol: A unified grayscale, 224 × 224 preprocessing pipeline with common normalization is applied across datasets, without dataset-specific contrast processing.
  • Leakage-Aware Evaluation Protocol: Patient- or study-disjoint grouping limits overlap and repeated-sample effects across transfer, multimodal, and source-proxy evaluations.
  • Label Mapping and Uncertainty Handling: Transfer uses an explicit shared pathology label space, retaining only categories with clear correspondence and applying a consistent uncertainty policy.

C.5. Training Regimes

The training-regime section defines source-only transfer, multimodal retrieval, and source-proxy probing as distinct diagnostics with explicit baselines and reporting rules. Together, these procedures relate task utility to alignment and recoverable shortcut information.

  • Training Regimes: The primary transfer table distinguishes frozen-backbone linear probing from partial fine-tuning of ResNet-18 layer4 and the classifier.
  • Training Regimes: CheXpert labels are excluded from source-only updates, validation, and early stopping; adaptation runs using unlabeled target features are reported separately.
  • Multimodal Alignment and Retrieval Protocol: Multimodal retrieval projects image and BioClinicalBERT text embeddings into a normalized shared space and evaluates designated-index Recall@K.
  • Multimodal Alignment and Retrieval Protocol: OpenI retrieval uses a candidate pool of N = 6800, with analytical chance baselines K/N and strict index matching rather than a clinical semantic-retrieval interpretation.
  • Source-Proxy Leakage Diagnostics: Frozen-embedding linear probes estimate recoverability of metadata-derived site_parent and site_folder source proxies, not verified hospital-site leakage.
  • Reporting Checklist: The reporting checklist places AUROC or Recall@K beside leakage accuracy, split specifications, and available seed dispersion for joint interpretation.

D. Reproducibility and Evidence Audit

The audit documents what the archived experiments support, what remains unrecoverable, and how retrieval and text construction are constrained. It preserves exact-index retrieval as a conservative diagnostic rather than a validated semantic metric.

  • Reproducibility and Evidence Audit: The archive retains point estimates but lacks seed-level values for reconstructing mean±standard deviation, so no statistical-significance claim is made.The reproducibility ledger also records unavailable run-level hyperparameters and translation snapshot identifiers.
  • Duplicate-Aware Multi-Positive Retrieval: A duplicate-aware multi-positive Recall@K is specified, but exact-index Recall@K remains the reported metric because validated positive sets are unavailable for every candidate.The multi-positive metric would avoid penalizing retrieval of validated report-level or equivalent-text positives.
  • Text Construction: PadChest label-derived English phrases provide controlled wording with minimal linguistic variability, focusing alignment on visual findings and structured descriptions.The phrases are generated from structured labels using fixed templates, concatenated findings, and standardized negation where available.
  • Text Construction: Translated PadChest reports use GPT-4 at temperature 0, with checks for preserved negation, absent added findings, and ambiguous or untranslated terms.Temperature 0 reduces variability but does not guarantee bitwise or API-level reproducibility; regenerated translations may differ slightly.
  • Text Construction: Retrieval is run with both textual formulations to assess sensitivity of multimodal alignment to linguistic variability and translation noise.The label-derived condition is treated as a text-construction ablation, not evidence for one-to-one semantic retrieval.
  • Diagnostic Scope: The appendix presents representation-similarity and candidate-pool diagnostics as reliability checks, not primary evidence of semantic grounding.Strict cross-dataset protocols and large candidate pools can make absolute retrieval values appear numerically small.

F.1. Representation Similarity Analysis

Representation-similarity analysis compares embeddings from different pretraining strategies using linear CKA, while retrieval diagnostics test candidate-pool sensitivity. The reported patterns indicate related but non-identical representations and lower Recall@K with larger pools.

  • F.1. Representation Similarity Analysis: Linear CKA compares feature embeddings from models trained under different pretraining regimes on the same images.Higher CKA values indicate stronger representation similarity, and representative values are computed on the NIH validation split.
  • F.1. Representation Similarity Analysis: Self-supervised and multimodal pretraining produce related but not identical feature structures, reflecting differences in their supervision signals.This comparison concerns visual representations rather than a universal ranking of pretraining strategies.
  • F.2. Retrieval Pool Size Sensitivity: Recall@K decreases as the OpenI candidate pool grows, consistent with theoretical scaling of retrieval difficulty.The analysis evaluates retrieval under different candidate-pool sizes and reports performance for several pool sizes.
Loading 2608.25251v1…