PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta-CLIP: Metadata-Conditioned Contrastive Learning for Zero-Shot Disease Inference

Haozhe Luo, Nirosh Sivanesan, Shelley Zixin Shu, Francesco Nitti, Mauricio Reyes

arXiv:2610.33674v1cs.AIcs.CL

TL;DR

Medical vision–language pretraining often enriches clinical semantics while leaving structured DICOM acquisition information underused, despite transfer challenges across scanners and protocols. Meta-CLIP uses acquisition similarity to reweight cross-modal negatives during pretraining. It improves CT-RATE macro-AUROC over CT-CLIP by 1.41 points and achieves the highest macro-AUROC on four of five CXR benchmarks.

  • Problem

    Structured DICOM acquisition information is underused in medical vision–language pretraining, while scanner and protocol differences can impair transfer.

    Source: §I · p. 1

  • Method

    Meta-CLIP uses a kernel over numerical and categorical DICOM fields to impose bounded margins that increase the contribution of acquisition-similar cross-modal negatives during pretraining.

    Source: §I · p. 1

  • Results

    Meta-CLIP improves CT-RATE macro-AUROC over CT-CLIP by 1.41 points and achieves the highest macro-AUROC on four of five CXR benchmarks.

    Source: §I · p. 1

  • Takeaways & Limitations

    The findings support acquisition metadata as a practical training-time prior for medical vision–language learning.

    Source: §VII · p. 8

    Prospective, multi-institutional evaluation is needed, and synthetic perturbations cover only part of real scanner and workflow variation.

    Source: C. Limitations · p. 7

Abstract

from arXiv · show

Medical vision--language pretraining learns transferable representations from paired images and radiology reports, yet differences in scanners and acquisition protocols can impair transfer. Existing approaches primarily enrich clinical semantics, leaving structured acquisition information in DICOM headers underused. We introduce Meta-CLIP, a metadata-conditioned contrastive framework that uses acquisition similarity to reweight cross-modal negatives during pretraining. A kernel over numerical and categorical DICOM fields defines a bounded pairwise margin. This margin increases the contribution of acquisition-similar negatives, emphasizing distinctions among studies that share imaging conditions. The kernel remains fixed during image--text optimization, and inference follows the standard dual-encoder architecture without requiring metadata. We evaluate Meta-CLIP on chest radiography and volumetric chest CT through zero-shot classification, appearance perturbations, grounding, and representation analyses. On CT-RATE, Meta-CLIP improves macro-AUROC over CT-CLIP by 1.41 points, with a 95\% paired bootstrap confidence interval of 0.96--1.85 points. It also achieves the highest macro-AUROC on four of five CXR benchmarks. An exploratory BrgSA evaluation extends the analysis to another CT backbone. These findings support acquisition metadata as a practical training-time prior for medical vision--language learning. Code will be made publicly available upon publication.

I. INTRODUCTION

Medical vision–language pretraining has strengthened clinical supervision, but transfer across acquisition settings remains challenging because scanners and protocols alter image appearance without changing pathology. Meta-CLIP uses acquisition similarity to reweight contrastive negatives during pretraining and improves zero-shot classification across CT and CXR benchmarks.

I. INTRODUCTION

  • Scanner vendors, exposure settings, and reconstruction protocols can change image appearance without changing pathology, and medical vision–language models remain sensitive to artifacts and distribution shifts.Standard contrastive learning weights unmatched image–report pairs by embedding similarity but lacks an explicit acquisition prior for emphasizing negatives.Source: §I · p. 1
  • Acquisition cues shared by two studies are less informative for matching reports to images, motivating acquisition similarity as a prior for weighting cross-modal negatives.The stated aim is to emphasize comparisons among acquisition-similar studies so alignment can rely on other image–report evidence.Source: §I · p. 1
  • Acquisition similarity from numerical and categorical DICOM fields defines bounded margins that raise acquisition-similar negative logits while leaving positive logits unchanged.The kernel remains fixed during image–text optimization; inference uses the standard dual encoder without metadata.Source: §I · p. 1
  • Meta-CLIP improves CT-RATE macro-AUROC over CT-CLIP by 1.41 points and achieves the highest macro-AUROC on four of five CXR benchmarks.Experiments also examine external transfer, robustness, grounding, and representation structure.Source: §I · p. 1
  • The work formulates acquisition similarity as a prior over cross-modal negatives and develops a symmetric contrastive objective with bounded pairwise margins.The objective preserves the underlying encoders and metadata-free inference procedure.Source: §I · p. 1

A. Medical Vision–Language Pretraining

Medical vision–language methods have expanded report supervision with clinical semantics and knowledge, while distribution shift and image artifacts remain challenges. Meta-CLIP complements these approaches by using acquisition context to shape cross-modal negatives during source pretraining.

A. Medical Vision–Language Pretraining

  • Medical vision–language pretraining progressed from global image–text alignment in ConVIRT to global and local alignment in GLoRIA, with BioViL strengthening biomedical text representations.Other developments use soft prompts [12], granular masked alignment [13], or anatomical consistency [14].Source: A. Medical Vision–Language Pretraining · pp. 1–2
  • KAD and MedKLIP associate visual features with disease entities and knowledge graphs,, while DeViDe decomposes disease descriptions into semantic facets; CT-CLIP extends pretraining to volumetric chest CT.Meta-CLIP complements clinical-supervision approaches by using acquisition context to determine cross-modal negative contributions.Source: A. Medical Vision–Language Pretraining · p. 2

B. Robustness and Vision–Language Adaptation

  • Medical image artifacts can substantially change vision–language model predictions, while target-task adaptation methods modify prompts, features, or model parameters –.These methods address a different learning stage from constructing contrastive relations during source pretraining.Source: B. Robustness and Vision–Language… · p. 2

C. Metadata-Aware Medical Representation Learning

  • Meta-CLIP uses source-image acquisition information to modify pretraining before target-domain inference, and can be evaluated zero-shot or combined with later adaptation when target data are available.It differs from approaches that use metadata as an input, prediction target, or positive-pair construction rule: acquisition similarity instead becomes a continuous margin on cross-modal negatives.Source: B. Robustness and Vision–Language… · p. 2, C. Metadata-Aware Medical… · p. 2
  • Medical representation learning has used metadata as weak targets, patient relations, or sample-pair definitions in unimodal pretraining –.Meta-CLIP instead applies acquisition relations to paired image–report negatives while preserving standard CLIP encoders and metadata-free inference.Source: C. Metadata-Aware Medical… · p. 2

III. METHOD

Meta-CLIP adds structured acquisition context to contrastive pretraining through a pairwise margin on cross-modal negatives, while retaining the original image and text encoder architectures. Its acquisition kernel is built from numerical and categorical DICOM fields.

III. METHOD

  • A pairwise acquisition kernel from numerical and categorical DICOM fields measures similarity between acquisition conditions and modulates logits in the contrastive objective.The image and text encoders retain their original architecture.Source: §III · p. 2, B. Acquisition Relation Modeling · p. 3

A. Problem Formulation

  • Each pretraining sample contains a CXR image or CT volume, its paired report, and structured DICOM metadata; image and text encoders produce ℓ2-normalized representations.The metadata describes acquisition conditions such as exposure, reconstruction, and scanner manufacturer.Source: A. Problem Formulation · pp. 2–3
  • The acquisition prior controls unmatched-pair contributions to the contrastive normalizer, while matched image–report pairs provide positive supervision.The prior is defined over unmatched pairs within each minibatch.Source: A. Problem Formulation · p. 3

B. Acquisition Relation Modeling

  • The acquisition kernel combines numerical and categorical DICOM fields to capture complementary aspects of image formation and scanner identity.The supplied Fig. 2 caption identifies disease-wise AUC comparisons, volume retrieval by top-k label similarity, and robustness under synthetic perturbations as evaluation views.Source: B. Acquisition Relation Modeling · p. 3, Figure 2 · p. 3

1) Numerical metadata:

Meta-CLIP measures acquisition similarity using numerical and categorical DICOM metadata, with pairwise comparisons accounting for which numerical fields are jointly observed. The resulting kernel combines field similarities into a bounded measure of imaging-condition similarity.

1) Numerical metadata:

  • For each sample pair, numerical similarity uses only jointly observed standardized DICOM fields in an RBF kernel.Missing values are excluded from that pairwise comparison; standardization makes fields with different units comparable.Source: 1) Numerical metadata · p. 3
  • The RBF bandwidth σ_d controls sensitivity to differences in numerical field d and how quickly similarity decays.For CXR, the Gaussian relation uses one normalized bandwidth; for CT, per-field similarities use normalized field weights.Source: 1) Numerical metadata · p. 3, 2) Categorical metadata · p. 3

2) Categorical metadata:

  • Categorical fields such as scanner manufacturer use fixed lookup embeddings, and numerical and categorical similarities are combined through normalized aggregation.The resulting acquisition kernel lies in [0, 1], with larger values indicating more similar imaging conditions.Source: 2) Categorical metadata · p. 3

3) Meta-CLIP relation:

Meta-CLIP converts acquisition similarity into bounded margins on unmatched image–report pairs, increasing the contribution of acquisition-similar negatives while leaving positive logits unchanged. The relation is applied in both retrieval directions, with a scheduled margin scale for CT and a static scale for CXR.

3) Meta-CLIP relation:

  • The kernel-derived correction lies in [−α, 0], with α controlling its strength; acquisition-similar negative pairs receive corrections of larger magnitude.The matched image–report pair is unchanged.Source: 3) Meta-CLIP relation · pp. 3–4

C. Metadata-Conditioned Contrastive Learning

D. Training and Inference

The acquisition relation is fixed and detached before it modulates the contrastive objective; each backbone retains its base training objective. At inference, the trained dual encoders require no DICOM metadata.

D. Training and Inference

  • The acquisition relation is computed during pretraining and detached before logit modulation, so image–text optimization updates encoders against a prescribed prior.Inference uses the trained image and text encoders without DICOM metadata.Source: D. Training and Inference · p. 4
  • CXR incorporates the metadata-conditioned term into KAD, while CT combines it with CT-CLIP’s symmetric InfoNCE objective.The acquisition relation is shared across modalities, and each backbone retains its base training objective.Source: D. Training and Inference · p. 5

IV. EXPERIMENTAL SETUP

The evaluation spans zero-shot classification and analyses of representation structure, grounding, and robustness for chest radiography and volumetric chest CT. It uses established CT and CXR backbones and datasets, with an exploratory portability comparison on BrgSA.

IV. EXPERIMENTAL SETUP

  • Evaluation covers 2D chest radiography and 3D chest CT, combining zero-shot classification with representation-structure, grounding, and appearance-robustness analyses.Zero-shot evaluation uses pretrained models without target-dataset parameter updates.Source: §IV · p. 5, A. Backbones and Evaluation Protocol · p. 5

A. Backbones and Evaluation Protocol

  • CT uses CT-ViT and CXR-BERT with CT-RATE for internal evaluation and RAD-ChestCT for external evaluation; CXR follows KAD across five benchmarks.CXR benchmarks are ChestXray14, CheXpert, PadChest-seen, RSNA Pneumonia, and SIIM-ACR.Source: A. Backbones and Evaluation Protocol · p. 5
  • The CXR and CT configurations use different training and margin settings, while kernel parameters remain fixed in both modalities.CXR uses a static schedule; CT increases the margin scale over training epochs.Source: A. Backbones and Evaluation Protocol · p. 5
  • The BrgSA portability comparison is exploratory because its metadata variant uses a different batch size and omits BrgSA’s local-alignment term.Both variants are trained on CT-RATE and evaluated on CT-RATE, RAD-ChestCT, and INSPECT.Source: A. Backbones and Evaluation Protocol · p. 5

B. Metadata Selection and Characterization

Metadata selection considers acquisition-field coverage and information content across CT and CXR settings. The CT field-set comparison varies acquisition attributes under a shared training configuration, while CXR dataset schemas differ in available fields.

B. Metadata Selection and Characterization

  • The seven selected CT DICOM attributes describe scanner identity or image formation; source-to-detector distance is available for only 47.9% of scans.The numerical kernel excludes unobserved fields from pairwise comparisons.Source: B. Metadata Selection and… · p. 5
  • The CT field-set ablation compares the default set with Physics-5, Alternative-5, and a variant adding XYSpacing under a shared CT-CLIP training configuration.The text encoder is frozen to isolate the field-set comparison from joint encoder fine-tuning in the main CT model.Source: B. Metadata Selection and… · p. 5
  • Across public CXR resources, dataset schemas expose different acquisition fields, and field completeness can vary across individual studies.This motivates selecting training fields according to source coverage; metadata-free inference accommodates evaluation datasets with different schemas.Source: B. Metadata Selection and… · p. 5
  • CT robustness evaluation applies histogram shift, contrast adjustment, and k-space spike noise; CXR tests metal artifacts, brightness shifts, and noise.Random retrieval provides the reference for retrieval analysis.Source: B. Metadata Selection and… · pp. 5–6

C. Ablations and Statistical Analysis

Meta-CLIP improves zero-shot classification across CT and CXR benchmarks, with gains also reported for grounding, robustness, and representation structure. Analyses examine acquisition-field information, metadata selection, acquisition strata, and exploratory transfer to another CT backbone.

V. RESULTS

  • CT-RATE macro-AUROC rises from 73.08% to 74.49% (1.41 points); Meta-CLIP leads on four of five CXR benchmarks.On ChestXray14 and CheXpert, gains over KAD are 4.16 and 2.08 points, respectively.Source: §V · p. 6
  • BrgSA macro-AUROC gains are 1.27 points on CT-RATE, 2.40 on RAD-ChestCT, and 0.98 on INSPECT.The authors describe this comparison as exploratory evidence for another volumetric backbone.Source: §V · p. 6, Figure 4 · p. 6
  • On ChestXray14-Det, gains over KAD are 12.80 points in Pointing Game accuracy, 5.48 in mean IoU, and 6.54 in mean Dice.These results indicate stronger spatial correspondence between textual findings and image evidence.Source: §V · p. 6
  • Under CT histogram shift, macro-AUROC decreases by 0.41 points for Meta-CLIP and 1.04 for CT-CLIP; CXR degradation is smaller than KAD's across three perturbations.The CXR comparisons cover metal artifacts, brightness shifts, and noise on ChestXray14, RSNA, and SIIM-ACR.Source: §V · p. 6
  • The seven selected CT fields yield mean balanced accuracy of 0.774 across acquisition proxies and mean disease AUROC of 0.593; the unrestricted 43-field pool scores 0.995 and 0.724.Source-to-detector distance is available for 47.9% of scans; the numerical kernel excludes unobserved fields from pairwise comparisons.Source: §V · p. 6, Table III · p. 7
  • The default seven-field set has the highest macro-AUROC point estimates on CT-RATE and RAD-ChestCT among the evaluated field sets.Adding XYSpacing decreases performance by 0.63 and 0.58 points, respectively; confidence intervals overlap, particularly on RAD-ChestCT.Source: §V · pp. 6–7, Table IV · p. 7
  • Across 999,921 sampled CT-RATE pairs, aggregate acquisition similarity is 0.657±0.216; ExposureTime similarity is concentrated near one (median 0.999).The other numerical fields provide a broader range of similarities; Manufacturer enters the categorical kernel.Source: §V · p. 7
  • Across 22 CT-RATE acquisition strata, 21 improve; the longest exposuretime stratum (n = 179) decreases by 0.54 points.All three manufacturer groups improve, including Siemens by 2.02 points and PNMS by 2.37 points.Source: §V · p. 7, Table V · p. 7

VI. DISCUSSION

Acquisition similarity is a training-time prior for weighting cross-modal negatives, with evidence of gains across CT and CXR and exploratory transfer to BrgSA. The mechanism does not explicitly align same-disease studies across acquisition conditions, so conditional acquisition invariance remains a separate property to evaluate.

VI. DISCUSSION

  • Meta-CLIP weights negative comparisons by acquisition similarity; it does not explicitly align same-disease studies across acquisition conditions.Conditional acquisition invariance remains a separate property to evaluate.Source: A. Acquisition Metadata as a… · p. 7

B. Practical Implications

Meta-CLIP preserves the dual encoder’s standard deployment procedure while using acquisition metadata during pretraining. The study reports improved classification, perturbation robustness, and grounding, but calls for prospective, multi-institutional evaluation.

B. Practical Implications

  • Meta-CLIP retains its dual encoder’s inference computation, memory footprint, and input requirements, so deployment needs images and text prompts but no DICOM input.The study favors source-specific selection of complementary acquisition fields; the compact CT set carries less direct disease signal than the unrestricted pool, and adding XYSpacing fails to improve evaluated transfer results.Source: B. Practical Implications · p. 7

C. Limitations

  • Prospective, multi-institutional evaluation is needed because the study uses retrospective public cohorts and synthetic perturbations cover only part of real-world scanner and workflow variation.DICOM fields may be missing, inconsistently encoded, or correlated with institution and patient population; unseen protocols and heterogeneous metadata schemas warrant further evaluation.Source: C. Limitations · p. 7

VII. CONCLUSION

  • The study reports improved zero-shot classification, smaller performance losses under appearance perturbations, and stronger disease grounding on chest radiography and volumetric CT.The CXR benchmark demonstrates gains over KAD, while external CT evaluation extends transfer evidence beyond the pretraining cohort.Source: §VII · p. 8, Figure 5 · p. 8

Related papers

Loading 2610.33674v1…