PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring scalable medical image encoders beyond text supervision

Fernando Pérez-García, Harshita Sharma, Sam Bond-Taylor, Kenza Bouzid, Valentina Salvatelli, Maximilian Ilse, Shruthi Bannur, Daniel C. Castro, Anton Schwaighofer, Matthew P. Lungren, Maria Teodora Wetscherek, Noel Codella, Stephanie L. Hyland, Javier Alvarez-Valle, Ozan Oktay

arXiv:2401.10815v3cs.CV

TL;DR

Biomedical image encoders often rely on incomplete, privacy-constrained image–text pairs, motivating an image-only alternative. The paper introduces RAD-DINO, which continually pre-trains DINOv2 on medical scans and achieves comparable or superior performance across diverse downstream benchmarks. Its performance scales with training-data quantity and diversity, although zero-shot image–text applications remain a limitation.

  • Problem

    Biomedical image–text pairs are scarce and radiology reports may omit visual information, limiting language-supervised representations for general-purpose medical imaging.

  • Method

    RAD-DINO continually pre-trains DINOv2 on unimodal medical imaging data using image-level and patch-level self-supervised objectives without external text supervision.

  • Results

    RAD-DINO achieves comparable or superior performance to state-of-the-art language-supervised biomedical encoders across diverse classification, segmentation, and report-generation benchmarks.

  • Takeaways & Limitations

    Image-only supervision can train a general-purpose biomedical image encoder while scaling with larger and more diverse imaging datasets.

  • Takeaways & Limitations

    RAD-DINO does not provide zero-shot image classification or text-to-image retrieval capabilities comparable to CLIP-style models.

Abstract

from arXiv · show

Language-supervised pre-training has proven to be a valuable method for extracting semantically meaningful features from images, serving as a foundational element in multimodal systems within the computer vision and medical imaging domains. However, the computed features are limited by the information contained in the text, which is particularly problematic in medical imaging, where the findings described by radiologists focus on specific observations. This challenge is compounded by the scarcity of paired imaging-text data due to concerns over leakage of personal health information. In this work, we fundamentally challenge the prevailing reliance on language supervision for learning general-purpose biomedical imaging encoders. We introduce RAD-DINO, a biomedical image encoder pre-trained solely on unimodal biomedical imaging data that obtains similar or greater performance than state-of-the-art biomedical language-supervised models on a diverse range of benchmarks. Specifically, the quality of learned representations is evaluated on standard imaging tasks (classification and semantic segmentation), and a vision-language alignment task (text report generation from images). To further demonstrate the drawback of language supervision, we show that features from RAD-DINO correlate with other medical records (e.g., sex or age) better than language-supervised models, which are generally not mentioned in radiology reports. Finally, we conduct a series of ablations determining the factors in RAD-DINO's performance; notably, we observe that RAD-DINO's downstream performance scales well with the quantity and diversity of training data, demonstrating that image-only supervision is a scalable approach for training a foundational biomedical image encoder. Model weights of RAD-DINO trained on publicly available datasets are available at https://huggingface.co/microsoft/rad-dino.

1 Introduction

RAD-DINO challenges reliance on paired image–text supervision for biomedical image encoders, whose textual descriptions can be incomplete and difficult to scale because of privacy constraints. It uses image-only self-supervision and achieves comparable or better downstream performance while scaling with training-data size and diversity.

  • Motivation: Paired medical image–text data are scarce because public multimodal datasets are limited and protected health information raises anonymity concerns.The lack of pixel-level textual supervision also constrains tasks requiring precise image analysis.
  • Motivation: Radiology text may omit visual details and include absent findings, potentially collapsing representations and discarding intra-class variation.The example “No cardiopulmonary process” can align many healthy chest X-rays with similar text despite image differences.
  • Approach: RAD-DINO continually pre-trains DINOv2 on medical scans using image-only self-supervised learning with complementary image-level and patch-level objectives.The approach is evaluated on image- and pixel-level predictive tasks and compared with text-supervised biomedical encoders.
  • Results: RAD-DINO achieves comparable or superior performance to state-of-the-art text-supervised methods across classification, segmentation, and report-generation benchmarks.The overview covers image classification, semantic segmentation, and report generation, with linear and UPerNet decoder variants for segmentation.
  • Scaling: Ablations show that RAD-DINO performance scales with training-data size, diversity, and input resolution, supporting image-only pre-training as a scalable encoder strategy.RAD-DINO was trained with 838k images and is intended to scale to additional image-only data.
  • Results: RAD-DINO outperforms all other models on aggregate on VinDr-CXR classification using linear probing with frozen backbones.The result holds despite comparisons with larger models trained on 10 or 100 times more data.
  • Results: RAD-DINO also outperforms other models on CANDID-PTX, with a significant margin for pneumothorax and chest-tube findings.The CANDID-PTX and RSNA-Pneumonia evaluations use subject-level splits and AUPRC on test sets.

2 Results

RAD-DINO produces transferable representations across classification, segmentation, and report generation, while image-only pre-training captures local and clinically relevant features that language-supervised models may miss.

  • Image classification: RAD-DINO outperforms multimodal methods for five of seven pathologies and on average, while multimodal methods lead only for cardiomegaly and aortic enlargement.The authors hypothesize that reports describe the large, clearly bounded heart and aorta more effectively than lower-contrast, texture-based pathologies.
  • Image classification: RAD-DINO scales with dataset size and diversity, while BiomedCLIP underperforms it across all evaluated benchmarks despite 15 million image–text pairs.The comparison emphasizes data quality and task relevance rather than pair count alone.
  • Representation analysis: RAD-DINO patch embeddings align local abnormalities such as consolidation and nodules across scans from different subjects, while also encoding anatomical correspondence.When anatomy overlaps with an abnormality, nearest-neighbour matches capture both types of information.
  • Report generation: RAD-DINO surpasses all other image encoders on every lexical and clinical metric for MIMIC-CXR report generation and all but one lexical metric for IU-Xray.The result holds for an in-domain dataset and an external dataset not used to train the encoder or language decoder.
  • Semantic segmentation: Masked image modelling improves segmentation representations because it captures local relationships and boosts performance across structures and datasets.The instance-discrimination objective emphasizes global relationships, whereas MIM is more suited to local textures required by dense tasks.
  • Semantic segmentation: Frozen RAD-DINO features are competitive with end-to-end task-specific networks, with intermediate activations and FPN-based decoders providing larger gains for smaller structures.The authors suggest hierarchical feature pyramids as a possible route to further improvement.

3 Methods and experimental setup

The experiments use DINOv2-based image-only self-supervision with complementary global and local objectives, diverse radiology data, frozen-backbone downstream evaluation, and demographic probes.

  • DINOv2: DINOv2 combines image-level instance discrimination with patch-level masked image modelling to learn features for both global and localized downstream tasks.Its siamese teacher–student design distils teacher predictions into a student network.
  • DINOv2: Multi-crop training helps the backbone learn distinctive local features for dense predictive tasks, while KoLeo regularization promotes uniformly distributed features for clustering and retrieval.The method also uses asymmetric augmentation, centring, and temperature choices across branches.
  • Training data: RAD-DINO continually trains a pre-trained DINOv2 ViT-B on diverse Multi-CXR radiology images for 60k steps with batch size 640 and fixed input resolution.Multi-CXR combines public and private sources spanning varied findings and demographics.
  • Downstream evaluation: The evaluation compares image–text, general-domain, and domain-specific backbones using linear probing across external classification, segmentation, report-generation, and demographic tasks.Segmentation trains a decoder from scratch, while frozen-backbone classification and demographic experiments assess transferable representations.
  • Data and evaluation: All images from each patient are assigned to one split to prevent data leakage, and classification uses external VinDr-CXR, CANDID-PTX, and RSNA-Pneumonia datasets.VinDr-CXR is selected for ablations because of its diverse findings and patient demographics.
  • Demographic evaluation: RAD-DINO significantly outperforms baselines for predicting sex, age, weight, and BMI from frozen features, indicating that image-only self-supervision captures broad imaging information.The demographic probes use five-fold cross-validation with five bins for age, weight, and BMI.

4 Discussion and conclusion

RAD-DINO shows that high-quality general-purpose biomedical image encoders can be trained solely on unimodal imaging data, achieving comparable or superior performance across diverse benchmarks. The approach also offers scalable access to imaging-only data, while retaining limitations for zero-shot multimodal applications and leaving alternative architectures for future work.

  • RAD-DINO achieves comparable or superior performance to state-of-the-art methods across multiple benchmarks using raw imaging data alone.
  • RAD-DINO features correlate better with patient medical records than CLIP-style models, while language-supervised representations may be limited by report descriptiveness.
  • RAD-DINO’s imaging-only training can leverage large medical imaging datasets, including domains where paired text is scarce or rarely available.
  • Image-only self-supervision supports reuse of one encoder with task-specific heads across segmentation and multimodal applications.
  • The study does not evaluate alternative encoder architectures such as Swin Transformers, which the authors expect could improve segmentation performance.
  • Zero-shot image classification and text-to-image retrieval remain limitations relative to CLIP-style models, with multimodal extensions left for future work.

5 Data availability

The study draws on a broad mixture of public and private biomedical datasets and situates RAD-DINO within prior image-only and multimodal representation-learning research. Its evaluations use expert-annotated benchmarks and cover classification, segmentation, and report generation contexts.

  • The study uses a mixture of public and private datasets, including CheXpert, ChestX-ray14, PadChest, USMix, VinDr-RibCXR, CANDID-PTX, RSNA-Pneumonia, IU-Xray, MIMIC-CXR, VinDr-CXR, Chest ImaGenome, and BRAX.
  • Prior medical foundation models commonly use multimodal contrastive learning, including CheXzero, GLoRIA, BioViL, BioViL-T, and Med-UniC.
  • Image-only pre-training research has studied contrastive and DINO approaches for transferable classification features and objectives tailored to segmentation.
  • RAD-DINO and its baselines are evaluated only on benchmarks containing expert annotations.
  • Medical image segmentation commonly uses U-Net and domain-specific approaches, while image backbones are also used for radiology report generation.

B Ablation studies

The ablation studies isolate image resolution, initialization, and training factors that may confound comparisons between RAD-DINO and baseline encoders. They examine resolution effects across large-scale findings and assess large-domain pre-training before in-domain adaptation.

  • Image resolution, training dataset, and weight initialization can confound comparisons between image networks and produce incomplete or misleading findings.
  • Dependence on image resolution: The resolution ablation evaluates VinDr-CXR linear probing for large or conspicuous findings across 224–518-pixel inputs, using each resolution for pre-training and inference.
  • Dependence on image resolution: RAD-DINO’s superior performance on large-scale findings is not driven solely by its ability to encode higher-resolution inputs.
  • Dependence on image resolution: Lower-resolution inputs may hinder VQA and text generation because ambiguity in the image signal can contribute to hallucinations and performance limits.
  • Domain transfer: The study also compares random initialization, ViT-B, and large-scale DINOv2 models to assess domain transfer before in-domain training.

B.3 Dependence on training dataset size

RAD-DINO performance is examined as training data size and diversity increase, alongside resolution effects for subtle findings. The ablations indicate that richer image-only data improves performance, while fine-grained findings require sufficient resolution.

  • Training dataset size and diversity: RAD-DINO training data is progressively expanded with more diverse examples, including outpatient studies, to compare image-only training with paired image–text baselines.
  • Training dataset size and diversity: 546k-sample training uses frontal chest X-rays because they match the exclusively frontal test set and empirically provide the maximal gain.
  • Training dataset size and diversity: Adding PadChest improves performance by increasing the diversity of findings in outpatient datasets.
  • Dependence on image resolution: RAD-DINO performance on pneumothorax and chest tubes diminishes at lower resolutions because reduced detail makes subtle findings more ambiguous.
  • Few-shot segmentation: Vision-only RAD-DINO transfers better to few-shot lung segmentation than BioViL-T and BiomedCLIP with a linear segmentation decoder.

C.3 Experiments with lateral chest X-ray scans

RAD-DINO is evaluated on lateral chest X-ray findings, where image–text alignment can be ambiguous because some findings are not visible or assessable from lateral scans alone. Its attention and patch embeddings capture support devices, lung regions, and anatomical correspondences across subjects while representing abnormalities alongside anatomy.

  • Lateral-scan image–text alignment can be difficult because some reported findings are not visible or assessable from lateral images alone.
  • The lateral-scan dataset uses selected findings chosen for detectability from lateral images, with 11.9k studies split 80/10/10% by subject.Testing subsets contain N = 373 for VDC, N = 542 for PE, and N = 503 for CAB.
  • Masked-modelling approaches, including RAD-DINO, deliver strong classification results on previously unseen lateral findings, whereas BioViL-T is nearly comparable to random classification for VDC.BiomedCLIP benefits from a more balanced mix of frontal and lateral scans than BioViL-T.
  • RAD-DINO attention accurately traces support devices and concentrates on lung fields, including basal and hilar regions, for images with pleural effusion and opacities.
  • Patch embeddings preserve anatomical correspondences across subjects despite pleural effusion and align abnormalities such as consolidation and nodules across scans.When anatomy overlaps an abnormality, matching reflects both anatomical and pathological information.

C.5.1 Qualitative segmentation results

RAD-DINO produces more detailed qualitative segmentation masks than image–text contrastive encoders across the evaluated tasks, especially for small structures and lung-zone boundaries.

  • RAD-DINO yields better qualitative segmentation results than BioViL-T and BiomedCLIP with linear decoder heads across all evaluated tasks.Its masks preserve more shape and edge detail, especially for chest tubes and lung zones.

C.6 Bias and fairness

The fairness analysis examines ethnicity prediction and downstream subgroup performance. RAD-DINO predicts ethnicity more accurately than comparison encoders, but the reported segmentation and report-generation analyses find no decreased fairness.

  • RAD-DINO outperforms BiomedCLIP and BioViL-T for ethnicity prediction using frozen-backbone features.The analysis uses MIMIC-CXR studies with “no findings” reports to reduce confounding with pathology.
  • Ethnicity classification is evaluated with 5-fold cross-validation, reporting mean accuracy and standard deviation.
  • Ethnicity may not be causally inferable from X-rays because predictions become nearly random after controlling for age and sex.
  • Across ethnicity groups, the worst segmentation group is white, while report generation performs worst for the Asian subgroup with a similar 7–8-point drop across encoders.The segmentation analysis covers lungs and lung zones; report generation uses average and worst-group accuracy.
  • The authors report no decreased fairness in RAD-DINO’s performance despite its stronger ethnicity-prediction ability.They perform stratified analyses for lung segmentation and report generation to assess this concern.

D.1 RAD-DINO pre-training

RAD-DINO is continually pre-trained on a large, diverse collection of chest X-ray datasets using publicly described imaging cohorts. The combined data span multiple patient populations, findings, locations, and time periods, with evaluation data excluded where applicable.

  • RAD-DINO is trained on combined chest X-ray datasets including BRAX, MIMIC-CXR, and PadChest.BRAX contributes 24,959 studies from 19,351 patients, while the other datasets add distinct cohorts and reported findings.
  • The combined pre-training data contain diverse radiological findings from patient cohorts across geographical locations and time periods.All available images are used except the recommended MIMIC-CXR training split, with evaluation subjects excluded where required.
  • Downstream classification uses VinDr-CXR, CANDID-PTX, and RSNA-Pneumonia, while segmentation evaluates lungs, lung zones, pneumothorax, and related targets.

E.1 RAD-DINO pre-training

RAD-DINO pre-training used distributed A100 hardware, specified optimization settings, and multi-scale crop augmentation. The appendix also documents downstream evaluation configurations and baseline implementation sources.

  • RAD-DINO pre-training: RAD-DINO pre-training used 4 nodes with 4 NVIDIA A100 GPUs each, batch size 640, AdamW, learning rate 0.001, and cosine scheduling with warmup.The ViT-B/14 encoder processed 518 × 518 inputs with random global and local crops that were upsampled to fixed sizes.
  • RAD-DINO pre-training: Global crops were sampled from U(259, 518), while local crops were sampled from U(104, 259) before upsampling.
  • Baseline implementation: Baseline encoders included CLIP@224, CLIP@336, BioViL-T, BiomedCLIP, MRM, and DINOv2, using public source code, weights, and corresponding preprocessing pipelines.Special pre-training tokens such as [CLS] were retained for linear probing when applicable.
  • Downstream evaluation: Classification evaluation used 8 NVIDIA V100 GPUs, batch size 96, AdamW, learning rate 5 × 10^-5, cosine scheduling, and image augmentations.Inputs were resized to 518 × 518 except for BiomedCLIP and CheXzero, which used 224 × 224.
  • Downstream evaluation: Segmentation evaluation used 8 NVIDIA V100 GPUs, batch size 80, Adam, learning rate 5 × 10^-4, cosine scheduling, and task-specific augmentations.Horizontal flipping was excluded for left–right lungs and lung zones.
  • Downstream evaluation: Report generation fine-tuning used frozen image encoders, updated the LLM and adaptor for three epochs, and decoded in 32-bit precision.Training used batch size 128, learning rate 2 × 10^-5, and warmup over 3% of training steps.

E.3.4 Experiments with patient demographics

The demographic experiment linked MIMIC-CXR images with anonymised MIMIC-IV medical records and evaluated whether image embeddings predicted patient variables. It used 60.1k AP/PA images and logistic regression with five-fold cross-validation.

  • Dataset construction: The experiment selected MIMIC-CXR radiographs whose reports noted “No findings” and linked anonymised subject information to MIMIC-IV medical records.
  • Data description: Table E.1 reports binned distributions of continuous variables for the demographic experiment.
  • Dataset construction: The resulting dataset contained 60.1k images with AP/PA views.
  • Prediction setup: Embeddings from BioViL-T, BiomedCLIP, and RAD-DINO were used as inputs to logistic regression models predicting sex, age, weight, and BMI.
  • Prediction setup: The prediction models used five-fold cross-validation with an 80/20 split and were trained for 100 epochs with default scikit-learn LogisticRegression settings.
Loading 2401.10815v3…