PaperlayerView exact revision on arXiv

Source-linked AI summary

Big Self-Supervised Models Advance Medical Image Classification

Shekoofeh Azizi, Basil Mustafa, Fiona Ryan, Zachary Beaver, Jan Freyberg, Jonathan Deaton, Aaron Loh, Alan Karthikesalingam, Simon Kornblith, Ting Chen, Vivek Natarajan, Mohammad Norouzi

arXiv:2101.05224v2eess.IVcs.CVcs.LG

TL;DR

Medical image classification often has limited labeled data, while self-supervised learning has received limited attention in this domain. The paper evaluates self-supervised pretraining with unlabeled natural and medical images, introduces MICLe for multi-image cases, and reports improved dermatology and chest X-ray performance over supervised ImageNet baselines.

  • Problem

    Medical image analysis needs methods for learning from limited labeled data because image annotation is time-consuming and expensive, while self-supervised approaches have received limited attention.

  • Method

    The paper combines self-supervised ImageNet pretraining, additional self-supervised pretraining on unlabeled medical images using MICLe when multiple case images exist, and supervised fine-tuning.

  • Results

    Self-supervised pretraining significantly outperforms supervised ImageNet pretraining across dermatology and chest X-ray classification, including gains of 6.7% in top-1 accuracy and 1.1% in mean AUC.

  • Takeaways & Limitations

    Self-supervised pretraining is more scalable than supervised pretraining because it does not require class-label annotation, and MICLe boosts classifier performance further.

Abstract

from arXiv · show

Self-supervised pretraining followed by supervised fine-tuning has seen success in image recognition, especially when labeled examples are scarce, but has received limited attention in medical image analysis. This paper studies the effectiveness of self-supervised learning as a pretraining strategy for medical image classification. We conduct experiments on two distinct tasks: dermatology skin condition classification from digital camera images and multi-label chest X-ray classification, and demonstrate that self-supervised learning on ImageNet, followed by additional self-supervised learning on unlabeled domain-specific medical images significantly improves the accuracy of medical image classifiers. We introduce a novel Multi-Instance Contrastive Learning (MICLe) method that uses multiple images of the underlying pathology per patient case, when available, to construct more informative positive pairs for self-supervised learning. Combining our contributions, we achieve an improvement of 6.7% in top-1 accuracy and an improvement of 1.1% in mean AUC on dermatology and chest X-ray classification respectively, outperforming strong supervised baselines pretrained on ImageNet. In addition, we show that big self-supervised models are robust to distribution shift and can learn efficiently with a small number of labeled medical images.

1. Introduction

The paper addresses limited labeled data in medical image analysis by evaluating self-supervised pretraining with unlabeled medical images across dermatology and chest X-ray classification. It introduces MICLe and reports stronger performance than supervised ImageNet pretraining.

  • Motivation: Limited labeled medical data motivates comparing supervised ImageNet pretraining with self-supervised pretraining on unlabeled images.Medical-image annotation is time-consuming and expensive, while self-supervised learning has received limited attention in this domain.
  • Approach: The training pipeline uses self-supervised ImageNet pretraining, additional self-supervised pretraining on unlabeled medical images, and supervised fine-tuning.The additional medical-image stage and fine-tuning are task- and dataset-specific.
  • Results: Self-supervised pretraining on unlabeled medical images significantly outperformed standard ImageNet pretraining and random initialization.The comparison was conducted across two distinct medical image classification tasks.
  • Approach: MICLe constructs positive pairs from distinct images of the same underlying pathology when multiple patient-case images are available.This exposes the representation to viewpoint, imaging-condition, and other confounding-factor changes without requiring class labels.
  • Results: 6.7% in top-1 accuracy was gained on dermatology classification, while self-supervised learning improved chest X-ray mean AUC by 1.1% over supervised ImageNet baselines.The experiments cover digital-camera dermatology images and multi-label chest X-ray classification.

2. Related Work

Related work situates the paper between conventional natural-image transfer learning and emerging medical self-supervision. It also identifies semi-supervised learning as complementary and MICLe as a generalization of multi-view representation learning.

  • Transfer Learning for Medical Image Analysis: Transfer learning from natural images is common in medical imaging despite differences in image statistics, scale, and task-relevant features.Prior investigations indicate that natural-image transfer does not always improve medical imaging performance.
  • Multi-Instance Contrastive Learning: MICLe generalizes representation learning from multiple views in video to image datasets containing sets of images with the same desired class information.The related video methods maximize agreement between adjacent frames or views of the same action.
  • Self-supervision for Medical Image Analysis: Medical self-supervision includes domain-specific pretext tasks and contrastive-learning adaptations, with this work most closely related to MoCo pretraining.The supplied passage frames these as recent applications within the medical domain.
  • Complementary Methods: Semi-supervised medical-imaging methods are complementary to self-supervised pretraining and may be combined in future work.The paper specifically identifies combining self-training and self-supervised pretraining as an interesting avenue.

3. Self-Supervised Pretraining

The framework learns representations with contrastive pretraining on unlabeled images, optionally using multiple images per patient case through MICLe, before supervised medical-image fine-tuning. MICLe improves robustness to viewpoint and lighting changes and raises dermatology accuracy.

  • Framework: Contrastive pretraining learns visual representations from unlabeled ImageNet and task-specific medical images before supervised fine-tuning.The pipeline first uses contrastive learning, then optionally performs MICLe-based medical pretraining, and finally fine-tunes on labeled medical images.
  • Contrastive Learning: SimCLR maximizes agreement between differently augmented views of the same image through a contrastive loss.Random crops, color distortion, and Gaussian blur create two views that are encoded and projected for the loss.
  • Multi-Instance Contrastive Learning: MICLe forms positive pairs from crops of distinct images belonging to the same patient and pathology rather than two augmentations of one image.Each patient case is treated as a bag of potentially varying numbers of images captured from different views.
  • Multi-Instance Contrastive Learning: Multi-instance contrastive learning helps representations become robust to viewpoint, lighting, and other confounding-factor changes.The method targets invariance across images of the same pathology, not only across augmentations of one image.
  • Results: MICLe significantly improves accuracy and supports state-of-the-art dermatology condition classification results.The reported improvement follows from leveraging multiple images of the same condition with contrastive learning.

4. Experiment Setup

The study evaluates self-supervised pretraining across dermatology and chest X-ray classification, using unlabeled medical data, MICLe, and supervised fine-tuning. It compares these strategies across datasets, architectures, and transfer settings.

  • Tasks and datasets: The experiments cover dermatology skin-condition classification from digital camera images and multi-label chest X-ray classification among five pathologies.The chest X-ray task uses the CheXpert dataset.
  • Multi-Instance Contrastive Learning: MICLe uses multiple images of the same pathology to learn representations invariant to both image augmentations and different images of that pathology.This design exploits complementary information from multiple patient images.
  • Tasks and datasets: The dermatology data contain heterogeneous consumer-camera images with variations in pose, lighting, blur, body parts, and background artifacts.Cases include between one and six images, with development and test sets split to avoid patient overlap.
  • Experimental comparisons: The study compares ResNet-50 (1×), ResNet-50 (4×), and ResNet-152 (2×) encoders under ImageNet, medical-only, and ImageNet-initialized medical pretraining.Table 1 reports top-1 accuracy and AUC across architectures; its caption states that bigger models perform better and ImageNet pretraining is complementary to medical pretraining.

5. Experiments & Results

Experiments across dermatology and chest X-ray classification evaluate self-supervised pretraining, MICLe, supervised baselines, robustness to distribution shift, and label efficiency.

  • Pretraining data: Combining ImageNet and task-specific unlabeled medical data achieved the best pretraining performance across dermatology and chest X-ray classification.The study compares ImageNet-only, medical-data-only, and combined self-supervised pretraining scenarios.
  • Multi-Instance Contrastive Learning: MICLe consistently improved dermatology classification accuracy over SimCLR across different pretraining datasets and architectures.MICLe uses multiple images per patient case to form more informative positive pairs.
  • Comparison with supervised transfer learning: 70.02 ± 0.22% top-1 accuracy was achieved with ResNet-152 (2×) and MICLe using ImageNet and Derm data.Longer training and a larger batch size further improved performance by providing more negative examples.
  • Comparison with supervised transfer learning: 6.7% top-1 accuracy and 1.1% mean AUC improvements over supervised ImageNet-pretrained baselines were observed for dermatology and chest X-ray classification, respectively.The dermatology comparison used MICLe, whereas the chest X-ray improvement was obtained without MICLe.
  • Self-supervised models generalize better: Self-supervised models generalized better under distribution shift, with combined ImageNet and in-domain pretraining producing further gains over single-source pretraining.The shifted-dataset evaluation used zero-shot transfer without further fine-tuning, and the improvement was more pronounced on the shifted dataset than on CheXpert.
  • Label efficiency: MICLe yielded proportionally larger gains with fewer labeled examples and matched baselines using 20% or 30% of training data, depending on architecture.The matching fractions were 20% for ResNet-50 (4×) and 30% for ResNet-152 (2×).

6. Conclusion

The paper finds that self-supervised pretraining on unlabeled natural and medical images improves medical image classification, with multiple images per case further enhancing performance.

  • Conclusion: Self-supervised pretraining significantly outperformed supervised pretraining for medical image classification.The approach used unlabeled natural and medical images and did not require class-label annotation.
  • Conclusion: Using multiple images per medical case enhanced self-supervised data augmentation and further boosted image-classifier performance.The paper identifies larger unlabeled medical datasets and cross-modality transfer as directions for future research.

A. Datasets

The experiments use dermatology and CheXpert datasets with distinct class distributions, multi-image patient cases, and custom data splits for evaluation.

  • Dermatology dataset: The Derm dataset contains 26 skin-condition classes with highly skewed prevalence across conditions.Conditions such as acne, eczema, and psoriasis exceed 10% prevalence, while lentigo, melanoma, and stasis dermatitis are below 1%.
  • Dermatology dataset: Dermatology cases may include multiple images from different viewpoints, body parts, and lighting conditions, enabling MICLe positive-pair construction.MICLe randomly selects two images from the same patient and condition when multiple images are available.
  • Dermatology dataset: The external dermatology dataset contains 8,563 de-identified multi-image cases and has higher prevalence of several skin cancers than the in-distribution dataset.The dataset was collected by skin-cancer clinics in Australia and New Zealand for out-of-distribution evaluation.
  • CheXpert dataset: CheXpert classification targets five thoracic pathologies using frontal and lateral chest radiographs.The default training set contains more than 200,000 images, while its validation set contains only 200 images.
  • CheXpert dataset: The study re-split CheXpert into 67,429 training, 22,240 validation, and 33,745 test images, making results incompatible with the original benchmark leaderboard.The authors consider the relative model performance representative and comparable with selected prior work.

B.1.1 Evaluation Details and Statistical Significance Testing

Evaluation uses top-k dermatology metrics, bootstrap-based significance analysis, and augmentation comparisons for MICLe.

  • Evaluation metrics: Dermatology performance is measured with top-k accuracy and average top-k sensitivity across 26 conditions.Average top-k sensitivity reduces bias toward more common conditions; the paper reports top-1 and top-3 metrics.
  • Statistical significance testing: Bootstrap confidence intervals were used to assess variability and statistical significance for selected model comparisons.The significance analysis selects the best fine-tuned model for each architecture rather than relying only on repeated-run confidence intervals.
  • Statistical significance testing: MICLe models significantly outperformed their SimCLR counterparts and the BiT supervised model on dermatology top-1 and top-3 accuracy.The reported significance threshold was p < 0.05.
  • Augmentation analysis: The augmentation study compares standard SimCLR transformations with lightweight random cropping combined with multi-instance pair selection.This isolates the contribution of pair selection from the contribution of conventional image augmentation.

B.1.3 Benefits of Longer Training

Longer self-supervised training improves dermatology classification and narrows the performance gap between ImageNet-initialized and medical-image-only models. The gains extend across label fractions and are more pronounced for larger architectures.

  • Training duration: Longer training provides more negative examples and improves dermatology top-1 accuracy.The experiments use fixed batch size and compare 150K with 450K training steps.
  • Training duration: ImageNet initialization facilitates faster convergence, while its performance advantage narrows with longer training.Medical-image-only models increasingly approach ImageNet-initialized models as training steps increase.
  • Label fractions: Longer training yields proportionally larger gains across dermatology label fractions.This pattern is reported for models pretrained with SimCLR on ImageNet→Derm data.
  • Architecture: The longer-training benefit is more pronounced for ResNet-152 (2×).For this architecture, longer self-supervised pretraining can match the supervised baseline using fewer labeled examples.

B.1.4 Detailed Performance Results

Detailed dermatology results show that larger models and combined ImageNet plus in-domain self-supervised pretraining improve performance. MICLe further improves classification over SimCLR across dataset settings.

  • Architecture and pretraining: Larger models yield better dermatology top-1 accuracy, top-3 accuracy, sensitivity, and AUC.The comparison spans architectures and pretrained representations from ImageNet, unlabeled medical data, or both.
  • Architecture and pretraining: Self-supervised pretraining on both ImageNet and in-domain Derm data outperforms self-supervised pretraining on either dataset alone.The benefit is reported in the detailed dermatology results.
  • Baseline comparison: Self-supervised models significantly outperform Random and Supervised pretraining baselines, even with ResNet-50 (1×).The reported statistical comparison has p < 0.05.
  • MICLe: MICLe consistently improves skin-condition classification over SimCLR, with significant top-1 accuracy gains for each dataset setting.The result is reported across representations learned from different unlabeled datasets.
  • Evaluation: Dermatology performance is evaluated with top-1 accuracy, top-3 accuracy, and AUC across MICLe, SimCLR, and supervised models.The comparisons vary the unlabeled pretraining dataset, label fraction, and architecture.

B.1.5 Detailed Label Efficiency Results

Self-supervised pretraining improves dermatology label efficiency across labeled-data fractions, with MICLe producing larger gains when labels are scarce. The subgroup analysis examines performance across four skin types.

  • Label efficiency: Self-supervised models outperform the supervised baseline across all dermatology labeled-data fractions.The results support improved label efficiency for medical image classification.
  • Label efficiency: MICLe yields proportionally larger gains when fine-tuning with fewer labeled examples.This pattern is consistent across top-1 and top-3 accuracy, sensitivity, and AUC.
  • Label fractions: The label-efficiency comparison includes SimCLR and MICLe fine-tuned on 10%, 50%, and 90% of labeled dermatology data.A supervised ImageNet model using 100% labels is included as a reference.
  • Skin-type subgroups: The dermatology subgroup analysis compares top-1 and top-3 accuracy across white, beige, brown, and dark brown skin types.The analysis assesses whether self-supervised gains are evenly distributed across subgroups.
  • Skin-type subgroups: Self-supervised models perform much better on rare skin-type subgroups.The figure reports subgroup performance differences between self-supervised and other models.

B.2.1 Detailed Performance Results

On CheXpert chest X-ray classification, combined ImageNet and in-domain self-supervised pretraining is beneficial, but label-efficiency gains are less pronounced than in dermatology. Performance varies across pathologies and architectures.

  • Pretraining: Self-supervised pretraining on both ImageNet and in-domain medical data outperforms pretraining on ImageNet or CheXpert alone.This result is reported for the five-pathology CheXpert classification task.
  • Task and metrics: CheXpert classification detects atelectasis, cardiomegaly, consolidation, edema, and pleural effusion.Performance is measured with AUC for each pathology.
  • Task and metrics: AUC gains across CheXpert pathologies are mixed for transfer learning with self-supervised and supervised models.The table reports pathology-specific AUC performance rather than a uniform improvement across all pathologies.
  • Label efficiency: Self-supervised models consistently outperform the supervised baseline for ResNet-50 (4×) across chest X-ray label fractions.The trend is less striking for ResNet-152 (2×) models.
  • Label efficiency: Chest X-ray label-efficiency improvement is less pronounced than dermatology label-efficiency improvement.The authors suggest that additional in-domain unlabeled data could improve chest X-ray classification.
Loading 2101.05224v2…