PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Learning of Medical Visual Representations from Paired Images and Text

Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D. Manning, Curtis P. Langlotz

arXiv:2010.00747v2cs.CVcs.CLcs.LG

TL;DR

Medical visual representation learning is limited by scarce annotations and by the mismatch or inefficiency of common pretraining and labeling strategies. ConVIRT pretrains medical image encoders by contrasting naturally paired images and text in both directions. It outperformed strong baselines across classification and retrieval evaluations, achieving comparable or better classification with 10% as much labeled data as ImageNet initialization.

  • Problem

    Medical image understanding lacks abundant expert annotations, while ImageNet transfer, rule-based report labels, and image-only contrastive learning are limited for this domain.

  • Method

    ConVIRT learns medical visual representations by maximizing agreement between paired image-text representations and random pairs with a bidirectional contrastive objective.

  • Results

    Across 4 classification tasks and 2 retrieval tasks, ConVIRT outperformed strong baselines and achieved better or comparable classification performance using 10% as much labeled data as ImageNet initialization.

  • Takeaways & Limitations

    Paired descriptive text can provide an effective, data-efficient source for medical visual pretraining without additional expert input.

  • Takeaways & Limitations

    The study did not compare ConVIRT with subsequent extensions such as LoVT or GloRIA because those comparisons were included in the later studies.

Abstract

from arXiv · show

Learning visual representations of medical images (e.g., X-rays) is core to medical image understanding but its progress has been held back by the scarcity of human annotations. Existing work commonly relies on fine-tuning weights transferred from ImageNet pretraining, which is suboptimal due to drastically different image characteristics, or rule-based label extraction from the textual report data paired with medical images, which is inaccurate and hard to generalize. Meanwhile, several recent studies show exciting results from unsupervised contrastive learning from natural images, but we find these methods help little on medical images because of their high inter-class similarity. We propose ConVIRT, an alternative unsupervised strategy to learn medical visual representations by exploiting naturally occurring paired descriptive text. Our new method of pretraining medical image encoders with the paired text data via a bidirectional contrastive objective between the two modalities is domain-agnostic, and requires no additional expert input. We test ConVIRT by transferring our pretrained weights to 4 medical image classification tasks and 2 zero-shot retrieval tasks, and show that it leads to image representations that considerably outperform strong baselines in most settings. Notably, in all 4 classification tasks, our method requires only 10\% as much labeled training data as an ImageNet initialized counterpart to achieve better or comparable performance, demonstrating superior data efficiency.

1. Introduction

Medical image understanding is constrained by subtle distinctions, scarce annotations, and limitations in existing transfer, rule-based, and image-only contrastive approaches. ConVIRT exploits naturally paired image-text data with bidirectional contrastive learning, outperforming baselines while improving labeled-data efficiency.

  • Medical image understanding remains difficult because visually similar images can differ by subtle abnormalities and expert annotations are extremely scarce.
  • Expert annotation is costly, producing datasets orders of magnitude smaller than natural-image resources and encouraging ImageNet weight transfer.
  • Rule-based report labeling scales through naturally produced text but is inaccurate, category-limited, and inefficient in its use of reports.
  • Image-view contrastive methods provide only marginal benefits over ImageNet pretraining on medical images, largely because of high inter-class similarity.
  • ConVIRT maximizes agreement between true image-text pairs and random pairs through a bidirectional contrastive objective, requiring no additional expert input.
  • Across four classification tasks and two zero-shot retrieval tasks, ConVIRT outperformed strong baselines; classification matched or exceeded ImageNet initialization using 10% as much labeled data.
  • ConVIRT reuses descriptive text naturally produced by experts through cross-modality learning, supporting downstream predictive tasks with reduced annotation cost.

2. Related Work

Related work includes ImageNet initialization, visual-linguistic pretraining, and general-domain image-text representation learning. ConVIRT instead applies cross-modal contrastive pretraining to medical visual representations, later inspiring larger-scale adaptations and medical-domain extensions.

  • Medical image studies commonly initialize encoders with ImageNet weights despite substantial differences between natural and medical image characteristics.
  • Prior visual-linguistic studies explored related cross-modality objectives for probing visual-linguistic models and phrase grounding, whereas ConVIRT targets medical visual representation learning.
  • Concurrent general-domain studies also investigated learning visual representations from text data, providing a broader context for ConVIRT-style approaches.
  • Larger-scale adaptations of ConVIRT, including CLIP and ALIGN, reported state-of-the-art general visual recognition capabilities.
  • ConVIRT was followed by numerous medical-domain studies applying multimodal contrastive learning to medical imaging.

3. Methods

ConVIRT learns medical image representations from naturally paired image-text data by mapping both modalities into a shared space and maximizing agreement between true pairs. It uses bidirectional contrastive losses with stochastic image and text transformations, while remaining agnostic to encoder and projection choices.

  • Representation learning: ConVIRT samples transformed image views and text spans before producing image and text representations.Image transformations include cropping, flipping, affine transformation, color jittering, and Gaussian blur; text uses uniform span sampling.
  • Representation learning: Paired images and text are encoded into d-dimensional vectors for cross-modal contrastive learning.Image and text projection functions map their encoder outputs into the same d-dimensional space.
  • Contrastive objective: The objective combines image-to-text and text-to-image contrastive losses over minibatches of paired representations.The image-to-text loss compares each true pair against other text representations using cosine similarity and a temperature parameter.
  • Contrastive objective: Minimizing the contrastive loss encourages representations that preserve mutual information between true image-text pairs.The image-to-text loss is asymmetric across modalities and has the form of an N-way classifier predicting the true pair.
  • Model realization: The framework is agnostic to the specific image and text encoders, transformations, and projection functions.The experiments use ResNet50 for images and ClinicalBERT with max pooling for text, fine-tuning only the last six transformer layers.
  • Model realization: Preliminary experiments found sampled image views outperform direct image inputs or fusion of multiple study images for pretraining.The authors conjecture that sampled views act as visual data augmentation and increase the effective number of unique image-text pairs.

4. Experiments

The experiments pretrain separate chest and bone image encoders from paired image-text datasets, then evaluate them across classification and zero-shot retrieval tasks against standard and competitive initialization baselines.

  • Pretraining data: ConVIRT is evaluated using separate chest and bone image encoders pretrained on paired medical image-text datasets.The chest encoder uses about 217k MIMIC-CXR image-text pairs, while the bone encoder uses 48k pairs from the Rhode Island Hospital system.
  • Evaluation tasks: The evaluation covers four medical image classification tasks, zero-shot image-image retrieval, and zero-shot text-image retrieval.The retrieval evaluation uses the CheXpert 8×200 Retrieval Dataset, covering eight chest abnormality categories with Precision@k metrics for k = 5, 10, 100.
  • Evaluation protocol: Classification is assessed with both frozen-weight linear evaluation and end-to-end fine-tuning of the pretrained CNN and classification head.These settings are evaluated with 1%, 10%, and all training data, except that COVIDx omits the 1% setting.
  • Baseline methods: Baselines include random initialization, ImageNet initialization, two captioning-based methods, and a binary image-text matching objective.Caption-LSTM and Caption-Transformer use paired reports for caption-based pretraining, while Contrastive-Binary-Loss classifies real versus randomly mismatched image-text pairs.
  • Baseline methods: Paired-image-text baselines use the same pretraining datasets as ConVIRT, and captioning methods are selected by best held-out validation CIDEr score.This protocol controls the paired-data comparison across pretraining approaches.

5. Results

ConVIRT outperforms the compared initialization methods across most classification and retrieval settings, while also showing strong performance with substantially less labeled data. Its representations produce better category clustering despite the high inter-class similarity of medical images.

  • Linear Classification: ConVIRT achieves the best overall linear-classification results in all settings, and with 1% training data surpasses ImageNet initialization using 100% data on three of four tasks.In-domain image-text initialization methods generally improve over ImageNet, with ConVIRT performing best overall.
  • Fine-tuning: ConVIRT achieves the best overall fine-tuning results in 10 of 11 settings, except for CheXpert with all training data, where it is similar to Caption-Transformer.With 10% labeled data on all datasets, ConVIRT is better or close to ImageNet initialization trained with 100% of the data.
  • Initialization comparison: ImageNet initialization remains superior to random initialization in most reported results, contrary to an earlier comparison that found comparable performance.The authors conjecture that the earlier result may reflect under-optimization of the compared models.
  • Zero-shot retrieval: ConVIRT achieves the best overall results on all zero-shot image-image and text-image retrieval metrics.All in-domain pretrained CNNs outperform ImageNet weights in retrieval; Contrastive-Binary-Loss performs better on image-image retrieval but trails ConVIRT on text-image retrieval.
  • Representation analysis: The t-SNE visualizations show better clustering for ConVIRT representations despite the high inter-class similarity of medical images.The comparison uses candidate images from five selected categories in the CheXpert 8×200 dataset.

6. Analysis and Discussion

ConVIRT substantially outperforms image-only contrastive methods by using paired text, while its learned representations focus on more relevant medical regions. Downstream performance correlates positively with pretraining performance, but results depend on hyperparameter choices and comparisons omit some later methods.

  • Image-only contrastive learning: ConVIRT substantially outperforms SimCLR and MoCo v2 across classification and retrieval tasks by exploiting paired text data.Both image-only methods provide marginal to moderate improvements over ImageNet initialization.
  • Representation analysis: ConVIRT saliency maps most closely match ground-truth abnormality regions across four CheXpert categories.Under 1% CheXpert linear evaluation, ImageNet focuses on trivial features, while SimCLR and MoCo v2 show scattered or incorrect attention.
  • Pretraining analysis: Pretraining performance shows a clear positive correlation with downstream performance on RSNA 1% linear evaluation and two retrieval tasks.The analysis transfers checkpoints every five epochs across 200 pretraining epochs.
  • Hyperparameter analysis: Temperature selection is critical: τ = 0.01 hurts retrieval, while τ = 1 notably hurts performance on all tasks.Changing batch size has little effect on classification, whereas linear projection heads moderately hurt retrieval results without notably affecting RSNA classification.
  • Limitations: The study compares ConVIRT with ImageNet initialization, image captioning-based initialization, and image-only contrastive learning, but not later extensions such as LoVT or GloRIA.Those subsequent studies reportedly include the relevant comparisons themselves.

7. Conclusion

ConVIRT learns medical visual representations by contrasting paired images and descriptive text. Across four classification and two retrieval tasks, it outperforms strong in-domain initialization methods and reaches ImageNet-comparable classification accuracy with an order of magnitude less labeled data.

  • Conclusion: ConVIRT is an unsupervised method that learns medical visual representations from paired descriptive text using a bidirectional cross-modal objective.The method contrasts image representations with their paired text.
  • Conclusion: ConVIRT outperforms other strong in-domain initialization methods on four medical image classification tasks and two image retrieval tasks.The conclusion characterizes the resulting representations as notably higher quality.
  • Conclusion: ConVIRT achieves the same classification accuracy as ImageNet pretraining with an order of magnitude less labeled data.The conclusion connects this result to data sparsity in healthcare.

Appendix A. Model Implementation and Pretraining Details

The implementation uses ResNet50 and ClinicalBERT encoders with standardized preprocessing and contrastive-learning hyperparameters. Pretraining uses held-out image-text validation pairs, Adam optimization, and ImageNet initialization for the image encoder.

  • Preprocessing: Radiology reports are tokenized with CoreNLP, and only the Findings and Impression sections are retained.The passage describes preprocessing for textual report data.
  • Encoders: Images use a torchvision ResNet50 encoder, while text uses BERT base initialized with ClinicalBERT pretrained on MIMIC clinical notes.Training a specialized BERT encoder on radiology notes made no substantial difference.
  • Hyperparameters: Contrastive pretraining uses projection dimension d = 512, temperature τ = 0.1, and loss weight λ = 0.75.These settings were selected using RSNA linear-evaluation validation scores.
  • Pretraining: Each dataset contributes 5k randomly sampled image-text pairs to held-out validation, with Adam optimization, learning rate 1e-4, weight decay 1e-6, and batch size 32.The image encoder is initialized with ImageNet weights before pretraining.

Appendix B. Image Classification Experiments

The classification experiments cover four medical imaging datasets with task-specific dataset splits and standardized training procedures. Evaluation uses ImageNet-pretrained weights where required, validation-based learning-rate annealing, and early stopping.

  • Datasets: The experiments use four classification datasets: RSNA Pneumonia Detection, CheXpert, COVIDx, and MURA bony abnormality detection.The datasets cover pneumonia, chest radiograph, COVIDx, and bone abnormality tasks.
  • Dataset splits: RSNA Pneumonia Detection contains 25,184/1,500/3,000 annotated training, validation, and test images.The dataset uses its original Kaggle version.
  • Dataset splits: CheXpert and MURA use their original validation sets as test sets, with sampled training images reserved for validation.CheXpert uses 5,000 sampled training images, while MURA uses 10% of training images.
  • Dataset splits: COVIDx version 4 is split into 13,598 training, 300 validation, and 300 test images.The validation set is randomly sampled from the training set.
  • Training procedure: Inputs are zero-padded to square and resized to 224×224, while validation scores control learning-rate annealing and stopping.The learning rate is halved after three unimproved epochs, and training stops after ten.

Appendix C. Image-image Retrieval Dataset Collection

The CheXpert 8×200 Retrieval Dataset covers eight chest-radiograph abnormality categories using rule-labeled and expert annotations. Candidate and query images are screened to isolate category-specific, well-posed examples.

  • The dataset contains 200 images for each of eight categories, including seven abnormalities and a no finding category.The categories are atelectasis, cardiomegaly, edema, fracture, pleural effusion, pneumonia, pneumothorax, and no finding.
  • Dataset construction reuses CheXpert rule-labeled annotations and adds expert annotations.
  • For each query category, 50 exclusively positive training images are pre-selected with all candidate images excluded.
  • A board-certified radiologist excludes query images with additional abnormalities, unusual color or contrast distortions, or poor capture positioning.The screening is intended to avoid images that could confound category-specific retrieval.

Appendix D. Text-image Retrieval Dataset Collection

The text-image retrieval dataset reuses the candidate images from the image-image benchmark and pairs each abnormality category with radiologist-written textual queries. Query sentences are designed to be unambiguous, diverse, and broadly applicable.

  • The text-image retrieval dataset reuses 200 candidate images for each of eight categories from the CheXpert image-image retrieval dataset.
  • For each abnormality category, a board-certified radiologist writes at least five sentences suitable for describing it in radiology reports.
  • The textual queries must describe only the target category without ambiguity or other categories.
  • The sentences must be diverse and avoid very specific anatomic locations or rare clinical details.

Appendix E. Experiments on Image-Only Contrastive Learning Methods

The appendix evaluates SimCLR and MoCo v2 as image-only contrastive learning baselines using the same MIMIC-CXR pretraining images as the proposed method. Both use ResNet50 encoders and medical-image-specific removal of saturation and hue jittering.

  • SimCLR and MoCo v2 are evaluated as image-only contrastive visual representation learning methods.
  • Both baselines use exactly the same MIMIC-CXR image set as the proposed method and other baselines for pretraining.
  • SimCLR: SimCLR uses a ResNet50 encoder, cosine similarity, temperature 0.1, and output dimension 128.
  • Both methods remove saturation and hue adjustments from color jittering because the medical images are monochrome.
  • MoCo v2: MoCo v2 uses a ResNet50 encoder, temperature 0.07, and output dimension 128 under its default setting.
Loading 2010.00747v2…