PaperlayerView exact revision on arXiv

Source-linked AI summary

MedAug: Contrastive learning leveraging patient metadata improves representations for chest X-ray interpretation

Yen Nhi Truong Vu, Richard Wang, Niranjan Balachandar, Can Liu, Andrew Y. Ng, Pranav Rajpurkar

arXiv:2102.10663v2eess.IVcs.CVcs.LG

TL;DR

Medical contrastive learning has limited systematic evidence on how patient metadata can guide positive-pair selection. MedAug uses metadata to pair different images during pretraining; metadata-matched pairing improved representations, with the best reported result 14.4% higher mean AUC than the ImageNet pretrained baseline.

  • Problem

    There has been limited work systematically investigating how patient metadata can guide positive-pair selection beyond augmentations of the same image in medical imaging.

    Source: §1 · p. 2

  • Method

    MedAug uses patient metadata to add different, metadata-matched images to the positive-pair choices during contrastive pretraining.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    14.4% increase in mean AUC over the ImageNet pretrained baseline was achieved by the best pretrained representation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Selecting positive pairs with study information to target shared disease information is needed for the best downstream disease-classification representation.

    Source: §5 · p. 10

    The approach is not applicable to datasets lacking patient metadata, and clinically relevant pair strategies for datasets with different metadata constraints remain future work.

    Source: §5 · p. 11

Abstract

from arXiv · show

Self-supervised contrastive learning between pairs of multiple views of the same image has been shown to successfully leverage unlabeled data to produce meaningful visual representations for both natural and medical images. However, there has been limited work on determining how to select pairs for medical images, where availability of patient metadata can be leveraged to improve representations. In this work, we develop a method to select positive pairs coming from views of possibly different images through the use of patient metadata. We compare strategies for selecting positive pairs for chest X-ray interpretation including requiring them to be from the same patient, imaging study or laterality. We evaluate downstream task performance by fine-tuning the linear layer on 1% of the labeled dataset for pleural effusion classification. Our best performing positive pair selection strategy, which involves using images from the same patient from the same study across all lateralities, achieves a performance increase of 14.4% in mean AUC from the ImageNet pretrained baseline. Our controlled experiments show that the keys to improving downstream performance on disease classification are (1) using patient metadata to appropriately create positive pairs from different images with the same underlying pathologies, and (2) maximizing the number of different images used in query pairing. In addition, we explore leveraging patient metadata to select hard negative pairs for contrastive learning, but do not find improvement over baselines that do not use metadata. Our method is broadly applicable to medical image interpretation and allows flexibility for incorporating medical insights in choosing pairs for contrastive learning.

1. Introduction

MedAug uses patient metadata to select positive pairs across different chest X-ray images, addressing limited systematic study of metadata-guided pairing. The paper reports improved representations and discusses how metadata-informed pairing may extend to other medical-imaging tasks.

1. Introduction

  • Despite early contrastive-learning successes in medical imaging, systematic investigation of metadata to select positive pairs beyond same-image augmentations remained limited.Prior work had applied contrastive learning to chest X-rays, dermatology classification, and MRI segmentation.Source: §1 · pp. 1–2
  • MedAug uses patient metadata, including patient, study, and laterality information, to select different images as positive pairs during contrastive pretraining.Unlike a second pretraining stage, these pairs are incorporated directly into the view-generation scheme.Source: §1 · p. 2
  • 14.4% increase in mean AUC over the ImageNet pretrained baseline was achieved by the best pretrained representation.The result is reported for the pleural effusion classification evaluation.Source: §1 · p. 2
  • Using different images from the same patients that share underlying pathologies, and increasing distinct images per query, improved pretrained representations.The paper also reports that metadata-based negative-pair selection did not improve over the default strategy.Source: §1 · p. 3

Generalizable Insights about Machine Learning in the Context of Healthcare

  • The authors suggest that metadata-informed positive-pair selection and larger image pools may benefit other medical-imaging tasks where such metadata is available.The approach is evaluated on chest X-ray classification; broader applicability is presented as a generalizable insight.Source: Generalizable Insights about Machine… · p. 3

2. Cohort

The cohort uses CheXpert chest X-rays for pretraining and a 1% labeled sample for fine-tuning pleural effusion classification. Evaluation uses a separate labeled test set and AUC.

2. Cohort

  • CheXpert contains 224,316 images from 65,240 patients, labeled for 14 radiological observations.The images are used for pretraining, with random samples of 1% used for fine-tuning.Source: §2 · p. 3
  • The test set comprises 500 labeled images from 500 studies excluded from training.Source: §2 · p. 4
  • Fine-tuning evaluates pleural effusion classification, selected for ground-truth quality, clinical importance, and prevalence, using test-set AUC.Source: §2 · p. 4

3. Methods

MedAug expands contrastive-learning positive pairs using metadata-matched images, with selection criteria chosen according to the downstream task. The study evaluates pretrained representations using linear classification and end-to-end fine-tuning under label scarcity.

3.1. Selecting positive pairs for contrastive learning with patient metadata

  • The contrastive setup treats two augmentations of an input image as a positive pair and augmentations of different images as negatives.Negatives come from a queue in MoCo or the minibatch in SimCLR.Source: §3.1 · p. 4
  • MedAug expands the augmentation set with transformed images that meet metadata criteria relative to the query, falling back to ordinary augmentations when no qualifying images exist.The criteria can be informed by clinical insights about the downstream task.Source: §3.1 · p. 4
  • For chest X-rays, the method samples a qualifying image from the same patient, applying study-number and laterality criteria, then pairs its augmentation with an augmented query.The models use ResNet-18 and MoCo v2 pretraining.Source: §3.1 · p. 4

3.2. Fine-tuning and evaluation

  • Representation quality is evaluated with a linear classifier on frozen encoder outputs and through end-to-end fine-tuning.Pretrained checkpoints are selected using a k-nearest-neighbors approach.Source: §3.2 · p. 5
  • Fine-tuning uses five randomly drawn 1% labeled-data splits, with mean AUC and standard deviation reported across splits.The design simulates label scarcity in medical settings.Source: §3.2 · p. 5

4. Experiments

The experiments compare metadata-based rules for selecting positive pairs, then test how pair labels, study membership, set size, and laterality relate to downstream performance. Metadata-based negative-pair strategies are also evaluated against the default strategy.

4.1. Positive pair selection

  • The same-patient, same-study, all-lateralities strategy gives the best result, with AUC gains of 0.029 (3.4%) for the linear model and 0.021 (2.4%) for the end-to-end model.Except when criteria include images from different studies, same-patient pairing improves AUC over the baseline.Source: §4.1 · p. 5, Table 1 · p. 6

4.2. Comparative Empirical Analysis

  • The comparative analysis examines how patient metadata criteria affect downstream pleural-effusion classification performance.It compares positive-pair selection criteria using label information.Source: §4.2 · p. 6

4.2.1. All studies v.s. same study

  • Using all studies while matching positive pairs by downstream label improves AUC over the all-studies strategy without label matching by 0.034 for linear and 0.022 for end-to-end models.The experiment supports the hypothesis that positive pairs from images with different downstream labels hurt performance.Source: §4.2.1 · p. 6, Table 2 · p. 7

4.2.2. All studies v.s. distinct studies

  • With pair-set size controlled, the same-patient, all-studies strategy achieves AUC 0.848 versus 0.792 for distinct-studies pairing.Almost 9% of queries have only different-label images in their distinct-studies sets, a scenario absent from all-studies sets.Source: §4.2.2 · p. 7, Table 3 · p. 7, Figure 2 · p. 8

4.2.3. All lateralities v.s. distinct lateralities v.s. same laterality

  • When pair-set size is controlled, same-laterality and all-laterality strategies have AUC 0.862 and 0.860, while using all available images across lateralities reaches 0.876.Different-laterality-only pairing gains 0.010 AUC for linear fine-tuning and a non-significant 0.004 for end-to-end fine-tuning.Source: Table 4 · p. 8, Table 5 · p. 8, §4.2.3 · p. 9

4.3. Negative pair selection

  • None of the metadata-based negative-pair strategies outperforms the default: the default AUC is 0.876, versus 0.872, 0.864, 0.875, and 0.870 for the tested variants.The authors note that further experiments with other metadata sources may be needed.Source: §4.3 · pp. 9–10, Table 6 · p. 10

5. Discussion

MedAug’s discussion finds that positive-pair selection needs appropriate patient and study metadata, while simply pairing images from the same patient can hurt performance. It also identifies limits on metadata-based negative pairing and on generalizing the approach beyond the evaluated setting.

5. Discussion

  • 3.4% higher linear fine-tuning AUC was achieved by the same-patient, same-study strategy versus the instance discrimination approach.The authors also note related same-patient pairing results in ECG and dermatology studies.Source: §5 · p. 10
  • Using same-patient images alone may hurt downstream performance; study metadata is needed to select pairs sharing underlying disease information.The authors identify age group and medical history, and tasks beyond disease classification, as future directions.Source: §5 · p. 10
  • The number of images used to form positive pairs matters, while the effect of mutual information is less clear.The authors suggest studying how the maximum number of distinct images per query affects downstream performance.Source: §5 · p. 10
  • Metadata-based hard-negative strategies did not outperform the baseline.The authors call for exploration of other metadata and similarity-based strategies for negative-pair selection.Source: §5 · pp. 10–11
  • The authors describe incorporating patient metadata into contrastive learning as potentially beneficial for medical images and extendable to broader tasks.They cite prior work on medical image interpretation as context for this possible extension.Source: §5 · p. 10
  • MedAug is not applicable to datasets without patient metadata, and clinically relevant positive-pair strategies remain to be established for datasets with different metadata constraints.The authors also leave assessment on other datasets for future work.Source: §5 · p. 11
  • Laterality-based negative-pair strategies did not improve pretrained representations; other metadata, including image view, age, and sex, remain to be tested.Similarity-based strategies are another proposed direction for future negative-pair selection.Source: §5 · p. 11

Appendix A. Repetition Across Tasks

Across all five CheXpert competition classification tasks, the same-patient, same-study positive-pair criterion achieved higher downstream classification performance than the ImageNet and MoCo v2 baselines.

Appendix A. Repetition Across Tasks

  • Across all five CheXpert competition tasks, same-patient, same-study positive pairing achieved higher downstream classification performance than the ImageNet and MoCo v2 baselines.The repeated tasks were Atelectasis, Cardiomegaly, Consolidation, and Edema, alongside Pleural Effusion.Source: Table 7 · p. 13, Appendix A · p. 13

Appendix B. Negative Pairs

The appendix describes four metadata-based negative-pair strategies that use laterality to filter, reweight, append, or synthesize queue keys. These strategies modify the negative-pair exponential sum used in the contrastive loss.

Appendix B. Negative Pairs

  • Same-laterality-only selection replaces the negative-pair sum with keys in the queue that match the query’s laterality.This excludes queue keys with different laterality.Source: Appendix B · p. 13
  • Same-laterality reweighting retains negative keys while assigning a target weight to those with the query’s laterality.The experiments set t = 0.1, allocating 90% of the weight to hard negatives with the same laterality.Source: Appendix B · p. 14
  • Same-laterality-appended selection adds a random sample of matching-laterality keys to the existing queue.The modified queue replaces the original negative-pair sum.Source: Appendix B · p. 14
  • Same-laterality-synthetic selection appends synthetic keys generated from randomly sampled pairs of matching-laterality keys.The synthetic representations are normalized, and the modified queue is used in place of the original negative-pair sum.Source: Appendix B · p. 14

Related papers

Loading 2102.10663v2…