Source-linked AI summary
Preservational Learning Improves Self-supervised Medical Image Models by Reconstructing Diverse Contexts
Hong-Yu Zhou, Chixiang Lu, Sibei Yang, Xiaoguang Han, Yizhou Yu
TL;DR
Contrastive learning preserves information implicitly through image-pair comparisons, but the authors argue explicit preservation is complementary. PCRL reconstructs diverse contexts alongside contrastive learning and reports substantial advantages over supervised and self-supervised counterparts across five classification and segmentation tasks.
Problem
Contrastive learning preserves information implicitly, and the authors argue that explicit preservation is also needed to capture more information in learned representations.
Source: §1 · p. 1Method
PCRL combines contrastive loss with reconstruction of diverse contexts using Transformation-conditioned Attention and Cross-model Mixup.
Source: §1 · p. 2Results
PCRL reports substantial advantages over self-supervised and supervised counterparts across five classification and segmentation tasks.
Source: §1 · p. 2Takeaways & Limitations
The authors present PCRL as a possible foundation for real-world medical imaging tasks, reporting positive results across varied tasks and datasets.
Source: §5 · p. 8The authors leave unresolved whether preserving more information is the only reason for improvements over contrastive learning.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Preserving maximal information is one of principles of designing self-supervised learning methodologies. To reach this goal, contrastive learning adopts an implicit way which is contrasting image pairs. However, we believe it is not fully optimal to simply use the contrastive estimation for preservation. Moreover, it is necessary and complemental to introduce an explicit solution to preserve more information. From this perspective, we introduce Preservational Learning to reconstruct diverse image contexts in order to preserve more information in learned representations. Together with the contrastive loss, we present Preservational Contrastive Representation Learning (PCRL) for learning self-supervised medical representations. PCRL provides very competitive results under the pretraining-finetuning protocol, outperforming both self-supervised and supervised counterparts in 5 classification/segmentation tasks substantially.
1. Introduction
The paper introduces PCRL to supplement contrastive learning’s implicit information preservation with explicit reconstruction of diverse contexts. It reports substantial advantages over self-supervised and supervised counterparts across five classification and segmentation tasks.
1. Introduction
- Contrastive learning preserves information implicitly through image-pair comparisons, but the authors argue explicit preservation is complementary.They report that simply adding a plain reconstruction branch for the original inputs did not significantly improve representations.Source: §1 · p. 1
- The paper also aims to provide a fairer, more thorough comparison of self-supervised methods across datasets and modalities.It describes extensive experiments to analyze performance across different datasets and data modalities.Source: §1 · p. 2
- PCRL reconstructs diverse contexts from contrastively learned representations to encode more information.The approach combines Transformation-conditioned Attention and Cross-model Mixup in a triple-encoder, single-decoder architecture.Source: §1 · p. 2
- PCRL shows substantial advantages over self-supervised and supervised counterparts across five classification and segmentation tasks.The paper reports extensive experiments and analyses across these tasks.Source: §1 · p. 2
2. Related Work
The related work covers pretext-based and contrastive self-supervision in natural and medical images, then positions PCRL’s cross-model feature mixing against existing mixup strategies.
Pretext-based self-supervised learning in natural images
- Natural-image pretext methods predict transformation-covariant properties, while contrastive methods learn invariant representations by contrasting image pairs.PCRL instead seeks representations that fully describe their source images.Source: §2 · p. 2
Self-supervised learning in medical image analysis
- Earlier medical-image work includes jigsaw and corrupted-image reconstruction tasks, as well as contrastive approaches for 2D radiographs and 3D volumes.The paper says two closely related reconstruction approaches [8] did not improve contrastive learning through context reconstruction, unlike its method.Source: §2 · p. 2
Mixup in medical imaging
3. Methodology
PCRL combines contrastive learning with reconstruction of diverse contexts in a U-Net-like, triple-encoder and shared-decoder framework. Its modules condition reconstruction on transformations and mix ordinary and momentum features, while the loss combines reconstruction and contrastive objectives.
3. Methodology
- PCRL uses ordinary, momentum, and hybrid encoders with a shared U-Net-like decoder, applying contrastive learning and reconstruction to multiple transformed image targets.Random crop, flip, and rotation generate encoder-specific batches; inpainting, outpainting, and Gaussian blur provide additional processing.Source: §3 · pp. 2–3, Figure 2 · p. 3
3.1. Transformation-conditioned Attention
- Transformation-conditioned Attention embeds a transformation indicator into high-level feature maps so the network can reconstruct different targets from a fixed input.It combines transformation-vector and pooled-feature information to produce channel-wise rescaling; for 2D inputs, the indicator omits F(z).Source: §3.1 · pp. 3–4
3.2. Cross-model Mixup
- Cross-model Mixup combines ordinary- and momentum-encoder feature maps to form a hybrid encoder, which reconstructs a mixed image target.The feature maps are interpolated using a sampled mixing coefficient λ.Source: §3.2 · p. 4
3.3. Loss Functions and Model Update
- The contrastive objective compares ordinary- and momentum-encoder representations against stored negative features, and the momentum encoder is updated by exponential moving average.The paper uses a queue for past features and updates momentum-encoder parameters using factor β.Source: §3.3 · p. 4
- The shared decoder uses mean squared error to reconstruct targets from multi-layer features, and the complete loss combines reconstruction and contrastive losses equally.The hybrid encoder has no separate encoder parameters; it uses mixed ordinary and momentum feature maps as decoder input.Source: §3.3 · p. 4
4. Experiments
The experiments compare self-supervised pretraining methods across 2D and 3D medical-image tasks, using dataset-specific pretraining and finetuning setups.
4. Experiments
- The study frames each task as source-dataset pretraining followed by target-dataset supervised finetuning.The section uses source→target notation for these stages.Source: §4 · p. 4
4.1. Baselines
- Baselines are grouped by input dimension, with separate 2D and 3D method sets.The 2D set includes train-from-scratch, ImageNet pretraining, Model Genesis, Semantic Genesis, and Comparing to Learn; the 3D set includes train-from-scratch, Model Genesis, Semantic Genesis, 3D-CPC, and Cube++.Source: §4.1 · p. 5
4.2. Datasets
- Evaluation spans Chest14 and CheXpert X-rays and LUNA, BraTS, and LiTS CT/MRI tasks, using AUC for classification and mean Dice for segmentation.CheXpert is pretraining-only, BraTS is finetuning-only for cross-modal transfer, and pretraining data are unlabeled training-set data.Source: §4.2 · p. 5
4.3. Implementation Details
The ablation study tests whether reconstructing diverse contexts adds information beyond contrastive learning and whether that information tracks downstream performance. PCRL combines these approaches and is reported to improve performance across labeled-data ratios, especially when labels are scarce.
4.3. Implementation Details
- Implementation uses 2D and 3D U-Net backbones, repeats each experiment three times, and averages the results.For 2D tasks, the U-Net encoder is replaced with ResNet-18.Source: §4.3 · p. 5
4.4. Ablation Study
- Figure 4 compares methods on 2D and 3D tasks, reporting LiTS liver Dice and BraTS mean Dice across whole tumor, tumor core, and enhancing tumor.Its ratios denote pretraining-to-finetuning data for Chest14→Chest14 and LUNA→LUNA, and source-dataset pretraining-data amounts for other tasks.Source: Figure 4 · p. 6
- Reconstructing diverse contexts improves representation information: TransAtt with rotation outperforms self-reconstruction on both pretext tasks, and PCRL achieves higher accuracy than the other variants.The comparisons use rotation-degree and relative-position prediction to assess representation information.Source: Table 1 · p. 5, §4.4 · p. 6
- Pretext-task accuracy and Chest14 performance follow similar trends, which the authors use to connect preserved information with downstream performance.The study treats rotation-degree and relative-position prediction as indicators of information in learned representations.Source: §4.4 · p. 6
- ContraLoss plus self-reconstruction yields only marginal gains, while adding rotation transformations to TransAtt improves overall performance.Reducing CrossMix's α by half slightly lowers overall performance.Source: Table 2 · p. 5, §4.4 · pp. 6–7
- PCRL surpasses ContraLoss by approximately 4 points across labeled-data ratios, with the largest improvement at 10%.The authors interpret the stronger low-label result as indicating greater usefulness of diverse-context reconstruction when labeled data are limited.Source: Table 2 · p. 5, §4.4 · p. 7
4.5. Comparison with State-of-the-Arts in 2D Tasks
Across the 2D tasks, PCRL significantly outperforms the compared pretraining baselines across labeled-data ratios. Its advantage is especially apparent with limited labels, and it remains ahead of C2L and ImageNet pretraining at a 100% labeled ratio.
4.5. Comparison with State-of-the-Arts in 2D Tasks
- PCRL significantly outperforms the compared baselines across labeled ratios in the evaluated 2D tasks, with particular advantages at small ratios.The evaluation covers Chest14→Chest14 and CheXpert→Chest14.Source: §4.5 · pp. 7–8
- In CheXpert→Chest14, Model Genesis and Semantic Genesis are comparable with train-from-scratch when the labeled ratio is at least 50%.The passage describes purely pretext-based approaches as potentially unstable across labeled ratios.Source: §4.5 · pp. 7–8
- Even at a 100% labeled ratio, PCRL outperforms C2L and ImageNet pretraining in CheXpert→Chest14.C2L outperforms ImageNet and other pretraining methods in almost all ratios in this task.Source: §4.5 · p. 8
4.6. Comparison with State-of-the-Arts in 3D Tasks
Across three 3D transfer tasks, PCRL consistently outperforms the prior baselines. The reported advantage is especially notable at lower labeled ratios in LUNA→LiTS and persists when other methods show similar results at larger ratios in LUNA→LUNA.
4.6. Comparison with State-of-the-Arts in 3D Tasks
- PCRL significantly outperforms other baselines across labeled ratios in LUNA→LUNA, including when baseline results converge at larger ratios.The performance gaps between train-from-scratch and self-supervised pretraining are smaller here than in Chest14.Source: §4.6 · p. 8
- In LUNA→LiTS, PCRL has apparent advantages over other baselines, especially at labeled ratios of 50% or less.3D-CPC outperforms Cube++ in almost all ratios among the cited baselines.Source: §4.6 · p. 8
- In LUNA→BraTS, PCRL consistently and remarkably surpasses previous baselines.The authors attribute the improvements to incorporating reconstruction of diverse contexts.Source: §4.6 · p. 8
4.7. Visual Analysis
In randomly selected LiTS and BraTS segmentation examples, PCRL handles details better than the other baselines, including accurate corners and isolated tumor regions.
4.7. Visual Analysis
- PCRL handles details better in the sampled LiTS and BraTS segmentation examples, delineating LiTS corners and detecting isolated BraTS tumor regions that other methods handle poorly.The examples were randomly selected.Source: §4.7 · p. 8
4.8. Comparison with State-of-the-Arts in Natural Image Segmentation and Detection Tasks
On ImageNet-1k-pretrained models transferred to natural-image downstream tasks, PCRL substantially outperforms MoCov2 and other popular self-supervised methods on Cityscapes segmentation and COCO detection.
4.8. Comparison with State-of-the-Arts in Natural Image Segmentation and Detection Tasks
- After ImageNet-1k pretraining and transfer to downstream tasks, PCRL substantially outperforms MoCov2 and other popular self-supervised methods on Cityscapes and COCO, widely adopted datasets in segmentation and detection.Source: §4.8 · p. 8
5. Discussion and Conclusion
The discussion reports that reconstructing diverse contexts improves contrastive representations across medical tasks and datasets, while leaving open whether information preservation alone explains the gains.
5. Discussion and Conclusion
- Reconstructing diverse contexts improves representations learned with contrastive loss in medical image analysis.The authors report positive results across a variety of medical tasks and datasets.Source: §5 · p. 8
- The authors leave open whether preserving more information is the only reason for improvements over contrastive loss.Source: §5 · p. 8
- The authors hope PCRL can lay foundations for real-world medical imaging tasks.Source: §5 · p. 8
7. Thorough Implementation Details (Supplementary Material)
The implementation details specify dataset splits, dataset-specific inputs and evaluation metrics, model backbones, comparison methods, and PCRL’s augmentation and training settings.
7. Thorough Implementation Details (Supplementary Material)
- 70%, 10%, and 20% of each dataset form the training, validation, and test sets, respectively.For semi-supervised pretraining and finetuning, pretraining draws a specific amount from the complete training set and the remainder is used for finetuning.Source: §7 · p. 9
- Chest14 contains 112,120 X-ray images from 30,805 patients and is used for 2D pretraining and finetuning with AUC evaluation.CheXpert contains 224,316 radiographs from 65,240 patients and is used mainly for 2D pretraining because it has uncertain labels.Source: §7 · p. 9
- LUNA contains 888 CT scans and supports 3D pretraining and finetuning evaluated by AUC; BraTS supplies 351 MRI scans for 3D finetuning evaluated by dice score.Source: §7 · p. 9
- LiTS contains 131 annotated CT scans and is mainly used for liver-only 3D finetuning, evaluated by Dice coefficient.The target volume is localized to the liver and expanded by 30 slices in each axis; inputs are 256×256×64, with HU range [-200,200].Source: §7 · p. 9
- ResNet-18 is used for the two 2D tasks, while ordinary 3D U-Net is the backbone for 3D tasks.Source: §7 · p. 9
- MG, SG, Cube++, and C2L use dataset-specific crops or augmentations, while 3D-CPC follows its published implementation.Cube++ divides volumes into 32 subvolumes of 16×16×16 and reconstructs the original volume; C2L applies inpainting to resized patches.Source: §7 · p. 9
- PCRL uses crop, flip, rotation, inpainting, outpainting, and Gaussian blur; 3D inputs are sampled as 16 pairs with IoU above 0.2.Each volume is resampled to 64×64×324, and 3D pretraining uses LUNA; online random cropping is omitted because its crop-and-resize cost is unacceptable on the available GPU.Source: §7 · pp. 9–10