Source-linked AI summary
Self-supervised learning methods and applications in medical imaging analysis: A survey
Saeed Shurrab, Rehab Duwairi
TL;DR
Medical imaging lacks sufficient high-quality annotated data, limiting supervised learning for robust model development. This survey reviews self-supervised learning methods and medical-imaging applications, organizing approaches by category and synthesizing evidence across recent studies. It reports effectiveness across several medical image-analysis tasks, while identifying limited public unlabeled data and constrained transferability as important boundaries.
Problem
Medical imaging requires large amounts of high-quality annotated data, but such annotations are scarce and difficult to produce, creating a major problem for supervised machine learning.
Method
The survey reviews self-supervised learning methods and applications, examines literature from 2017–2021, and categorizes approaches and 40 medical-imaging works by method and task.
Results
Self-supervised learning approaches generally outperform training from scratch or ImageNet initialization, with medical-image-aware designs often outperforming direct computer-vision task adoption.
Takeaways & Limitations
Self-supervised learning can alleviate annotated-data scarcity by learning robust representations from unlabeled medical images for subsequent analysis tasks.
Takeaways & Limitations
Public datasets are usually small, private datasets are difficult to access, and incorporating medical knowledge may limit an approach’s transferability across modalities and tasks.
Abstract
from arXiv · showhide
The scarcity of high-quality annotated medical imaging datasets is a major problem that collides with machine learning applications in the field of medical imaging analysis and impedes its advancement. Self-supervised learning is a recent training paradigm that enables learning robust representations without the need for human annotation which can be considered an effective solution for the scarcity of annotated medical data. This article reviews the state-of-the-art research directions in self-supervised learning approaches for image data with a concentration on their applications in the field of medical imaging analysis. The article covers a set of the most recent self-supervised learning methods from the computer vision field as they are applicable to the medical imaging analysis and categorize them as predictive, generative, and contrastive approaches. Moreover, the article covers 40 of the most recent research papers in the field of self-supervised learning in medical imaging analysis aiming at shedding the light on the recent innovation in the field. Finally, the article concludes with possible future research directions in the field.
INTRODUCTION
Medical image analysis uses learning-based models to extract information for diagnosis, but CNNs require substantial annotated data and medical annotation is costly, expert-dependent, time-consuming, and privacy-sensitive. Self-supervised learning addresses this scarcity by learning representations from unlabeled data through a pretext task, then transferring them to downstream medical tasks; the survey reviews methods, applications, and categories bridging computer vision and medical imaging.
- Medical image analysis includes classification, detection and localization, segmentation, and registration for extracting diagnostic information.
- CNNs learn hierarchical image features but commonly require large annotated datasets because they contain millions of trainable parameters.Convolutional and pooling layers extract and aggregate features, while optimization minimizes the difference between outputs and ground-truth labels.
- High-quality medical annotation is expensive, time-consuming, dependent on domain experts, and subject to patient-privacy concerns.
- ImageNet transfer learning is controversial for medical imaging because natural-image features, model scale, and 2D training may mismatch medical data and 3D modalities.Medical tasks may have fewer than 10 classes, whereas ImageNet models are designed for 1,000 labels; CT, MRI, and OCT are predominantly 3D.
- Self-supervised learning creates supervisory signals from unlabeled data, learns representations in a pretext task, and transfers them as initial weights for downstream fine-tuning.The workflow uses pseudo-labels generated from a large unlabeled dataset and then applies the learned representations to tasks with limited ground-truth data.
- The survey reviews state-of-the-art computer-vision methods, categorizes them as predictive, generative, and contrastive, and covers 40 recent medical-imaging studies.It aims to bridge computer vision and medical imaging while highlighting recent self-supervised-learning innovations.
SURVEY METHODOLOGY
The survey searches selected literature using defined sources, keywords, and eligibility criteria, then reviews 40 medical-imaging papers alongside 15 methods developed for natural images. It organizes the selected evidence by publication year and category.
- Sources and keywords: The search used three primary bibliographic databases and two secondary sources, including related-work sections in selected papers.
- Sources and keywords: The search used keywords covering self-supervised learning, pretext tasks, representation learning, and contrastive learning in medical imaging.
- Inclusion/exclusion criteria: Eligible studies either applied self-supervised learning to medical-imaging tasks or introduced a novel medical-imaging approach; frequently used computer-vision methods were also selected.
- Papers selection: The survey selected 15 natural-image self-supervised approaches and 40 papers directly addressing self-supervised learning applications in medical imaging.
- Papers selection: Figure 2 presents the distribution of the 40 medical-imaging papers by publication year and category.
SELF-SUPERVISED LEARNING APPROACHES
The survey presents self-supervised learning as representation learning through reusable pretext tasks, grouped into predictive, generative, and contrastive approaches. These tasks create supervisory signals from unlabeled images and transfer learned representations to downstream tasks.
- Overview: Early self-supervised learning trained layer-specific auto-encoders sequentially and used their weights to initialize fine-tuning on a target task.The survey notes that this greedy approach was later superseded by end-to-end deep models.
- Overview: Pretext tasks provide a shared representation-learning mechanism that can support different downstream tasks and datasets.
- Predictive approaches: Predictive methods assign categorical or numerical pseudo-labels generated from each unlabeled image and train models through classification or regression.
- Exemplar CNN: Exemplar CNN creates 32×32 exemplary patches and transformed variants, then learns representations by classifying the resulting surrogate classes.
- Relative position prediction: Relative position prediction divides an image into a 3×3 patch grid and predicts each query patch’s position relative to the central anchor.
- Jigsaw puzzle: Jigsaw pretext learning shuffles image patches using predefined permutations and classifies the permutation index with a shared-weight siamese network.For nine patches, the unrestricted space contains 9! = 362,880 permutations, motivating a predefined subset.
- Rotation prediction: Rotation prediction generates supervisory signals by rotating images through 0°, 90°, 180°, or 270° and classifying the applied transformation.
- Generative approaches: Generative methods learn representations by reconstructing inputs or generating examples, using architectures such as auto-encoders and adversarial models.
SELF-SUPERVISED METHODS IN MEDICAL IMAGING
Self-supervised medical imaging methods are organized around predictive, generative, contrastive, and multi-task strategies, often adapting computer-vision pretext tasks to medical data. These methods create supervisory signals from medical images or combine complementary objectives to learn representations for downstream analysis.
- Researchers follow two paths: directly adopting or modifying computer-vision pretext tasks, or designing new tasks using medical and computer-vision knowledge.
- Multi-task methods: Multi-task methods combine distinct objectives, including generative modeling with instance discrimination for sequential medical images.
- Predictive methods: Predictive methods use medical-image structure, including slice order, spatial relations, anatomical positions, rotations, and multimodal patch arrangements, as auxiliary supervision.
- Predictive methods: Three-dimensional Rubik cube recovery tasks learn representations by reconstructing rearranged and rotated sub-cubes, with Rubik cube+ adding masking identification.
- Generative methods: Generative approaches train models to restore or generate medical images after colorization, context corruption, distortion, or simulated craniectomy.
- Contrastive methods: Contrastive learning adaptations improve SimCLR for 3D medical-image segmentation through domain-specific and problem-specific modifications.
PERFORMANCE COMPARISON
The survey compares self-supervised approaches with training from scratch and ImageNet transfer learning across classification and segmentation tasks. Across these evaluations, self-supervised methods generally perform better, especially when pretext tasks incorporate medical-image properties.
- The comparison evaluates classification and semantic segmentation after fine-tuning the whole model, using LUNA, CheXpert, brain hemorrhage, and BraTS datasets.
- Classification: Self-supervised approaches outperform both training from scratch and ImageNet-pretrained models across the reported classification comparisons.
- Classification: Contrastive algorithms tend to achieve better classification results than other self-supervised categories, although performance varies by dataset and backbone.
- Classification: Directly adopted computer-vision pretext tasks provide marginal improvements over ImageNet pretraining, whereas medically adapted tasks provide significant improvements in the reported CheXpert comparison.
- Semantic segmentation: Self-supervised methods significantly outperform training from scratch on BraTS semantic segmentation, while medical-specific designs outperform methods adopted from computer vision.
DISCUSSION AND FUTURE RESEARCH DIRECTIONS
The discussion emphasizes that medical images differ fundamentally from natural images, so self-supervised pretext tasks often require medical-specific adaptation. It identifies medical knowledge, multimodal learning, and broader unbiased unlabeled datasets as important research directions.
- Computer vision task in medical imaging: Medical images differ from natural images in channels, intensity, location, scale, and orientation, affecting how pretext tasks should be designed.Medical data may be 2D grayscale, 3D, or 4D, and intensity and location can carry diagnostic meaning.
- Computer vision task in medical imaging: Directly adopting computer-vision pretext tasks may not produce the same performance in medical image analysis.The survey therefore recommends accounting for medical images’ unique properties when adapting these tasks.
- Computer vision task in medical imaging: Several studies modify computer-vision methods to handle volumetric medical images and other modality-specific characteristics.These adaptations address the mismatch between methods developed for 2D natural images and medical imaging data.
- Pretext tasks based on medical knowledge: Medical-knowledge-based pretext tasks may provide medically informed representations but can limit applicability to particular modalities, diseases, or downstream tasks.Using image properties and image manipulation generally offers a wider range of applications across modalities with common attributes.
- Pretext tasks design with multiple imaging modalities: Multiple imaging modalities provide complementary information, and incorporating them into pretext tasks can offer richer representations.The survey cites multimodal designs involving modalities such as OCT scans and fundus color photographs as a direction needing further research.
- Data availability: The field needs large unlabeled data pools spanning imaging modalities while avoiding bias toward healthy cases.Reducing healthy-case bias is presented as necessary for representations rich in pathological features.
CONCLUSION
The conclusion presents self-supervised learning as a response to scarce annotated medical images and surveys its methods and applications. It organizes the literature, compares approaches on a unified benchmark, and identifies patterns and open issues.
- CONCLUSION: Annotated medical images are scarce, while machine-learning applications require large amounts of high-quality labeled data.This scarcity is identified as a major problem in medical imaging analysis.
- CONCLUSION: Self-supervised learning can alleviate annotation scarcity by learning robust representations from unlabeled medical images.The approach does not require human annotation for representation learning.
- CONCLUSION: The survey categorizes recent methods into predictive, generative, contrastive, and multi-tasking approaches and reviews 40 medical-imaging applications.It also reviews 15 computer-vision self-supervised learning methods used in medical imaging analysis.
- CONCLUSION: A unified-benchmark comparison highlights the best-performing reviewed self-supervised approaches in medical imaging.The survey also summarizes major patterns and open issues requiring research attention.
DECLARATION OF COMPETING INTERESTS
The supplied material states that the authors declare no relevant conflicts of interest and lists implementation-code resources for reviewed works.
- DECLARATION OF COMPETING INTERESTS: The authors declare no conflicts of interest relevant to the article’s content.This is the paper’s competing-interests statement.
- DECLARATION OF COMPETING INTERESTS: Table A1 lists publicly available implementations for selected computer-vision and medical-image-analysis works.Starred implementations represent the authors’ official code.
- DECLARATION OF COMPETING INTERESTS: The listed implementations include Exemplar CNN, relative-position prediction, jigsaw puzzle, rotation prediction, denoising auto-encoder, image inpainting, and image colorization.The table names implementations in caffe, pytorch, and theano, with some works listing multiple frameworks.