PaperlayerView exact revision on arXiv

Source-linked AI summary

A scoping review of transfer learning research on medical image analysis using ImageNet

Mohammad Amin Morid, Alireza Borjali, Guilherme Del Fiol

arXiv:2004.13175v5eess.IVcs.CVcs.LG

TL;DR

Medical-image transfer learning using ImageNet lacked a focused synthesis despite limited labeled data making CNN training from scratch difficult. This scoping review identified and characterized the relevant literature, finding generally reasonable target-task performance and several research gaps.

  • Problem

    Limited labeled medical datasets make full CNN training difficult, while prior reviews had not focused on ImageNet-based transfer learning for medical image analysis.

  • Method

    The review identified ImageNet transfer-learning studies and extracted their inputs, CNN models, transferred parameters, performance measures, and research-question characteristics.

  • Results

    Studies generally achieved reasonable performance in their target tasks, suggesting ImageNet transfer learning might be effective for medical tasks.

  • Takeaways & Limitations

    The review identifies prevalent implementation tracks for data preparation, methodology selection, and output evaluation in medical image analysis.

  • Takeaways & Limitations

    The review identifies critical gaps concerning transfer-learning choices, dataset size and augmentation, and classifier performance visualization.

Abstract

from arXiv · show

Objective: Employing transfer learning (TL) with convolutional neural networks (CNNs), well-trained on non-medical ImageNet dataset, has shown promising results for medical image analysis in recent years. We aimed to conduct a scoping review to identify these studies and summarize their characteristics in terms of the problem description, input, methodology, and outcome. Materials and Methods: To identify relevant studies, MEDLINE, IEEE, and ACM digital library were searched. Two investigators independently reviewed articles to determine eligibility and to extract data according to a study protocol defined a priori. Results: After screening of 8,421 articles, 102 met the inclusion criteria. Of 22 anatomical areas, eye (18%), breast (14%), and brain (12%) were the most commonly studied. Data augmentation was performed in 72% of fine-tuning TL studies versus 15% of the feature-extracting TL studies. Inception models were the most commonly used in breast related studies (50%), while VGGNet was the common in eye (44%), skin (50%) and tooth (57%) studies. AlexNet for brain (42%) and DenseNet for lung studies (38%) were the most frequently used models. Inception models were the most frequently used for studies that analyzed ultrasound (55%), endoscopy (57%), and skeletal system X-rays (57%). VGGNet was the most common for fundus (42%) and optical coherence tomography images (50%). AlexNet was the most frequent model for brain MRIs (36%) and breast X-Rays (50%). 35% of the studies compared their model with other well-trained CNN models and 33% of them provided visualization for interpretation. Discussion: This study identified the most prevalent tracks of implementation in the literature for data preparation, methodology selection and output evaluation for medical image analysis. Also, we identified several critical research gaps existing in the TL studies on medical image analysis.

1. Introduction

This introduction frames ImageNet-based transfer learning as an emerging approach for medical image analysis that addresses limited labeled data and reviews its study characteristics and outcomes.

  • Full CNN training requires large labeled datasets plus extensive computational and memory resources, which clinical practice may lack.
  • Transfer learning leverages knowledge from large non-medical datasets to solve targeted medical imaging problems.
  • ImageNet-trained CNN transfer learning became an emerging medical imaging trend that earlier reviews had scarcely addressed.
  • The scoping review summarizes input data, CNN models, transferred parameters, and performance measures across relevant studies.
  • The review asks which medical tasks benefit from ImageNet transfer learning, what inputs they use, which transfer processes are followed, and what outcomes result.

2.1. Convolutional Neural Networks

This section introduces CNNs and ImageNet as the foundation for medical-image transfer learning, then outlines major ImageNet architectures and their distinguishing design choices.

  • Convolutional Neural Networks: CNNs commonly comprise an input layer, convolution layers, pooling operations, and fully connected layers.
  • ImageNet: ImageNet contains over 15 million labeled images and supports large-scale object-recognition challenges.
  • ImageNet architectures: AlexNet, VGGNet, CaffeNet, ZFNet, and GoogLeNet are described as successful or influential ImageNet CNN architectures.
  • VGGNet: VGG architectures replace large filters with multiple smaller filters, producing deeper convolutional networks than AlexNet.
  • GoogLeNet: GoogLeNet introduced Inception modules to approximate sparse connections while improving efficiency without reducing accuracy.
  • Later architectures: ResNet learns residuals through identity shortcut connections, while DenseNet passes feature maps between all preceding and subsequent layers.

2.2. Transfer Learning

Transfer learning addresses the lack of large labeled medical datasets by initializing CNNs with learned weights, using either frozen feature extraction or trainable fine-tuning.

  • Motivation: The lack of large labeled datasets is a central challenge in training CNNs from scratch for medical image analysis.
  • Motivation: Transfer learning uses parameters from well-trained CNNs on large datasets such as ImageNet to solve medical image analysis problems.
  • Rationale: Transferred low-level features may reduce the need for large datasets, training time, and memory cost.
  • Feature extraction: Feature-extracting transfer learning freezes convolutional layers, removes the original classifier, and trains only a new classifier.
  • Fine-tuning: Fine-tuning replaces the classifier while allowing pretrained convolutional weights to update during training of the entire network.

2.3. Data Augmentation

Data augmentation generates training variations from existing images, with online and offline approaches trading lower memory use for faster training.

  • Purpose: Data augmentation artificially generates training data by creating variations of the original dataset.
  • Transformations: Common image transformations include rotation, translation, scaling, and flipping.
  • Approaches: Online augmentation occurs during training, whereas offline augmentation generates and stores data beforehand.
  • Trade-off: Online augmentation saves memory but slows training, while offline augmentation speeds training but consumes substantial memory.

2.4. Visualization of Convolutional Neural Networks

Visualization methods help inspect what CNNs learn and which image regions drive decisions, supporting interpretation and trust in medical image analysis.

  • Motivation: CNN black-box behavior makes it difficult to understand decision-making and verify that networks attend to appropriate image regions.Visualization can also reveal new domain knowledge.
  • Visualization methods: Preferred-input visualization synthesizes an image that maximally activates a neuron, revealing features learned at each convolutional layer.The synthesized input represents the neuron’s preferred input.
  • Visualization methods: Deconvolutional visualization maps a convolutional feature map back into image-pixel space using a DeconvNet and up-sampling.The process reverses convolution and pooling operations for a given feature map.
  • Visualization methods: Class Activation Mapping, or heatmaps, identifies discriminative image regions used by a CNN for classification.It visualizes activation-function triggers in intermediate network layers.

3. Method

The review followed established scoping-review procedures to identify ImageNet-based CNN transfer-learning studies, extract standardized characteristics, and describe prevalent methods and research gaps.

  • Study design: The review used the Arksey scoping-review framework and followed PRISMA guidelines.Its goals were to identify prevalent approaches and research gaps.
  • Interpretation: The review explicitly cautioned that method prevalence does not imply superior efficacy without direct benchmarking on the same dataset.Optimal-method claims require comparisons across methods under matched data conditions.
  • Search and eligibility: MEDLINE, IEEE, and the ACM digital library were searched for studies published from June 1, 2012, through January 2, 2020.The start date followed ImageNet’s initial release in 2012.
  • Search and eligibility: Eligible studies were original research on classification of macroscopic medical images using unmodified CNNs trained on non-medical ImageNet images.Studies of microscopic images and those lacking key study characteristics were excluded.
  • Screening: Two reviewers independently screened titles, abstracts, and full texts, resolving disagreements by consensus.Cohen’s kappa was 0.81 for title/abstract screening and 0.86 for full-text screening.
  • Data analysis: Thirteen study features covering problem description, input, methodology, and output were extracted and summarized with descriptive statistics and graphics.The analysis included task, anatomical site, image type, dataset size, augmentation, transfer-learning approach, and evaluation characteristics.

4. Results

Among 8,421 screened studies, 102 met inclusion criteria; the review found recurring patterns in anatomy, imaging modality, CNN choice, augmentation, transfer-learning strategy, and visualization.

  • Study characteristics: 102 studies met the inclusion criteria after screening 8,421 studies, and 85% were published after 2018.The screening process selected 689 articles for full-text review.
  • Study characteristics: Eye, breast, and brain were the most studied anatomical sites, representing 18%, 14%, and 12% of studies, respectively.X-ray and MRI were the most common image types, at 29% and 17%.
  • Transfer-learning methods: Inception-V3, VGG-16, AlexNet, and ResNet-50 were the most frequently used CNN models overall, at 19%, 18%, 15%, and 13%.Most studies used fully connected layers for final classification, while 67% fine-tuned weights and 33% used feature extraction.
  • Transfer-learning methods: 54% of studies performed data augmentation, whereas 65% did not benchmark their CNN against another model.Most medical transfer-learning studies used binary classification.
  • Model patterns: By anatomical site, Inception was most frequent for breast, VGGNet for eye, skin, and tooth, AlexNet for brain, and DenseNet for lung studies.The corresponding frequencies were 50%, 44%, 50%, 57%, 42%, and 38%.
  • Data augmentation: Data augmentation appeared in 72% of fine-tuning studies versus 15% of feature-extracting studies.For datasets under 1,000 images, augmentation occurred in 77% of fine-tuning studies versus 22% of feature-extracting studies.
  • Interpretability: 33% of reviewed studies provided CNN visualization, most commonly heatmaps, which represented 67% of visualization methods.Visualization was most frequent in brain, lung, and tooth studies.

5. Discussion

The review maps prevalent ImageNet transfer-learning practices across medical imaging modalities and anatomical sites, while identifying gaps in benchmarking, dataset strategy, model choice, and interpretability.

  • Scope and contribution: 85% of reviewed studies were published after 2018, providing a critical update of ImageNet transfer learning for medical image analysis.The review summarizes model selection, data augmentation, and visualization according to imaging modality and anatomical site.
  • Model selection: Inception models were most prevalent for skeletal-system X-rays, ultrasound, and endoscopy, while VGGNet was common for fundus and OCT images.Inception models were especially prevalent for skeletal-system X-rays; VGGNet appeared frequently in fundus and OCT studies.
  • Model selection: AlexNet was frequently used for MRI and breast X-ray studies, whereas DenseNet was most frequent for lung X-rays.The review associated these patterns with shallow models for some image types and deeper models for lung X-rays, while noting that further investigation is needed.
  • Benchmarking: 65% of studies did not benchmark their CNN against another model, motivating stronger systematic benchmarking with standardized methods.Seven benchmarking studies found that deep CNN models outperformed other CNN models, but deep networks were otherwise understudied across modalities and anatomical sites.
  • Interpretability and research gaps: Only 33% of studies addressed CNN visualization, mostly through heat maps, leaving model decision-making as an important research gap.The review also highlights uncertainty about whether dataset size or CNN model choice is more important for optimizing accuracy, time, and memory.

6. Conclusion

This scoping review identified prevalent ImageNet transfer-learning approaches for medical image analysis and highlighted important gaps in benchmarking, dataset-size assessment, augmentation, visualization, and task-specific evaluation.

  • ImageNet transfer learning generally achieved reasonable performance on target medical tasks, suggesting it may be effective for medical image analysis.
  • The review identified prevalent approaches for model selection, data augmentation, and output evaluation across medical image analysis tasks.
  • Inception models were prevalent for ultrasound, endoscopy, and skeletal-system X-rays; AlexNet for brain MRIs and breast X-rays; DenseNet for lung X-rays; and VGGNet for eye images.
  • Feature-extracting TL was most prevalent with smaller datasets, whereas fine-tuning TL required larger datasets, sometimes through data augmentation.
  • Most studies did not benchmark CNN models against alternatives, limiting understanding of optimal models for each medical imaging task.
  • Few studies compared feature-extracting and fine-tuning TL on the same task, leaving dataset size and CNN-model choice unresolved as determinants of accuracy, time, and memory.
  • Further research should examine dataset-size thresholds, alternative augmentation methods such as GANs, and methods within individual imaging modalities or anatomical sites.
  • Most studies did not apply visualization, although practitioner understanding of algorithmic rationale was described as important for healthcare integration.

Database

The database section documents the MEDLINE search strategy, abbreviation list, and supplementary tables of extracted study features organized by imaging modality and study type.

  • The MEDLINE query combined transfer-learning and CNN terms with medical-imaging terms including MRI, CT, and ultrasound.
  • The supplementary material included a dedicated list of abbreviations.
  • Supplementary tables catalogued extracted features for CT, MRI, ultrasound, skin-lesion, fundus, OCT, X-ray, and endoscopy studies.
Loading 2004.13175v5…