A Survey on Deep Learning in Medical Image Analysis

Geert Litjens; Thijs Kooi; Babak Ehteshami Bejnordi; Arnaud Arindra Adiyoso Setio; Francesco Ciompi; Mohsen Ghafoorian; Jeroen A. W. M. van der Laak; Bram van Ginneken; Clara I. Sánchez

Source-linked AI summary

A Survey on Deep Learning in Medical Image Analysis

Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen A. W. M. van der Laak, Bram van Ginneken, Clara I. Sánchez

arXiv:1702.05747v2Feb 19, 2017cs.CV

TL;DR

Medical image analysis lacked a comprehensive review spanning nearly all application and methodological areas. This survey synthesizes 308 contributions across major tasks and finds that deep learning rapidly pervaded the field, with 242 papers published in 2016 or early 2017.

Problem
A prior review omitted important medical imaging areas, motivating a more comprehensive application- and methodology-driven overview.
Method
The paper reviews major deep learning concepts and synthesizes 308 contributions across classification, detection, segmentation, registration, and other medical-image tasks.
Results
242 of the 308 reviewed papers were published in 2016 or the first month of 2017, showing rapid adoption across medical image analysis.
Takeaways & Limitations
Convolutional networks generally perform strongly, but task-specific expert knowledge may matter more than selecting an exact architecture.
Takeaways & Limitations
Expert annotations can contain substantial label noise, limiting the development of medical-image algorithms.

Abstract

from arXiv · show

Deep learning algorithms, in particular convolutional networks, have rapidly become a methodology of choice for analyzing medical images. This paper reviews the major deep learning concepts pertinent to medical image analysis and summarizes over 300 contributions to the field, most of which appeared in the last year. We survey the use of deep learning for image classification, object detection, segmentation, registration, and other tasks and provide concise overviews of studies per application area. Open challenges and directions for future research are discussed.

1. Introduction

Medical image analysis has shifted from handcrafted, rule-based systems toward supervised models and deep learning that learn increasingly higher-level features from data. This survey reviews over 300 recent contributions, spanning major analysis tasks, and identifies application challenges and directions for future research.

Historical development: Medical image analysis evolved from sequential low-level processing and mathematical modeling toward supervised systems trained on example data with extracted feature vectors.Earlier systems used edge and line detectors, region growing, and fitted geometric models, whereas supervised methods learned decision boundaries in high-dimensional feature spaces.
Historical development: Deep learning extends this progression by learning features that optimally represent the problem through multilayer networks transforming images into outputs.These networks learn increasingly higher-level features rather than relying entirely on manually extracted discriminant features.
Deep learning transition: AlexNet’s 2012 ImageNet victory, enabled by improved training techniques and computing systems, accelerated interest in deeper convolutional architectures.CNNs had earlier succeeded in LeNet, but their broader momentum followed advances that made deep networks more efficiently trainable.
Field growth: Deep learning applications in medical image analysis grew rapidly in 2015 and 2016 and became dominant at major conferences.The field also received a dedicated IEEE Transactions on Medical Imaging special issue in May 2016.
Survey scope and aims: The survey covers over 300 mostly recent papers across classification, detection, segmentation, registration, retrieval, generation, and enhancement, while addressing challenges and future research directions.Its stated aims are to show deep learning’s penetration across medical image analysis, identify challenges, and highlight contributions that solve or circumvent them.

2. Overview of deep learning methods · 2.1. Learning algorithms · 2.2. Neural Networks

This section introduces deep learning concepts, learning algorithms, and neural-network architectures relevant to medical image analysis. It distinguishes supervised and unsupervised learning, explains neural-network structure and training, and identifies CNNs as the most widely used architecture in medical image analysis.

2. Overview of deep learning methods: The overview formally introduces deep learning concepts, techniques, and architectures found in the surveyed medical image analysis papers.
2.1. Learning algorithms: Supervised learning uses input-label pairs and fits parameters Θ to predict outputs by minimizing a loss function L(y, ˆy).The model output is ˆy = f(x; Θ).
2.1. Learning algorithms: Unsupervised learning processes unlabeled data to discover patterns such as latent subspaces, using objectives including reconstruction loss L(x, ˆx).Examples include principal component analysis and clustering, with reconstruction often based on lower-dimensional or noisy representations.
2.2. Neural Networks: Neural networks are learning algorithms built from neurons with activations and parameters Θ = {W, B}, where W contains weights and B contains biases.Each neuron applies a linear combination followed by an element-wise nonlinearity σ(·).
2.2. Neural Networks: Multiple hidden layers make a neural network deep, which motivates the term deep learning; traditional MLPs stack successive transformations.Sigmoid and hyperbolic tangent are typical transfer functions for traditional neural networks.
2.2. Neural Networks: Neural-network outputs can be mapped to a class distribution P(y|x; Θ) at the final layer.The output node for class i is associated with weight vector wi.
2.2. Neural Networks: Maximum likelihood with stochastic gradient descent fits Θ using mini-batches, yielding binary or categorical cross-entropy losses.These losses may not directly optimize target measures such as area under the ROC curve or the Dice coefficient.
2.2. Neural Networks: Current deep-learning models are predominantly trained end-to-end with supervision, with CNNs most widely used in medical image analysis and RNNs gaining popularity.Earlier layer-wise unsupervised pre-training followed by supervised fine-tuning enabled effective DNN training and included SAEs and DBNs.

2.3. Convolutional Neural Networks (CNNs) · 2.4. Deep CNN Architectures

CNNs use shared convolutions, nonlinear feature extraction, and pooling to reduce parameters while providing translation equivariance and invariance. Deep CNN architectures extend these principles through deeper, multi-stream, and fully convolutional designs adapted to medical imaging tasks.

2.3. Convolutional Neural Networks (CNNs): CNNs share convolutional weights across image positions, avoiding separate detectors for identical objects and reducing parameter counts.Weight sharing makes CNNs equivariant to input translations, and the number of weights no longer depends on image size.
2.3. Convolutional Neural Networks (CNNs): Pooling aggregates neighborhood values with permutation-invariant operations such as max or mean, inducing translation invariance and further reducing parameters.Convolutional layers generate feature maps through kernels, biases, and element-wise nonlinear transformations before pooling and later fully connected layers.
2.4.1. General classification architectures: After 2012, CNN research increasingly favored deeper models that stack smaller kernels, using fewer parameters and often less inference memory than large-receptive-field alternatives.These properties can enable deployment on mobile devices, although lower inference memory is typically less important for medical applications.
2.4.1. General classification architectures: GoogLeNet introduced a 22-layer architecture with inception blocks that combine convolutions of different sizes to improve training efficiency and reduce parameters.Despite ImageNet performance saturation after 2014, AlexNet, VGG, and other simple models remained popular for medical data.
2.4.2. Multi-stream architectures: Multi-stream architectures combine information at different network stages and support multi-scale image analysis and 2.5D classification.For volumetric medical data, multiple streams can avoid the large parameter counts of full 3D convolutions by processing volume slices separately.
2.4.2. Multi-stream architectures: Multi-stream designs can add down-scaled contextual information alongside high-resolution local information without relying solely on larger input patches.Larger patches increase network parameters and memory requirements, while context is important for abnormality detection.
2.4.3. Segmentation Architectures: Fully convolutional networks apply shared convolutions to entire images or volumes, avoiding repeated computations inherent in sliding-window pixel classification.Pooling can reduce output resolution; shift-and-stitch was proposed to recover a higher-resolution output.
2.4.3. Segmentation Architectures: U-net extends fully convolutional segmentation with contractive and expansive paths, learned up-convolutions, and skip-connections linking opposing paths.The expansive path increases image size while skip-connections directly connect corresponding contracting and expanding representations.

2.5. Recurrent Neural Networks (RNNs)

Recurrent neural networks analyze variable-length sequences by maintaining a hidden state over time and mapping the final state to class posteriors. Because temporal backpropagation makes them difficult to train, LSTM and GRU memory units are commonly used, and RNNs are increasingly applied to medical-image segmentation.

Core formulation: RNNs model variable-length input and output sequences, learning class distributions conditioned on an ordered sequence rather than a single input vector.They were traditionally developed for discrete sequence analysis and generalized MLPs to handle varying sequence lengths.
Core formulation: A plain RNN updates a hidden state from the current input and previous state, with weight matrices shared across time, then uses fully connected layers and softmax for classification.The sequence is mapped to a posterior over classes using the final hidden state.
Training challenges: Temporal backpropagation makes RNNs inherently deep in time and causes training difficulties analogous to those in regular deep neural networks.Specialized memory units address these training problems.
Memory units: LSTM was the earliest and most popular specialized memory unit, while the GRU provides a more recent simplification and is also commonly used.Both units were developed to mitigate the training problems associated with plain RNNs.
Medical-image applications: RNNs, initially proposed for one-dimensional inputs, are increasingly applied to images, including promising medical-image segmentation results in the MRBrainS challenge.In natural images, pixelRNNs are also used as autoregressive generative models.

2.6. Unsupervised models

Unsupervised models learn latent representations or generate data without labels, with auto-encoders and restricted Boltzmann machines providing foundational architectures. Variational auto-encoders and generative adversarial networks were promising but had not yet been peer-reviewed in medical imaging applications.

Auto-encoders: Auto-encoders reconstruct input x as x′ through a hidden representation h using learned weights, biases, and a nonlinear activation.The hidden activation is computed from the input-to-hidden and hidden-to-reconstruction parameters.
Auto-encoders: A hidden layer smaller than the input projects data into a lower-dimensional subspace capturing dominant latent structure, while regularization or sparsity can improve discovery.Without dimensionality reduction or additional nonlinearities, the model could learn the identity function.
Auto-encoders: Denoising auto-encoders reconstruct inputs from noise-corrupted versions, while stacked auto-encoders layer auto-encoders and are often greedily pretrained before supervised fine-tuning.The surveyed applications typically used salt-and-pepper noise for corruption.
Restricted Boltzmann machines: Restricted Boltzmann machines connect visible inputs and hidden latent features bidirectionally, enabling latent representation inference and generation of new data points.They are generative models formulated as Markov Random Fields.
Deep belief networks and newer architectures: Deep belief networks replace stacked auto-encoder layers with restricted Boltzmann machines, train layers unsupervised, and add a linear classifier for supervised fine-tuning.Variational auto-encoders and generative adversarial networks had promising natural-image applications but no peer-reviewed medical-image applications yet.

2.7. Hardware and Software

Deep learning’s growth has been driven by widespread GPU availability and open-source software packages that provide efficient implementations of neural-network operations. Current GPUs typically make deep learning 10 to 30 times faster than CPUs, while frameworks enable high-level implementation without low-level optimization.

Hardware and Software: 10 to 30 times faster than CPUs is the typical speed advantage of current GPUs for deep learning.GPUs provide highly parallel computing with an order of magnitude more execution threads than CPUs.
Hardware and Software: Open-source libraries provide efficient GPU implementations of neural-network operations, such as convolutions, enabling high-level implementation without low-level optimization.The passage identifies software availability as another driving force behind deep learning’s popularity.
Hardware and Software: Popular packages included Caffe, Tensorflow, Theano, and Torch, offering C++ and Python or Python and Lua interfaces.Caffe and Tensorflow provide C++ and Python interfaces; Theano provides Python; Torch provides Lua.
Hardware and Software: Third-party packages such as Lasagne and Keras were built on top of one or more of these frameworks.The paper does not discuss all such packages in detail.

3. Deep Learning Uses in Medical Imaging

Deep learning is used across medical imaging tasks including classification, localization, detection, segmentation, registration, retrieval, and image generation. CNNs, particularly pretrained models, are prominent, while adaptation to medical structure and task-specific challenges remains important.

Exam classification: Medical imaging studies adapt deep networks to intrinsic data structure using 3D convolutions and layers designed for brain connectivity graphs.Examples include Alzheimer classification with 3D convolutions and MRI diffusion-tensor imaging with edge-to-edge, edge-to-node, and node-to-graph layers.
Exam classification: CNNs are the current standard for exam classification, and pretrained natural-image CNNs have achieved near-human expert performance in some tasks.Transfer learning uses pretrained networks either as feature extractors or through fine-tuning; evidence comparing the strategies is conflicting, while fine-tuning has outperformed feature extraction in some studies.
Object classification: Multiple instance learning outperformed handcrafted features and closely approached fully supervised performance when object annotation is expensive.The passage motivates MIL because obtaining high-quality annotated medical data is challenging.
Localization and detection: 2D CNN classification is the most popular localization strategy, while newer methods directly emphasize accurate localization during learning.Object detection remains labor-intensive and faces challenges including class imbalance, hard-negative mining, and efficient pixel- or voxel-wise processing.
Segmentation: Lesion segmentation combines approaches from object detection and organ segmentation because the underlying challenges are largely similar.Developments in detection and organ segmentation are expected to propagate naturally to lesion segmentation.
Registration: 1500x speed-up for 2D and 66x speed-up for 3D were reported for registration, but no best integration strategy has yet emerged.The registration literature remains limited, with existing studies using distinctly different approaches.
Retrieval and image generation: Deep learning has enabled rich multilevel feature learning for content-based image retrieval, but successful applications remain limited.Generated images have also been used in Alzheimer’s disease computer-aided diagnosis when original data are missing or unavailable.

4. Anatomical application areas

Deep learning has been applied across diverse anatomical imaging areas, with CNNs achieving leading performance in several brain, eye, thoracic, pathology, organ, and dermatological challenges. Applications commonly involve classification, detection, and segmentation, while methods address limited context, candidate generation, and small datasets through architectural and learning strategies.

Brain imaging: Brain imaging studies prominently target Alzheimer’s classification, tissue and anatomical-structure segmentation, and lesion detection or segmentation.Most methods learn mappings from local patches to representations and labels, but larger anatomical context may require non-uniform sampling or multiscale representation fusion.
Brain imaging: CNNs have led the 2014–2015 BRATS, 2015 MS lesion, 2015 ISLES, and 2013 MRBrains brain-imaging challenges.The surveyed challenge-leading systems primarily analyzed brain MR images.
Ophthalmic imaging: Four teams exceeded human performance in Kaggle’s diabetic-retinopathy competition, which provided over 35,000 training images and 53,000 test images.All four teams used end-to-end CNNs; a Google Inception v3 system later performed comparably to seven certified ophthalmologists.
Thoracic imaging: CNNs were used by all top-performing systems in the LUNA16 CT nodule-detection challenge.The best systems still relied on rule-based nodule candidates, although deep networks such as U-net also performed very well for candidate detection.
Pathology: CNN-based methods outperformed competing algorithms in mitosis-detection and neuronal-process-segmentation challenges, while deep models also dominated the top five CAMELYON16 systems.CAMELYON16’s large annotated dataset enabled 22-layer GoogLeNet, 16-layer VGG-Net, and 101-layer ResNet models.
Other anatomical applications: Deep learning is expanding into organ and skin imaging, with a 3D fCNN recently taking first place in PROMISE12 and CNN systems producing promising dermoscopic skin-cancer results.Traditional methods had dominated PROMISE12 and SLIVER07 until 2016; another system trained on 12K discs reported near-human performance across four radiological scoring tasks.

5. Discussion · Overview

Deep learning has rapidly spread across every aspect of medical image analysis, with 242 of 308 surveyed papers published in 2016 or January 2017. The field has shifted from pretrained CNN feature extraction toward end-to-end trained CNNs as the preferred approach.

Overview: Deep learning has pervaded every aspect of medical image analysis covered by the survey.The survey reviewed 308 papers.
Overview: 242 of 308 surveyed papers were published in 2016 or the first month of 2017.This timing demonstrates the field’s rapid expansion.
Overview: The survey covers a large diversity of deep architectures.The passage introduces architectural variety without enumerating specific categories.
Overview: Early studies used pre-trained CNNs as feature extractors.Pre-trained networks could be downloaded and directly applied to medical images.
Overview: Downloaded pre-trained networks facilitated deep-learning adoption for medical images.They could also extend existing systems based on handcrafted features.
Overview: End-to-end trained CNNs became the preferred approach for medical imaging interpretation in the last two years.This represents a shift from the earlier feature-extraction approach.

Key aspects of successful deep learning methods

Successful medical image analysis methods depend less on selecting an exact architecture than on training data quality, augmentation and preprocessing, task-specific design, and appropriate inputs. Hyper-parameter optimization can improve performance but remains secondary and largely empirical.

Convolutional neural networks and derivatives are top performers in most medical image analysis competitions, but exact architecture is not the main determinant of solution quality.The survey emphasizes that no single perfect architecture can be distilled for every task or application area.
Augmentation, preprocessing, and architectures incorporating task-specific properties can outperform straightforward CNNs.Examples include multi-view and multi-scale networks.
Input size should reflect the resolution and context required by the task, because enlarging patches without changing the receptive field may not improve performance.Visual assessment by researchers or domain experts can serve as a sanity check for whether the input or architecture needs modification.
Hyper-parameter optimization, including learning and dropout rates, can yield extra performance but is secondary to the previously discussed factors and training-data quality.The survey reports no clear recipe for selecting hyper-parameters; intuition-based random search is commonly used and often works sufficiently well.

Unique challenges in medical image analysis

Deep learning in medical image analysis faces challenges beyond image availability, especially obtaining accurate expert annotations, handling label noise and heterogeneous or imbalanced classes, and incorporating clinical context and anatomical location.

Data and annotations: Millions of medical images are digitally available through PACS, but relevant annotations and structured labels remain the central data challenge.Free-text radiology reports require sophisticated text-mining methods to convert them into accurate annotations.
Data and annotations: Expert annotation is time-consuming, particularly for 3D radiology segmentation requiring slice-by-slice labeling.Researchers often rely on radiologists or pathologists for task-specific annotations because automated extraction from reports is complex.
Label quality: Label noise can substantially limit algorithm development because medical-image annotations may vary across expert readers.The LIDC-IDRI lung-CT nodule dataset illustrates this issue through independent annotations by four radiologists and subsequent reader review.
Class structure and imbalance: Medical-image classes are heterogeneous, so binary normal-versus-abnormal or object-versus-background formulations can miss rare subclasses and artifacts.Expanding to detailed multiclass labels is often infeasible because it further increases expert annotation requirements.
Class structure and imbalance: Class imbalance is prevalent because medical datasets often contain many normal images and relatively few abnormal or suspicious cases.Breast-screening mammogram databases exemplify this pattern, with most images normal and suspicious lesions often noncancerous.
Clinical context and spatial information: Medical-image models must also account for patient context and anatomical location, which patch-based classification may omit.Potential remedies include integrating patient history and demographics or using whole-image learning with evaluation driven by a Dice-coefficient loss.

Outlook

The outlook highlights both demonstrated successes and unresolved challenges for deep learning in medical imaging. Future progress may depend on task-specific methods, unsupervised learning from abundant unlabeled data, and improved interpretability for medically accountable systems.

Outlook: Deep learning has outperformed medical experts in certain dermatology and ophthalmology tasks, despite many challenges remaining inadequately addressed.The cited successes include Esteva et al. (2017) and Gulshan et al. (2016).
Outlook: Directly predicting nuclei center locations can outperform classification-based center localization, which may require non-deep-learning post-processing.Relevant post-processing includes counting, segmentation, or regression.
Outlook: Unsupervised learning is a key emerging area for medical imaging after supervised methods became standard following AlexNet’s success.The passage describes renewed interest in unsupervised learning and its earlier role in greedy layer-wise pre-training.
Outlook: Unsupervised methods can initialize networks using abundant unlabeled data, which is generally easier to obtain than labeled medical data.VAEs and GANs could optimally leverage this information.
Outlook: Medical deep-learning systems must become more interpretable because accurate predictions alone may be insufficient where accountability and legal consequences matter.Strategies such as deconvolution networks and guided back-propagation have been developed to understand intermediate convolutional-layer responses.

Appendix A: Literature selection

The literature search combined PubMed, arXiv, and conference proceedings using deep-learning and medical-imaging terms. Retrieved papers were manually screened to exclude non-medical-imaging studies and studies relying only on handcrafted features.

Search strategy: Over 700 hits were initially retrieved from PubMed using “convolutional” or “deep learning” in any field.The search deliberately omitted “neural network” to avoid numerous false-positive brain-research papers.
Search strategy: ArXiv was searched with a medical-imaging query combining modalities such as MRI, CT, ultrasound, pathology, radiography, mammography, fundus, and OCT with deep-learning terms.The query included “deep learning,” convolutional, CNN, and “neural network” expressions.
Screening criteria: Conference proceedings from MICCAI, SPIE, ISBI, and EMBC were screened, excluding papers unrelated to medical imaging or using only handcrafted features with neural networks.Examples of excluded applications included genetics and chemistry.