PaperlayerView exact revision on arXiv

Source-linked AI summary

Going Deep in Medical Image Analysis: Concepts, Methods, Challenges and Future Directions

Fouzia Altaf, Syed M. S. Islam, Naveed Akhtar, Naeem K. Janjua

arXiv:1902.05655v1cs.CV

TL;DR

Medical Imaging lacks appropriately annotated large-scale datasets, limiting the evidence base for exploiting Deep Learning. This paper reviews recent methods through pattern-recognition and anatomical taxonomies, explains core concepts, and uses Computer Vision, Pattern Recognition, and Machine Learning perspectives to identify challenges and directions. Its central conclusion is that Medical Imaging can benefit significantly more through collaboration with those research communities.

  • Problem

    Medical Imaging generally has small public datasets compared with general Computer Vision, creating a major challenge for Deep Learning.

  • Method

    The paper surveys recent Medical Imaging Deep Learning methods by pattern-recognition task and human anatomy, adds an accessible tutorial, and draws on related fields.

  • Results

    The review identifies the lack of large-scale annotated datasets as the major problem in Deep Learning for Medical Image Analysis.

  • Takeaways & Limitations

    Medical Imaging can benefit significantly more from Deep Learning by encouraging collaborative research with Computer Vision and Machine Learning communities.

  • Takeaways & Limitations

    Recurrent Neural Networks face vanishing or exploding gradients on long sequences, motivating designs such as LSTM networks for longer-range handling.

Abstract

from arXiv · show

Medical Image Analysis is currently experiencing a paradigm shift due to Deep Learning. This technology has recently attracted so much interest of the Medical Imaging community that it led to a specialized conference in `Medical Imaging with Deep Learning' in the year 2018. This article surveys the recent developments in this direction, and provides a critical review of the related major aspects. We organize the reviewed literature according to the underlying Pattern Recognition tasks, and further sub-categorize it following a taxonomy based on human anatomy. This article does not assume prior knowledge of Deep Learning and makes a significant contribution in explaining the core Deep Learning concepts to the non-experts in the Medical community. Unique to this study is the Computer Vision/Machine Learning perspective taken on the advances of Deep Learning in Medical Imaging. This enables us to single out `lack of appropriately annotated large-scale datasets' as the core challenge (among other challenges) in this research direction. We draw on the insights from the sister research fields of Computer Vision, Pattern Recognition and Machine Learning etc.; where the techniques of dealing with such challenges have already matured, to provide promising directions for the Medical Imaging community to fully harness Deep Learning in the future.

1 INTRODUCTION

Deep Learning has rapidly expanded into Medical Imaging following advances across scientific fields and Computer Vision. This review organizes recent methods, identifies data scarcity as a fundamental challenge, and draws on neighboring fields for future directions.

  • Deep Learning has attracted notable interest in Medical Imaging after breakthroughs across Computer Vision, Natural Language Processing, Particle Physics, and other fields.
  • Its recent rise was enabled by large training datasets, powerful computational resources, and public libraries implementing Deep Learning algorithms.
  • Medical Imaging methods began exploiting contemporary deep neural networks around 2013, after earlier work was limited to two- or three-layer networks by data and computational constraints.
  • The review categorizes methods by pattern-recognition task and human anatomy, while explaining core Deep Learning concepts for Medical Imaging researchers.
  • The authors identify a lack of appropriately annotated large-scale datasets as the fundamental challenge and propose guidance informed by Computer Vision, Pattern Recognition, and Machine Learning.

2 BACKGROUND CONCEPTS

Machine Learning methods are distinguished by the supervision available during training and by the type of output they learn. Deep Learning methods can be used within these learning categories for tasks including classification, regression, and clustering.

  • Supervised Learning: Supervised learning uses training pairs (x, y), where x is an input example and y is its label.
  • Supervised Learning: Supervised models learn to predict labels for previously unseen test samples by estimating parameters that minimize a specified objective.
  • Classification and Regression: Classification maps inputs to discrete class labels, whereas regression maps examples to continuous scalar or vector outputs such as tumor malignancy probabilities.
  • Unsupervised Learning: Unsupervised learning operates without sample labels and commonly groups data according to similarities in intrinsic characteristics.
  • Other Learning Types: Semi-supervised learning uses labels for only a subset of training data, while reinforcement learning uses rewards or punishments as a form of supervision.

2.1 Standard Artificial Neural Networks

A standard Artificial Neural Network hierarchically composes neurons across layers to transform inputs into activations. Its learned parameters and manually selected architecture choices determine the resulting computational model.

  • An Artificial Neural Network is a hierarchical composition of neurons, and using many layers makes the network deep.
  • Each neuron weights input features, adds a bias, applies an activation function, and produces an activation signal.
  • Back-propagation requires differentiable activation functions, with ReLU widely used in modern Deep Learning, especially for CNNs.
  • A layer’s neuron weights can be represented by a matrix, allowing all neuron activations to be computed simultaneously.
  • An L-layer network composes layer transformations into a model M(x, Θ), with weights and biases across layers collectively denoted Θ.
  • Network architecture varies in layer count, neurons per layer, and activation functions, while model parameters are learned and hyper-parameters are usually selected manually.
  • Softmax normalizes activations across a layer so its outputs can be interpreted as a probability vector representing class confidence.

2.2 Convolutional Neural Networks

CNNs use convolutional, pooling, and fully connected layers to learn image representations. Convolution learns shared kernel weights, while pooling reduces activation-map dimensions.

  • CNNs comprise convolutional, pooling, and fully connected layers, with convolutional layers providing their primary strength for image analysis.
  • Convolutional layers: CNNs learn convolutional kernel weights as network parameters instead of requiring manually designed filter weights.The learned kernels can be optimized using the same tools as standard ANN weights.
  • Convolutional layers: In 2D, a moving kernel multiplies and sums corresponding grid elements to produce a scalar at each step, forming a 2D feature map.
  • Convolutional layers: For 3D volumes, multiple kernels produce separate output channels that together form an output volume.
  • Convolutional layers: Parameter sharing connects every input feature through the same kernel weights across locations.
  • Pooling layers: Pooling layers reduce activation-map width and height using maximum or average values without learnable parameters.

2.3 Recurrent Neural Networks

RNNs model sequential data by maintaining a state that is updated from the current input and previous state. Shared weights across time enable recurrence but make long-sequence training vulnerable to gradient problems.

  • RNNs maintain an internal memory state to model dependencies among sequential inputs.At each timestamp, the state is computed from the current input and preceding state.
  • The state update uses the current input and previous state, while the network may produce an output from the current state.
  • RNNs reuse the same weight matrices U, V, and W at every timestamp.This recursively applies the same operations over the input sequence.
  • Back-propagation through time recursively propagates errors across timestamps, creating difficulties for long sequences because of vanishing or exploding gradients.
  • LSTM networks retain the RNN architecture but compute hidden states with cells that regulate what to remember and erase.

2.4 Using Neural Networks for Unsupervised Learning

Neural networks can learn representations without labeled data through unsupervised frameworks. Autoencoders reconstruct inputs to learn latent representations, while GANs model data distributions using competing networks.

  • Unsupervised neural-network frameworks model data without requiring labels.
  • Autoencoders: Autoencoders map an input signal to itself while learning a latent representation intended to be more useful for a task than the raw input.
  • Autoencoders: Undercomplete autoencoders prevent trivial identity mappings by using a latent representation with substantially lower dimensionality than the input.
  • Autoencoders: Regularized autoencoders encourage useful representations through sparse connections or reconstruction from noisy inputs.
  • Generative Adversarial Networks: GANs use a generator to create samples and a discriminator to classify samples as real or fake, aiming to mimic the data-generating distribution.

2.5 Best practices in using CNNs for image analysis

CNNs for image analysis may be trained from scratch, fine-tuned from pretraining, or used as feature extractors. Computer Vision practice recommends matching the strategy to training-data size and task similarity, with FCNs and U-Net commonly used for segmentation.

  • CNN-based image-analysis models are commonly trained from scratch, fine-tuned from large-scale pretraining, or used as feature extractors.
  • Training from scratch is recommended only when the available dataset is very large, such as 50K images or more.
  • Transfer learning is preferred when the dataset is smaller, and pretrained data or tasks should be as similar as possible to the target problem.
  • With only a few hundred images, CNNs may be better used as feature extractors rather than trained end to end.
  • VGG-16 and VGG-19 have similar representation power, although VGG-16 remains frequently applied while AlexNet is no longer state of the art.
  • FCNs and U-Net are among the most popular CNN architectures for image segmentation.

2.6 Deep Learning Programming Frameworks

The paper reviews widely used Deep Learning programming frameworks for image analysis, while noting that the list is not exhaustive and that legacy frameworks remain important because of public implementations.

  • High-level programming frameworks provide pre-implemented Deep Learning algorithms and support image-analysis development.The frameworks are maintained continuously, and new findings are incorporated rapidly.
  • TensorFlow, PyTorch, Caffe2, Keras, MatConvNet, Caffe, Theano, and Torch are listed among frameworks used for Deep Learning image analysis.The list is ordered primarily by current popularity, with Theano and Torch included as important earlier frameworks.
  • Keras offers a high-level interface for TensorFlow and Theano and is popular for quickly developing and testing networks.The paper describes Keras as less flexible than some other frameworks and as a gateway for new users.
  • Caffe remains in use because public implementations of many popular networks are available in that framework.Caffe2 is described as rapidly replacing Caffe, but legacy implementations preserve Caffe’s practical relevance.
  • The framework list is not exhaustive, and implementations of networks proposed in 2012–2015 are often originally found in Torch, Theano, or Caffe.Legacy implementations may also be available in newer frameworks such as TensorFlow, although not necessarily from the original authors.

3 DEEP LEARNING METHODS IN MEDICAL IMAGE ANALYSIS

This section reviews Deep Learning methods in Medical Image Analysis by Pattern Recognition task and anatomical region. It surveys detection and localization work across brain, breast, eye, lung, and other medical imaging applications.

  • The review focuses mainly on papers published after December 2017, especially recent 2018 contributions, while briefly covering influential earlier work.Earlier literature is covered more comprehensively by cited review articles.
  • The literature is organized first by Pattern Recognition task and then by human anatomical region.The resulting taxonomy is depicted in Fig. 6.
  • Detection/Localization: Detection identifies a region of interest and places a bounding box around it; in medical imaging, this task is commonly called Computer Aided Detection.Examples include brain-tumor, lung-cancer, and breast-cancer detection.
  • Detection/Localization: 95% sensitivity was achieved for breast-tumor detection using a 3D CNN applied to volumes of interest from Automated whole Breast Ultrasound images.The experiment tested 171 tumors, with candidates selected from sliding-window volumes according to CNN probabilities.
  • Detection/Localization: 98.6% prediction detection accuracy was reported for detecting and classifying age-related macular degeneration and diabetic macular edema in retinal OCT images.The study used 207,130 images and fine-tuned a VGG-16 model pretrained on ImageNet.
  • Detection/Localization: 98% accuracy with 99% sensitivity was reported for detection in retinal images using a CNN-based method.
  • Detection/Localization: 98.51% accuracy and 2.45mm localization error were reported for inner-ear detection in CT images using 3D U-Net.The method used false-positive suppression and a shape-based training constraint.

3.3 Registration

Deep Learning methods are being applied to medical image registration, a task that aligns source and target images into a common anatomical space. The reviewed work spans reconstruction, deformable registration, and multimodal alignment across anatomical regions.

  • Image registration aligns images to a common anatomical space by transforming a source image toward a target image.It was an established medical image-analysis task before the Deep Learning era, but neural networks have increasingly entered this area.
  • Deep Learning registration studies include 3D reconstruction from 4D spatio-temporal data, CT and MRI reconstruction, and deformable registration.Reported approaches include stacked bidirectional convolutional LSTMs, CNNs, recurrent inference machines, and learning-based registration.
  • Deep Learning registration has been explored for fetal MRI, retinal OCT, chest CT, abdominal MRI, and MR–TRUS imaging.The studies use architectures and procedures including CNNs, U-Nets, feature-point methods, and adversarial networks.
  • Registration methods have been evaluated for computational efficiency, spatial transformation, and multimodal alignment quality.The AIR method jointly learns a registration network and a metric network for MR–TRUS registration.
  • Reconstruction time was reduced from 1h to 1 minute in one abdominal MRI registration study.The authors compared their CNN-based method with three other approaches.

3.4 Classification

Classification is a longstanding medical image-analysis task and a foundation of Computer Aided Diagnosis. The reviewed Deep Learning studies apply CNNs, transfer learning, recurrent networks, GANs, and multiple-instance learning across anatomical regions and diseases.

  • Medical image classification supports Computer Aided Diagnosis, motivating extensive use of recent Deep Learning advances for this task.
  • Brain: Brain-classification studies also used two-stage CNNs for autism-spectrum-disorder detection and Full-BiLSTM networks for mild-cognitive-impairment classification.The reviewed literature includes MRI-based methods for Alzheimer’s Disease and mild cognitive impairment, with some studies predicting later conversion to Alzheimer’s Disease.
  • Brain: Up to 99% accuracy was reported for Alzheimer’s Disease classification on ADNI using transfer learning with a 3D convolutional model.The approach used a model pretrained on the CAD Dementia dataset to address data scarcity.
  • Breast: 85.6 accuracy was reported for an estrogen-receptor classification task, improving from 68.6 using a CNN with quantile-function multiple-instance learning.The study used 1,713 breast-histology images from the Carolina Breast Cancer Study, Phase 3.
  • Eye: Diabetic-retinopathy classification achieved AUC scores of 0.94 and 0.95 on MESSIDOR 2 and E-ophtha test partitions, respectively.
  • Abdomen: GAN-based augmentation demonstrated up to 7% improvement in classification of 182 liver lesions in CT images.The study used synthetic medical-image generation as augmentation for the classification task.

4 DATASETS

Medical imaging deep learning depends on good-quality data and substantial training volume. Although public datasets are emerging, most remain small compared with general computer-vision datasets.

  • Good-quality data and large training sets are primary requirements for learning reliable deep models in medical imaging.Many public medical-imaging datasets have begun to emerge, but the article does not exhaustively review them.
  • Most public medical-imaging datasets contain relatively few samples and patients compared with general computer-vision datasets.Computer-vision datasets typically range from hundreds of thousands to millions of annotated images, whereas medical-imaging datasets are generally much smaller.
  • Table 5 provides representative examples of commonly used medical-imaging datasets rather than an exhaustive catalogue.

5 CHALLENGES IN GOING DEEP

The paper identifies limited, difficult-to-obtain annotations as the fundamental obstacle to exploiting deep learning in medical imaging, alongside imbalance and uncertainty concerns. It therefore calls for both large public datasets and methods that can learn from less data.

  • The section frames these issues as fundamental challenges rather than problems confined to individual medical-imaging tasks.
  • 5.0.0.1 Lack of appropriately annotated data: Limited appropriately annotated training data is identified as the fundamental challenge for applying deep learning to medical imaging.Deep networks learn many parameters and require large amounts of data to generalize well.
  • 5.0.0.1 Lack of appropriately annotated data: Medical-image annotation requires specialized expertise, carries high stakes, and makes large-scale dataset creation difficult.Concrete labels may not be easily inferred from reports, and annotations may not align with routine medical practice.
  • Future directions: The paper recommends developing large public medical-imaging datasets while simultaneously pursuing techniques that exploit deep learning with less data.
  • 5.0.0.2 Imbalanced data: Imbalanced medical-imaging datasets can bias deep networks toward abundant negative samples.For example, mammography datasets may contain few positive breast-cancer cases but many negative samples.
  • 5.0.0.3 Lack of confidence interval: Deep-learning outputs often provide a single probability rather than a confidence interval around the prediction.The paper questions whether end-to-end learning is always appropriate for medical imaging and links this uncertainty to limited adoption.

6 FUTURE DIRECTIONS

The paper proposes future directions centered on mitigating limited annotated data, building large-scale datasets, and connecting Medical Imaging with Computer Vision and Machine Learning communities.

  • 6.1 Limited annotated data: Transfer learning can address limited medical annotations by adapting models trained on large-scale source-domain data to target medical tasks.The paper highlights taskonomy-style source-target selection and quantifying transfer suitability as promising directions.
  • 6.1 Limited annotated data: Using pretrained networks as feature extractors, followed by task-specific feature processing, is presented as more promising than end-to-end training when datasets are small.Deep features can be post-processed, such as through Dictionary Learning, before classification.
  • 6.1 Limited annotated data: Freezing selected layers, generating synthetic images with GANs, and applying data augmentation are proposed to reduce the data requirements of medical deep models.These approaches respectively reduce trainable parameters or increase the effective training set.
  • 6.2 Enhancing dataset sizes: The root problem of small datasets requires Deep Learning-compatible large-scale annotated medical datasets, supported by protocols that convert reports into training formats.Document Analysis and Natural Language Processing may reduce the extra burden on medical experts.
  • 6.3 Broader outlook: Progress can come from stronger collaboration with Computer Vision and Machine Learning, aided by workshops, tutorials, and clearer terminology for medical tasks and competitions.The paper identifies medical jargon as a hindrance and recommends translating problems into terms familiar to neighboring communities.

7 CONCLUSION

The article combines an intuitive tutorial, a recent literature review, and a Computer Vision/Machine Learning perspective to examine Deep Learning in Medical Imaging. It identifies the lack of large-scale annotated datasets as the major problem and recommends cross-disciplinary strategies and collaboration as future directions.

  • 7 CONCLUSION: The article introduces core Deep Learning concepts intuitively for Medical Imaging researchers and reviews approaches published mainly in 2018.It also discusses major challenges and future directions for Deep Learning in Medical Image Analysis.
  • 7 CONCLUSION: Its Computer Vision/Machine Learning perspective distinguishes the survey from existing related literature surveys.This perspective supports both conceptual explanation and analysis of challenges in Medical Image Analysis.
  • 7 CONCLUSION: The review identifies the lack of large-scale annotated datasets as the major problem for Deep Learning in Medical Image Analysis.The article recommends strategies adopted in sister scientific fields to address similar problems.
  • 7 CONCLUSION: Medical Imaging can benefit significantly more from Deep Learning by encouraging collaborative research with Computer Vision and Machine Learning communities.The recommendation follows the survey’s use of insights from multiple scientific fields to address identified challenges.
Loading 1902.05655v1…