Source-linked AI summary
3D Deep Learning on Medical Images: A Review
Satya P. Singh, Lipo Wang, Sukrit Gupta, Haveesh Goli, Parasuraman Padmanabhan, Balázs Gulyás
TL;DR
Medical imaging analysis remains dependent on trained radiologists, while the applicability of deep learning to limited medical datasets remains uncertain. This review traces 3D CNN development, describes their mathematics and preprocessing, and synthesizes applications across major medical imaging tasks. The reviewed research demonstrates strong results across classification and detection while identifying computational, information-preservation, overfitting, and generalization challenges.
Problem
Medical images still require analysis by trained radiologists, while it remains ambiguous whether deep learning models work successfully with the smaller datasets typical of medical imaging.
Method
The review traces 3D CNN development, describes their mathematical basis and preprocessing, and surveys applications in classification, segmentation, detection, and localization.
Results
The reviewed studies report 86.3% AUC using 3D VggNet, 90% sensitivity with false positives reduced from 35 to 5, and 89.3% detection on test data.
Takeaways & Limitations
3D CNNs are used for volumetric medical image analysis across classification, segmentation, detection, localization, and registration applications.
Takeaways & Limitations
3D medical imaging models require many training parameters, and downsampling high-resolution volumes to reduce computational cost can cause significant information loss.
Abstract
from arXiv · showhide
The rapid advancements in machine learning, graphics processing technologies and the availability of medical imaging data have led to a rapid increase in the use of deep learning models in the medical domain. This was exacerbated by the rapid advancements in convolutional neural network (CNN) based architectures, which were adopted by the medical imaging community to assist clinicians in disease diagnosis. Since the grand success of AlexNet in 2012, CNNs have been increasingly used in medical image analysis to improve the efficiency of human clinicians. In recent years, three-dimensional (3D) CNNs have been employed for the analysis of medical images. In this paper, we trace the history of how the 3D CNN was developed from its machine learning roots, we provide a brief mathematical description of 3D CNN and provide the preprocessing steps required for medical images before feeding them to 3D CNNs. We review the significant research in the field of 3D medical imaging analysis using 3D CNNs (and its variants) in different medical areas such as classification, segmentation, detection and localization. We conclude by discussing the challenges associated with the use of 3D CNNs in the medical imaging domain (and the use of deep learning models in general) and possible future trends in the field.
1. Introduction
Medical imaging still depends on trained radiologists despite major advances in acquisition technology. Deep learning and CNNs offer automated analysis, while 3D CNNs extend this capability to volumetric medical scans.
- Radiological images require subsequent analysis and diagnosis by trained human radiologists despite improvements in acquisition time, quality, resolution, and affordability.
- Traditional automated image classification relied on time-consuming rule-based algorithms or handcrafted features with poor generalization and substantial domain-knowledge requirements.
- CNNs learn progressively higher-level image features through successive filter layers, avoiding manual feature handcrafting and requiring little preprocessing.
- Advances in computational power and GPUs have enabled 3D deep learning for volumetric CT, DTI, fMRI, ultrasound, and MRI scans.
2. Materials and Methods
The review describes the growth and selection of 3D deep-learning literature, explains 3D CNN architecture and training components, and outlines preprocessing for medical images. It also situates 3D CNNs within major CNN architectural advances and highlights practical constraints in medical imaging.
- Publication activity in deep learning for both 2D and 3D medical images accelerated after limited earlier work.
- 2.1. A Typical Architecture of 3D CNN: A 3D CNN extends 2D convolution by adding depth, allowing simultaneous extraction of spectral and spatial features from an input volume.The convolution uses a 3D kernel over an M × N × D input volume, with layer outputs and bias terms.
- 2.1. A Typical Architecture of 3D CNN: Pooling aggregates voxel activations through averaging or maximization and introduces translational invariance, while padding can avoid boundary problems during error calculation.The output of a pooling layer becomes the input to the next layer.
- 2.1. A Typical Architecture of 3D CNN: Deep CNNs are trained by calculating loss, backpropagating error, and updating weights and parameters in earlier layers.Batch normalization addresses changing hidden-layer inputs during training, while dropout and other regularization strategies target overfitting.
- 2.2. Breakthroughs in CNN Architectural Advances: CNN architectural advances include AlexNet’s multi-GPU training and regularization strategies, while ResNet uses skip connections to facilitate very deep networks.ResNet architectures described in the review range from 52 to 10,000 layers.
- Medical-image preprocessing begins with artifact removal, such as skull stripping, extracerebral-tissue removal, or principal-component analysis for fMRI spike artifacts.The review also notes sequential slice-acquisition delays when constructing volumetric brain representations.
4. Applications in 3D Medical Imaging
The review surveys 3D CNN applications across medical-image segmentation, classification, detection, and localization. Reported studies address challenging volumetric tasks including small-lesion segmentation, disease classification, microbleed and lung-nodule detection, and multimodal image analysis.
- 4.1. Segmentation: Segmentation studies apply 3D CNNs to brain lesions, tumors, brain structures, liver, heart, vessels, knee structures, and cervical tumors.DeepMedic uses multiresolution processing for brain-lesion segmentation, while other studies use FusionNet variants, fully connected 3D CNNs, 3D U-Net, and hybrid shape-model approaches.
- 4.1. Segmentation: Small lesion and tumor sizes relative to whole scan volumes create class-imbalance challenges for medical-image segmentation.The review identifies lesion size variation across scans and limited diseased-patient samples as contributors to imbalanced training data.
- 4.2. Classification: 3D CNN classification studies primarily detect diseases from neuroimaging, including dementia and Alzheimer’s disease across MRI, fMRI, and DTI data.The reviewed work also converts established 2D architectures into 3D counterparts and uses visual inspection methods to examine learned Alzheimer’s-related features.
- 4.3. Detection and Localization: 93% sensitivity was reported for fully 3D CNN detection of cerebral microbleeds, while lung-nodule systems reported 89.3% detection with a 1.79 false-positive rate.Other lung-nodule results included 90% sensitivity with false positives reduced from 35 to 5, and 92.9% sensitivity with four false positives per scan.
- 4.3. Detection and Localization: Localization methods use volumetric information to address anatomical variation, while multimodal approaches support matching CT and multiple MRI views for the same subject.The review distinguishes genuine 3D localization from methods that process 2D orthogonal slices and notes the need to identify corresponding structures across modalities and orientations.
5. Challenges and Conclusions
The review identifies data, computational, generalization, labeling, and interpretability challenges for 3D medical deep learning, while noting transfer learning and GAN-based synthesis as possible responses.
- Data requirements: Medical-image datasets are smaller and less variable than natural-image datasets, leaving the effectiveness of deep learning on limited medical data ambiguous.The review notes that fine-tuning 3D CNNs pretrained on natural-image datasets can be applied to medical images.
- Computational complexity: 3D medical-image volumes demand many training parameters, and downsampling high-resolution scans reduces computational cost but can cause significant information loss.Scan depth may range from 20 to 400 slices, while high-resolution volumes are commonly 512 × 512.
- Computational complexity: Without downsampling, fully connected layers can create intractable 3D CNN models with many trainable weights, making training harder and more GPU intensive than for 2D images.The review identifies the 3D extension of GoogLeNet’s inception module as a possible direction for addressing computational complexity.
- Generalization and interpretability: Medical-image models may overfit dataset-specific features because scanner settings, acquisition procedures, demographics, and disease heterogeneity vary across subjects and datasets.The review recommends interpretability methods, feature validation, and testing with external datasets.
- Data generation: Limited labeled images, high labeling costs, and disagreement among experts motivate reliable data augmentation and ground-truth sample generation using GANs such as CycleGANs.CycleGANs are discussed for cross-modal image synthesis and producing pseudo-images similar to the original dataset.