Medical Image Segmentation with 3D Convolutional Neural Networks: A Survey

• Segmentation is crucial for detecting abnormalities and estimating organ or lesion extent, yet medical-image research emphasizes classification and 3D-image studies often use 2D CNNs.Source: §1 · pp. 2–3
• The review addresses the absence of a comprehensive survey of 3D deep-learning methods for medical-image segmentation and discusses future perspectives.Source: §1 · p. 4
• It reviews state-of-the-art 3D deep-learning approaches for medical-image segmentation and excludes papers with overlapping techniques to reduce redundancy.Source: §1 · p. 4
• The review also provides an overview of 3D deep learning for medical-image processing and discusses research gaps and future directions in 3D segmentation.Source: §1 · p. 4

2.1. Convolutional Neural Networks: An overview

• CNN kernels learn image features, while pooling downsamples feature maps, reduces later-layer parameters, expands receptive fields, and provides invariance to small shifts.CNNs reuse filters across image locations, making the number of weight parameters independent of image size.Source: §2.1 · pp. 4–5

2.2. 2D CNN vs 3D CNN in Image Segmentation

• 3D CNN segmentation processes volumetric input with 3D kernels and pooling to learn inter-slice spatial features, but requires more computation and memory than 2D CNNs.Encoder-decoder models use upsampling and skip connections to restore feature-space dimensions and preserve fine features in the prediction mask.Source: §2.2 · pp. 6, 8

2.3. An Overview on 3D Medical Data

• Most 3D medical images are voxel-based volumes formed by registering and stacking 2D slices acquired across axial, coronal, or sagittal planes.Compared with 2D imaging, 3D imaging requires significantly more storage and is often more expensive.Source: §2.3 · pp. 8–9

2.4. Why 3D CNN is Important in 3D Medical Image Segmentation?

• Almost half the per-epoch training time was required by 3D CNN versus 2D CNN under matched frame counts and comparable trainable parameters.The comparison used 224 volumes of size 256×256×32 and reshaped them into 7,168 2D samples; 3D CNNs also require more graphical memory and may need more samples for high segmentation accuracy.Source: §2.4 · p. 9

2.5. 3D CNN in Medical Image Segmentation

• The survey classifies 3D CNN medical image segmentation studies published from 2015 through 2021 into subgroups based on learning approaches.The supplied passage introduces the categories but does not name them.Source: §2.5 · p. 9

1. Fully supervised 3D CNN models

• The fully supervised models are divided into direct 3D CNN models, 3D patch-wise segmentation models, and multi-task learning models.Source: §1 · p. 10

• The survey also identifies 3D CNN models with semi-supervised and weakly supervised learning.Source: §1 · p. 10

4. Cost-effective approximations of 3D CNN

• Fully supervised 3D CNNs train on image volumes with labeled masks; the approach is straightforward and has less training complexity.The survey further classifies these models by architecture implementation and learning paradigm.Source: §4 · p. 10
• Because labeling large medical image datasets is challenging, research also examines 3D CNNs for semi-supervised and weakly supervised segmentation with limited labeled samples.The survey includes a few such papers in its review.Source: §4 · p. 10
• 3D CNNs demand substantially more memory, motivating cost-effective approaches that simulate them to extract inter-slice features.The survey reviews these efforts and compares state-of-the-art 3D medical image segmentation methods within each category.Source: §4 · p. 10

• V-Net uses residual blocks, strided convolution, and Dice-based loss, while Bui et al. [46] fuse multiscale features for infant brain MRI segmentation.V-Net results on PROMISE2012 were very close to state-of-the-art 2D models; Bui et al.'s direct decoder feature merging may create semantic gaps.Source: §1 · p. 11
• Deep-supervised 3D encoder-decoders merge features across decoder levels, with Dou et al. also using CRF post-processing to improve segmentation.Kayalibay et al. [48] use element-wise summation to speed convergence, though semantic gaps may limit learning; Dou et al.'s approach can work on relatively small datasets and uses CRF.Source: §1 · p. 11
• Li et al. use increasing dilation and residual connections to retain spatial resolution and combine multiscale features without max-pooling.The lack of max-pooling reduces translation invariance and increases computational complexity.Source: §1 · p. 11
• Residual 3D models use multiscale feature fusion, with VoxResNet incorporating multimodal MRI context and Niyas et al. targeting focal cortical dysplasia.VoxResNet uses T1, T1-IR, and T2-FLAIR sequences [55]; Niyas et al. use sliced stacking to generate more samples and a shallow residual U-Net to extract multiscale features with fewer trainable parameters.Source: §1 · pp. 11–12
• Attention-gated 3D U-Net routes skip-connection features through local grid-based gates and consistently improves on U-Net across imaging datasets,.The authors note that shallow-depth attention may fail to pass salient features to upsampling layers.Source: §1 · p. 12
• Wang et al. combine recursive residual blocks with pyramid pooling to improve multi-class segmentation of white matter, gray matter, and cerebrospinal fluid.The model reports better performance on CANDI, IBSR18, and IBSR20 datasets.Source: §1 · p. 12
• Peng et al. use multiple 3D U-Net modules and Xception blocks to capture spatial information at different scales and fuse the resulting feature maps.Depth-wise separable convolutions reduce computation complexity, while the deep, complex network requires substantial memory and computation.Source: §1 · pp. 12–13
• OM-Net combines segmentation tasks in one pass and uses cross-task guided attention to address model-cascade complexity and class imbalance.The method correlates task predictions and reports significant performance improvements on multiple public datasets.Source: §1 · p. 13

2. 3D Patch-wise segmentation models

• Patch-wise methods process small 3D sub-samples separately and merge predictions into a full 3D map, reducing resource demands but potentially missing global, region-specific features.Patch analysis also generates many samples that favor learning, but limited global feature extraction may constrain performance when abnormalities are region-specific.Source: §2 · pp. 13, 15
• Yu et al. extended FractalNet [71] with auxiliary classifiers for deep supervision, using 64 × 64 × 64 patches and reporting good HVSMR 2016 results.The auxiliary classifiers were deployed at expanding layers to reduce the vanishing-gradient problem.Source: §2 · p. 13
• Two-scale dual-pathway CNN improved brain-lesion segmentation over 2D CNNs on BRATS 2015 and ISLES.It classified center voxels using features from different receptive fields and used a 3D CRF to reduce false alarms; a residual-connection version reported better performance at BRATS 2016.Source: §2 · p. 14
• EMMA combined DeepMedic, three 3D FCNNs, and two 3D U-Nets, winning BRATS 2017 but increasing computation and training time.It aggregated model confidence maps and assigned each voxel the class with the highest average confidence.Source: §2 · p. 14
• Chen et al. used two patch scales and hierarchical dense convolutions without pooling, reducing class imbalance and training cost while outperforming similar patch-wise methods on BRATS 2017.The approach segmented glioma from multimodal brain MRI volumes.Source: §2 · p. 14

3. Multi-task learning models

• K-Net simultaneously segments and quantifies the left ventricle from paired echo views, and its reported results show superiority in heterogeneous learning.The method uses 2D convolutions at different stages to quantify the 3D structure.Source: §3 · p. 16

2.5.2. Semi-supervised Learning

• Sparse-label 3D segmentation includes a modified 3D U-Net, GAN-based labeled/unlabeled training, and DMPCT teacher–student pseudo-labeling.The sparse-label U-Net study reported improvements over state-of-the-art 2D CNN architectures without considerable segmentation-performance degradation; the GAN approach notes that synthetic samples may not match actual data distributions.Source: §2.5.2 · pp. 16–17
• Semi-supervised learning still requires accurately labeled examples, and its performance depends on annotation accuracy and feedback between labeled and unlabeled data.These requirements make SSL challenging for large 3D image datasets and motivate weakly supervised alternatives.Source: §2.5.2 · pp. 16, 18

2.5.3. Weakly-supervised Learning

• Weak supervision can make labeling cheaper and more practical through less accurate annotations, but noisy pseudo-labels can substantially reduce segmentation performance.The review highlights image-level or abstract annotations as easier to obtain, sometimes with minimal radiologist support.Source: §2.5.3 · pp. 19–20
• Weakly supervised methods include bounding-box-guided catheter segmentation, attention-based brain-lesion representation learning, and CIVA-Net pseudo-localization for cryo-ET segmentation [99, 100, 101].CIVA-Net combines seed generation, cross-image consensus, and inter-voxel affinity learning before VoxResNet predicts the segmentation map.Source: §2.5.3 · pp. 19–20

2.5.4. Cost-effective approximations of 3D CNN models

• Orthogonal-plane approaches combine 2D views to predict 3D segmentations, but some lose contextual or location information or incur resource costs.The review also notes that patch-wise analysis may fail to learn region-specific features.Source: §2.5.4 · pp. 20–21
• OBELISK reduces parameters and memory, while cascaded HNN, separable 3D U-Net, and CPR designs target efficient segmentation; approximation performance depends on data and task [105, 106, 109, 43].The review also cautions that approximation-specificity limits generic use, leaving a universal model an open challenge.Source: §2.5.4 · pp. 21–22
• Med3D and Models Genesis provide pretrained 3D models that can improve transfer performance or convergence, but modality and abnormality differences can limit transfer learning.Med3D is reported as useful for similar modalities and small datasets with less training time.Source: §2.5.4 · p. 22

Direct 3D CNN models

• Milletari et al. use a 3D V-Net with residual blocks and strided convolution for PROMISE2012 MRI.Source: Direct 3D CNN models · p. 23
• Dou et al. apply 3D U-Net with deep supervision, including a variant with CRF-based post-processing, to liver CT.Source: Direct 3D CNN models · p. 23
• Li et al. use dilated convolution in a 3D Res-Net, while Chen et al. present VoxResNet for learning from small datasets.The table lists brain MRI datasets for both approaches, including ADNI for Li et al. and a multi-modal dataset for Chen et al.Source: Direct 3D CNN models · p. 23
• The surveyed approaches also include patch-wise 3D segmentation and multi-modal 3D CNNs using patches at two scales.One listed ensemble combines two DeepMedic, three 3D FCNN, and two 3D U-Net models.Source: Direct 3D CNN models · p. 23
• Other U-Net variants include region-specific attention, pyramid pooling to fuse decoding-level feature maps, Xception blocks, and cross-task guided attention in OM-Net.Source: Direct 3D CNN models · p. 23
• DeepMedic is a multi-scale 3D CNN with residual connections and a 3D fully connected CRF.The table associates this model with Kamnitsas et al. and brain MRI (BRATS 2015).Source: Direct 3D CNN models · p. 23

Multi-task learning models

• Zhou et al. describe a multi-task network for classification and segmentation in 3D breast ultrasound.The dataset name in the passage is truncated after “Pri-”.Source: Multi-task learning models · p. 24
• Gordaliza et al. [83] are listed with chest CT from a private dataset and a self-normalized neural network approach.The approach description is truncated in the passage.Source: Multi-task learning models · p. 24
• Ge et al. are listed with A4C and A2C echocardiography images.Source: Multi-task learning models · p. 24

Semi-supervised Learning

• K-Net is listed for multi-view echo sequence segmentation and quantification, alongside deep multi-planar co-training (DMPCT).Source: Semi-supervised Learning · p. 24
• A 3D GAN generates synthetic samples for training with labeled and unlabeled 3D volumes.Source: Semi-supervised Learning · p. 24
• Li et al. used a 3D V-Net with a signed distance map for 3D heart MRI segmentation.An SSL framework using labeled images and a signed distance map is also listed.Source: Semi-supervised Learning · p. 24
• Wang et al. proposed FocalMix, a thoracic CT model built on the MixMatch semi-supervised learning method.Source: Semi-supervised Learning · p. 24
• A mean teacher framework uses hierarchical consistency regularization and multi-scale deep supervision.Source: Semi-supervised Learning · p. 24

3.1. Summary

• The review classifies nearly 40 articles by their 3D deep-learning approaches and notes a remarkable recent rise in publications, although 3D methods remain less used than 2D approaches.The authors believe deep learning significantly contributes to medical image segmentation with 3D images.Source: §3.1 · p. 25
• Converting 2D CNN kernels and pooling to 3D eases implementation but causes optimization issues and limits the full potential of 3D data analysis.The review also covers alternatives addressing computation cost, memory demands, and the need for large volumetric datasets.Source: §3.1 · p. 26

3.2. Challenges in 3D Medical Image Segmentation

• Large training datasets are difficult to assemble because medical data may be inconsistently acquired or annotated, while slice-by-slice labeling is time-consuming and subjective.Multiple experts may be needed to label large volumes, potentially introducing noise that reduces segmentation efficiency.Source: §3.2 · p. 26
• Acquisition variation, contrast, and noise hinder feature generalization, while class imbalance can skew segmentation toward high-voxel-share classes, especially in lesion and multiclass tasks.Large volumetric datasets also demand substantial memory and computation.Source: §3.2 · p. 26

3.3. Future Directions

• Semi-supervised and weakly supervised methods can learn from partial or coarse annotations and reduce label error and subjectivity, but remain early-stage research.Developing sophisticated semi-supervised 3D CNNs for volumetric segmentation remains an open challenge.Source: §3.3 · p. 27
• Asymmetric loss functions with custom voxel-class weights have been explored, but selective sampling and adaptive augmentation fail on highly imbalanced datasets.The authors identify class-imbalance methods, multitask learning with 3D CNNs, and GAN-generated realistic medical volumes as future research opportunities.Source: §3.3 · p. 27