Source-linked AI summary
CNN-based Segmentation of Medical Imaging Data
Baris Kayalibay, Grady Jensen, Patrick van der Smagt
TL;DR
Medical image segmentation must handle scarce labels, class imbalance, and the memory demands of 3D images. This paper develops a 3D-filter CNN method based on a U-Net-like architecture, applies it to hand and brain MRI, and reports good-quality results despite limited labelled data.
Problem
Medical image segmentation faces scarce labelled data, severe class imbalance, and high memory demand from three-dimensional images.
Method
The paper uses a U-Net-like CNN with three-dimensional filters, combines multi-scale segmentation maps, compares skip-connection summation with concatenation, and downsamples oversized images.
Results
3D CNN architectures achieved good-quality results despite the relative scarcity of labelled medical images.
Takeaways & Limitations
The method is validated for bone segmentation in hand MRI and tumor segmentation in brain MRI.
Takeaways & Limitations
The Jaccard-based loss is poorly defined for samples with very few or no foreground instances and has limitations for multiclass maps.
Abstract
from arXiv · showhide
Convolutional neural networks have been applied to a wide variety of computer vision tasks. Recent advances in semantic segmentation have enabled their application to medical image segmentation. While most CNNs use two-dimensional kernels, recent CNN-based publications on medical image segmentation featured three-dimensional kernels, allowing full access to the three-dimensional structure of medical images. Though closely related to semantic segmentation, medical image segmentation includes specific challenges that need to be addressed, such as the scarcity of labelled data, the high class imbalance found in the ground truth and the high memory demand of three-dimensional images. In this work, a CNN-based method with three-dimensional filters is demonstrated and applied to hand and brain MRI. Two modifications to an existing CNN architecture are discussed, along with methods on addressing the aforementioned challenges. While most of the existing literature on medical image segmentation focuses on soft tissue and the major organs, this work is validated on data both from the central nervous system as well as the bones of the hand.
1 Introduction and Related Work
Medical image segmentation adapts CNN-based semantic segmentation to challenges including limited labelled data, severe class imbalance, and high 3D memory demands. The paper examines a U-Net-like approach and two architectural modifications for hand and brain MRI.
- The work applies a CNN-based method to bone segmentation in hand MRI and tumor segmentation in brain MRI.
- The proposed architecture is similar to U-Net and tests combining segmentation maps from different scales.
- The second modification forwards feature maps between network stages using element-wise summation.
- CNN segmentation architectures replace fully connected layers with convolutions to predict multiple pixels efficiently, but downsampling can produce low-resolution outputs.
- Medical segmentation commonly uses patch-based sliding inference or fully convolutional networks trained on full images or large sections.
- Full-image or large-section training must address scarce labelled data, while pixel-wise training inherits severe foreground–background class imbalance.
2 Method
The method uses a fully convolutional 3D CNN with modified U-Net-style skip connections and multi-resolution segmentation maps for MRI segmentation. It addresses 3D memory demands and class imbalance through downsampling and a Jaccard-distance-based loss.
- Network architecture: The baseline network uses contracting and expanding stages with long skip connections between corresponding feature maps.
- Network architecture: Element-wise summation combines contracting-stage and expanding-stage feature maps while forming residual blocks across the intervening layers.
- Multi-resolution outputs: Three segmentation maps are produced at input resolution, half resolution in each dimension, and quarter resolution in each dimension, then aligned and combined by interpolation and summation.
- Multi-resolution outputs: Combining segmentation maps from different network depths encourages earlier layers to produce useful segmentations and is intended to speed convergence rather than refine the final output.
- Memory management: For images too large for the network, the method uses downsampled images for training and inference, then upsamples the low-resolution segmentation.
- Implementation: The network uses zero-padding, PReLU activations, batch normalization, and strided convolutions instead of max-pooling.
- Loss metric: Training minimizes the Jaccard distance, 1 − Jacc, accumulating the loss across classes to address class imbalance.
- Loss metric: The loss is poorly defined for samples with very few or no foreground instances, and the Jaccard and Dice formulations are inherently binary before extension to multiclass segmentation.
3 Experiments
The experiments evaluate the proposed 3D CNN on hand-bone and brain-tumor MRI, using separate data partitions, augmentation, and modality or architecture comparisons. They examine class imbalance, skip connections, loss functions, segmentation-map combination, and MRI modality selection.
- 3.1 Data: The hand MRI task uses 30 volumes with four bone classes, while BRATS 2015 contains 274 multimodal images labeled for four tumor regions.Hand images are divided into training, validation, and test sets; BRATS images use Flair, T1, T1C, and T2 modalities.
- 3.1 Data: Background dominates both datasets, reaching approximately 96% in hand MRI and 99% in brain MRI.Middle and distal phalanxes are also nearly one order of magnitude less frequent than other foreground hand classes.
- 3.2 Training: Training uses random transformations to address limited data, with 20 hand volumes for training and BRATS partitions of 220, 27, and 27 images.BRATS training runs for 50 epochs, and one 160 × 144 × 128 image takes roughly 3 seconds to segment.
- 3.2.1 Training: Weights use a zero-mean Gaussian initialization with standard deviation 0.01; Xavier initialization was tested on hand MRI but produced worse results.The reported setup distinguishes the initialization used for results from the alternative tested initialization.
- 3.2 Experiments: The experiments compare Jaccard loss with categorical cross-entropy, long skip connections, multiple versus single segmentation maps, and summation versus concatenation.The hand comparisons include six-fold cross-validation for selected experiments, while BRATS experiments assess individual and combined MRI modalities and final network performance.
4 Results
The experiments compare loss functions and architectural modifications for 3D CNN segmentation on hand and brain MRI. Jaccard loss addressed infrequent classes, while multi-scale maps and concatenation improved or supported network performance, alongside observed false-positive limitations.
- Hand MRI: Categorical cross-entropy was far less capable than Jaccard distance at detecting the infrequent middle and distal hand-bone classes.Both networks performed well on the two relatively common classes, whereas cross-entropy favored frequent labels.
- Hand MRI: Combining three segmentation maps produced similar validation loss but better test-set Dice scores and faster convergence than producing one final map.The modified network was retained for subsequent experiments because faster convergence is desirable when training time is limiting.
- Hand MRI: Concatenation outperformed element-wise summation across all six cross-validation folds.The averaged validation-loss curves and test-set Dice scores supported the same comparison.
- Hand MRI: Feature visualizations suggest concatenation preserves separate global and local-information streams, whereas summation reintroduces irrelevant detail.The summation network produced fewer global-only feature maps and more maps combining global information with substantial local detail.
- BRATS brain MRI: T1C was the best individual BRATS modality for every class except edema, while Flair and then T2 produced better edema predictions.Combining Flair and T1C achieved Dice scores close to using all four modalities.
- BRATS brain MRI: BRATS leaderboard performance was generally worse than on the other evaluation sets, with greater Dice and precision variability for the enhanced region.Sensitivity remained similar across the four evaluation sets.
- BRATS brain MRI: Visual inspection revealed many false positives despite apparently high mean specificity, especially when segmenting images without visible unhealthy tissue.The authors suggest adding healthy-brain MRI, but could not acquire matching modalities for this work.
5 Conclusion
The work demonstrated a 3D CNN medical-image segmentation method on hand and brain MRI using two U-Net modifications. Summation performed slightly worse, while multi-scale segmentation maps accelerated convergence without reducing final performance; remaining limitations motivate further comparisons and broader validation.
- 5.1 Overview of Results: The method was demonstrated on hand-bone and brain-tumor segmentation using 3D CNNs applied to hand and brain MRI.The architecture was similar to U-Net.
- 5.1 Overview of Results: The two tested modifications combined multiple segmentation maps at different scales and forwarded feature maps by element-wise summation.These modifications altered the U-Net-style architecture used throughout the experiments.
- 5.1 Overview of Results: Summation produced slightly worse results, whereas combining multiple segmentation maps accelerated convergence without hurting final performance.The conclusion reports these as the principal architectural findings.
- 5.1 Overview of Results: The Jaccard-based loss function addressed class imbalance, while downsampling reduced the memory burden of 3D images.Downsampling was used instead of dividing images into multiple sections.
- 5.1 Overview of Results: The results indicate that 3D CNN architectures can achieve good-quality medical-image segmentation despite relatively scarce labelled data.The conclusion states this within the scope of the demonstrated hand and brain MRI tasks.
- 5.2 Future Work: Future work should test healthy-brain MRI inclusion and compare Jaccard loss with weighted or ordinary categorical cross-entropy under less severe imbalance.The authors also propose applying the network to other imaging technologies and structures and using deeper architectures.