Source-linked AI summary
Hough-CNN: Deep Learning for Segmentation of Deep Brain Regions in MRI and Ultrasound
Fausto Milletari, Seyed-Ahmad Ahmadi, Christine Kroll, Annika Plate, Verena Rozanski, Juliana Maiostre, Johannes Levin, Olaf Dietrich, Birgit Ertl-Wagner, Kai Bötzel, Nassir Navab
TL;DR
The paper addresses medical-image segmentation under limited annotated data, including midbrain segmentation relevant to computer-assisted diagnosis of Parkinson’s disease. It proposes Hough-CNN, a CNN-based Hough-voting approach, and reports stronger segmentation than voxel-wise CNN semantic segmentation while using less training data.
Problem
Limited large annotated datasets constrain clinical environments, while midbrain segmentation is a crucial step toward computer-assisted diagnosis of Parkinson’s disease.
Method
Hough-CNN performs patch-wise multi-atlas segmentation using Hough voting and features from the deepest portion of a CNN.
Results
Hough-CNN outperformed voxel-wise CNN semantic segmentation in all parameter settings while using less training data; one model used 1.35 million rather than 13.5 million 3D patches.
Takeaways & Limitations
The proposed approach supports segmentation across regions and modalities while reducing the amount of training data used.
Takeaways & Limitations
The approach delivers sub-optimal results in the setting discussed around Figure 7, and future work will investigate formation from multiple experts.
Abstract
from arXiv · showhide
In this work we propose a novel approach to perform segmentation by leveraging the abstraction capabilities of convolutional neural networks (CNNs). Our method is based on Hough voting, a strategy that allows for fully automatic localisation and segmentation of the anatomies of interest. This approach does not only use the CNN classification outcomes, but it also implements voting by exploiting the features produced by the deepest portion of the network. We show that this learning-based segmentation method is robust, multi-region, flexible and can be easily adapted to different modalities. In the attempt to show the capabilities and the behaviour of CNNs when they are applied to medical image analysis, we perform a systematic study of the performances of six different network architectures, conceived according to state-of-the-art criteria, in various situations. We evaluate the impact of both different amount of training data and different data dimensionality (2D, 2.5D and 3D) on the final results. We show results on both MRI and transcranial US volumes depicting respectively 26 regions of the basal ganglia and the midbrain.
1. Introduction
The paper investigates CNN-based segmentation of single and multiple deep-brain regions across MRI and ultrasound. It proposes Hough-CNN and evaluates architectures, training-data amounts, and input dimensionalities.
- Motivation: Clinical segmentation of brain structures in MRI and ultrasound is relevant but challenging because of weak contrast, artefacts, and missing boundaries.MRI basal-ganglia segmentation supports diagnosis, treatment, research, and DBS planning; TCUS midbrain segmentation is relevant to computer-assisted Parkinson’s diagnosis.
- Motivation: Manual 3D labeling is extremely time-consuming, while CNNs typically require massive annotated datasets and substantial computational resources.Clinical environments may lack large annotated datasets and may restrict computation outsourcing because of data-protection policies.
- Approach: The method is registration-free, robust to artefacts, and can segment structures that are only partially visible or corrupted by artefacts.Its patch-based voting strategy is designed to support localisation and segmentation despite incomplete or degraded appearances.
- Evaluation: The study compares six CNN architectures across different training-data amounts and 2D, 2.5D, and 3D convolutional settings.The experiments examine how architecture complexity and data dimensionality affect segmentation results.
2. Related Works
Prior medical-imaging work often relied on 2D processing, limited training data, or CNNs embedded within other inference frameworks. Hough-CNN is presented as a flexible voting-based alternative that incorporates anatomical context and evaluates multiple dimensionalities and architectures.
- Prior CNN-based methods: Medical-imaging studies often use few training images, augment patches, or embed CNN outputs within global inference, graph, or shape-based methods.Examples include graph cuts, affinity-graph partitioning, active-shape models, and shape-regularised level sets.
- Hough-CNN: Hough-CNN combines voxel-wise classifications with high-level feature-based votes retrieved from annotated training images.Its neighborhood-based feature-space voting allows a variable number of votes and avoids regression-based vote behavior on unusual data.
- Hough-CNN: The approach is designed to support multi-region segmentation while allowing diagnostic or therapeutic information to be modified without retraining.The paper describes this flexibility as a consequence of associating votes with segmentation patches and related information.
- Evaluation: The study benchmarks six network architectures, including an eight-convolutional-layer network, and evaluates 2D, 2.5D, and 3D patches.The 3D patches retain a large amount of anatomical context.
3. Method
Hough-CNN combines CNN patch classification and learned intermediate features with Hough voting to localise and segment brain structures. The method uses six architectures and supports 2D, 2.5D, and 3D medical-volume inputs while limiting pooling to preserve localisation accuracy.
- CNN design and training: Six CNN architectures are trained on patches extracted from annotated medical volumes acquired in MRI and ultrasound modalities.The volumes depict deep structures of the human brain.
- CNN design and training: Pooling is minimised because its spatial invariance conflicts with the localisation accuracy required for segmentation, while max-pooling otherwise reduces response dimensionality.The restriction is also constrained by existing hardware requirements.
- Hough voting: During testing, foreground classifications cast votes into a map, whose maximum identifies the anatomy centroid; nearby votes then assemble contour evidence by projecting segmentation patches.A confidence map is thresholded to retrieve the final contour.
- CNN design and training: CNNs classify patches voxel-wise, but this semantic segmentation can be sub-optimal because it lacks regularisation and statistical priors.The authors introduce a method based on simultaneous localisation and robust contour extraction to address this limitation.
- Hough-CNN representation: Hough-CNN maps each patch to a foreground/background label and an intermediate feature vector from the second-last fully connected layer.These learned features are used in addition to categorical CNN predictions.
- Hough voting: During training, foreground patches are paired with displacement votes toward anatomy centroids and binary segmentation patches in a database.The vote joins a patch location to the centroid of the anatomy in its training volume.
4. Experiments and Results
The experiments evaluate CNN-based Hough-voting segmentation across MRI and transcranial ultrasound brain volumes, varying architecture, training data, and patch dimensionality. Hough-CNN robustly localises and segments structures across modalities, although small and low-contrast regions remain harder to segment.
- Datasets: The study evaluates CNN segmentation on complementary MRI and ultrasound brain datasets, including basal ganglia structures and the midbrain.The MRI and ultrasound modalities differ inherently, and the experiments assess Hough-voting-based segmentation on both.
- Generalisation: The experiments show that the method generalises well to previously unseen data, a property described as desirable for clinical settings.Ultrasound testing uses previously unseen subjects and 114 test sweeps, while the broader evaluation reports results across MRI and ultrasound volumes.
- Experimental design: Six architectures are tested with 2D, 2.5D, and 3D patches while varying the amount of training data.Each architecture is trained across dimensionality and training-set combinations to assess how added contextual information affects performance.
- Ultrasound results: Hough-CNN accurately localises and segments the ultrasound midbrain across left and right bone windows and varying bone-window quality.The method remains robust despite visibility problems, signal drop, and blurring in transcranial ultrasound.
- Performance patterns: Hough-CNN yields better Dice coefficients for bigger and high-contrast regions, whereas small and low-contrast regions are harder to segment.Small low-contrast regions may still be correctly localised but are generally more difficult to segment.
- MRI results: Hough-CNN correctly localises and segments multiple MRI structures despite large anatomical variability, including cortical atrophy and enlarged lateral ventricles.Visual MRI examples compare semantic segmentation results with Hough-CNN outputs against ground-truth annotations.
5. Discussion
The discussion finds Hough-CNN robust across MRI and ultrasound, with performance shaped by modality, input dimensionality, architecture, and training-set size. Its voting and shape-prior mechanisms support accurate, registration-free segmentation even when anatomy is difficult to see.
- Training data and dimensionality: 3D data produced the best segmentation results overall, but ultrasound showed the opposite dimensionality trend from MRI.In MRI, 3D processing adds useful information; in ultrasound, sagittal and coronal planes can provide misleading information.
- Training data and dimensionality: 90% less training data was required for 3D patches to surpass performance obtained with 13.5 million 2D and 2.5D patches.The 3D configuration required only 1.35 million training patches.
- Overall performance: Hough-CNN outperforms voxel-wise semantic segmentation across the evaluated settings.The approach embeds CNN classification within a higher-level voting method that encodes anatomic shape and appearance priors.
- Architecture: The simplest architecture, “7-5-3”, performed best in MRI, whereas deeper architectures performed best in challenging ultrasound.“SmallAlex” and “3-3-3-3-3” were among the best-performing ultrasound networks.
- Practical implications: The method is faster than atlas segmentation, registration-free, and flexible because votes and segmentation patches can be substituted or augmented without retraining.Reported runtimes were 30 seconds for ultrasound and 3–4 minutes for MRI on the testing machine.
6. Conclusion
The paper proposes Hough-CNN, a patch-wise multi-atlas method that applies CNNs to medical image segmentation across modalities and studies key CNN design factors. It outperformed voxel-wise semantic segmentation across parameter settings while using less training data and producing smooth contours without post-processing.
- The study examines CNN architectures, patch dimensionality, and training-set size under limited data and computational resources.
- Hough-CNN is a patch-wise multi-atlas segmentation method that implicitly encodes anatomical shape and context.
- Hough-CNN outperformed CNN voxel-wise semantic segmentation in all tested parameter settings.
- The method used less training data and produced smooth segmentation contours without post-processing.
- The method is modality-independent and scalable to multiple regions, supporting clinical applications of CNN-based deep learning.