Source-linked AI summary
Location Sensitive Deep Convolutional Neural Networks for Segmentation of White Matter Hyperintensities
Mohsen Ghafoorian, Nico Karssemeijer, Tom Heskes, Inge van Uden, Clara Sanchez, Geert Litjens, Frank-Erik de Leeuw, Bram van Ginneken, Elena Marchiori, Bram Platel
TL;DR
CNNs do not naturally incorporate anatomical location, despite its importance for accurate medical image segmentation. The paper compares CNN architectures that use multiscale patches or explicit spatial features for WMH segmentation. Location-aware CNNs substantially outperform conventional hand-crafted-feature segmentation and approach independent-human-observer performance.
Problem
CNNs lack a natural ability to incorporate anatomical location, although location is important for accurate WMH voxel classification and segmentation.
Method
The study compares CNN architectures using multiscale patches, explicit spatial features, or both, trained and evaluated on a dataset of more than 500 patients.
Results
Location-aware CNNs substantially outperform conventional hand-crafted-feature segmentation and CNNs without location information; the best network reaches test Dice 0.791 versus 0.797 for an independent human observer.
Takeaways & Limitations
Incorporating anatomical location enables WMH CNN segmentation to approach the performance level of an independent human observer.
Takeaways & Limitations
The system produces prevalent false positives around lacunes because insufficient similar negative samples were included during training.
Abstract
from arXiv · showhide
The anatomical location of imaging features is of crucial importance for accurate diagnosis in many medical tasks. Convolutional neural networks (CNN) have had huge successes in computer vision, but they lack the natural ability to incorporate the anatomical location in their decision making process, hindering success in some medical image analysis tasks. In this paper, to integrate the anatomical location information into the network, we propose several deep CNN architectures that consider multi-scale patches or take explicit location features while training. We apply and compare the proposed architectures for segmentation of white matter hyperintensities in brain MR images on a large dataset. As a result, we observe that the CNNs that incorporate location information substantially outperform a conventional segmentation method with hand-crafted features as well as CNNs that do not integrate location information. On a test set of 46 scans, the best configuration of our networks obtained a Dice score of 0.791, compared to 0.797 for an independent human observer. Performance levels of the machine and the independent human observer were not statistically significantly different (p-value=0.17).
1. Introduction
WMH quantification is clinically relevant but manual segmentation is laborious and variable, while reliable fully automated methods have remained elusive. Because anatomical location informs WMH classification, the study evaluates CNN architectures that incorporate spatial context.
- Clinical motivation: WMHs are common in several neurological conditions and are best observed as high-intensity signals on FLAIR MR images.Reported prevalence among small vessel disease patients reaches up to 95%, depending on population and imaging technique.
- Clinical motivation: Accurate WMH volume and distribution assessment is clinically important for prognosis, disease tracking, and treatment-effectiveness evaluation.
- Clinical motivation: Manual WMH segmentation is time-consuming and shows considerable inter- and intra-rater variability.
- Existing methods: Existing automated approaches include supervised models with hand-crafted or learned features and unsupervised methods, but no reliably human-level fully automated method had been identified.
- Study rationale: Anatomical location is important for voxel classification in biomedical segmentation, unlike many general computer-vision segmentation benchmarks.
- Study rationale: The study compares CNN strategies using multi-scale patches or explicit spatial features on a dataset of more than 500 patients.
2. Materials
The study uses RUN DMC brain MRI data from selected participants, with T1 and FLAIR imaging, manual reference annotations, preprocessing, and separate training, validation, and independent test sets.
- Study cohort: RUN DMC participants were selected at baseline in 2006 and underwent rescanning in 2011/2012, with a third follow-up being acquired.
- Study cohort: Participants were aged 50–85 years and had cerebral small vessel disease on neuroimaging, while multiple neurological and medical conditions were exclusion criteria.
- MRI acquisition: Baseline imaging used a single 1.5 Tesla scanner with T1 and FLAIR modalities.
- Reference annotations: Two experienced raters manually contoured hyperintense FLAIR lesions lacking corresponding cerebrospinal-fluid-like hypointensity on T1 images.
- Preprocessing: Preprocessing included brain extraction, rigid T1-to-FLAIR registration, nonlinear registration to the ICBM152 atlas, bias-field correction, and per-patient intensity normalization.
- Data partitioning: After exclusions, 378 single-annotated cases were used for training, 42 for validation, and 46 double-annotated cases for independent testing.
3. Methods
The methods compare single-scale and multi-scale CNNs with explicit spatial-location features for WMH segmentation. Multi-scale inputs provide context, while explicit features encode atlas-based and anatomical position information.
- Patch preparation: Patches of 32×32, 64×64, and 128×128 were extracted from T1 and FLAIR neighborhoods, with larger patches downsampled to 32×32.
- Single-scale model: The single-scale baseline uses only 32×32 patches and forms the basis for the other location-sensitive architectures.
- Single-scale model: The CNN baseline uses four convolutional layers followed by fully connected layers and a two-class softmax output.
- Multi-scale early fusion: Multi-scale early fusion combines the three downsampled scales as input channels, aiming to balance local localization with broader contextual information.
- Multi-scale fusion: Alternative multi-scale architectures use separate streams that fuse scale-specific representations, with weight sharing proposed to reduce parameters and overfitting.
- Explicit spatial features: Explicit spatial features are added at the first fully connected layer and include MNI coordinates, anatomical distances, and prior WMH probability.
- Training procedure: Networks are trained with mini-batch stochastic gradient descent using cross-entropy, RMSPROP, ReLU activations, Glorot initialization, dropout, and validation-based model selection.
4. Experimental Evaluation
The evaluation compares CNN architectures, a conventional hand-crafted-feature segmentation system, and human observers using validation Az and test Dice. Dice thresholds are optimized on validation data, while spatial-location integration is evaluated at several network locations.
- All brain-mask voxels are classified with trained networks using a sliding-window procedure.
- Dice similarity measures segmentation agreement from 0 for complete disagreement to 1 for complete agreement.A Dice score of 0.7 or higher is usually considered good segmentation.
- Binary CNN probability maps use method-specific thresholds selected to maximize validation-set Dice.
- The conventional comparator uses 22 hand-crafted features, including FLAIR and T1 intensities and multiscale image filters.
- Spatial features are compared at the last convolutional, first fully connected, and second fully connected layers.Table 5 defines the abbreviations LCL, FFCL, and SFCL for these locations.
5. Experimental Results
The experiments compare CNN architectures, the conventional method, and human observers on validation and independent test data. Adding spatial location at the first fully connected layer performs significantly better than the other tested insertion points.
- Table 2 reports validation Az and test Dice for CNN methods with and without spatial location features.
- Table 3 compares the conventional method, MSWS+Loc, and two human observers on the independent test set.
- Adding spatial location information at the first fully connected layer yields significantly better Dice than the other two integration options (p-value < 0.01).
- ROC curves focus on low false-positive rates, while threshold plots show Dice as a function of binary-mask threshold.
- Test Dice is also examined as a function of training-set size.
6. Discussion
Discussion results indicate that explicit location and multiscale context improve CNN WMH segmentation, with explicit location outperforming learned location from multiscale patches. The authors also examine training-set size and qualitative failure patterns.
- Adding explicit location to single-scale CNNs significantly improves Dice (p-value < 0.01).
- Multi-scale CNNs also significantly outperform single-scale CNNs, indicating that multiscale input provides useful context information (p-value < 0.01).
- Explicit location features outperform location learned from multiscale patches in the reported comparison.
- Figure 5 and Table 6 relate test Dice to training-set size.
6.2. Early fusion vs. late fusion, independent weights vs. weight sharing
Late fusion architectures performed better than early fusion, while MSWS+Loc achieved performance close to independent human observers and exceeded the conventional method. Qualitative examples show that location and larger context can reduce false positives and improve missed-lesion detection.
- Early fusion vs. late fusion: Late fusion architectures showed comparable good performance, whereas early fusion produced the least improvement over the single-scale approach.The authors attribute this to weak correspondence between scales and early summation of their responses.
- Comparison with conventional method and observers: MSWS+Loc achieved a Dice score of 0.79 versus 0.71 for the conventional segmentation method (p-value<0.01).Its Dice score closely resembled inter-observer variability, and the independent observer did not significantly outperform it (p-value = 0.17).
- Qualitative examples: Location and larger-context information avoided a septum pellucidum false positive produced by the single-scale CNN.A second qualitative example showed improved false negatives for the single-scale method.
- Qualitative examples: Hyperintense voxels around lacunes were a prevalent class of false positive segmentation.The authors linked this error to insufficient negative training examples resembling lacune surroundings.
- Qualitative examples: MSWS+Loc detected a small right temporal-lobe lesion missed by both human observers.The authors suggest some apparent false positives may be small lesions missed by one or both observers, though further research is required.
6.5. Integration of location features
The study found that integrating spatial location features at the first fully connected layer was the most effective of the investigated integration options. The broader approach trains location information jointly with the network representations rather than adding it only afterward.
- Integration of location features: Adding spatial location features to the first fully connected layer produced significantly better Dice performance than the other two investigated options (p-value < 0.01).Adding them near the last convolutional layer made the eight location features almost insignificant among approximately 35K representation features.
- Integration of location features: The results indicate that integrating location information into a CNN can play an important role in obtaining accurate segmentation.The location features are integrated while the network learns its representations.
- Single-stage versus two-stage model: The study’s single-stage approach jointly trains representations with location information instead of training a representation network and then a separate classifier.The authors argue that a two-stage system may learn filters without access to the location information available during later classification.
6.7. 2D vs. 3D patches
The study used 2D patches from T1 and FLAIR images rather than 3D patches. The authors note that slice correspondence and computational cost constrain the usefulness of a 3D approach in this dataset.
- 2D vs. 3D patches: The networks sampled 2D patches from the T1 and FLAIR modalities, while consecutive slices might provide useful additional information through 3D patches.The dataset had 5 mm slice thickness and a 1 mm interslice gap, so consecutive slices did not highly correspond.
- 2D vs. 3D patches: Extensively incorporating 3D patches would increase computational costs during both training and segmentation.This computational consideration is part of the authors’ rationale for using 2D patches.
6.8. Fully convolutional segmentation network
The paper trained its networks in a patch-based manner but notes that the architecture could be reformulated as a fully convolutional segmentation network. This reformulation would support dense prediction with fewer repetitive computations.
- Fully convolutional segmentation network: Fully convolutional networks replace fully connected layers with 1×1 convolutions that perform the same functionality.The reformulation preserves the fully connected layers’ role while implementing it with convolutions.
- Fully convolutional segmentation network: A fully convolutional reformulation could speed segmentation by processing larger input images and producing dense predictions for the whole image.It would also avoid repetitive computations associated with patch-based processing.
7. Conclusions
Location information added significant value to CNN-based WMH segmentation, bringing performance close to that of an independent human observer.
- Location information significantly improved CNN-based WMH segmentation.
- CNN-based segmentation outperformed a conventional segmentation method.
- Incorporating location information brought CNN performance close to an independent human observer.