Source-linked AI summary
Fully Convolutional Network Ensembles for White Matter Hyperintensities Segmentation in MR Images
Hongwei Li, Gongfa Jiang, Jianguo Zhang, Ruixuan Wang, Zhaolei Wang, Wei-Shi Zheng, Bjoern Menze
TL;DR
WMH are common in older adults and associated with neurological and geriatric disorders, while manual delineation is laborious and variable. The paper develops and evaluates an ensemble of deep fully convolutional networks for WMH segmentation, using benchmark datasets and studies of modalities, augmentation, scanners, and ensemble size. Its system won the MICCAI 2017 WMH Segmentation Challenge, and the authors report good adaptability across scanners and protocols.
Problem
WMH are associated with neurological and geriatric disorders, but manual delineation is laborious, time-consuming, and variable.
Method
The paper uses an ensemble of convolution-deconvolution networks with long-range connections and evaluates modality combinations, augmentation, scanner adaptability, and ensemble size.
Results
The proposed system won the MICCAI 2017 WMH Segmentation Challenge and achieved good adaptability across various scanners and protocols.
Takeaways & Limitations
FLAIR and T1 provide complementary information, while ensemble modeling helps reduce over-fitting and boost segmentation results.
Takeaways & Limitations
Intensity normalization based on brain-wide voxel intensities can be biased by patients’ varying disease burden.
Abstract
from arXiv · showhide
White matter hyperintensities (WMH) are commonly found in the brains of healthy elderly individuals and have been associated with various neurological and geriatric disorders. In this paper, we present a study using deep fully convolutional network and ensemble models to automatically detect such WMH using fluid attenuation inversion recovery (FLAIR) and T1 magnetic resonance (MR) scans. The algorithm was evaluated and ranked 1 st in the WMH Segmentation Challenge at MICCAI 2017. In the evaluation stage, the implementation of the algorithm was submitted to the challenge organizers, who then independently tested it on a hidden set of 110 cases from 5 scanners. Averaged dice score, precision and robust Hausdorff distance obtained on held-out test datasets were 80%, 84% and 6.30mm respectively. These were the highest achieved in the challenge, suggesting the proposed method is the state-of-the-art. In this paper, we provide detailed descriptions and quantitative analysis on key components of the system. Furthermore, a study of cross-scanner evaluation is presented to discuss how the combination of modalities and data augmentation affect the generalization capability of the system. The adaptability of the system to different scanners and protocols is also investigated. A quantitative study is further presented to test the effect of ensemble size. Additionally, software and models of our method are made publicly available. The effectiveness and generalization capability of the proposed system show its potential for real-world clinical practice.
1. Introduction
WMH are common in older adults and associated with neurological and geriatric disorders, but manual delineation is laborious and variable. The paper addresses this challenge through a benchmarked deep-learning system and examines its generalization across scanners and protocols.
- WMH commonly appear as bilateral, mostly symmetrical FLAIR hyperintensities in clinically healthy elderly people and are associated with neurological and geriatric disorders.
- Manual WMH delineation is reliable for assessing white-matter abnormalities but laborious, time-consuming, and subject to intra-rater and inter-rater variability.
- WMH segmentation is difficult to compare across studies because reported results for brain-lesion tasks are largely incomparable.
- WMH differ from several benchmarked lesion tasks through symmetry, scattered distribution, very small regions, and irregular shapes.
- The paper presents a winning MICCAI 2017 challenge entry based on an ensemble of convolution-deconvolution networks with long-range connections.
- The study evaluates public and hidden datasets while examining modality combinations, augmentation, scanner adaptability, and ensemble size.
2. Materials
The WMH Segmentation Challenge benchmarked automatic WMH segmentation across heterogeneous MR datasets using five evaluation metrics and an organizer-controlled ranking procedure.
- Datasets: The dataset contained FLAIR and T1 MR images with binary WMH masks acquired across five scanners and three hospitals.T1 images were resampled to match the FLAIR images before preprocessing.
- Datasets: The challenge compared automatic WMH segmentation methods using public training data and hidden test cases from multiple scanners.The dataset included 60 training subjects from three scanners and 110 hidden test cases from five scanners.
- Evaluation metrics: Five metrics evaluated different aspects of segmentation performance against a ground-truth map and an algorithm-generated map.The metrics included overlap, boundary distance, volume difference, lesion recall, and lesion-level F1-score.
- Evaluation metrics: The robust Hausdorff distance used the 95th percentile distance rather than the maximum distance between segmentation boundaries.This modification reduced sensitivity to the single largest boundary discrepancy.
- Evaluation metrics: The lesion-level F1-score counted correctly and wrongly detected lesions, with each lesion defined as a 3D connected component.The supplied evaluation description identifies NP as correctly detected lesions and NF as wrongly detected lesions.
- Ranking method: For each metric, organizer-calculated performances were normalized to rank scores from zero to one and averaged into an overall score.The best team received one and the worst team received zero for each metric.
3. Methods
The method combines preprocessing, augmentation, a two-channel 2D U-Net, Dice loss, and an ensemble testing pipeline for WMH segmentation.
- Preprocessing: Preprocessing crops or pads axial slices to 200×200 and normalizes scan intensities, while augmentation is applied only during training.The preprocessing targets uniform input size, reduced intensity variation, and robustness; most steps apply to FLAIR and T1.
- Data augmentation: Rotation, shear mapping, and scaling augment each axial slice to improve invariance and robustness across subjects and scanners.Training scaling used the range (0.9, 1.1), while some test sets had larger voxel-size ratios.
- 2-D convolutional network: The 2D U-Net receives FLAIR and T1 axial slices jointly, using shrinking and expansive parts to detect and locate WMH.The architecture contains repeated convolution, ReLU, and pooling operations, with 19 convolutional layers overall.
- Dice loss: Dice loss addresses the strong imbalance between positive and negative voxels in WMH segmentation.The formulation uses ground-truth and predicted probabilistic maps, with a stability term set to 1 in the experiments.
- Ensemble FCNs: The ensemble trains same-architecture U-Nets with different random initializations and shuffled batches, averages their probability maps, then thresholds the result.Post-processing restores the original image size and removes anatomically unreasonable detections.
4. Results
The system was evaluated on public data and hidden cases, including subject-held-out and cross-scanner settings. It won the challenge and showed strong segmentation performance, including on unseen scanners.
- Held-out test dataset: The submitted system won after independent organizer testing on 110 hidden cases from five scanners.The ranking averaged rank scores across five segmentation metrics.
- Held-out test dataset: The proposed ensemble achieved the highest average Dice similarity coefficient, smallest Hausdorff distance, and best lesion recall among the compared methods.For unseen AMS GE1.5T and AMS PETMR scanners, lesion recalls were 90% and 84%, respectively.
- Leave-one-subject-out evaluation: 87% averaged Dice similarity coefficient, 3.6mm Hausdorff distance, and 85% lesion recall were achieved across the 60 public cases in leave-one-subject-out evaluation.The results summarize overlap, localization accuracy, and overall lesion detection across the public training dataset.
- Cross-scanner evaluation: Cross-scanner evaluation trained on two scanners and tested on the third, repeating the procedure until each scanner served as the validation set.The figure distributions summarize performance for each testing scanner using five metrics.
- Cross-scanner evaluation: 74.5% Dice similarity coefficient and 87% lesion recall were achieved on two unseen scanners, with runner-up performance on the other three metrics.Performance across the three testing scanners was described as comparable, although GE3T was relatively poorer because of voxel-size differences.
5. Discussion
The discussion emphasizes the system’s cross-scanner behavior and the motivation for using a 2D architecture for WMH segmentation.
- Cross-scanner evaluation: Cross-scanner results were comparable overall, while GE3T performance was relatively poorer because its voxel size differed substantially from the other scanners.The comparison is presented for cases from each scanner tested in turn.
- 2D versus 3D architectures: 2D architectures are motivated by WMH characteristics and imaging constraints, including small discontinuous lesions, low contrast, poor z-direction resolution, and variable spatial resolution.These factors restrict the spatial and contextual information available for WMH segmentation.
5.2. Analysis of U-Net hyper parameters
Training stability depended on hyperparameter selection, while the study also examined modality complementarity and segmentation behavior across slices. Learning-rate and batch-size choices influenced optimization, and FLAIR–T1 combination improved several evaluation measures.
- Hyperparameter selection: Learning rates above 10^-3 caused training loss to collapse near zero early and stop updating, indicating unstable optimization.Batch size and learning rate influenced gradient magnitude and could lead to gradient exposure issues.
- Imaging modalities: FLAIR and T1 were treated as complementary modalities because FLAIR highlights white-matter pathology while T1 contrasts healthy brain tissues.The paper investigated whether their combination improved generalization to unseen scanners.
- Imaging modalities: Combining FLAIR and T1 slightly outperformed FLAIR alone on most metrics, with significant improvements in H95 and F1-score.Wilcoxon signed-rank tests on 60 cases produced p-values smaller than 1 ×10^-4 for H95 and F1-score.
5.4. Influence of data augmentation
Data augmentation was evaluated through cross-scanner experiments to test whether training diversity improves robustness. It slightly improved most metrics, significantly improved selected measures, and reduced some small-volume false positives.
- Evaluation setup: Data augmentation generated training samples with different distributions to teach invariance and robustness.The technique was evaluated using the cross-scanner experimental setting.
- Quantitative results: Data augmentation slightly improved segmentation on most metrics, with statistically significant gains in H95, Recall and F1-score.Wilcoxon signed-rank tests on 60 cases gave p-values smaller than 1 ×10^-4 for these improvements.
- Qualitative results: In a Utrecht case tested with a Singapore-and-GE3T-trained model, augmentation removed some small-volume false positives.The comparison was between segmentation without and with data augmentation.
- Qualitative results: The modality comparison was visualized with FLAIR images, ground truth, and segmentation outputs with and without T1.Green marks overlap, red marks false negatives, and black marks false positives in the segmentation columns.
5.5. Adaptability to different scanners
The study tested whether adding data from multiple scanners improves adaptation to unseen scanner domains. Multi-scanner training improved four metrics, while single-scanner training retained an advantage for lesion recall and small-lesion sensitivity.
- Evaluation design: The system’s adaptability was evaluated on imaging data from different scanners and protocols.Experiments compared models trained on a single scanner with models trained on multiple scanners.
- Evaluation design: The cross-scanner comparison used leave-one-subject-out evaluation and contrasted each scanner-specific model with a model trained on the other scanners’ data.For Utrecht, 20 subjects were split into training and test sets before comparison.
- Results: Models trained with data from three scanners significantly improved Dice, H95, average volume difference and lesion F1-score over single-scanner models.The comparison used box plots across datasets from Utrecht, Singapore and GE3T.
- Results: Single-scanner models achieved slightly better lesion recall because they detected fewer small lesions as undetected.The paper attributes this result to scanner- or sequence-specific sensitivity.
5.6. Effect of the size of ensembles
Ensemble size affected both segmentation performance and result variability. Ensembles of at least three models outperformed single models, while larger ensembles showed saturation and reduced variation.
- Performance: Ensembles with three or more models clearly outperformed single-model ensembles on all five evaluation metrics.The final challenge submission used an ensemble of three models.
- Performance: An ensemble of five models significantly improved over an ensemble of three on all five metrics.The reported improvements had small p-values.
- Performance: Increasing ensemble size further caused performance to saturate, with minor gains in some measures accompanied by small decreases in others.This indicates that larger ensembles did not uniformly improve every measure.
- Robustness: Larger ensembles reduced variation in the main evaluation metrics and produced more robust segmentation results.Combining models also avoided the worst result among models with different initializations and shuffled data.
- Analysis setup: The analysis plotted segmentation performance against the number of models in the ensemble.The horizontal axis represented ensemble size.
5.7. Statistical Analysis
The framework’s components were compared statistically on a validation split, with results indicating benefits from augmentation, preprocessing, ensemble modeling, and multimodal inputs. Increasing ensemble size also reduced performance variation.
- Ensemble size: Increasing ensemble size reduced the standard deviation of segmentation performance on the validation set.The figure reports reduced performance variation as ensemble size increased.
- Statistical evaluation: 48 training cases and 12 validation cases were used to compare frameworks with and without individual components across five segmentation metrics.The public 60-patient dataset was split 4:1 before component-wise evaluation.
- Component effects: Data augmentation significantly improved performance, while preprocessing produced statistically significant improvements on H95 and AVD.The reported p-values for augmentation were very small; preprocessing significance was observed specifically for H95 and AVD.
- Component effects: Ensembling improved all five metrics, with statistically significant gains for DSC, H95, and AVD.The comparison evaluated the ensemble against frameworks omitting the ensemble component.
5.8. Computational Complexity
The proposed two-channel FLAIR-and-T1 model was trained and tested on a single-GPU Linux server, with runtime reported for both training and inference.
- Model and hardware: 8,748,609 trainable parameters were used in the proposed model with two-channel FLAIR and T1 inputs.The model was trained on a single NVIDIA Titan-Xp GPU with 12GB RAM.
- Training cost: Training one model for 50 epochs on 10,000 200×200 images took around 180 minutes.Experiments ran on a GNU/Linux server using Ubuntu 16.04 and 32GB RAM.
6. Conclusions
The paper concludes that complementary modalities, scanner adaptability, and ensemble modeling supported a high-performing WMH segmentation system. It also identifies small-volume WMH detection as an unresolved challenge and notes a normalization concern related to disease burden.
- Conclusions: FLAIR and T1 provide complementary information for judging WMH, while the system showed adaptability across scanners and protocols.These factors were identified as important for building a generalizable WMH segmentation system.
- Conclusions: Ensemble modeling helped reduce over-fitting and boost segmentation results.The conclusion presents ensemble modeling as an important factor in the system’s performance and generalization capability.
- Limitations and future work: Small-volume WMH detection in MR images remains a challenging problem and is identified as a direction for future research.The authors connect this challenge to the system’s potential for real-world clinical use without claiming that it has been solved.
- Limitations and future work: Intensity normalization across scanners can be biased by patients’ varying disease burden, motivating robust measures or iterative lesion exclusion.The passage mentions robust mean, median absolute deviance, and excluding previously identified lesion areas during later normalization.