Source-linked AI summary
Standardized Assessment of Automatic Segmentation of White Matter Hyperintensities and Results of the WMH Segmentation Challenge
Hugo J. Kuijf, J. Matthijs Biesbroek, Jeroen de Bresser, Rutger Heinen, Simon Andermatt, Mariana Bento, Matt Berseth, Mikhail Belyaev, M. Jorge Cardoso, Adrià Casamitjana, D. Louis Collins, Mahsa Dadar, Achilleas Georgiou, Mohsen Ghafoorian, Dakai Jin, April Khademi, Jesse Knight, Hongwei Li, Xavier Lladó, Miguel Luna, Qaiser Mahmood, Richard McKinley, Alireza Mehrtash, Sébastien Ourselin, Bo-yong Park, Hyunjin Park, Sang Hyun Park, Simon Pezold, Elodie Puybareau, Leticia Rittner, Carole H. Sudre, Sergi Valverde, Verónica Vilaplana, Roland Wiest, Yongchao Xu, Ziyue Xu, Guodong Zeng, Jianguo Zhang, Guoyan Zheng, Christopher Chen, Wiesje van der Flier, Frederik Barkhof, Max A. Viergever, Geert Jan Biessels
TL;DR
WMH quantification is important but manual segmentation is laborious, and automatic methods lack standardized comparison. The paper organizes a multi-center, multi-scanner challenge with independent evaluation and metrics spanning overlap, distance, volume, and lesion detection. Four methods rank significantly above the others with one clear winner, while inter-scanner rankings show that generalization to unseen scanners varies.
Problem
WMH quantification supports neurological research, but manual segmentation is laborious and existing automatic methods lack standardized comparison.
Method
The paper evaluates containerized automatic WMH segmentation methods on standardized multi-center data using independent test-set assessment and multiple metrics, including inter-scanner robustness.
Results
Four methods rank significantly better than the others, with one clear winner, while inter-scanner rankings show that methods differ in generalization to unseen scanners.
Takeaways & Limitations
The challenge provides a public platform for standardized evaluation of automatic WMH segmentation methods.
Abstract
from arXiv · showhide
Quantification of cerebral white matter hyperintensities (WMH) of presumed vascular origin is of key importance in many neurological research studies. Currently, measurements are often still obtained from manual segmentations on brain MR images, which is a laborious procedure. Automatic WMH segmentation methods exist, but a standardized comparison of the performance of such methods is lacking. We organized a scientific challenge, in which developers could evaluate their method on a standardized multi-center/-scanner image dataset, giving an objective comparison: the WMH Segmentation Challenge (https://wmh.isi.uu.nl/). Sixty T1+FLAIR images from three MR scanners were released with manual WMH segmentations for training. A test set of 110 images from five MR scanners was used for evaluation. Segmentation methods had to be containerized and submitted to the challenge organizers. Five evaluation metrics were used to rank the methods: (1) Dice similarity coefficient, (2) modified Hausdorff distance (95th percentile), (3) absolute log-transformed volume difference, (4) sensitivity for detecting individual lesions, and (5) F1-score for individual lesions. Additionally, methods were ranked on their inter-scanner robustness. Twenty participants submitted their method for evaluation. This paper provides a detailed analysis of the results. In brief, there is a cluster of four methods that rank significantly better than the other methods, with one clear winner. The inter-scanner robustness ranking shows that not all methods generalize to unseen scanners. The challenge remains open for future submissions and provides a public platform for method evaluation.
I. INTRODUCTION
WMH quantification matters for neurological research but manual delineation is laborious and observer-dependent, while automatic methods lack standardized comparison. The challenge addresses this gap by evaluating competing methods on shared data, including unseen scanners for generalizability.
- WMH volume, shape, and location measures support clinical research and are associated with cognitive impairment and gait disturbances.
- Manual WMH delineation is time-consuming and observer-dependent.
- Automatic methods are difficult to compare because studies use different ground truths, subject cohorts, experts, protocols, and evaluation criteria.
- Deployment across institutes is challenging because scanners and imaging protocols differ, often requiring transfer learning or fine-tuning.
- The WMH Segmentation Challenge standardizes assessment through independent evaluation on a common test set containing data from scanners absent from training.
- The paper presents the challenge organization, its results, and a detailed evaluation.
A. Training and test data
The challenge used 60 training images and 110 test images drawn from five scanners, three vendors, and three institutes. Test data included images from scanners not represented in training.
- 60 training and 110 test images were acquired from five scanners, three vendors, and three institutes.
- The dataset provided 3D T1-weighted and 2D multi-slice FLAIR images for each subject.
- The training set contained 60 images, with 20 3 T images from one scanner at each institute.
- The test set contained 90 images from the training scanners and 20 images from scanners absent from training.
- Subjects came from memory-clinic cohorts in the Netherlands and Singapore.
- Subjects were randomly selected and assigned to training or test sets separately for each scanner.
1) MRI parameters:
MRI data included T1-weighted and FLAIR acquisitions across scanners and field strengths. Figure 1 illustrates WMH appearance on T1 and FLAIR images alongside the manual segmentation.
- All 3D sequences were acquired sagittally, whereas all 2D multi-slice sequences were acquired transversally.
- The dataset included T1-weighted and FLAIR sequences from Philips, Siemens, and GE scanners.
- Scanner configurations covered 1.5 T and 3 T acquisitions across multiple institutions and vendors.
- In Figure 1, WMH appear hypointense on T1-weighted images and hyperintense on FLAIR images.
- Figure 1(c) shows the corresponding manual WMH segmentation for the example subject.
2) Data pre-processing:
Pre-processing corrected intensity bias, registered T1-weighted images to resampled FLAIR images, and removed facial information. The challenge website provided data before and after preprocessing to registered participants.
- All images were bias-corrected using SPM12.
- Elastix registration aligned the 3D T1-weighted images with the resampled FLAIR images.
- Subject faces were manually removed from all sequences, with the corresponding masks provided.
- Preprocessed and unprocessed data were available through the challenge website for registered participants.
3) Manual reference standard:
The manual reference standard was created through expert WMH and pathology delineation, peer review, consensus correction, and binary-mask conversion. Additional observers provided inter-observer agreement measures.
- Manual delineation: WMH and other pathologies were manually segmented according to STRIVE criteria by an experienced observer and peer-reviewed by a second expert.The primary observer had segmented more than 1000 cases; the reviewer had eleven years of quantitative neuroimaging experience.
- Reference standard: The reference standard was the primary observer’s corrected segmentation after peer review and consensus discussion.Corrections addressed errors and delineations inconsistent with STRIVE criteria.
- Mask conversion: Manual contours were converted into binary masks using a >50% voxel-volume inclusion rule, with labels for background, WMH, and other pathology.Masks were then dilated in-plane by 1 pixel, with WMH taking precedence where labels overlapped.
- Agreement: Two additional observers segmented the 60 training images to measure inter-observer agreement.One observer lacked extensive prior experience, whereas the other had prior experience with WMH segmentation.
B. Set-up of the challenge
Challenge methods were submitted as Docker containers and evaluated subject by subject under scanner-blinded conditions. Containerization supported reproducible execution across platforms, with GPU hardware available when needed.
- Submission and reproducibility: Participants submitted Docker-containerized methods for organizer-run evaluation.Containerization was intended to ease deployment and produce identical outputs on different platforms.
- Testing procedure: During testing, each container processed test subjects individually without scanner identifiers, and was destroyed and reloaded after every subject.This setup prevented methods from using explicit scanner identity during evaluation.
- Computing resources: An NVIDIA Titan Xp GPU was available for methods requiring GPU computation.
C. Participants
Twenty teams submitted diverse automatic WMH segmentation methods, including deep neural networks, random-forest classifiers, and probabilistic models. The paper summarizes representative architectural and feature-engineering choices, with detailed descriptions available online.
- Participation: Twenty teams submitted methods before the challenge deadline.The paper provides a brief alphabetical summary of each submitted method.
- Neural methods: Submitted approaches included neural networks using atrous convolutions, spatial pyramid pooling, residual connections, patches, and augmentation.The described achilles method trained only on FLAIR images.
- Hybrid segmentation: One approach combined FLAIR-based U-Net WMH segmentation with T1-based white-matter segmentation to remove false positives.The U-Net was trimmed to retain three pooling layers.
- Classical and probabilistic methods: Other methods used random forests, texture features, Gaussian mixture models, iterative refinement, and location-based false-positive pruning.The summarized methods included nist, text class, and tig.
- Method documentation: Detailed information on each method was made available through the challenge results website.
D. Evaluation and Ranking
Methods were ranked using five complementary segmentation and lesion-detection metrics, with bootstrap confidence intervals and a separate inter-scanner robustness analysis. The evaluation also examined lesion size and a fused STAPLE segmentation.
- Evaluation metrics: Five criteria were used: Dice similarity coefficient, modified Hausdorff distance at the 95th percentile, lAVD, individual-lesion recall, and individual-lesion F1-score.Individual lesions were defined as 3D connected components within an image.
- Metric revision: lAVD replaced AVD because AVD bounded undersegmentation at 100% but allowed nearly unlimited oversegmentation.The replacement was the absolute log-transformed volume difference.
- Overall ranking: Overall ranking averaged relative ranks across the five metrics after each metric was averaged over all test scans.The best method received rank 0 and the worst rank 1 for each metric.
- Statistical uncertainty: 95% confidence intervals were estimated by bootstrapping 2,000 test-set samples with replacement; non-overlap indicated significance at α = 0.05.
- Lesion-size analysis: Recall was computed separately for lesions at or below versus above the subject-specific median lesion size.
- Inter-scanner robustness: A separate ranking measured robustness from inter-scanner performance differences using scanner-specific median metrics and their standard deviations.
- Segmentation fusion: STAPLE fused segmentations from all methods and top-ranking methods, then evaluated the combined segmentation separately.The paper notes that fusion had outperformed individual methods in other applications.
III. RESULTS
The challenge results show strong performance differences across segmentation metrics, lesion sizes, and scanner robustness. Four methods ranked significantly above the others, while ensemble STAPLE results improved several metrics further.
- The dataset had a mean WMH volume of 16.9±21.6 ml and a mean WMH count of 62 ± 35 lesions.WMH volume ranged from 0.78 ml to 195.15 ml, and lesion count ranged from 12 to 194 lesions.
- sysu media performed best on DSC, H95, and recall; cian on lAVD; and nlp logix on F1.These were the best-performing methods on the individual metrics, with 95% confidence intervals reported.
- All methods recalled small lesions worse than large lesions, with relative recall drops ranging from −20% to −87%.sysu media recalled 94% of large lesions versus 76% of small lesions, a relative difference of −20%.
- ipmi-bern achieved the highest inter-scanner robustness rank, followed by sysu media; achilles and knight entered the top four.Lower standard deviation of median performance across scanners indicated better inter-scanner performance.
- STAPLE applied to the top four methods ranked second overall and achieved the best DSC and lAVD, while STAPLE applied to all methods ranked fourth and achieved the best H95.Both STAPLE variants outperformed all other methods when inter-scanner robustness was recomputed.
- Training-to-test performance showed excellent correlation, with R-squared: 0.94 and p < 0.001, providing no indication of training-set overfitting.The analysis compared each method's performance on the training and test data.
IV. DISCUSSION
The challenge found strong overall automatic WMH segmentation performance, but rankings varied across metrics and unseen scanners. Small-lesion recall and inter-scanner robustness remain key areas for improvement.
- Overall performance: The top-ranking methods achieved performance similar or superior to individual observers, suggesting automatic segmentation may replace individual observers.The reference standard was created by two experienced observers, while two additional observers provided inter-observer comparisons.
- Overall performance: sysu media ranked significantly higher than all other participants, while cian, nlp logix, and nic-vicorob formed the next significantly higher group.The top methods also included ensemble approaches, including sysu media and nlp logix.
- Inter-scanner robustness: STAPLE combinations achieved the best inter-scanner robustness, although sysu media remained the overall winner because of stronger recall, especially for small lesions.Both STAPLE variants performed less well in recalling lesions below the median size.
- Method characteristics: Dropout and ensemble strategies characterized several high-ranking methods, but dropout-trained deep models showed lower inter-scanner ranks and may generalize poorly to unseen scanners.This pattern suggests better performance on unseen data from known scanners than on data from unseen scanners.
- Remaining challenges: Small WMH were frequently missed, particularly in regions with fewer training examples, reducing recall and F1 more than DSC, H95, or lAVD.Potential responses included dedicated small-lesion reclassification, selective sampling, augmentation, and less location-sensitive decisions.
- Inter-scanner robustness: The inter-scanner ranking changed substantially: ipmi-bern became first, while nic-vicorob and lrde dropped on the unseen Philips scanner.Only 10 of 110 test images came from that scanner, limiting its effect on the overall ranking.
APPENDIX C SUMMARIES OF RESULTS
Appendix C presents participant-by-participant results across five scanners and five evaluation criteria, ordered by the final ranking. The figures also report false-negative and false-positive rates, with several boxplots containing out-of-range values.
- Evaluation criteria: Five criteria are reported for each method: DSC, H95, lAVD, recall, and F1 for individual lesions.Recall and F1 treat individual lesions as 3D connected components.
- Figure organization: The detailed participant figures are presented in final-ranking order and show performance across the five scanners.
- Per-method results: Each participant figure includes false-negative and false-positive rate columns alongside the five evaluation criteria.
- Out-of-range values: Out-of-range values also occur in k2, misp, nih cidi, skkumedneuro, knight, upc dlmi, nist, neuro.ml, and hadi results.For hadi, some output was empty and H95 and lAVD were not evaluated.
- Out-of-range values: Several H95 and lAVD boxplots contain values outside the plotted range, including maxima of 152.66 mm for H95 and 5.60 for lAVD.The 152.66 mm H95 maximum occurs for text class; the 5.60 lAVD maximum occurs for lrde.