PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluation of Algorithms for Multi-Modality Whole Heart Segmentation: An Open-Access Grand Challenge

Xiahai Zhuang, Lei Li, Christian Payer, Darko Stern, Martin Urschler, Mattias P. Heinrich, Julien Oster, Chunliang Wang, Orjan Smedby, Cheng Bian, Xin Yang, Pheng-Ann Heng, Aliasghar Mortazi, Ulas Bagci, Guanyu Yang, Chenchen Sun, Gaetan Galisot, Jean-Yves Ramel, Thierry Brouard, Qianqian Tong, Weixin Si, Xiangyun Liao, Guodong Zeng, Zenglin Shi, Guoyan Zheng, Chengjia Wang, Tom MacGillivray, David Newby, Kawal Rhode, Sebastien Ourselin, Raad Mohiaddin, Jennifer Keegan, David Firmin, Guang Yang

arXiv:1902.07880v1cs.CV

TL;DR

Whole-heart segmentation is clinically useful but difficult to automate and compare consistently across modalities. The MM-WHS challenge benchmarks submitted algorithms on shared clinical CT and MRI data using blinded evaluation. Deep-learning methods can achieve high accuracy, while multi-atlas methods are more stable; MRI remains harder and some deep-learning results vary substantially.

  • Problem

    Whole-heart segmentation is difficult because heart shape varies, clinical image quality differs, and prior comparisons used differing datasets and evaluation metrics.

  • Method

    The MM-WHS challenge provides 120 manually delineated clinical images, common training and test data, and blinded evaluation of submitted CT and MRI algorithms.

  • Results

    Across submissions, CT segmentation was more successful than MRI; deep-learning methods could achieve high accuracy but varied widely, whereas multi-atlas methods were more robust and stable.

  • Takeaways & Limitations

    The shared dataset and blinded evaluation establish an ongoing benchmark for comparing whole-heart segmentation methods across CT and MRI.

  • Takeaways & Limitations

    MRI whole-heart segmentation remains harder because of low image quality, inconsistent appearance, and large heart-shape variation, while whole-heart training data are difficult to obtain.

Abstract

from arXiv · show

Knowledge of whole heart anatomy is a prerequisite for many clinical applications. Whole heart segmentation (WHS), which delineates substructures of the heart, can be very valuable for modeling and analysis of the anatomy and functions of the heart. However, automating this segmentation can be arduous due to the large variation of the heart shape, and different image qualities of the clinical data. To achieve this goal, a set of training data is generally needed for constructing priors or for training. In addition, it is difficult to perform comparisons between different methods, largely due to differences in the datasets and evaluation metrics used. This manuscript presents the methodologies and evaluation results for the WHS algorithms selected from the submissions to the Multi-Modality Whole Heart Segmentation (MM-WHS) challenge, in conjunction with MICCAI 2017. The challenge provides 120 three-dimensional cardiac images covering the whole heart, including 60 CT and 60 MRI volumes, all acquired in clinical environments with manual delineation. Ten algorithms for CT data and eleven algorithms for MRI data, submitted from twelve groups, have been evaluated. The results show that many of the deep learning (DL) based methods achieved high accuracy, even though the number of training datasets was limited. A number of them also reported poor results in the blinded evaluation, probably due to overfitting in their training. The conventional algorithms, mainly based on multi-atlas segmentation, demonstrated robust and stable performance, even though the accuracy is not as good as the best DL method in CT segmentation. The challenge, including the provision of the annotated training data and the blinded evaluation for submitted algorithms on the test data, continues as an ongoing benchmarking resource via its homepage (\url{www.sdspeople.fudan.edu.cn/zhuangxiahai/0/mmwhs/}).

1. Introduction

Whole heart segmentation extracts clinically important cardiac substructures, but automation and cross-method comparison remain difficult because anatomy, image quality, datasets, and metrics vary.

  • Clinical motivation: WHS delineates the ventricles, atria, LV myocardium, aorta, and pulmonary artery for morphological analysis and clinical applications.Its outputs can support functional indices such as ejection fraction and surgical guidance, including radio-frequency ablation planning.
  • Clinical motivation: Manual whole-heart delineation is labor-intensive and tedious, motivating automated methods.
  • Existing methods: Prior WHS methods span boundary-driven, region-based, graph-cuts, and model-fitting techniques.
  • Existing methods: Multi-atlas segmentation propagates and fuses labels from annotated atlases, but registration and fusion are generally computationally expensive.
  • Benchmarking gap: Recent challenges and datasets have often focused on specific cardiac substructures rather than the whole heart across modalities.
  • Benchmarking gap: The MM-WHS competition provides a shared benchmark for developing, validating, and comparing whole-heart algorithms across CT and MRI.

2. Materials and setup

MM-WHS uses 120 clinical whole-heart images with seven labeled structures, manual expert-reviewed reference segmentations, standardized metrics, and separate training and test sets.

  • Data: The dataset contains 60 cardiac CT and 60 cardiac MRI images collected from multiple clinical sites, with variable image quality.
  • Labels: The WHS target comprises seven structures: four blood-pool cavities, LV myocardium and epicardium, the ascending aorta, and the pulmonary artery.
  • Reference standard: Manual segmentations were produced slice-by-slice and reviewed by senior cardiac-imaging researchers, with revisions when necessary.
  • Evaluation: Accuracy was evaluated using normalized generalized Dice, Jaccard, surface-to-surface distance, and Hausdorff Distance metrics.
  • Evaluation: Each modality was split into 20 training images and 40 test images, with images and gold standards released only for training.
  • Participants: Twelve teams were selected, while three methods marked with an asterisk were submitted after the challenge deadline for performance ranking.

3. Evaluated Methods

The benchmark covers twelve submitted WHS algorithms, including deep-learning and multi-atlas approaches, with UOL’s method exemplifying registration, label fusion, and probability-map regularization.

  • Benchmark scope: The paper introduces twelve benchmarked algorithms and summarizes their approaches for comparative evaluation.
  • Deep-learning methods: Payer et al. use separate localization and segmentation CNNs to identify the heart and segment fine structures within a small ROI.
  • Multi-atlas methods: Heinrich and Oster propose multi-atlas MRI segmentation using discrete registration and contrast-invariant features to handle shape and intensity variation.
  • Multi-atlas methods: The multi-atlas pipeline registers training atlases to the test image, fuses warped labels, and smooths probability maps with a multi-label random walk.

3.3. KTH Royal Institute of Technology (KTH)

The KTH method combines CNN-based segmentation with statistical shape information and multiple architectural enhancements for whole-heart CT segmentation.

  • Shape-informed CNN: Wang and Smedby combine CNN predictions with statistical shape priors to provide explicit three-dimensional shape information.
  • Shape-informed CNN: A random-forest landmark detector identifies the region of interest, while shape models are learned from 20 annotated CT training images.
  • 3D FCN enhancements: Yang et al. use a 3D fully convolutional network initialized with knowledge transferred from the Sports-1M video dataset.
  • 3D FCN enhancements: The 3D FCN shortens the back-propagation path and adds auxiliary losses to improve gradient flow and training.
  • Architecture: The schematic’s digits indicate feature-volume counts in each layer, and the dotted volume denotes concatenation.

3.5. University of Central Florida (UCF)

The UCF methods use multi-view CNNs, connected-component analysis, and a deeply supervised 3D FCN for whole-heart segmentation.

  • Multi-object multi-planar CNN: MO-MP-CNN trains encoder-decoder CNNs from axial, sagittal, and coronal views, then adaptively fuses their outputs.The method applies connected-component analysis to estimate reliable and unreliable regions in the final segmentation.
  • Connected-component analysis: Connected-component analysis compares reliable and unreliable regions to guide segmentation reliability.A larger difference between the regions indicates greater reliability.
  • Deeply supervised 3D FCN: The 3D FCN performs end-to-end dense labeling with auxiliary losses and deep supervision to address gradient vanishing and class imbalance.It also transfers spatial-temporal knowledge from a large-scale video dataset to initialize shallow down-sampling layers.
  • Evaluation: Table 4 and Table 5 report quantitative results for ten CT algorithms and eleven MRI algorithms, respectively.The tables distinguish deep learning-based methods from multi-atlas segmentation methods and identify late submissions with an asterisk.

3.7. Southeast University (SEU)

The SEU approaches include atlas-based CT segmentation and an incremental interactive method built from probabilistic atlases and topological graphs.

  • Atlas-based CT segmentation: The SEU CT method detects and normalizes a heart ROI before affine and non-rigid registration of atlas images.Atlas ranking is also applied, although the supplied passage truncates the description of that step.
  • Probabilistic atlas method: The incremental interactive method constructs local probabilistic atlases for each heart substructure using training images.These atlases support incremental extraction of different regions of interest.
  • Topological graph: A topological graph stores shape and intensity features at nodes and spatial relationships between anatomical structures on edges.Separate graphs can be constructed for CT and MRI in multi-modality data.

3.9. Shenzhen Institutes of Advanced Technology (SIAT)

The SIAT approaches use a deeply supervised 3D U-Net for multi-modality whole-heart segmentation and a Bayesian dilated residual network for MRI volumes.

  • Deeply supervised 3D U-Net: The SIAT 3D U-Net uses independently augmented heart-substructure ROIs and first produces a coarse whole-heart segmentation to reduce false positives.The detected ROI is subsequently refined through pixel-wise classification.
  • Multi-modality processing: Cardiac CT and MRI images are normalized and fused so the network can exploit multi-modality information during feature extraction.Both image size and intensity range are normalized before training.
  • Evaluation: Figure 4 presents CT whole-heart Dice-score boxplots for the ten evaluated methods.The figure supports comparison of the methods’ CT segmentation score distributions.
  • Bayesian VoxDRN: Bayesian VoxDRN uses pixel-wise dilated residual networks to segment whole-heart structures from 3D MRI images.After training, it can generate semantic segmentations for arbitrarily sized data volumes.

3.11. University of Bern, Method No. 2 (UB2*)

UB2* combines multi-scale fully convolutional Dense-Nets with U-Net-style encoding and decoding, while the evaluation includes MRI Dice-score distributions across eleven methods.

  • Network architecture: The network uses an encoder-decoder architecture, with the encoder learning features and the decoder generating segmentation results.Skip connections recover spatial context lost during down-sampling, while multi-scale context further supports feature learning.
  • Two-stage concatenated U-Net: The two-stage concatenated U-Net simultaneously detects the heart ROI and classifies pixels into substructures without losing original resolution.A coarse prediction from a down-sampled volume is resampled before the second U-Net refines the labels.
  • MSVoxFCDN: MSVoxFCDN directly maps an entire 3D MRI volume to volume-wise whole-heart labels after training.The method uses multi-scale context and multi-scale deep supervision to enhance feature learning.
  • Evaluation: Figure 5 shows MRI whole-heart Dice-score boxplots for the eleven evaluated methods.The boxplots provide a distribution-based comparison of MRI segmentation performance.

4. Results

The evaluated methods achieved stronger quantitative performance on CT than MRI overall, but performance varied substantially across algorithms and cases. Deep learning methods often led accuracy metrics, while multi-atlas methods showed advantages in some metrics and more realistic shapes.

  • 0.91±0.09 was the best CT Dice score, achieved by GUT, a deep-learning algorithm with anatomical label configurations.
  • Deep-learning methods generally outperformed multi-atlas approaches on CT for Jaccard, Dice, and SD, but not for HD.The mean HD of the two multi-atlas methods was not worse than that of the eight deep-learning approaches.
  • 0.87 ± 0.04 was the best MRI Dice score from UB2∗, whereas UOL achieved the best MRI HD at 28.535 ± 13.220 mm.Across all evaluation metrics, the two multi-atlas teams averaged better accuracy than the nine deep-learning teams.
  • The top four MRI deep-learning methods obtained accuracy comparable to UOL, while other deep-learning approaches produced much poorer results.
  • CT segmentation was generally better than MRI segmentation, and CT methods achieved reasonable pulmonary-artery delineation only in selected cases.For GUT, KTH, CUHK1, UB1∗, and CUHK2, the pulmonary-artery mean Dice score exceeded 0.8.
  • Worst CT cases included incomplete right ventricles, myocardial misclassification, atrium–ventricle confusion, single-structure segmentation, and incorrect orientation.
  • MRI median cases were often poor, and several worst cases missed substructures or contained no complete whole-heart substructure.The visual findings were consistent with the quantitative results showing better CT than MRI segmentation.

5. Discussion

The benchmark found higher mean WHS accuracy for DL methods, while MAS methods produced more realistic shapes and more stable results. CT segmentation generally outperformed MRI, with the pulmonary artery and other non-chamber structures remaining difficult.

  • 0.872 ± 0.087 (CT) versus 0.824 ± 0.102 (MRI) in mean Dice scores, confirming higher benchmark accuracy for CT.
  • MAS versus DL-based segmentation: 0.875 ± 0.083 (CT) versus 0.859 ± 0.108 (CT) was reported for DL-based and MAS-based methods, respectively.
  • MAS versus DL-based segmentation: DL methods achieved better mean accuracies, whereas MAS approaches tended to generate more realistic heart shapes.
  • Overall performance: Four-chamber segmentation was generally good, but pulmonary artery segmentation in CT and myocardium, aorta, and pulmonary artery segmentation in MRI were more challenging.
  • MAS versus DL-based segmentation: 9 of 11 CT methods and 8 of 10 MRI methods used deep neural networks, but some DL methods produced large variability and unrealistic heart shapes.
  • Progress and challenges: The challenge offers an open-access dataset and ongoing evaluation framework for developing and comparing WHS algorithms.

6. Conclusion

The conclusion frames automated multi-modality WHS as clinically needed but difficult because of image quality, cardiac complexity, and anatomical variation. The MM-WHS challenge standardizes evaluation across submitted methods and shows distinct strengths and remaining challenges across modalities and approaches.

  • Manual WHS is labor-intensive and poorly reproducible, motivating demand for fully automated multi-modality WHS.
  • The challenge evaluates twelve methods on 120 clinical CT and MRI images using shared training and test data with blinded comparison.
  • CT WHS was more successful than MRI WHS, while the four chambers were generally easier to segment than the great vessels.
  • DL methods could achieve high accuracy but also produce unrealistic shapes, whereas conventional atlas-based approaches generally performed stably.
  • Hybrid methods combining deep learning with atlas or shape information were identified as promising for future WHS development.

Authors contributions

The authors’ contributions covered challenge organization, data provision, manual segmentations, evaluation, and manuscript preparation. The supplied contribution passage explicitly identifies these roles and contributors.

  • XZ initiated the challenge and provided 60 CT images, 41 MRI images, and manual segmentations for all 120 images.
  • GY, RM, JK, and DF provided the other 19 MRI images.
  • XZ, GY, and LL organized the challenge, while LL evaluated all submitted segmentation results.
  • GY generated the first draft, and XZ and LL restructured and rewrote the manuscript.
Loading 1902.07880v1…