Source-linked AI summary
AssemblyNet: A large ensemble of CNNs for 3D Whole Brain MRI Segmentation
Pierrick Coupé, Boris Mansencal, Michaël Clément, Rémi Giraud, Baudouin Denis de Senneville, Vinh-Thong Ta, Vincent Lepetit, José V. Manjon
TL;DR
Whole brain segmentation is challenging because many anatomical labels must be learned from limited annotated MRI data. The paper proposes AssemblyNet, a large ensemble of localized U-Nets organized into two communicating assemblies with refinement and majority voting. Validation found competitive performance, strong scan-rescan consistency, reduced sensitivity to Alzheimer’s pathology relative to SLANT-27, and improved results from semi-supervised learning.
Problem
Whole brain segmentation must predict many anatomical labels from limited annotated MRI data, making manual labeling and large training datasets difficult to obtain.
Method
AssemblyNet uses 250 localized 3D U-Nets in two assemblies with neighboring knowledge transfer, multiscale refinement, atlas priors, and majority voting.
Results
AssemblyNet showed competitive state-of-the-art performance, 92.8% intra-method scan-rescan consistency, and lower pathology impact than SLANT-27.
Takeaways & Limitations
Semi-supervised learning with a few hundred well-balanced unlabeled images significantly improved AssemblyNet across unseen acquisition protocols, age periods, and pathology cases.
Takeaways & Limitations
Scan-rescan consistency results used only 4 subjects, and the Morphometric dataset lacked data for repeated segmentation by the same expert.
Abstract
from arXiv · showhide
Whole brain segmentation using deep learning (DL) is a very challenging task since the number of anatomical labels is very high compared to the number of available training images. To address this problem, previous DL methods proposed to use a single convolution neural network (CNN) or few independent CNNs. In this paper, we present a novel ensemble method based on a large number of CNNs processing different overlapping brain areas. Inspired by parliamentary decision-making systems, we propose a framework called AssemblyNet, made of two "assemblies" of U-Nets. Such a parliamentary system is capable of dealing with complex decisions, unseen problem and reaching a consensus quickly. AssemblyNet introduces sharing of knowledge among neighboring U-Nets, an "amendment" procedure made by the second assembly at higher-resolution to refine the decision taken by the first one, and a final decision obtained by majority voting. During our validation, AssemblyNet showed competitive performance compared to state-of-the-art methods such as U-Net, Joint label fusion and SLANT. Moreover, we investigated the scan-rescan consistency and the robustness to disease effects of our method. These experiences demonstrated the reliability of AssemblyNet. Finally, we showed the interest of using semi-supervised learning to improve the performance of our method.
1 Introduction
Whole brain segmentation is difficult because many anatomical labels must be learned from few annotated scans, while existing approaches trade computational cost against local or volumetric coverage. AssemblyNet addresses this setting by organizing a much larger ensemble of localized 3D U-Nets in a two-assembly framework.
- Motivation: Whole brain segmentation supports quantitative brain and pathology analysis but requires many anatomical labels despite limited training data.Manual whole-brain annotation is tedious and difficult, limiting the creation of large labeled datasets.
- Existing approaches: Multi-atlas methods handle limited training data but require multiple nonlinear registrations that can be computationally expensive.Their pipeline registers labeled templates to the target image and fuses the transformed segmentations.
- Existing approaches: Patch-based methods were introduced to reduce multi-atlas computation and better capture local anatomy, with joint label fusion remaining a major state-of-the-art reference.Patch-based methods use nonlocal patch estimators or machine-learning alternatives for label fusion.
- Deep learning: Early deep-learning methods used patchwise or 2D processing, while later 3D methods used reduced volumes or overlapping U-Net tiles such as SLANT.GPU-memory limits motivated these localized or reduced-volume strategies.
- Proposed direction: AssemblyNet extends tiled 3D segmentation to 250 simpler U-Nets while aiming to keep processing time similar, using two communicating assemblies.The study also evaluates semi-supervised learning, scan-rescan reliability, and robustness to disease effects.
2 Materials and Methods
AssemblyNet combines localized overlapping U-Nets, atlas priors, neighboring-network transfer learning, and a multiscale two-assembly cascade. The study evaluates this framework with manually labeled MRI datasets, preprocessing, augmentation, training, and semi-supervised learning using unlabeled data.
- Method overview: Each AssemblyNet member processes an overlapping brain sub-volume, shares initialization knowledge with its nearest neighbor, and contributes to a global majority vote.This produces an overcomplete aggregation of local predictions for whole-brain segmentation.
- Method overview: A nonlinearly registered atlas supplies prior knowledge about the expected final segmentation.The atlas prior is treated as prior decision information for the assemblies.
- Method overview: The first assembly generates a coarse 2×2×2 mm3 decision, which the second assembly analyzes at 1×1×1 mm3 for refinement.The multiscale cascade models communication between assemblies through an amendment-like refinement step.
- Method overview: The framework contributes atlas registration, nearest-neighbor transfer learning, multiscale iterative refinement, and student-teacher semi-supervised learning.These are the paper’s four stated methodological contributions.
- Datasets: Training used 45 manually labeled OASIS T1w MRI with 132 anatomical labels, while testing used 19 manually labeled images from OASIS, Colin27, and CANDI.The testing data span adult, child, and single-subject cohorts with different age ranges.
- Preprocessing: Preprocessing included denoising, inhomogeneity correction, affine MNI registration, intensity normalization, and brain extraction before within-mask normalization.Atlas priors were obtained through nonlinear registration of a MICCAI 2012 multi-atlas labeling resource.
- Semi-supervised learning: Teacher-student semi-supervised learning used an AssemblyNet trained on 45 labeled images to pseudo-label 360 unlabeled lifespan images for subsequent training.The approach targets settings where expert manual segmentations are limited.
- Training: Training used sagittal flipping, MixUp augmentation, reduced-filter 3D U-Nets, and temporal averaging of model weights to improve generalization.The supplied training description also specifies moving-average optimization after the main training epochs.
3 Results
AssemblyNet improved segmentation accuracy over its baselines and state-of-the-art alternatives, including under limited training data, unseen acquisition settings, scan-rescan testing, and disease-related evaluation. Semi-supervised learning further improved performance, while the method remained computationally practical compared with a large-library SLANT approach.
- 3.1 AssemblyNet performance: 5.9 pp higher mean Dice than baseline Assembly at 2×2×2 mm3, with significantly better Dice than all considered alternatives using 45 training cases.Atlas priors, transfer learning, and the multiscale cascade contributed additional gains of 0.3, 0.5, and 1.1 pp, respectively.
- 3.2 Impact of semi-supervised learning: 0.6 pp mean Dice improvement came from teacher-student semi-supervised learning, followed by a further 0.3 pp gain from fine-tuning.A second fine-tuning iteration reached 74.0% mean Dice but produced only marginal, nonsignificant improvement, so the first student generation was retained.
- 3.3 Comparison with state-of-the-art methods: 16.3 pp, 9.9 pp, and 7.2 pp mean Dice gains over U-Net, JLF, and SLANT-27, respectively, were obtained when comparing methods trained with 45 images.AssemblyNet also had training and testing times comparable to SLANT-based methods; SLANT-27 trained on 5111+45 images required a 21-CPU-year library extension.
- 3.3 Comparison with state-of-the-art methods: 19.7 pp, 19.2 pp, and 9 pp mean Dice gains over U-Net, JLF, and SLANT-27 were observed on child scans acquired with a different protocol.Performance dropped dramatically for most methods on CANDI and Colin27, whereas AssemblyNet retained the best segmentation accuracy on both datasets.
- 3.4 Scan-rescan consistency: More than 90% scan-rescan Dice consistency was achieved by automatic methods, compared with 76.8% for the expert; AssemblyNet was more consistent than SLANT-27, especially with SSL.AssemblyNet with SSL also had significantly higher Expert-Method consistency than SLANT-27, while automatic methods were not significantly below intra-expert consistency.
- 3.5 Robustness to disease effects: 73.6% global Dice with SSL exceeded 73.1% without SSL and 71.6% for SLANT-27 trained on 5111+45 images, with significant improvement over both alternatives.For AD scans, the SSL improvement over SLANT-27 was 2.2 pp, compared with 1.7 pp for CN scans; differences between CN and AD groups were not significant for any method.
4 Discussion
AssemblyNet achieved competitive segmentation performance, improved results with semi-supervised learning, and showed strong consistency and pathology robustness, while validation remained limited in some settings.
- 250 U-Nets, atlas prior, nearest-neighbor transfer learning, and multiscale assemblies improved global segmentation accuracy.
- Semi-supervised learning with a few hundred well-balanced unlabeled images significantly improved results across unseen acquisition protocols, age periods, and pathologies.The teacher-student approach was more practical in computational time and resources than methods using larger classically labeled auxiliary datasets.
- AssemblyNet showed high segmentation accuracy and computational efficiency, with larger ensembles of simpler CNNs outperforming a single U-Net within a multiscale framework.Testing could be further accelerated through multiple GPUs because the method is fully parallelizable.
- Accuracy on the CANDI dataset remained limited, potentially reflecting lower image quality and the distance between adult training data and child images.The authors identify these factors for deeper investigation in future whole brain segmentation methods.
- 92.8% intra-method scan-rescan consistency exceeded the 76.8% intra-rater consistency reported for human experts.Expert-method consistency was not significantly lower than intra-rater consistency, although only four scan-rescan subjects were evaluated.
- Pathology caused a small but non-significant Dice decrease in Alzheimer’s disease relative to cognitively normal subjects, and AssemblyNet was less affected than SLANT-27.The authors describe this as a first step toward validation on additional pathologies.
5 Conclusion
The paper introduces AssemblyNet, a large ensemble of 3D CNNs organized to segment the whole brain accurately and efficiently despite limited training data. Validation showed competitive performance, while semi-supervised learning improved performance on unseen acquisition protocols, age periods, and pathologies.
- AssemblyNet organizes 250 3D U-Nets into two assemblies for accurate and efficient whole brain segmentation.The framework is designed to handle limited training data within practical training and testing times.
- Semi-supervised learning improved AssemblyNet performance on unseen acquisition protocols, age periods, and pathologies.