Source-linked AI summary
Ensembles of Multiple Models and Architectures for Robust Brain Tumour Segmentation
Konstantinos Kamnitsas, Wenjia Bai, Enzo Ferrante, Steven McDonagh, Matthew Sinclair, Nick Pawlowski, Martin Rajchl, Matthew Lee, Bernhard Kainz, Daniel Rueckert, Ben Glocker
TL;DR
Brain-tumour segmentation models differ in performance and behaviour because of architectural and training choices, creating concerns about configuration bias and generalisation. The paper introduces EMMA, which aggregates diverse CNNs to marginalize configuration effects and improve robustness. EMMA won first place in BRATS 2017 among 50+ competing teams.
Problem
Architectural and training choices influence CNN behaviour, while configurations optimized on one database may overfit and fail to generalise.
Method
EMMA averages predictions from diverse CNN architectures and configurations to marginalize the effects of meta-parameters.
Results
EMMA won first place in the BRATS 2017 testing phase among 50+ competing teams.
Takeaways & Limitations
EMMA provides a model intended to be robust to independent component failures and suitable for more objective analysis with less configuration-specific behaviour.
Takeaways & Limitations
The paper presents reusability across different tasks, domain-shift analysis, training-data estimation, and uncertainty-based patient analysis as future work rather than established results.
Abstract
from arXiv · showhide
Deep learning approaches such as convolutional neural nets have consistently outperformed previous methods on challenging tasks such as dense, semantic segmentation. However, the various proposed networks perform differently, with behaviour largely influenced by architectural choices and training settings. This paper explores Ensembles of Multiple Models and Architectures (EMMA) for robust performance through aggregation of predictions from a wide range of methods. The approach reduces the influence of the meta-parameters of individual models and the risk of overfitting the configuration to a particular database. EMMA can be seen as an unbiased, generic deep learning model which is shown to yield excellent performance, winning the first position in the BRATS 2017 competition among 50+ participating teams.
1 Introduction
Brain-tumour delineation is clinically important but difficult, motivating automatic segmentation. CNNs perform well, yet architectural and training choices can alter behaviour, bias findings, and reduce generalisation; EMMA addresses this by combining diverse models.
- Clinical motivation: MRI supports tumour diagnosis, monitoring, treatment planning, and treatment assessment, but accurate delineation is difficult because tumours and imaging data are complex.Manual segmentation is time-consuming and subject to inter- and intra-rater variability.
- Automatic segmentation: CNNs have recently achieved very promising results for brain-tumour segmentation.They followed earlier atlas-based, registration-based, and Random Forest approaches.
- Model variability: Different CNN architectures and numerous meta-parameters influence both segmentation performance and model behaviour.Choices include preprocessing, architecture, class-imbalance strategy, and loss function.
- Model variability: A configuration highly optimized for one database may overfit and fail to generalise to other data or tasks.This makes conclusions about model behaviour potentially database-specific.
- EMMA: EMMA combines CNN architectures configured and trained in diverse ways to average away model- and configuration-specific behaviour.The authors report robustness to independent component failures and first place in BRATS 2017 among 50+ teams.
2 Background: Model bias, variance and ensembling
Neural networks can achieve strong performance but exhibit high-variance, unpredictable errors, which is problematic in reliability-critical biomedical applications. Ensembling addresses this by combining models, while EMMA extends the idea toward configuration-invariant objectivity.
- Model bias and variance: Unregularized neural networks can overfit noise, while optimization stochasticity and multiple local minima produce inconsistent errors between instances.These effects constitute high-variance models.
- Ensembling: Ensembling combines multiple models to seek higher performance with lower variance by averaging inconsistent errors.Only inconsistent errors can be averaged out; consistent failures remain.
- Reliability: N-version programming frames ensembling as reliability through redundancy, using independently failing versions and majority voting.This perspective formalizes reliability requirements for ensemble construction.
- Objective analysis: Because biomedical applications are reliability-critical, the paper investigates diverse ensembles to improve robustness and create an objective, configuration-invariant model.This departs from treating ensembling solely as a performance-enhancement technique.
3 Ensembles of Multiple Models and Architectures
EMMA ensembles diverse CNN architectures and configurations to reduce model- and configuration-specific bias, improving robustness and generalisation. The ensemble combines separately trained models by averaging voxel-wise class-confidence maps.
- Motivation: CNN architectural and meta-parameter choices influence model behaviour, findings, and generalisation across datasets or tasks.Examples include receptive-field size, class-imbalance strategies, loss functions, preprocessing, optimisation, regularisation, and augmentation.
- Ensemble formulation: EMMA treats meta-parameter configurations as a stochastic variable and marginalizes their effect to approximate P(y|x) with reduced bias.The ensemble uses a uniform prior over the subspace covered by its models, whereas a single model corresponds to a concentrated prior.
- Ensemble formulation: The approach requires a relatively small collection of relatively high-quality models that covers diverse configurations.Diversity is important because averaging can address inconsistent errors, while consistent failures cannot be averaged out.
- Model collection: EMMA includes two fully 3D DeepMedic models, including a residual model and a wider variant with double the feature maps per layer.DeepMedic uses parallel pathways with down-sampled context for efficient processing of 3D images; both models use cross-entropy training.
- Model collection: The collection also includes three 3D FCNs and two adapted 3D U-Nets with differing residual blocks, down-sampling, skip connections, and training settings.The U-Nets use 64×64×64 patches, while the FCNs use patches of width 64 or 80 voxels.
- Aggregation: At test time, independently trained models produce class-confidence maps that EMMA averages voxel-wise before assigning each voxel the highest-confidence class.The implementation combines models only after each has segmented the unseen image individually.
4 Evaluation
EMMA was evaluated on BRATS 2017 using models trained with diverse intensity normalisation schemes. It achieved the competition’s best testing performance and showed similar validation and test performance despite different data sources.
- Preprocessing: Three instances of each network were trained on data processed with different normalisation methods to average away normalisation effects.The evaluated preprocessing variants included Z-score normalisation, bias-field correction, and piece-wise linear normalisation.
- Preprocessing: Results are affected by normalisation, motivating EMMA models trained on differently normalised data.
- Results: EMMA achieved the overall best testing performance in BRATS 2017 based on Dice score and Hausdorff distance.Validation results were also compared with the teams ranked second and third in testing, whose testing metrics were unavailable.
- Results: EMMA achieved similar performance on validation and test sets, although the test set contained data from different sources.The paper contrasts this with competing methods that fit the validation set well but did not retain the same levels on testing.
- Results: Ensembling corrected inconsistent mistakes made by individual models, while a consistent mistake remained.The preliminary EMMA illustration used six models and marked these two error patterns separately.
5 Conclusion
The paper concludes that EMMA improves robustness and supports more objective analysis by combining heterogeneous CNNs. It won BRATS 2017 despite using straightforward, non-task-optimized component networks, while reuse across tasks remains future work.
- Conclusion: EMMA combines widely varying CNNs to reduce configuration-specific behaviour and remain insensitive to independent component failures.
- Conclusion: EMMA won first place in the BRATS 2017 final testing stage among more than 50 teams despite straightforward, non-optimized individual networks.
- Conclusion: Ensembling marginalizes configuration-induced bias, making EMMA more suitable for objective analysis.
- Conclusion: Reuse on different tasks is a possible application that the authors identify for future exploration.