Source-linked AI summary
HeMIS: Hetero-Modal Image Segmentation
Mohammad Havaei, Nicolas Guizard, Nicolas Chapados, Yoshua Bengio
TL;DR
Incomplete multi-modal MRI data make segmentation difficult because conventional approaches often require missing modalities or modality-specific imputation. HeMIS independently embeds each modality, fuses available embeddings through moments, and achieves state-of-the-art results with graceful degradation as modalities are removed, without combinatorial imputation models.
Problem
Multi-modal medical image segmentation often requires modalities that may be missing because of acquisition and patient-related hurdles.
Method
HeMIS independently embeds each modality into a shared latent space, computes moments over available modalities, and processes the fused representation for segmentation.
Results
HeMIS achieves state-of-the-art segmentation results on two neurological pathology tasks and degrades gracefully as modalities go missing compared with popular imputation approaches.
Takeaways & Limitations
The framework supports any available subset of modalities without requiring models for every potential missing-modality combination.
Abstract
from arXiv · showhide
We introduce a deep learning image segmentation framework that is extremely robust to missing imaging modalities. Instead of attempting to impute or synthesize missing data, the proposed approach learns, for each modality, an embedding of the input image into a single latent vector space for which arithmetic operations (such as taking the mean) are well defined. Points in that space, which are averaged over modalities available at inference time, can then be further processed to yield the desired segmentation. As such, any combinatorial subset of available modalities can be provided as input, without having to learn a combinatorial number of imputation models. Evaluated on two neurological MRI datasets (brain tumors and MS lesions), the approach yields state-of-the-art segmentation results when provided with all modalities; moreover, its performance degrades remarkably gracefully when modalities are removed, significantly more so than alternative mean-filling or other synthesis approaches.
1 Introduction
Medical image segmentation supports clinical visualization and quantification, but multi-modal approaches often depend on complete modality sets that are difficult to acquire. HeMIS addresses this missing-modality problem by embedding each modality into a shared latent space and fusing available modalities without combinatorial imputation models.
- Multi-modality imaging provides complementary information for discriminating tissues, anatomies, and pathologies, while manual segmentation is lengthy and subject to human variability.
- Existing segmentation strategies are typically optimized for a specific multi-modal image set and usually require those modalities to be available.
- Clinical acquisition and patient artifacts commonly leave one or more modalities missing for a given instance.
- HeMIS learns modality-specific embeddings in a shared latent space, averages well-defined moments over available modalities, and avoids training models for every missing-modality combination.
2 Method
HeMIS processes each modality independently, fuses available modalities through feature-map statistics, and then predicts pixelwise segmentation. Training randomly drops modalities after a warmup period so the network learns robustness to incomplete inputs.
- Each available modality enters an independent convolutional pipeline before modality fusion.
- A missing modality does not alter the fusion computation; it increases uncertainty in the estimated mean and variance instead.
- The abstraction layer computes first and second moments across available modalities, then concatenates the resulting feature maps for front-end processing.
- The architecture produces pixelwise posterior probabilities with a softmax and selects the most likely class for each pixel, without postprocessing.
- During pseudo-curriculum training, modalities are randomly dropped after warmup, with zero- or one-modality drops given higher probability.
- The back end learns a common embedding for all modalities, enabling vector-algebra fusion through summary statistics and end-to-end optimization.
3 Data and Implementation details
The evaluation covers multiple sclerosis and glioma MRI datasets with varying modality counts, scanner settings, and available ground truth. Images undergo bias correction, intensity normalization, registration, and patch-wise training to address class imbalance.
- The study evaluates HeMIS on MSGC and RRMS multiple-sclerosis cohorts and the BRATS glioma dataset.
- MSGC contains 20 training cases with manual lesion masks and 23 test cases evaluated through an automated online system.
- RRMS includes 300 patients scanned with sagittal T1W, T2W, and T1C MRI on 1.5T scanners from multiple manufacturers.
- BRATS-2015 contains 220 high-grade and 54 low-grade tumor subjects with four MRI modalities and five voxel-level segmentation labels.
- Preprocessing applies bias-field correction, intensity normalization, co-registration to T1W, and interpolation to 1mm isotropic resolution.
- Patch-wise training uses balanced data for feature learning before adapting the final classification layer toward ground-truth class priors.
4 Experiments and Results
HeMIS is evaluated on BRATS and MS datasets, including full-modality benchmarks and all modality-drop configurations. It remains competitive against state-of-the-art and imputation-based methods while degrading more gradually as modalities disappear.
- HeMIS outperforms Tustison et al., the BRATS 2013 challenge winner, on most tumor region categories.
- 83.2% combined score on MSGC, compared with 90.0% representing human performance under interrater variability.
- HeMIS leads BRATS Core segmentation in 14 of 15 cases and leads Complete and Enhancing categories in 10 and 9 cases, respectively.
- HeMIS outperforms other imputation approaches in 9 of 15 MS lesion segmentation cases and usually exceeds MLP-imputation when one or two modalities are missing.
- HeMIS performance drops gradually as modalities go missing, whereas MLP-imputation and mean-filling show much more severe declines.
- Adding modalities improves HeMIS segmentation by removing false positives and refining lesion or tumor outlines; FLAIR is most relevant for Complete tumor identification, while T1C is most relevant for Core and Enhancing categories.
5 Conclusion
The paper proposes HeMIS, a fully automatic convolutional framework for heterogeneous multi-modal MRI that embeds modalities to support segmentation with missing inputs. Across MS and glioma tasks, it achieves state-of-the-art results and degrades gracefully without requiring models for every missing-modality combination.
- HeMIS is a fully automatic convolutional deep neural network framework for heterogeneous multi-modal MRI segmentation with missing modalities.
- HeMIS achieves state-of-the-art segmentation results on two challenging neurological pathology image-processing tasks.
- HeMIS degrades gracefully as modalities go missing without requiring training specific models for every potential missing-modality combination.
- Future work should extend HeMIS beyond MRI to modalities such as CT, PET, and ultrasound.