Source-linked AI summary
Hetero-Modal Variational Encoder-Decoder for Joint Modality Completion and Segmentation
Reuben Dorent, Samuel Joutard, Marc Modat, Sébastien Ourselin, Tom Vercauteren
TL;DR
Missing imaging modalities make tumour segmentation difficult because standard deep-learning methods often require the full modality set. The paper introduces a hetero-modal variational encoder-decoder with mixture sampling and a shared latent representation, achieving better segmentation than HeMIS and similar performance to subset-specific models.
Problem
Tumour segmentation must handle clinically common cases where only an arbitrary subset of imaging modalities is available, while many methods require all n modalities.
Method
The method independently embeds observed modalities into a shared latent representation using a 3D U-Net and MVAE-based variational encoder-decoder with mixture-sampling optimisation.
Results
The model outperforms HeMIS for tumour segmentation and achieves similar performance to subset-specific models, while also generating missing modalities.
Takeaways & Limitations
A shared variational representation supports segmentation with missing modalities while avoiding a separate model for every modality subset.
Takeaways & Limitations
The prior MVAE sampling procedure is memory-consuming for 3D images and does not accommodate clinically unequal frequencies of imaging subsets; training may also assume complete modalities for full reconstruction evaluation.
Abstract
from arXiv · showhide
We propose a new deep learning method for tumour segmentation when dealing with missing imaging modalities. Instead of producing one network for each possible subset of observed modalities or using arithmetic operations to combine feature maps, our hetero-modal variational 3D encoder-decoder independently embeds all observed modalities into a shared latent representation. Missing data and tumour segmentation can be then generated from this embedding. In our scenario, the input is a random subset of modalities. We demonstrate that the optimisation problem can be seen as a mixture sampling. In addition to this, we introduce a new network architecture building upon both the 3D U-Net and the Multi-Modal Variational Auto-Encoder (MVAE). Finally, we evaluate our method on BraTS2018 using subsets of the imaging modalities as input. Our model outperforms the current state-of-the-art method for dealing with missing modalities and achieves similar performance to the subset-specific equivalent networks.
1 Introduction
The paper targets tumour segmentation from arbitrary subsets of imaging modalities, avoiding the cost of separate subset-specific networks. It proposes a hetero-modal variational encoder-decoder that learns a shared latent representation for segmentation and modality completion.
- Clinical tumour assessment uses variable subsets of T1, T1c, T2, and FLAIR images, motivating segmentation from arbitrary modality sets.
- Most deep-learning segmentation methods require all n modalities, despite missing modalities being common in practice.
- Training separate networks or synthesising missing modalities requires handling 2^n −1 modality combinations, whereas a common feature space enables one shared model.
- HeMIS extracts modality-specific features and combines their first and second moments, but this arithmetic operation does not enforce a shared latent representation.
- The proposed model uses auto-encoding and modality completion to promote an informative latent space that reconstructs observed modalities and limits information loss from missing ones.
- The hetero-modal variational encoder-decoder extends MVAE to 3D tumour segmentation, uses mixture-sampling optimisation, adapts 3D U-Net, and outperforms HeMIS while matching subset-specific models.
2 Method
The method combines independently encoded observed modalities into a shared variational latent representation, samples modality subsets according to clinical frequencies, and decodes both missing modalities and tumour segmentation. Its architecture integrates MVAE principles with a fully convolutional 3D U-Net using variational skip-connections and multi-scale latent variables.
- Multi-modal Variational Auto-Encoders: MVAE models modalities as conditionally independent given a latent variable z and encodes each modality with a modality-specific variational posterior.The posterior is represented as a Gaussian whose parameters are produced by an encoder.
- Motivation: The MVAE training procedure described in prior work is memory-intensive for 3D images and does not account for clinically uneven frequencies of modality subsets.The proposed mixture-sampling procedure incorporates those subset probabilities instead.
- Modality Completion and Segmentation: The model uses four encoders for input modalities and five decoders to reconstruct imaging modalities and produce tumour segmentation.Segmentation is decoded rather than encoded because it is not observed at inference time.
- Mixture Sampling: Training samples a non-empty modality subset π with probability απ, treating the optimisation as ancestral sampling from a mixture model.The subset probabilities are selected to represent the clinical scenario, and the lower bound is a weighted sum of subset-specific lower bounds.
- Mixture Sampling: A single Gaussian prior promotes consistency between latent embeddings inferred from different modality subsets and the full modality set.At each iteration, a subset is randomly selected and its corresponding ELBO is optimised.
- Network Architecture: The proposed U-HVED combines a 3D U-Net and MVAE through variational skip-connections and multi-level latent variables.Each modality is independently encoded into four multi-scale means and variances, fused across the observed subset using a product of Gaussians, then decoded for modalities and segmentation.
3 Data and implementation details.
The method was evaluated on preprocessed BRATS18 scans, with training and optimisation choices designed for multimodal 3D segmentation and reconstruction.
- Data: BRATS18 training data contained 285 patients, including 210 with high-grade glioma and 75 with low-grade glioma.Each patient had T1, T1c, T2, and FLAIR sequences that were skull-stripped, resampled to isotropic 1 mm resolution, and co-registered.
- Implementation details: Preprocessing used histogram-based scale standardisation followed by zero-mean, unit-variance normalisation.Training augmentation randomly flipped axes and applied rotations between −10° and 10°.
- Implementation details: The networks were implemented in TensorFlow with NiftyNet and optimised using Adam from a 10^-3 learning rate, batch size 1, and up to 60k iterations.The learning rate was divided by 4 every 10^4 iterations, with early stopping applied during training.
4 Experiments and results
Experiments compared U-HVED with missing-modality baselines and subset-specific networks, assessing both modality completion and tumour segmentation across input subsets. U-HVED produced qualitative reconstructions and competitive segmentation performance while using a shared model.
- Model comparison: U-HVED was compared with HeMIS, U-HeMIS, and Single, the subset-specific approach that trains a separate U-Net for each modality subset.The three-fold validation setup required training 45 Single networks.
- Missing modalities completion: U-HVED generated missing modalities, whereas the comparison approaches focused on segmentation.T1 and FLAIR completion was evaluated qualitatively because completion served as a means rather than the primary endpoint.
- Missing modalities completion: The reported T1 and FLAIR reconstructions were good quality despite the blurriness commonly associated with VAEs.Comparable reconstructions were obtained using three and four input modalities, including attempts to recover unclear tumour information.
- Tumour segmentation: U-HVED significantly outperformed U-HeMIS in 13 of 15 complete-tumour, 10 of 15 core-tumour, and 11 of 15 enhancing-tumour cases.Segmentation was evaluated using Dice Similarity across all possible input subsets.
- Tumour segmentation: U-HVED achieved similar performance to the 15 subset-specific Single models.The comparison used Dice percentages for different combinations of available modalities, with significance marked by a Wilcoxon test at p < 0.05.
5 Discussion and conclusion
The paper concludes that a multimodal variational approach can support tumour segmentation with missing modalities and outperform HeMIS. It also identifies analysis of the learned multiscale embedding as future work.
- Discussion: U-HVED outperformed HeMIS by extending the variational formulation beyond HeMIS’s mean-based, identity-covariance, segmentation-only setup.The discussion characterises HeMIS as a non-variational version that independently encodes and averages modalities.
- Conclusion: The method offers a potential way to leverage large but incomplete datasets for multimodal tumour segmentation.The authors frame this as promising insight rather than a completed analysis of the learned representation.
- Future work: Future work will analyse the learned embedding, which the authors identify as challenging because the hidden variable has a multiscale representation.The paper does not report that analysis as part of the present work.