Source-linked AI summary
Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion
Cheng Chen, Qi Dou, Yueming Jin, Hao Chen, Jing Qin, Pheng-Ann Heng
TL;DR
Missing MRI modalities complicate multimodal brain tumor segmentation, despite the complementary information that full modality sets provide. The paper disentangles modality-specific appearance from modality-invariant content and fuses content with learned location-dependent gating. On BRATS, it reports competitive full-modality performance and strong robustness across missing-modality combinations, including a reported average Dice improvement of over 16% for whole-tumor segmentation.
Problem
Clinical brain tumor segmentation cannot always rely on complete MRI modality sets, although multimodal images provide complementary information for the task.
Method
The framework disentangles each modality into appearance and content codes, reconstructs modalities from shared content and appearance codes, and fuses content using learned gated weights.
Results
The method achieves competitive full-modality performance and outstanding robustness under various missing-modality situations, exceeding the state-of-the-art method by over 16% in average whole-tumor Dice.
Takeaways & Limitations
The framework is presented as a robust multimodal segmentation method for brain tumor segmentation across full and missing-modality settings.
Abstract
from arXiv · showhide
Accurate medical image segmentation commonly requires effective learning of the complementary information from multimodal data. However, in clinical practice, we often encounter the problem of missing imaging modalities. We tackle this challenge and propose a novel multimodal segmentation framework which is robust to the absence of imaging modalities. Our network uses feature disentanglement to decompose the input modalities into the modality-specific appearance code, which uniquely sticks to each modality, and the modality-invariant content code, which absorbs multimodal information for the segmentation task. With enhanced modality-invariance, the disentangled content code from each modality is fused into a shared representation which gains robustness to missing data. The fusion is achieved via a learning-based strategy to gate the contribution of different modalities at different locations. We validate our method on the important yet challenging multimodal brain tumor segmentation task with the BRATS challenge dataset. With competitive performance to the state-of-the-art approaches for full modality, our method achieves outstanding robustness under various missing modality(ies) situations, significantly exceeding the state-of-the-art method by over 16% in average for Dice on whole tumor segmentation.
1 Introduction
Brain tumor segmentation benefits from complementary information across MRI modalities, but complete modality sets are not guaranteed in real-world clinical scenarios. Existing solutions address missing modalities with either modality-specific synthesis models or latent-space fusion, while feature disentanglement remains unexplored for robust multimodal segmentation at arbitrary modality numbers.
- Brain tumor segmentation supports quantitative assessment of tumor progression and preoperative treatment planning.Complementary biological information is provided by FLAIR, T1, T1c, and T2 MRI modalities.
- Full MRI modality availability is not guaranteed because of varied scanning protocols and patient conditions.Robustness to one or more missing modalities is therefore identified as essential for widely applicable multimodal learning.
- Missing-modality synthesis requires a separate model for each possible combination of available modalities, making the approach complicated.HeMIS instead fuses modality features through mean and variance statistics, adapting to varying numbers of modalities.
- Feature disentanglement may extract modality-invariant tumor content by canceling modality-specific information.Prior medical-imaging studies addressed uni-modal or bi-modal data, leaving robust multimodal segmentation at arbitrary modality numbers unstudied.
- The proposed framework combines feature disentanglement with gated feature fusion to remain robust when modalities are missing.It separates modality-specific appearance codes from modality-invariant content codes, fuses the content codes, and learns location-dependent modality contributions.
2 Method
The framework disentangles each MRI modality into appearance and content codes, then learns gated fusion of content codes to support segmentation when modalities are missing. Reconstruction, segmentation, and regularization objectives jointly train the end-to-end model.
- Feature Disentanglement: Each modality is encoded into a modality-specific appearance code and a modality-invariant content code.Appearance encoders model modality-specific factors, while content codes are fused into an integrated representation for tumor segmentation.
- Feature Disentanglement: The fused content representation is trained to reconstruct each original modality using any corresponding appearance code, including when content inputs are dropped.A pseudo-cycle-consistency reconstruction loss uses L1 distance, while latent-space modality dropout improves robustness to missing data.
- Learning Process and Network Architecture: Training combines Dice and weighted cross-entropy segmentation losses to address unbalanced multi-class tumor segmentation.The segmentation decoder predicts from the fused representation, and class weights are calculated online for cross-entropy.
- Multimodal Content Fusion with Learned Gating: Learned gating assigns voxel-wise modality weights before concatenation and bottleneck convolution, allowing location-dependent integration of complementary content.The gating matrix is produced by a convolution with M output channels and sigmoid activation; each content code is re-weighted element-wise.
- Learning Process and Network Architecture: The architecture uses separate modality encoders within a 3D U-Net-style segmentation network and residual-block reconstruction components.Convolutions use 3×3×3 kernels; the segmentation and content encoders use four residual blocks, while appearance encoders use global average pooling and fully connected layers.
3 Experiments
Experiments evaluate the proposed method on BRATS 2015 using full-modality segmentation, missing-modality robustness, ablations, and reconstruction examples. The method performs competitively with full modalities and substantially outperforms comparison methods across missing-modality combinations.
- Dataset and Preprocessing: The BRATS 2015 dataset contains four MRI modalities and three tumor classes, with 274 training cases and 110 test cases.The modalities are FLAIR, T1, T1c, and T2; the classes are complete, core, and enhancing tumor.
- Performance of Robust Brain Tumor Segmentation: The proposed method achieves the highest Dice scores for core and enhancing tumor on the BRATS 2015 test set, while other evaluations remain highly competitive with OM-Net.Results were obtained through the online evaluation system without postprocessing.
- Performance of Robust Brain Tumor Segmentation: Across all 15 missing-modality combinations and all three tumor classes, the method significantly outperforms HeMIS and imputation MLP.Missing modalities are implemented by setting the corresponding gating variable δ_i to zero during inference.
- Ablation Study: Feature disentanglement and gated fusion each improve Dice performance across tumor parts when evaluated over the 15 possible modality combinations.The ablation compares an average-fusion baseline without disentanglement against sequential additions of feature disentanglement and gated fusion.
- Ablation Study: The shared representation reconstructs missing FLAIR and T2 modalities even when some inputs are unavailable, indicating that it captures essential tumor content.Figure 3 presents example reconstructions for different combinations of input modalities.
4 Conclusion
The paper concludes that jointly applying feature disentanglement and gated feature fusion produces a modality-invariant, discriminative representation for multimodal brain tumor segmentation. Experiments on BRATS show state-of-the-art results under full and missing-modality settings, supporting robustness to inference variation.
- 4 Conclusion: The framework jointly uses feature disentanglement and gated feature fusion to obtain a modality-invariant and discriminative representation.
- 4 Conclusion: The method achieves new state-of-the-art results on BRATS under full modalities and various combinations of missing modalities.
- 4 Conclusion: Its robustness to inference variations supports applicability in real-world clinical scenarios.