Source-linked AI summary
mmFormer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation
Yao Zhang, Nanjun He, Jiawei Yang, Yuexiang Li, Dong Wei, Yawen Huang, Yang Zhang, Zhiqiang He, Yefeng Zheng
TL;DR
Missing MRI modalities hinder robust multimodal brain tumor segmentation, while existing approaches can require complete training data or modality-specific models. mmFormer uses convolutional and Transformer encoders to model within- and across-modality dependencies, achieving stronger incomplete-modality segmentation on BraTS 2018, including a 19.07% average Dice improvement with one available modality.
Problem
Incomplete MRI acquisitions make it difficult to build a unified brain tumor segmentation model robust to arbitrary subsets of available modalities.
Method
mmFormer combines hybrid modality-specific CNN–intra-modal Transformer encoders, an inter-modal Transformer, a convolutional decoder, and auxiliary regularizers.
Results
mmFormer outperformed state-of-the-art methods on BraTS 2018 incomplete-modality segmentation, with a 19.07% average Dice improvement for enhancing tumor segmentation using one available modality.
Takeaways & Limitations
The results support Transformer-based modality-invariant representations as a robust approach for brain tumor segmentation when MRI modalities are missing.
Abstract
from arXiv · showhide
Accurate brain tumor segmentation from Magnetic Resonance Imaging (MRI) is desirable to joint learning of multimodal images. However, in clinical practice, it is not always possible to acquire a complete set of MRIs, and the problem of missing modalities causes severe performance degradation in existing multimodal segmentation methods. In this work, we present the first attempt to exploit the Transformer for multimodal brain tumor segmentation that is robust to any combinatorial subset of available modalities. Concretely, we propose a novel multimodal Medical Transformer (mmFormer) for incomplete multimodal learning with three main components: the hybrid modality-specific encoders that bridge a convolutional encoder and an intra-modal Transformer for both local and global context modeling within each modality; an inter-modal Transformer to build and align the long-range correlations across modalities for modality-invariant features with global semantics corresponding to tumor region; a decoder that performs a progressive up-sampling and fusion with the modality-invariant features to generate robust segmentation. Besides, auxiliary regularizers are introduced in both encoder and decoder to further enhance the model's robustness to incomplete modalities. We conduct extensive experiments on the public BraTS $2018$ dataset for brain tumor segmentation. The results demonstrate that the proposed mmFormer outperforms the state-of-the-art methods for incomplete multimodal brain tumor segmentation on almost all subsets of incomplete modalities, especially by an average 19.07% improvement of Dice on tumor segmentation with only one available modality. The code is available at https://github.com/YaoZhang93/mmFormer.
1 Introduction
Brain tumor segmentation benefits from jointly learning multiple MRI modalities, but missing scans make existing multimodal methods difficult to use robustly. mmFormer addresses this gap with Transformer-based within- and across-modality modeling and reports stronger incomplete-modality performance.
- Motivation: Clinical brain tumor segmentation uses complementary T1, T1c, T2, and FLAIR MRI contrasts, making joint multimodal learning valuable.Each modality provides distinctive information about brain structure and pathology.
- Research gap: Incomplete multimodal learning seeks models that remain robust for any subset of available modalities during inference.Existing approaches may require complete modalities during training or separate models for different missing-modality subsets.
- Research gap: A modality-invariant embedding with global tumor-region semantics may improve segmentation robustness when one or more modalities are missing.This motivates modeling correlations across modalities rather than relying only on discriminative modalities.
- Proposed approach: mmFormer combines hybrid modality-specific encoders with a modality-correlated encoder to model long-range dependencies within and across MRI modalities.The architecture uses convolutional and Transformer components for multimodal brain tumor segmentation.
- Proposed approach: Auxiliary regularizers are added to the encoder and decoder to encourage discriminative features when modalities are missing.The model is designed as a unified approach for incomplete multimodal learning rather than separate models for each missing-modality condition.
- Results: 19.07% average Dice improvement was reported for enhancing tumor segmentation with only one available modality.On BraTS 2018, mmFormer outperformed state-of-the-art methods across missing-modality settings, especially in the one-modality condition.
2 Method
mmFormer uses an encoder–decoder architecture combining modality-specific convolutional and intra-modal Transformer processing with an inter-modal Transformer. Auxiliary decoding and deep-supervision regularizers are added to support segmentation when modalities are missing.
- mmFormer combines a hybrid modality-specific encoder, a modality-correlated encoder, and a convolutional decoder in an encoder–decoder architecture.The modality-specific encoders process each imaging modality, while the modality-correlated encoder integrates multimodal representations.
- Hybrid Modality-specific Encoder: The hybrid modality-specific encoder bridges convolutional encoding for local context with an intra-modal Transformer for long-range context within each modality.Convolutional feature maps are flattened, projected into token space, augmented with learnable positional embeddings, and processed by self-attention and a feed-forward network.
- Hybrid Modality-specific Encoder: The convolutional encoder uses five stages with two convolutional blocks per stage and progressively downsamples feature maps using stride-2 convolutions.The stage filter counts are 16, 32, 64, 128, and 256, respectively.
- Modality-correlated Encoder: The modality-correlated encoder concatenates embeddings from modality-specific encoders and uses an inter-modal Transformer to build cross-modality correlations.A Bernoulli modality-level dropout indicator randomly removes modalities during training, replacing missing-modality tokens with zero vectors before inter-modal processing.
- Auxiliary Regularizer: The convolutional decoder progressively restores spatial resolution with encoder–decoder skip connections, while shared-weight decoding and deep supervision provide auxiliary segmentation regularization.Auxiliary regularizers encourage modality-specific encoders and decoder stages to produce accurate tumor segmentations when modalities are missing; Dice loss is used.
3 Experiments and Results
Experiments on BraTS 2018 evaluate mmFormer under incomplete and complete multimodal MRI settings. The method is robust across missing-modality combinations, compares favorably with prior methods, and benefits from its critical components.
- Dataset and Implementation: BraTS 2018 contains 285 co-registered multi-contrast MRI scans with T1, T1c, T2, and FLAIR modalities, evaluated using DSC.Tumor regions are assessed as whole tumor, tumor core, and enhancing tumor.
- Performance of Incomplete Multimodal Segmentation: mmFormer significantly outperforms HeMIS and U-HVED for enhancing tumor and tumor core across all 15 available-modality combinations, and for whole tumor in 12 combinations.With one modality available, mmFormer still achieves proper brain-tumor segmentation.
- Performance of Incomplete Multimodal Segmentation: With more missing modalities, mmFormer obtains larger average improvements over HeMIS and U-HVED, particularly for more difficult tumor targets.Table 2 reports average improvements using DSC across different numbers of missing modalities.
- Comparison with ACN: mmFormer requires 25 hours and 106M parameters, compared with ACN's 672 hours and 144M parameters, while achieving similar incomplete-modality DSC.mmFormer's average DSC values are 59.85, 72.97, and 82.94 for enhancing tumor, tumor core, and whole tumor, versus ACN's 61.21, 77.62, and 85.92.
- Performance of Complete Multimodal Segmentation: For complete multimodal segmentation, mmFormer outperforms TransBTS on enhancing tumor, tumor core, and whole tumor DSC.The respective mmFormer DSC values are 77.61%, 85.78%, and 89.64%, compared with TransBTS values of 72.66%, 72.69%, and 79.99%.
- Ablation Study: Ablation results show that the intra-modal Transformer, inter-modal Transformer, and auxiliary regularizer improve performance across all tumor subregions.The comparison averages DSC over 15 possible input-modality combinations.
4 Conclusion
The conclusion presents mmFormer as a Transformer-based approach for incomplete multimodal brain-tumor segmentation. Its modality-invariant representation supports performance across missing-modality combinations on the BraTS benchmark.
- 4 Conclusion: mmFormer bridges Transformer and CNN components to model long-range dependencies within and across MRI modalities.The resulting representation is modality-invariant.
- 4 Conclusion: mmFormer outperforms state-of-the-art methods on the BraTS benchmark under various combinations of missing modalities.The method gains more improvement when more modalities are missing or when the target is more difficult to segment.