Source-linked AI summary
NestedFormer: Nested Modality-Aware Transformer for Brain Tumor Segmentation
Zhaohu Xing, Lequan Yu, Liang Wan, Tong Han, Lei Zhu
TL;DR
Multi-modal MRI segmentation methods often rely on early or middle fusion that inadequately explores nonlinear relationships between modalities. NestedFormer uses nested intra- and inter-modality transformers with modality-sensitive gating, and experiments on BraTS2020 and MeniSeg report clear outperformance over state-of-the-art methods.
Problem
Existing multi-modal MRI segmentation methods commonly use early or layer fusion, which can hardly explore nonlinear relationships between modalities.
Method
NestedFormer uses multi-encoder feature extraction, NMaFA for nested intra- and inter-modality fusion, and modality-sensitive gating for low-resolution skip connections.
Results
NestedFormer clearly outperforms state-of-the-art methods on the BraTS2020 benchmark and private MeniSeg dataset.
Takeaways & Limitations
The framework hierarchically fuses modality-specific features and is modality-agnostic for extension to other multimodal medical data.
Abstract
from arXiv · showhide
Multi-modal MR imaging is routinely used in clinical practice to diagnose and investigate brain tumors by providing rich complementary information. Previous multi-modal MRI segmentation methods usually perform modal fusion by concatenating multi-modal MRIs at an early/middle stage of the network, which hardly explores non-linear dependencies between modalities. In this work, we propose a novel Nested Modality-Aware Transformer (NestedFormer) to explicitly explore the intra-modality and inter-modality relationships of multi-modal MRIs for brain tumor segmentation. Built on the transformer-based multi-encoder and single-decoder structure, we perform nested multi-modal fusion for high-level representations of different modalities and apply modality-sensitive gating (MSG) at lower scales for more effective skip connections. Specifically, the multi-modal fusion is conducted in our proposed Nested Modality-aware Feature Aggregation (NMaFA) module, which enhances long-term dependencies within individual modalities via a tri-orientated spatial-attention transformer, and further complements key contextual information among modalities via a cross-modality attention transformer. Extensive experiments on BraTS2020 benchmark and a private meningiomas segmentation (MeniSeg) dataset show that the NestedFormer clearly outperforms the state-of-the-arts. The code is available at https://github.com/920232796/NestedFormer.
1 Introduction
Multi-modal MRI provides complementary information for distinguishing brain-tumor structures, but common fusion strategies inadequately model nonlinear relationships between modalities. NestedFormer addresses this gap with nested modality-aware fusion and modality-sensitive gating for brain-tumor segmentation.
- Clinical motivation: Multi-modal MRI sequences provide complementary information for distinguishing tumor, edema, and tumor core in gliomas and meningiomas.Glioma segmentation commonly uses T1, T1Gd, T2, and T2-FLAIR, while meningioma imaging uses T1Gd and contrast-enhanced T2-FLAIR.
- Related methods: CNN encoder-decoder models and transformer-based methods have been applied to brain-tumor segmentation by learning local and long-range image dependencies.Examples include S3D-UNet, SegResNet, HPU-Net, TransBTS, and UNETR.
- Research gap: Early fusion concatenates modalities at the input, but it can hardly explore nonlinear relationships between different modalities.Layer-fusion methods instead combine modality-specific features from separate encoders in middle network layers.
- Proposed approach: NestedFormer proposes a transformer-based framework that explicitly models intra-modality and inter-modality relationships for multi-modal brain-tumor segmentation.Its NMaFA module performs modality-aware feature aggregation, while modality-sensitive gating supports lower-scale feature reuse.
- Proposed approach: NMaFA combines single-modality spatial coherence and cross-modality coherence through nested transformers, while tri-orientated spatial attention accelerates 3D spatial-coherence calculation.The framework also uses modality-sensitive gating to dynamically filter modality-aware low-resolution features for skip connections.
2 Method
NestedFormer uses modality-specific encoders, nested transformer fusion for high-level features, and modality-sensitive gating for lower-resolution skip connections. Its NMaFA module combines within-modality spatial dependencies with cross-modality relationships, while the decoder reconstructs segmentation maps from fused representations.
- NestedFormer uses multiple encoders for multi-scale modality representations, an NMaFA fusion module for high-level features, and gating to transfer modality-sensitive low-resolution features.The architecture combines a multi-encoder with a single decoder.
- Global Poolformer encoders use learnable global pooling and MLP blocks to extract modality-specific representations with enhanced global information.Each encoder progressively maps a modality image into high-level features.
- NMaFA nests tri-orientated spatial attention and cross-modality attention to model intra-modality spatial coherence and inter-modality relationships.The spatial transformer operates on concatenated modality features, while the cross-modality transformer uses modality tokens for fusion.
- Unlike conventional channel-spatial attention, NMaFA fuses its two transformers in a nested rather than serial or parallel arrangement.The design relies on transformer-based relational modeling instead of channel-wise and spatial-wise feature reweighting.
- Tri-orientated spatial attention combines axial-wise, plane-wise, and window-wise attention to capture volumetric dependencies with reduced computation.These operations model vertical, within-slice, and local 3D-window relationships.
- The decoder folds fused tokens into a high-level feature map, progressively upsamples it, and uses modality-sensitive gating to filter encoder skip features.The gating map is learned from the NMaFA output before the filtered features enter decoding.
3 Experiment
Experiments evaluate NestedFormer on BraTS2020 and MeniSeg using Dice and HD95, including comparisons with state-of-the-art methods, visual results, and module ablations. NestedFormer achieves strong quantitative performance and ablations indicate benefits from NMaFA, GPB, and MSG.
- Datasets and metrics: BraTS2020 evaluation uses Dice and HD95 across whole tumor, tumor core, and enhancing tumor regions.The dataset contains 369 aligned four-modal MRI cases, and Table 1 reports regional and averaged scores.
- Comparison with SOTA methods: NestedFormer achieves averaged Dice of 0.861 and averaged HD95 of 5.051 on BraTS2020.It ranks first for Dice on WT and TC and for HD95 on TC, while ranking second on several ET and WT measures.
- Comparison with SOTA methods: NestedFormer improves average Dice by 1.0% on MeniSeg, with 1.5% improvement for tumor and 0.2% for edema.It also obtains the smallest edema HD95, 6.173, while its tumor HD95 is the fourth smallest at 2.647.
- Visual comparisons: Visual comparisons report more accurate brain-tumor and peritumoral-edema segmentation than the compared methods on BraTS2020 and MeniSeg.The reported explanation is improved fusion of multi-modal MRIs through explicit intra-modality and inter-modality relationship modeling.
- Ablation study: Ablations show that NMaFA improves long-distance dependency extraction, GPB benefits from global information, and MSG further improves segmentation.Together, the modules produce the best average MeniSeg Dice of 0.765.
4 Conclusion
NestedFormer hierarchically fuses multi-modal features with NMaFA and MSG, and its effectiveness is validated on BraTS2020 and MeniSeg. The framework is modality-agnostic and may extend to other multimodal medical data, while future work targets more efficient low-level fusion.
- Conclusion: NestedFormer uses multiple Global Poolformer Encoders, NMaFA for high-level fusion, and MSG to select low-level features for decoder skip connections.These modules extract and hierarchically fuse features from different modalities.
- Conclusion: The framework is validated on the BraTS2020 and MeniSeg datasets.The conclusion presents the framework as modality-agnostic and extendable to other multimodal medical data.
- Conclusion: Future work will explore more efficient low-level feature fusion to further improve segmentation performance.