Source-linked AI summary
TransBTS: Multimodal Brain Tumor Segmentation Using Transformer
Wenxuan Wang, Chen Chen, Meng Ding, Jiangyun Li, Hong Yu, Sen Zha
TL;DR
3D MRI tumor segmentation needs both local volumetric context and global dependencies, which conventional CNNs and direct patch tokenization handle incompletely. TransBTS combines 3D CNN encoding, Transformer global modeling, and progressive decoder upsampling, achieving comparable or higher results than prior state-of-the-art 3D methods on BraTS 2019 and 2020.
Problem
3D MRI segmentation requires both local and global features, while CNNs have limited long-distance modeling and patch tokenization can ignore local volumetric structure.
Method
TransBTS uses a 3D CNN encoder for local volumetric features, reformulates feature maps as Transformer tokens for global modeling, and progressively upsamples them in a decoder.
Results
TransBTS achieves comparable or higher results than previous state-of-the-art 3D methods on BraTS 2019 and 2020 brain-tumor segmentation.
Takeaways & Limitations
TransBTS provides a strong 3D baseline that combines local 3D context modeling with Transformer-based global semantic correlations for volumetric segmentation.
Abstract
from arXiv · showhide
Transformer, which can benefit from global (long-range) information modeling using self-attention mechanisms, has been successful in natural language processing and 2D image classification recently. However, both local and global features are crucial for dense prediction tasks, especially for 3D medical image segmentation. In this paper, we for the first time exploit Transformer in 3D CNN for MRI Brain Tumor Segmentation and propose a novel network named TransBTS based on the encoder-decoder structure. To capture the local 3D context information, the encoder first utilizes 3D CNN to extract the volumetric spatial feature maps. Meanwhile, the feature maps are reformed elaborately for tokens that are fed into Transformer for global feature modeling. The decoder leverages the features embedded by Transformer and performs progressive upsampling to predict the detailed segmentation map. Extensive experimental results on both BraTS 2019 and 2020 datasets show that TransBTS achieves comparable or higher results than previous state-of-the-art 3D methods for brain tumor segmentation on 3D MRI scans. The source code is available at https://github.com/Wenxuan-1119/TransBTS
1 Introduction
TransBTS addresses the need to model both local and global information in 3D MRI brain-tumor segmentation by combining 3D CNNs with a Transformer. The proposed encoder-decoder design captures volumetric context and global dependencies, with experiments showing comparable or higher results than prior state-of-the-art 3D methods.
- Accurate automated glioma segmentation from MRI is important for clinical diagnosis.
- CNN-based methods represent images effectively but struggle to model explicit long-distance dependencies because convolution kernels have limited receptive fields.
- Dense prediction of volumetric MRI data requires modeling local structures, global relationships, and continuous information across the depth dimension.
- TransBTS is presented as the first approach in this paper to exploit Transformer within a 3D CNN for 3D MRI brain-tumor segmentation.
- The architecture uses 3D CNNs for local volumetric features, Transformer tokens for global modeling, and a decoder for progressive upsampling to full-resolution segmentation.
- Experiments on BraTS 2019 and 2020 report comparable or higher results than previous state-of-the-art 3D brain-tumor segmentation methods.
2 Method
TransBTS combines 3D CNN feature extraction with Transformer-based global modeling in an encoder-decoder pipeline. Its decoder maps Transformer features back to volumetric space and progressively upsamples them into detailed segmentation masks.
- The encoder uses 3D CNNs to extract compact features containing spatial and depth information before Transformer processing.
- Because Transformer complexity grows quadratically with token count, volumetric inputs are represented through compact feature maps and tokens rather than fully flattened images.
- Learnable position embeddings are added to projected feature tokens so the Transformer retains location information needed for segmentation.
- The Transformer encoder stacks layers containing Multi-Head Attention and Feed Forward Network blocks, with layer normalization and residual outputs.
- The decoder progressively upsamples mapped features to full resolution and concatenates encoder skip-connections to recover spatial detail.
- A feature-mapping module reshapes Transformer output sequences into 3D feature maps and reduces their channel dimension for decoding.
- Unlike TransUNet, TransBTS processes all slices jointly with 3D CNNs, modeling spatial and depth dependencies without relying on pretrained ViT weights.
3 Experiments
Experiments on BraTS 2019 and 2020 evaluate TransBTS against established 3D methods and examine its architecture through ablations. Results report competitive segmentation performance, reduced complexity in a lightweight variant, and gains from conventional CNN skip-connections.
- Main Results: Five-fold cross-validation on BraTS 2019 training data yields average Dice scores of 78.69%, 90.98%, 82.85% for ET, WT and TC.These evaluations follow a conventional setting used by many existing works.
- Main Results: TransBTS achieves Dice scores of 78.93%, 90.00%, 81.94% on ET, WT, TC, respectively, on the BraTS 2019 validation set.The study compares these results with previous state-of-the-art 3D approaches and also reports improvement in Hausdorff distance.
- Main Results: TransBTS achieves Dice scores of 78.73%, 90.09%, 81.73% and HD of 17.947mm, 4.964mm, 9.769mm on ET, WT, TC on BraTS 2020.The reported comparison shows superiority over 3D U-Net, V-Net and Residual 3D U-Net in both metrics.
- Model Complexity: The lightweight TransBTS has 15.14M parameters and 208G FLOPs, reducing parameters by 54.11% and FLOPs by 37.54% with only marginal performance loss.Its BraTS 2019 validation Dice scores are 78.94%, 90.36%, 81.76% for ET, WT and TC.
- Ablation Study: Using skip-connections from the 3D convolutional layers achieves gains of 3.96% and 1.23% for ET and TC by recovering low-level spatial detail.Attaching skip-connections to the first three Transformer layers instead resembles adjacent-layer feature aggregation without compensating for lost spatial details.
4 Conclusion
TransBTS incorporates Transformer into a 3D CNN for multimodal MRI brain tumor segmentation, combining local-context modeling with global semantic correlations. Results on BraTS 2019 and 2020 validate its effectiveness.
- TransBTS combines 3D CNN local-context modeling with Transformer-based global semantic-correlation learning for multimodal MRI segmentation.
- Experimental results on BraTS 2019 and 2020 validate the effectiveness of the proposed TransBTS framework.
- Future work will investigate computationally and memory-efficient Transformer attention for volumetric segmentation.
Appendix
The appendix provides TransBTS network design details, including layer terminology, residual-block structure, and the input-image size.
- Table 6 documents the design details of the proposed TransBTS network.It specifies the architecture’s implementation components and configuration.
- Conv3 denotes a 3×3×3 convolutional layer, BN denotes Batch Normalization, and DeConv denotes a deconvolution layer.
- Each encoder and decoder block is a residual block.
- The input image size is 4 × 128 × 128 × 128.