PaperlayerView exact revision on arXiv

Source-linked AI summary

CKD-TransBTS: Clinical Knowledge-Driven Hybrid Transformer with Modality-Correlated Cross-Attention for Brain Tumor Segmentation

Jianwei Lin, Jiatai Lin, Cheng Lu, Hao Chen, Huan Lin, Bingchao Zhao, Zhenwei Shi, Bingjiang Qiu, Xipeng Pan, Zeyan Xu, Biao Huang, Changhong Liang, Guoqiang Han, Zaiyi Liu, Chu Han

arXiv:2207.07370v1eess.IVcs.CV

TL;DR

Brain tumor segmentation needs a more clinically grounded way to fuse multiple MRI modalities than direct concatenation. CKD-TransBTS groups correlated sequences, combines Transformer and CNN processing with MCCA and TCFC blocks, and achieves state-of-the-art performance against eleven comparison models on BraTS 2021.

  • Problem

    Existing brain tumor segmentation methods do not adequately account for the structural correlation among aligned MRI modalities during fusion.

  • Method

    CKD-TransBTS groups T1 with T1Gd and T2 with T2FLAIR, then uses a dual-branch hybrid encoder with MCCA and a TCFC decoder.

  • Results

    The model achieves state-of-the-art brain tumor segmentation performance against five CNN-based and six Transformer-based models on BraTS 2021.

  • Takeaways & Limitations

    Clinical knowledge about MRI sequence relationships can guide multimodal fusion and support more precise boundaries with fewer false positives.

Abstract

from arXiv · show

Brain tumor segmentation (BTS) in magnetic resonance image (MRI) is crucial for brain tumor diagnosis, cancer management and research purposes. With the great success of the ten-year BraTS challenges as well as the advances of CNN and Transformer algorithms, a lot of outstanding BTS models have been proposed to tackle the difficulties of BTS in different technical aspects. However, existing studies hardly consider how to fuse the multi-modality images in a reasonable manner. In this paper, we leverage the clinical knowledge of how radiologists diagnose brain tumors from multiple MRI modalities and propose a clinical knowledge-driven brain tumor segmentation model, called CKD-TransBTS. Instead of directly concatenating all the modalities, we re-organize the input modalities by separating them into two groups according to the imaging principle of MRI. A dual-branch hybrid encoder with the proposed modality-correlated cross-attention block (MCCA) is designed to extract the multi-modality image features. The proposed model inherits the strengths from both Transformer and CNN with the local feature representation ability for precise lesion boundaries and long-range feature extraction for 3D volumetric images. To bridge the gap between Transformer and CNN features, we propose a Trans&CNN Feature Calibration block (TCFC) in the decoder. We compare the proposed model with five CNN-based models and six transformer-based models on the BraTS 2021 challenge dataset. Extensive experiments demonstrate that the proposed model achieves state-of-the-art brain tumor segmentation performance compared with all the competitors.

I. INTRODUCTION

Brain tumor segmentation must integrate complementary MRI modalities, yet prior BTS work often used simple modality concatenation. CKD-TransBTS incorporates clinical modality relationships into a hybrid Transformer-CNN architecture and reports state-of-the-art performance on BraTS 2021.

  • Motivation: MRI provides complementary information for glioma diagnosis, treatment planning, follow-up, and precise lesion segmentation.Radiologists attend particularly to enhancing regions, necrosis, and peritumoral edema.
  • Motivation: CNNs provide local feature representation, whereas Transformers capture long-range information suited to 3D volumetric MRI.The paper motivates combining both architectures for brain tumor segmentation.
  • Research gap: MRI modalities have strong structural correlations, making their fusion distinct from multimodal data with large semantic and structural gaps.T1Gd derives from T1, while T2 and T2FLAIR are often jointly interpreted clinically.
  • Proposed approach: CKD-TransBTS groups modalities as T1 with T1Gd and T2 with T2FLAIR instead of directly concatenating all inputs.Its dual-branch hybrid encoder uses MCCA for multimodal fusion, while TCFC calibrates Transformer and CNN features.
  • Evaluation: The model achieves state-of-the-art BTS performance against five CNN-based and six Transformer-based models on BraTS 2021.The study also reports ablation studies evaluating its technical novelties and clinical-knowledge formulation.

B. Transformer-based BTS Models

Transformer-based BTS models address 3D context and long-range dependencies, while CKD-TransBTS further integrates clinical knowledge into multimodal fusion and combines Transformer and CNN representations.

  • Transformer-based BTS Models: Transformers use multi-head self-attention to model long-range information in medical images.ViT extends Transformer architectures to computer vision by tokenizing images.
  • Transformer-based BTS Models: Existing BTS models combine Transformer and CNN components through architectures such as TransUNet, TransBTS, CoTr, nnFormer, and Swin-UNETR.These approaches target global features, encoder-decoder dependencies, contextual representations, or long-range dependencies.
  • Transformer-based BTS Models: CKD-TransBTS introduces a hybrid design that formulates clinical knowledge in multimodal fusion and adds CNN locality inside Transformer modules.It also calibrates Transformer and CNN features to bridge their representation gap.
  • Multimodal fusion: Brain MRI modalities are pixel-aligned and structurally correlated, so multimodal fusion methods designed for heterogeneous modalities cannot be applied directly without adaptation.The paper identifies multimodal fusion as a distinct BTS challenge.
  • Multimodal fusion: Earlier BTS models often concatenated all modalities, whereas later methods learned modality weights or complementary information with specialized branches.CKD-TransBTS instead uses clinical imaging principles to guide modality grouping.
  • Multimodal fusion: The paper organizes its method around clinical-knowledge-driven fusion, hybrid Transformer-CNN design, and feature calibration.These elements are presented as the principal design perspectives of CKD-TransBTS.

A. Formulation and Model Architecture

CKD-TransBTS mirrors radiologists’ pairing of MRI sequences by using a dual-branch hybrid encoder and calibrated decoder. Its formulation groups correlated modalities to learn cross-modal representations while combining Transformer context with CNN locality.

  • A. Formulation and Model Architecture: Radiologists integrate four MRI sequences, pairing T1 with T1Gd for tumor core assessment and T2 with T2FLAIR for water-related tissue interpretation.T1Gd highlights vascular structures and blood-brain barrier breakdown, while T2 and T2FLAIR distinguish free- and bound-water patterns.
  • 2) Clinical Knowledge-Driven Formulation:: The formulation replaces all-modality concatenation with grouped inputs {T1, T1Gd} and {T2, T2FLAIR}.The grouping is explicitly motivated by clinical knowledge of MRI imaging principles.
  • 2) Clinical Knowledge-Driven Formulation:: Grouping correlated modalities enables the model to learn their inherent correlation and improve cross-modal feature representation.This is the stated rationale for the clinical knowledge-driven input organization.
  • 3) Model Architecture:: The U-Net-like architecture uses a Swin Transformer backbone with convolutional layers to combine long-range information and local inductive bias.Skip connections are retained for segmentation.
  • 3) Model Architecture:: A dual-branch hybrid encoder processes the two modality groups with a convolutional stem and MCCA blocks.The branches correspond to the paired inputs and exchange information through modality-correlated attention.
  • 3) Model Architecture:: The complete figure depicts grouped inputs, MCCA-based encoding, TCFC-based decoding, and omitted downsampling or upsampling operations for simplification.Feature-map resolutions are indicated by scaling factors.
  • 3) Model Architecture:: TCFC blocks calibrate Transformer skip features against CNN decoder features to bridge their semantic gap before final prediction.The decoder then uses convolutional layers to produce segmentation results.

B. Dual-Branch Hybrid Encoder

The dual-branch hybrid encoder processes paired MRI modalities through separate, identically structured branches. Convolutional stems provide softer multiscale downsampling before feature extraction and skip connections.

  • The encoder uses two identical, non-shared-weight branches to process modality pairs {XT1, XT1Gd} and {XT2, XT2FLAIR}.
  • 1) Convolutional Stem (CS):: Each modality receives a convolutional stem that reduces input dimensions through several convolutional blocks rather than direct fourfold downsampling.The stem is intended to reduce information loss during downsampling.
  • 1) Convolutional Stem (CS):: The convolutional stem produces feature volumes at 1/2 and 1/4 scales for skip connections and MCCA feed-forward processing.
  • 1) Convolutional Stem (CS):: Multiscale stem outputs help recover information during decoding, while early convolution can improve ViT optimization stability.

2) Modality-Correlated Cross-Attention (MCCA) Block:

The MCCA block extracts features from paired, structurally correlated modalities by combining per-modality hybrid processing with cross-modal attention. Transformer operations capture long-range information, while convolution supplies locality and inductive bias.

  • MCCA accepts paired inputs and generates paired outputs through two identical branches, each containing self-modal and cross-modal modules.
  • Self-modal module:: The self-modal module uses Transformer processing for long-range information and convolution layers instead of MLP layers to introduce locality and inductive bias.
  • MSA denotes window-partitioned multi-head self-attention, LN denotes layer normalization, and MBConv denotes the EfficientNet module.
  • Cross-modal module:: The cross-modal module uses shifted-window partitioning, MBConv, and cross-modal attention to exchange information between correlated modalities.

3) Bottleneck Layer:

After three MCCA blocks, the encoder concatenates features from all four modalities and passes them to a bottleneck layer that connects the encoder with the decoder.

  • After three MCCA blocks, features from the four modalities are concatenated and processed by a bottleneck layer.The bottleneck shares the single-branch MCCA structure but omits cross-modal attention.

C. Feature Calibration Decoder

The feature calibration decoder combines encoder skip features with CNN decoding through TCFC blocks. TCFC applies pixel-wise spatial attention to bridge the semantic gap between Transformer and CNN features.

  • The decoder uses three consecutive TCFC blocks, convolutional blocks, and a segmentation head to predict final segmentation results.
  • 1) Trans&CNN Feature Calibration Block (TCFC):: TCFC bridges the semantic gap between encoder and decoder features by applying pixel-wise spatial attention to MCCA-derived features.
  • 1) Trans&CNN Feature Calibration Block (TCFC):: In the first TCFC block, the feed-forward feature tensor is the bottleneck output FBNL, while transformer tensors come from the dual-branch encoder.
  • 1) Trans&CNN Feature Calibration Block (TCFC):: Average pooling is applied separately along three directions to both feed-forward and Transformer feature tensors to use 3D volumetric information.
  • 1) Trans&CNN Feature Calibration Block (TCFC):: Direction-wise Transformer and CNN features are reshaped, concatenated, channel-compressed with a 1 × 1 × 1 convolution, and aggregated.
  • 1) Trans&CNN Feature Calibration Block (TCFC):: Matrix multiplication of the three directional vectors produces a calibrated attention tensor A, whose calibrated Transformer features are concatenated with the main stream.
  • Table I presents quantitative comparisons with state-of-the-art methods on the BraTS21 dataset and marks CNN-based models with a dagger.

D. Implementation Details

The experiments use a PyTorch and MONAI implementation trained for 500 epochs with cosine-annealed learning rate adjustment and Dice loss on an NVIDIA 3090 GPU.

  • Training used PyTorch and MONAI on a workstation equipped with an NVIDIA 3090 GPU.
  • The learning rate was initialized at 1e-4 and adjusted with cosine annealing over 500 training epochs.
  • Dice loss served as the training objective, with an initial sub-volume resolution and embedding size of 4 × 4 × 4 and 32, respectively.

IV. EXPERIMENT

Experiments evaluate CKD-TransBTS on BraTS 2021 using Dice and HD95, comparing it with CNN and Transformer baselines through quantitative and qualitative analyses. The model achieves the strongest overall segmentation results, especially for enhanced-tumor boundaries and false-positive suppression.

  • Dataset and Metrics: The BraTS 2021 training set was split into 834 training, 208 validation, and 209 test samples after skull stripping and 1mm3 resampling.
  • Dataset and Metrics: Segmentation performance was evaluated using Dice score and 95% Hausdorff distance (HD95).
  • Comparison with SOTA Models: CKD-TransBTS was compared with five CNN-based and six Transformer-based models, including DynUNet, TransBTS, UNETR, and Swin UNETR.
  • Comparison with SOTA Models: CKD-TransBTS outperformed all compared models in Dice for ET, TC, and WT, while ET HD95 reached 5.93 mm, 3 mm below the second-best 9.01 mm.
  • Comparison with SOTA Models: Against DynUNet, the BraTS 2021 CNN winner, CKD-TransBTS improved ET Dice from 0.8581 to 0.8850 and ET HD95 from 13.03 to 5.93.
  • Comparison with SOTA Models: Qualitative comparisons show fewer baseline false positives, less noise, and more complete or precise segmentation boundaries for CKD-TransBTS.

C. Ablation Studies

Ablation studies isolate the effects of clinical multimodal fusion, the hybrid Transformer-CNN encoder, and TCFC feature calibration. Each novelty improves segmentation, while their combination yields stronger Dice and HD95 results.

  • Ablation Design: The ablations compare a baseline without cross-modal fusion, hybrid encoding, or feature calibration against configurations containing each novelty individually and in combination.
  • Effectiveness of Multi-modal Fusion: Clinical knowledge-driven multimodal fusion improved both Dice and HD95 over the baseline, reaching mean HD95 7.74 and mean Dice 0.8949.
  • Effectiveness of the Hybrid Encoder: The hybrid encoder generally improved Dice, but without multimodal fusion its HD95 remained larger than the fusion-only configuration.
  • Effectiveness of the Hybrid Encoder: Adding multimodal fusion to the hybrid encoder improved Dice from 0.8885 to 0.8959 and HD95 from 9.36 to 7.60.
  • Effectiveness of Feature Calibration: TCFC calibrates Transformer skip features with CNN decoder features, and the hybrid encoder plus TCFC achieved Dice 0.9029 and HD95 7.92.
  • Summary: Overall, multimodal fusion primarily improves boundary precision, while the hybrid encoder and TCFC combine complementary Transformer and CNN representations.

D. Comparison with Different Fusion Strategies

The study compares conventional feature-level and input-level fusion with modality-grouped cross-modal attention. Grouping clinically correlated modalities, especially T1 with T1Gd, provides the strongest contour and enhanced-tumor results.

  • Fusion Strategy Comparison: Table III compares multimodal fusion strategies, including separate modality branches, direct four-modality concatenation, alternative modality groups, and the final CKD-TransBTS configuration.
  • Fusion Strategy Comparison: Conventional feature-level and input-level fusion achieve reasonable Dice but are less effective for HD95, which emphasizes contour correctness.
  • Fusion Strategy Comparison: Alternative modality groupings reduce HD95, indicating more precise contours than conventional feature-level and input-level fusion.
  • Clinical Modality Grouping: Grouping T1 with T1Gd achieved the best enhanced-tumor results, with Dice 0.8850 and HD95 5.93.
  • Qualitative Analysis: Qualitative examples show the clinical fusion strategy identifying an additional small enhanced-tumor lesion when T1 and T1Gd are jointly interpreted.
  • Conclusion: The model combines MCCA multimodal fusion, a hybrid Transformer-CNN encoder, and TCFC feature calibration to improve cross-modal representation and decoder compatibility.
Loading 2207.07370v1…