PaperlayerView exact revision on arXiv

Source-linked AI summary

CKD-TransBTS: Clinical Knowledge-Driven Hybrid Transformer with Modality-Correlated Cross-Attention for Brain Tumor Segmentation

Jianwei Lin, Jiatai Lin, Cheng Lu, Hao Chen, Huan Lin, Bingchao Zhao, Zhenwei Shi, Bingjiang Qiu, Xipeng Pan, Zeyan Xu, Biao Huang, Changhong Liang, Guoqiang Han, Zaiyi Liu, Chu Han

arXiv:2207.07370v1eess.IVcs.CV

TL;DR

Brain tumor segmentation needs a more clinically grounded way to fuse multiple MRI modalities than direct concatenation. CKD-TransBTS groups correlated sequences, combines Transformer and CNN processing with MCCA and TCFC blocks, and achieves state-of-the-art performance against eleven comparison models on BraTS 2021.

  • Problem

    Existing brain tumor segmentation methods do not adequately account for the structural correlation among aligned MRI modalities during fusion.

    Source: §I · p. 2, C. Multi-modal Fusion Models · p. 3

  • Method

    CKD-TransBTS groups T1 with T1Gd and T2 with T2FLAIR, then uses a dual-branch hybrid encoder with MCCA and a TCFC decoder.

    Source: §I · p. 2

  • Results

    The model achieves state-of-the-art brain tumor segmentation performance against five CNN-based and six Transformer-based models on BraTS 2021.

    Source: §I · p. 2

  • Takeaways & Limitations

    Clinical knowledge about MRI sequence relationships can guide multimodal fusion and support more precise boundaries with fewer false positives.

    Source: §V · p. 9

Abstract

from arXiv · show

Brain tumor segmentation (BTS) in magnetic resonance image (MRI) is crucial for brain tumor diagnosis, cancer management and research purposes. With the great success of the ten-year BraTS challenges as well as the advances of CNN and Transformer algorithms, a lot of outstanding BTS models have been proposed to tackle the difficulties of BTS in different technical aspects. However, existing studies hardly consider how to fuse the multi-modality images in a reasonable manner. In this paper, we leverage the clinical knowledge of how radiologists diagnose brain tumors from multiple MRI modalities and propose a clinical knowledge-driven brain tumor segmentation model, called CKD-TransBTS. Instead of directly concatenating all the modalities, we re-organize the input modalities by separating them into two groups according to the imaging principle of MRI. A dual-branch hybrid encoder with the proposed modality-correlated cross-attention block (MCCA) is designed to extract the multi-modality image features. The proposed model inherits the strengths from both Transformer and CNN with the local feature representation ability for precise lesion boundaries and long-range feature extraction for 3D volumetric images. To bridge the gap between Transformer and CNN features, we propose a Trans&CNN Feature Calibration block (TCFC) in the decoder. We compare the proposed model with five CNN-based models and six transformer-based models on the BraTS 2021 challenge dataset. Extensive experiments demonstrate that the proposed model achieves state-of-the-art brain tumor segmentation performance compared with all the competitors.

I. INTRODUCTION

Brain tumor segmentation must integrate complementary MRI modalities, yet prior BTS work often used simple modality concatenation. CKD-TransBTS incorporates clinical modality relationships into a hybrid Transformer-CNN architecture and reports state-of-the-art performance on BraTS 2021.

Motivation

  • MRI provides complementary information for glioma diagnosis, treatment planning, follow-up, and precise lesion segmentation.Radiologists attend particularly to enhancing regions, necrosis, and peritumoral edema.Source: §I · p. 1
  • CNNs provide local feature representation, whereas Transformers capture long-range information suited to 3D volumetric MRI.The paper motivates combining both architectures for brain tumor segmentation.Source: §I · p. 1

Research gap

  • MRI modalities have strong structural correlations, making their fusion distinct from multimodal data with large semantic and structural gaps.T1Gd derives from T1, while T2 and T2FLAIR are often jointly interpreted clinically.Source: §I · pp. 1–2

Proposed approach

  • CKD-TransBTS groups modalities as T1 with T1Gd and T2 with T2FLAIR instead of directly concatenating all inputs.Its dual-branch hybrid encoder uses MCCA for multimodal fusion, while TCFC calibrates Transformer and CNN features.Source: §I · p. 2

Evaluation

  • The model achieves state-of-the-art BTS performance against five CNN-based and six Transformer-based models on BraTS 2021.The study also reports ablation studies evaluating its technical novelties and clinical-knowledge formulation.Source: §I · p. 2

B. Transformer-based BTS Models

Transformer-based BTS models address 3D context and long-range dependencies, while CKD-TransBTS further integrates clinical knowledge into multimodal fusion and combines Transformer and CNN representations.

Transformer-based BTS Models

  • Transformers use multi-head self-attention to model long-range information in medical images.ViT extends Transformer architectures to computer vision by tokenizing images.Source: B. Transformer-based BTS Models · p. 2
  • Existing BTS models combine Transformer and CNN components through architectures such as TransUNet, TransBTS, CoTr, nnFormer, and Swin-UNETR.These approaches target global features, encoder-decoder dependencies, contextual representations, or long-range dependencies.Source: B. Transformer-based BTS Models · pp. 2–3
  • CKD-TransBTS introduces a hybrid design that formulates clinical knowledge in multimodal fusion and adds CNN locality inside Transformer modules.It also calibrates Transformer and CNN features to bridge their representation gap.Source: B. Transformer-based BTS Models · p. 3

Multimodal fusion

  • Brain MRI modalities are pixel-aligned and structurally correlated, so multimodal fusion methods designed for heterogeneous modalities cannot be applied directly without adaptation.The paper identifies multimodal fusion as a distinct BTS challenge.Source: C. Multi-modal Fusion Models · p. 3
  • Earlier BTS models often concatenated all modalities, whereas later methods learned modality weights or complementary information with specialized branches.CKD-TransBTS instead uses clinical imaging principles to guide modality grouping.Source: C. Multi-modal Fusion Models · p. 3
  • The paper organizes its method around clinical-knowledge-driven fusion, hybrid Transformer-CNN design, and feature calibration.These elements are presented as the principal design perspectives of CKD-TransBTS.Source: C. Multi-modal Fusion Models · p. 3, §III · p. 3

A. Formulation and Model Architecture

CKD-TransBTS mirrors radiologists’ pairing of MRI sequences by using a dual-branch hybrid encoder and calibrated decoder. Its formulation groups correlated modalities to learn cross-modal representations while combining Transformer context with CNN locality.

A. Formulation and Model Architecture

  • Radiologists integrate four MRI sequences, pairing T1 with T1Gd for tumor core assessment and T2 with T2FLAIR for water-related tissue interpretation.T1Gd highlights vascular structures and blood-brain barrier breakdown, while T2 and T2FLAIR distinguish free- and bound-water patterns.Source: 1) How Radiologists Diagnose Brain… · p. 3

2) Clinical Knowledge-Driven Formulation:

  • The formulation replaces all-modality concatenation with grouped inputs {T1, T1Gd} and {T2, T2FLAIR}.The grouping is explicitly motivated by clinical knowledge of MRI imaging principles.Source: 2) Clinical Knowledge-Driven Formulation · p. 3
  • Grouping correlated modalities enables the model to learn their inherent correlation and improve cross-modal feature representation.This is the stated rationale for the clinical knowledge-driven input organization.Source: 2) Clinical Knowledge-Driven Formulation · p. 3

3) Model Architecture:

  • The U-Net-like architecture uses a Swin Transformer backbone with convolutional layers to combine long-range information and local inductive bias.Skip connections are retained for segmentation.Source: 3) Model Architecture · p. 3
  • A dual-branch hybrid encoder processes the two modality groups with a convolutional stem and MCCA blocks.The branches correspond to the paired inputs and exchange information through modality-correlated attention.Source: 3) Model Architecture · p. 3, Figure 1 · p. 4
  • The complete figure depicts grouped inputs, MCCA-based encoding, TCFC-based decoding, and omitted downsampling or upsampling operations for simplification.Feature-map resolutions are indicated by scaling factors.Source: Figure 1 · p. 4
  • TCFC blocks calibrate Transformer skip features against CNN decoder features to bridge their semantic gap before final prediction.The decoder then uses convolutional layers to produce segmentation results.Source: 3) Model Architecture · p. 4

B. Dual-Branch Hybrid Encoder

The dual-branch hybrid encoder processes paired MRI modalities through separate, identically structured branches. Convolutional stems provide softer multiscale downsampling before feature extraction and skip connections.

1) Convolutional Stem (CS):

  • Each modality receives a convolutional stem that reduces input dimensions through several convolutional blocks rather than direct fourfold downsampling.The stem is intended to reduce information loss during downsampling.Source: 1) Convolutional Stem (CS) · p. 4
  • The convolutional stem produces feature volumes at 1/2 and 1/4 scales for skip connections and MCCA feed-forward processing.Source: Figure 2 · p. 4, 1) Convolutional Stem (CS) · p. 4
  • Multiscale stem outputs help recover information during decoding, while early convolution can improve ViT optimization stability.Source: 1) Convolutional Stem (CS) · pp. 4–5

2) Modality-Correlated Cross-Attention (MCCA) Block:

The MCCA block extracts features from paired, structurally correlated modalities by combining per-modality hybrid processing with cross-modal attention. Transformer operations capture long-range information, while convolution supplies locality and inductive bias.

Self-modal module:

Cross-modal module:

3) Bottleneck Layer:

After three MCCA blocks, the encoder concatenates features from all four modalities and passes them to a bottleneck layer that connects the encoder with the decoder.

  • After three MCCA blocks, features from the four modalities are concatenated and processed by a bottleneck layer.The bottleneck shares the single-branch MCCA structure but omits cross-modal attention.Source: 3) Bottleneck Layer · p. 5

C. Feature Calibration Decoder

The feature calibration decoder combines encoder skip features with CNN decoding through TCFC blocks. TCFC applies pixel-wise spatial attention to bridge the semantic gap between Transformer and CNN features.

1) Trans&CNN Feature Calibration Block (TCFC):

  • Table I presents quantitative comparisons with state-of-the-art methods on the BraTS21 dataset and marks CNN-based models with a dagger.Source: Table I · p. 6

D. Implementation Details

The experiments use a PyTorch and MONAI implementation trained for 500 epochs with cosine-annealed learning rate adjustment and Dice loss on an NVIDIA 3090 GPU.

IV. EXPERIMENT

Experiments evaluate CKD-TransBTS on BraTS 2021 using Dice and HD95, comparing it with CNN and Transformer baselines through quantitative and qualitative analyses. The model achieves the strongest overall segmentation results, especially for enhanced-tumor boundaries and false-positive suppression.

Dataset and Metrics

Comparison with SOTA Models

C. Ablation Studies

Ablation studies isolate the effects of clinical multimodal fusion, the hybrid Transformer-CNN encoder, and TCFC feature calibration. Each novelty improves segmentation, while their combination yields stronger Dice and HD95 results.

Ablation Design

  • The ablations compare a baseline without cross-modal fusion, hybrid encoding, or feature calibration against configurations containing each novelty individually and in combination.Source: C. Ablation Studies · p. 7

Effectiveness of Multi-modal Fusion

Effectiveness of the Hybrid Encoder

Effectiveness of Feature Calibration

Summary

D. Comparison with Different Fusion Strategies

The study compares conventional feature-level and input-level fusion with modality-grouped cross-modal attention. Grouping clinically correlated modalities, especially T1 with T1Gd, provides the strongest contour and enhanced-tumor results.

Fusion Strategy Comparison

Clinical Modality Grouping

Qualitative Analysis

Conclusion

  • The model combines MCCA multimodal fusion, a hybrid Transformer-CNN encoder, and TCFC feature calibration to improve cross-modal representation and decoder compatibility.Source: §V · p. 9

Related papers

Loading 2207.07370v1…