Multi-Compound Transformer for Accurate Biomedical Image Segmentation

1 Introduction

• CNN segmentation models such as FCN struggle to model long-term dependencies, while TransUNet [4] models global context at one scale and omits category correlations and feature consistency.The introduction identifies cross-scale dependency and consistency as important for segmenting lesions with dramatic size changes.Source: §1 · pp. 1–2
• MCTrans combines cross-scale contextual dependencies with semantic relationship mining for biomedical image segmentation.Its Transformer-Self-Attention module models cross-scale pixel context, while proxy embeddings support category-dependency learning and feature enhancement through self- and cross-attention.Source: §1 · pp. 2–3
• A learnable proxy embedding models category dependencies and interacts with feature representations, while auxiliary loss improves within-category correlation and between-class discriminability.The proxy embedding is used with self-attention to learn semantic correspondence and cross-attention to interact with feature representations.Source: §1 · pp. 2–3
• MCTrans is plugged into a UNet-like network and reportedly outperforms state-of-the-art methods across six challenging datasets with a slight computation increase.The paper attributes the results to the effectiveness of its proposed network components.Source: §1 · p. 3

Attention Mechanisms

• Earlier attention methods model pixel relationships at a single scale, whereas this work uses Transformers to connect multi-scale features and exchange information across scales.Prior methods include attention gates [15], feature-channel attention, and non-local operations; they capture semantic and spatial relationships but neglect information from other scales.Source: §2 · p. 3

Transformer

• Vaswani et al. [19] introduced Transformers for machine translation, using self-attention to exchange information among inputs; vision applications include image classification [7] and semantic segmentation [21].Transformer methods have also been applied to object detection and tracking [25].Source: §2 · p. 3
• TransUnet [4] applies a Transformer encoder to UNet’s highest-level feature to collect long-range dependencies, whereas this work develops an approach specifically for medical image segmentation.The paper frames its approach as leveraging attention mechanisms more thoroughly for medical image segmentation.Source: §2 · pp. 3–4

3 Multi-Compound Transformer Network

• MCTrans projects and concatenates multi-scale CNN features as tokens, processes them with TSA and TCA, then folds them back into pyramid features for prediction.It uses features from levels 2–4, with each feature-map location treated as a patch, and adds positional embeddings before attention.Source: §3 · p. 4

3.1 Transformer-Self-Attention

• TSA models pixel-level contextual dependencies across multi-scale features, using deformable sparse attention to reduce the cost of processing long token sequences.The module stacks self-attention and feed-forward layers; deformable self-attention attends to a sparse, data-dependent subset of elements.Source: §3.1 · pp. 4–5

3.2 Transformer-Cross-Attention

• TCA uses proxy embeddings to model category relationships and interact with tokens, improving feature consistency and discriminability, with auxiliary classification loss supervising proxy learning.Self-attention connects category pairs; cross-attention lets proxy embeddings interact with enhanced tokens, and the auxiliary prediction is supervised using unique labels from the ground-truth mask.Source: §3.2 · pp. 5–6

4.1 Datasets and Settings

• Experiments cover six datasets across cell, polyp, and skin-lesion segmentation, reporting DSC with 3-fold or 5-fold cross-validation.Pannuke uses official 3-fold cross-validation; other datasets use 5-fold cross-validation because test annotations are not publicly available. Training uses VGG-Style or ResNet-34 backbones, cross-entropy and Dice losses, flipping augmentation, and Adam with initial learning rate 3e-4.Source: §4.1 · pp. 6–7

Analysis of the Network Components

• On Pannuke, MCTrans reached 68.40% DSC versus UNet’s 64.92%; adding TSA raised UNet to 67.93%, while TCA raised baseline and MCTrans to 67.16% and 68.40%.Removing auxiliary loss reduced performance to 67.87%; single-scale Non-local and Transformer-Encoder variants performed far below MCTrans.Source: Analysis of the Network Components · p. 7

Sensitivity to the Setting

• On the small Pannuke dataset, DSC first increased and then decreased as TSA module count grew, while TCA performance peaked at Nc = 4 and then fell.The authors interpret this pattern as indicating that transformer-model capacity is not always better when training on a small dataset.Source: Table 2 · p. 7, Sensitivity to the Setting · p. 8

4.3 Comparisons with State-of-the-art Methods

• MCTrans outperformed top methods on Pannuke with VGG-Style and ResNet-34 feature extractors and on five lesion datasets, demonstrating versatility.The comparisons cover Pannuke and CVC-Clinic, CVC-ColonDB, ETIS-Larib, Kvasir, and ISIC2018.Source: §4.3 · p. 8, Table 3 · p. 8, Table 4 · p. 8
• On Pannuke, MCTrans improved over UNet by 3.64% with almost identical parameters and a slight computation increase.The passage also reports that UNet++ uses much more computation while achieving lower performance than MCTrans.Source: §4.3 · p. 8, Table 3 · p. 8

• The proposed transformer-based network targets medical image segmentation.Source: §5 · p. 9
• Attention mechanisms model rich context and semantic relationships, addressing cross-scale dependencies and correspondence among categories.Source: §5 · p. 9
• The approach is reported effective and outperforms state-of-the-art methods such as TransUnet on several public datasets.Source: §5 · p. 9