Source-linked AI summary
Attention Swin U-Net: Cross-Contextual Attention Mechanism for Skin Lesion Segmentation
Ehsan Khodapanah Aghdam, Reza Azad, Maral Zarvani, Dorit Merhof
TL;DR
Skin lesion segmentation is important for melanoma diagnosis, but CNN locality limits global and structural representation in medical images. Att-SwinU-Net extends Swin U-Net with encoder-derived spatial attention and cross-contextual feature recalibration on skip connections. Across three skin lesion segmentation datasets, the authors report improvements over CNN and Transformer-based architectures and smoother masks than comparison methods.
Problem
CNN-based U-Net models are limited by local receptive fields when modeling long-range dependencies and lesion structure in medical images [10] [12].
Method
Att-SwinU-Net enhances Swin U-Net skip connections with encoder-derived spatial attention and cross-contextual attention for feature recalibration.
Results
The proposed method outperforms CNN- and Transformer-based approaches across three skin lesion segmentation datasets and evaluation metrics, with smoother segmentation results than comparison methods.
Takeaways & Limitations
Attention-enhanced skip connections improve the reported performance of the Swin U-Net design for skin lesion segmentation.
Abstract
from arXiv · showhide
Melanoma is caused by the abnormal growth of melanocytes in human skin. Like other cancers, this life-threatening skin cancer can be treated with early diagnosis. To support a diagnosis by automatic skin lesion segmentation, several Fully Convolutional Network (FCN) approaches, specifically the U-Net architecture, have been proposed. The U-Net model with a symmetrical architecture has exhibited superior performance in the segmentation task. However, the locality restriction of the convolutional operation incorporated in the U-Net architecture limits its performance in capturing long-range dependency, which is crucial for the segmentation task in medical images. To address this limitation, recently a Transformer based U-Net architecture that replaces the CNN blocks with the Swin Transformer module has been proposed to capture both local and global representation. In this paper, we propose Att-SwinU-Net, an attention-based Swin U-Net extension, for medical image segmentation. In our design, we seek to enhance the feature re-usability of the network by carefully designing the skip connection path. We argue that the classical concatenation operation utilized in the skip connection path can be further improved by incorporating an attention mechanism. By performing a comprehensive ablation study on several skin lesion segmentation datasets, we demonstrate the effectiveness of our proposed attention mechanism.
1. INTRODUCTION
Skin lesion segmentation supports earlier melanoma diagnosis but remains difficult because dermoscopic images contain varied appearance, boundaries, and artifacts. The paper motivates Att-SwinU-Net by combining Swin Transformer context modeling with attention-enhanced skip connections.
- Clinical motivation: Melanoma survival falls from 99% to 25% when diagnosis occurs at advanced stages, motivating automated lesion segmentation.Dermatologist diagnosis accuracy varies from 24% to 77% with clinician expertise.
- Clinical motivation: Skin lesion segmentation is challenging because dermoscopic images vary in illumination, contrast, texture, position, shape, color, and lesion boundaries, while also containing visual artifacts.
- Related work: U-Net extensions commonly redesign skip connections to improve feature re-usability, including U-Net++ [4], frequency attention UNet, U2-Net, H-DenseUNet, and BCDU-Net.H-DenseUNet uses residual encoding and dense skips, while BCDU-Net incorporates an LSTM module for nonlinear feature representation.
- Related work: CNN-based U-Net approaches have weak global representation because convolution uses local receptive fields, limiting modeling of lesion structure and boundary information [10] [12].
- Related work: Transformer U-shaped models address global context, with TransUNet combining CNN spatial detail and Transformer context [14], followed by pure-transformer Swin U-Net without convolutional operations [16].
- Proposed contribution: Att-SwinU-Net enhances Swin U-Net skip connections by applying encoder-derived attention to informative tokens and introducing cross-contextual attention for feature recalibration.The proposed design applies attention at encoder-decoder scales and avoids extra memory or computation from separately generating attention with a transformer block.
2. PROPOSED METHOD
Att-SwinU-Net extends Swin U-Net with a symmetric encoder-decoder and a two-level attention mechanism on skip connections. It transfers encoder attention for spatial weighting and uses cross-contextual fusion to recalibrate encoder-decoder features.
- Swin U-Net backbone: Att-SwinU-Net uses stacked Swin Transformer blocks in the encoder and decoder, with patch merging for downsampling and patch expanding for upsampling.Three blocks progressively reduce spatial dimensions in the encoder, while the decoder reconstructs the prediction mask symmetrically.
- Swin Transformer block: The Swin block uses successive window-based and shifted-window multi-head self-attention blocks with layer normalization, residual connections, and two-layer MLPs.Relative-position attention uses query, key, and value matrices together with a learned bias matrix B [17].
- Spatial attention: The first skip-connection attention level transfers encoder attention weights into each decoder scale to emphasize spatially important tokens.This reuses attention generated inside the Swin encoder rather than adding a separate attention-producing transformer block.
- Attention module: The proposed module is designed to improve skip-connection feature fusion by combining localization-aware spatial attention with cross-contextual interaction between encoder and decoder features.Figure 1 presents the module as performing spatial and channel attention on the skip connection.
- Cross-contextual attention: The second level averages token representations into a global encoder representation, fuses it with decoder tokens, and applies Swin attention for cross-contextual feature recalibration.The resulting interaction between encoder and decoder series is nonlinear and uses trainable projections W_j, W_k, and W_l.
3. EXPERIMENTS
The experiments evaluate Att-SwinU-Net on three skin-lesion datasets and report improvements over CNN- and Transformer-based approaches, with smoother qualitative segmentations.
- Experimental setup: Att-SwinU-Net was evaluated on ISIC 2017, ISIC 2018, and PH2 datasets using 224 × 224 inputs and a standardized preprocessing protocol.Training used Adam with a 1e-4 learning rate, batch size 24, and 100 epochs.
- Comparative results: The proposed method outperformed CNN- and Transformer-based approaches across all three datasets and evaluation metrics.It also surpassed Transformer-based peers [14] in nearly all skin-segmentation benchmarks.
- Qualitative results: The proposed method produced smoother ISIC2017 segmentations than competing methods, especially TransUNet [14].The qualitative comparison includes the input, ground truth, TransUNet [14], Swin U-Net [16], and the proposed method.
- Qualitative results: The attention mechanism slightly reduced false segmentation results produced by Swin U-Net.This comparison isolates a qualitative benefit attributed to incorporating attention into the proposed architecture.
- Ablation study: The experiments examined attention placement at 1/4, 1/8, and 1/16 resolution scales and varied the number of attention modules in skip connections.The ablation study also considered input sizes, model scales, and the presence of the proposed attention mechanism.
4. CONCLUSION
The conclusion attributes the reported improvements to a two-level attention mechanism added to Swin U-Net and supported by comparative and ablation analyses.
- Conclusion: The proposed two-level attention mechanism integrates encoder attention weights and performs cross-contextual attention to enhance Swin U-Net.The first level highlights important tokens, while the second level recalibrates features through cross-contextual attention.
- Conclusion: The ISIC17 ablation study examines how attention usage in skip connections affects segmentation performance.The study varies attention placement across multiple resolution scales and module counts.
- Conclusion: Comparative results across skin-lesion segmentation architectures support improvements from the proposed attention-based design.The reported evidence includes the performance-comparison table and the paper’s stated improvements over many architectures.