Source-linked AI summary
Dual Cross-Attention for Medical Image Segmentation
Gorkem Can Ates, Prasoon Mohan, Emrah Celik
TL;DR
U-Net-based segmentation models face limitations from local convolutions and the semantic gap introduced by skip-connections. DCA strengthens those connections by sequentially applying channel and spatial cross-attention to multi-scale encoder features. Across six architectures and five datasets, DCA improved segmentation performance with small parameter increases.
Problem
U-Net skip-connections can leave a semantic gap between encoder and decoder features, while local convolutions do not directly capture long-range dependencies.
Method
DCA sequentially applies Channel Cross-Attention and Spatial Cross-Attention to fuse multi-scale encoder features before connecting enhanced representations to decoder stages.
Results
DCA significantly improved segmentation performance across six U-Net-based architectures and five benchmark datasets, with Dice Score improvements up to 2.74% on MoNuSeg.
Takeaways & Limitations
DCA provides a lightweight mechanism for strengthening skip-connections in U-Net-based medical image segmentation models.
Abstract
from arXiv · showhide
We propose Dual Cross-Attention (DCA), a simple yet effective attention module that is able to enhance skip-connections in U-Net-based architectures for medical image segmentation. DCA addresses the semantic gap between encoder and decoder features by sequentially capturing channel and spatial dependencies across multi-scale encoder features. First, the Channel Cross-Attention (CCA) extracts global channel-wise dependencies by utilizing cross-attention across channel tokens of multi-scale encoder features. Then, the Spatial Cross-Attention (SCA) module performs cross-attention to capture spatial dependencies across spatial tokens. Finally, these fine-grained encoder features are up-sampled and connected to their corresponding decoder parts to form the skip-connection scheme. Our proposed DCA module can be integrated into any encoder-decoder architecture with skip-connections such as U-Net and its variants. We test our DCA module by integrating it into six U-Net-based architectures such as U-Net, V-Net, R2Unet, ResUnet++, DoubleUnet and MultiResUnet. Our DCA module shows Dice Score improvements up to 2.05% on GlaS, 2.74% on MoNuSeg, 1.37% on CVC-ClinicDB, 1.12% on Kvasir-Seg and 1.44% on Synapse datasets. Our codes are available at: https://github.com/gorkemcanates/Dual-Cross-Attention
1. Introduction
U-Net skip-connections help recover contextual information but remain limited by local convolutions and a semantic gap between encoder and decoder features. DCA addresses these issues by applying sequential channel and spatial cross-attention across multi-scale encoder features before reconnecting them to the decoder.
- U-Net skip-connections concatenate multi-scale encoder features with corresponding decoder features to recover contextual information lost during encoder down-sampling.
- Local convolutions limit the capture of long-range dependencies because they build receptive fields progressively from local kernels.
- Plain skip-connections create a semantic gap between encoder and decoder features, motivating effective fusion of low-level multi-scale encoder features before decoding.
- DCA extracts channel-wise and spatial-wise interdependencies across multi-scale encoder features through sequential cross-attention to narrow this semantic gap.Channel cross-attention captures global channel context, while spatial cross-attention captures global spatial context.
- DCA is designed as a lightweight bridge between encoder and decoder rather than an extensively parameterized end-to-end network.The stated design goal is a powerful skip-connection bridge with only a slight parameter increase.
2. Dual Cross-Attention (DCA)
DCA enhances skip connections by sequentially applying channel and spatial cross-attention to multi-scale encoder features before connecting them to decoder stages.
- DCA overview: DCA takes features from the first n encoder stages, produces enhanced representations, and connects them to corresponding decoder stages.Its design is invariant to the number of encoder stages.
- Patch embedding: Patch embedding extracts flattened patches with a shared patch count P across multi-scale encoder features, enabling cross-attention across tokens.Average pooling and 1×1 depth-wise convolutions are applied to the encoder features.
- Channel Cross-Attention (CCA): Channel Cross-Attention normalizes each token, uses each scale's token as queries, and concatenates multi-scale tokens to form keys and values.The resulting attention captures channel-wise dependencies across encoder features.
- Spatial Cross-Attention (SCA): Spatial Cross-Attention uses concatenated CCA outputs as queries and keys while each scale's output supplies values to capture spatial dependencies.Its projected outputs are further processed before forming DCA outputs.
- Decoder integration: DCA outputs are normalized and activated, then upsampled and passed through 1×1 convolution, batch normalization, and ReLU before decoder connections.Cross-attention fuses multi-scale encoder features rather than treating each encoder stage independently, supporting long-range dependencies between stages.
3. Experiments
Experiments evaluate DCA across five medical image segmentation datasets and six U-Net-based models, comparing plain and DCA-integrated variants under shared settings. DCA improves quantitative and visual segmentation outcomes with small parameter increases.
- Experimental Setup: Experiments cover GlaS, MoNuSeg, CVC-ClinicDB, Kvasir-SEG, and Synapse using six U-Net-based models.The tested models are U-Net, V-Net, R2Unet, ResUnet++, DoubleUnet, and MultiResUnet.
- Experimental Setup: Plain and DCA-integrated models are compared with the same training settings using DSC and IoU metrics.Bold-faced table results denote better performance.
- Quantitative Results: DCA increases parameters by less than 0.7% for ResUnet++, 0.3%–1.5% for U-Net, MultiResUnet, R2Unet, and V-Net, and 3.4% for DoubleUnet.The increase depends on the number of skip-connection layers and model capacity.
- Quantitative Results: Up to 2.05% DSC improvement occurs on GlaS, 2.74% on MoNuSeg, 1.37% on CVC-ClinicDB, 1.12% on Kvasir-Seg, and 1.44% on Synapse.These gains are reported alongside small additional parameter costs.
- Visual Results: Visual comparisons show DCA models producing more consistent boundaries, preserving shape information, and eliminating false-positive predictions relative to plain models.The figure compares predictions across all five datasets.
4. Ablation Study
The ablation study selects the sequential CCA-SCA layout and sequential fusion as the strongest design choices, while 2D average pooling outperforms convolutional patch embedding with fewer parameters.
- DCA Layout: CCA-SCA sequencing improves DSC by 0.79% on GlaS, while SCA-CCA improves it by 0.16%.On MoNuSeg, SCA-CCA performs slightly better than the individual modules, but CCA-SCA remains best overall.
- DCA Layout: The CCA-SCA sequence provides the best performance on both GlaS and MoNuSeg, indicating complementary channel and spatial cross-attention mechanisms.Individual CCA and SCA modules also outperform U-Net on both datasets.
- Fusion of CCA and SCA: Sequential fusion outperforms parallel summation and concatenation fusion for CCA and SCA on both datasets.Summation and concatenation also improve U-Net, but achieve weaker results than sequential fusion.
- Patch Embedding: DCA with convolutional patch embedding performs slightly worse than DCA with 2D average pooling.Convolutional patch embedding adds approximately 260K parameters, whereas average pooling is parameter-free.
- Patch Embedding: 2D average pooling performs better when combined with 1×1 depth-wise convolutional projections.This design uses a parameter-free pooling operation for multi-scale encoder features.
5. Conclusion
DCA strengthens skip-connections in U-Net-based architectures by combining channel and spatial cross-attention to narrow the semantic gap. Experiments across six architectures and five datasets show improved segmentation performance.
- Channel and Spatial Cross-Attention sequentially capture long-range dependencies in channel and spatial dimensions, respectively.Cross-attention fuses low-level multi-scale encoder features into fine-grained representations that narrow the semantic gap.
- Six U-Net-based architectures and five benchmark datasets were used to evaluate DCA.
- DCA significantly improves segmentation performance for models with skip-connections.