Source-linked AI summary
UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformer
Haonan Wang, Peng Cao, Jiaqi Wang, Osmar R. Zaiane
TL;DR
U-Net skip connections can inadequately fuse multi-scale encoder and decoder features, with some connections even harming segmentation. UCTransNet replaces them with CTrans, combining channel-wise multi-scale Transformer fusion and cross-attention. The method reports improved segmentation across multiple datasets and architectures, including absolute Dice gains over U-Net on GlaS, MoNuSeg and Synapse.
Problem
Simple U-Net skip connections may mismatch encoder and decoder features, inadequately model global multi-scale context, and sometimes reduce segmentation performance.
Method
UCTransNet replaces U-Net skip connections with CTrans, which combines channel-wise multi-scale fusion through CCT and decoder-guiding cross-attention through CCA.
Results
4.05% Dice, 7.98% Dice and 9.00% Dice are the reported absolute gains over U-Net on GlaS, MoNuSeg and Synapse, respectively.
Takeaways & Limitations
UCTransNet provides a channel-wise alternative to conventional skip connections and improves conventional medical image segmentation pipelines across the reported datasets.
Abstract
from arXiv · showhide
Most recent semantic segmentation methods adopt a U-Net framework with an encoder-decoder architecture. It is still challenging for U-Net with a simple skip connection scheme to model the global multi-scale context: 1) Not each skip connection setting is effective due to the issue of incompatible feature sets of encoder and decoder stage, even some skip connection negatively influence the segmentation performance; 2) The original U-Net is worse than the one without any skip connection on some datasets. Based on our findings, we propose a new segmentation framework, named UCTransNet (with a proposed CTrans module in U-Net), from the channel perspective with attention mechanism. Specifically, the CTrans module is an alternate of the U-Net skip connections, which consists of a sub-module to conduct the multi-scale Channel Cross fusion with Transformer (named CCT) and a sub-module Channel-wise Cross-Attention (named CCA) to guide the fused multi-scale channel-wise information to effectively connect to the decoder features for eliminating the ambiguity. Hence, the proposed connection consisting of the CCT and CCA is able to replace the original skip connection to solve the semantic gaps for an accurate automatic medical image segmentation. The experimental results suggest that our UCTransNet produces more precise segmentation performance and achieves consistent improvements over the state-of-the-art for semantic segmentation across different datasets and conventional architectures involving transformer or U-shaped framework. Code: https://github.com/McGregorWwww/UCTransNet.
Introduction
Medical image segmentation relies on U-Net-style encoder-decoder models, but simple skip connections can fail to bridge semantic and resolution gaps across scales. UCTransNet replaces them with channel-wise Transformer-based fusion and attention, improving segmentation performance across three datasets.
- Motivation: U-Net skip connections restore spatial information but may not model global multi-scale context when encoder and decoder features have semantic gaps.The study reports that some skip connections can negatively affect segmentation, and that effective feature fusion remains necessary.
- Existing approaches: UNet++ narrows encoder-decoder semantic gaps through dense connectivity, while other methods add nonlinear transformations to propagated encoder features.These approaches improve restrictive same-scale skip connections but remain limited in exploring sufficient information from all scales.
- Proposed method: UCTransNet introduces CCT to fuse multi-scale channel-wise features through cross attention and CCA to connect the fused features with decoder features.Together, CCT and CCA form CTrans, which replaces the original U-Net skip connections and addresses inconsistent semantic levels.
- Results: 4.05% Dice, 7.98% Dice and 9.00% Dice are the reported absolute gains over U-Net on GlaS, MoNuSeg and Synapse, respectively.The paper also describes the method as applicable to U-shaped medical segmentation networks with lower computational cost and higher performance than the compared combination.
Related Works
Transformer-based medical segmentation methods use long-range modeling but often modify the encoder or introduce independent Transformer branches. U-Net variants improve skip connections, yet equal contribution across connections is not supported and some connections can harm performance.
- Transformers for Medical Image Segmentation: Transformer-based methods such as TransUNet, MedT and Swin-Unet apply Transformer mechanisms to medical image segmentation, often by modifying U-Net components.The paper notes that these designs may introduce structural redundancy and prohibitive computational cost.
- U-shaped Networks: U-Net-derived models including UNet++, Attention U-Net, DenseUNet, R2U-Net and UNet 3+ extend the original skip-connection mechanism for medical segmentation.These methods are designed to recover fine-grained target details through encoder-decoder feature connections.
- Skip Connections in U-shaped Nets: Different skip connections contribute unequally, and some may harm final segmentation performance.This challenges the assumption that every skip connection makes an equal contribution.
The Analysis of Skip connection
The analysis shows that skip connections are not uniformly beneficial: their effects vary across connections and datasets, motivating alternatives to simple feature copying.
- Finding 1: Skip connections are not always beneficial: U-Net without any skip connection can outperform the original U-Net on some datasets.On GlaS, U-Net-none remains competitive with U-Net-all, whereas it performs worst on almost all metrics for MoNuSeg.
- Finding 2: Simple copying makes skip-connection contributions uneven, with some connections negatively affecting segmentation because encoder and decoder feature sets are incompatible.On MoNuSeg, individual connections range from 67.5% to 76.44% Dice and from 52.2% to 62.73% IOU; L1 underperforms U-Net-none on GlaS.
- Attention comparison: Figure 4 contrasts conventional self-attention with the proposed channel-wise cross-attention.The figure is part of the analysis motivating a channel-oriented alternative to simple skip connections.
- Finding 3: The best combination of skip contributions differs by dataset and depends on lesion scale and appearance.The model without L4 performs best on MoNuSeg, while L3 alone performs best on GlaS.
UCTransNet for Medical Image Segmentation
UCTransNet replaces U-Net’s simple skip connections with a channel-wise Transformer connection that fuses multi-scale encoder features and guides their integration with decoder features. Its CCT and CCA modules use channel-focused cross-attention to model dependencies and reduce semantic inconsistency.
- UCTransNet framework: UCTransNet inserts a channel-wise Transformer between the vanilla U-Net encoder and decoder to address semantic gaps in skip connections.The framework targets inconsistent encoder and decoder features, especially shallow encoder features with less semantic information.
- Channel Cross Fusion Transformer: CCT fuses multi-scale encoder features through feature embedding, multi-head channel-wise cross-attention, and a residual MLP.Tokens from four encoder scales are concatenated as keys and values, while each scale supplies its own queries.
- Channel Cross Fusion Transformer: Unlike original self-attention, CCT performs attention along the channel axis rather than the patch axis and applies instance normalization to similarity maps.The authors state that instance normalization enables smoother gradient propagation.
- Channel-wise Cross Attention: CCA uses Transformer outputs and decoder feature maps as inputs, then recalibrates Transformer channels before concatenating them with up-sampled decoder features.Global average pooling produces channel descriptors, and the resulting channel attention indicates channel importance.
- Evaluation: The method is evaluated through three-times five-fold cross-validation on GlaS and MoNuSeg and compared with state-of-the-art methods on Synapse.The supplied table captions identify Dice and IoU reporting for GlaS and MoNuSeg and Synapse comparisons involving R50-U-Net and R50-AttUNet.
Experiments
Experiments evaluate UCTransNet across three datasets and compare it with U-Net-based and transformer-based segmentation methods. Results consistently favor the proposed CCT and CCA combination, while ablations support multi-scale feature fusion and cross-attention.
- Overall evaluation: UCTransNet is evaluated on GlaS, MoNuSeg, and Synapse using comparisons with three U-Net-based and three transformer-based methods.The experiments use published settings and originally released codes for competing methods.
- Overall evaluation: UCTransNet achieves consistent improvements over prior methods and offers a favorable trade-off between segmentation effectiveness and computational efficiency.The authors also report parameter counts and GFLOPs in the comparison.
- Qualitative results: Qualitative results show fewer confusing false-positive lesions, more coherent boundaries, and finer segmentation while preserving detailed shape information.The comparisons are visualized on GlaS, MoNuSeg, and Synapse.
- Ablation studies: Base+CCT+CCA is generally better than the other ablation baselines across all datasets, supporting the combined use of both proposed modules.The ablation table identifies CCT as Channel Transformer and CCA as Channel-wise Cross Attention.
- Ablation studies: Performance improves as more encoder-level skip connections are used, with multi-scale features improving until four scales.The results support transforming features from more scales into queries and concatenating multi-scale representations as keys.
- Ablation studies: Cross-attention visualizations show that K2 and K3 have more confident correlations with other encoder levels on GlaS and MoNuSeg, respectively.These findings are consistent with the paper’s skip-connection analysis.
Conclusion
UCTransNet combines channel-wise multi-scale fusion with cross-attention to improve medical image segmentation. The authors report that it narrows the encoder-decoder semantic gap and uses multi-scale encoder features effectively.
- Conclusion: UCTransNet combines multi-scale Channel-wise Cross fusion Transformer and Channel-wise Cross-Attention modules in an end-to-end segmentation network.The proposed design is presented as a channel-wise alternative for connecting encoder and decoder features.
- Conclusion: The method significantly improves state-of-the-art medical image segmentation results across multiple benchmark datasets.The conclusion attributes the reported improvements to the proposed UCTransNet model.
- Conclusion: UCTransNet narrows the semantic gap and takes full advantage of multi-scale features in the encoding stage.These conclusions are supported by the paper’s empirical analysis.