Source-linked AI summary
U-Net v2: Rethinking the Skip Connections of U-Net for Medical Image Segmentation
Yaopeng Peng, Milan Sonka, Danny Z. Chen
TL;DR
Medical image segmentation needs skip connections that combine low-level detail with high-level semantics without imposing excessive complexity or resource costs. U-Net v2 uses Hadamard-product semantics-and-detail infusion across encoder levels, and experiments on skin-lesion and polyp datasets report improved performance with preserved efficiency.
Problem
Conventional feature fusion may inadequately combine detailed low-level features with semantic high-level features, while concatenation can introduce noise and increase resource demands.
Method
U-Net v2 refines each encoder-level feature map by infusing higher-level semantics and lower-level details through Hadamard products before decoder processing.
Results
U-Net v2 consistently outperforms state-of-the-art methods on skin-lesion and polyp segmentation while preserving FLOPs and GPU-memory efficiency.
Takeaways & Limitations
The proposed skip connections provide a straightforward, efficient framework that integrates semantic information and fine details across feature levels.
Abstract
from arXiv · showhide
In this paper, we introduce U-Net v2, a new robust and efficient U-Net variant for medical image segmentation. It aims to augment the infusion of semantic information into low-level features while simultaneously refining high-level features with finer details. For an input image, we begin by extracting multi-level features with a deep neural network encoder. Next, we enhance the feature map of each level by infusing semantic information from higher-level features and integrating finer details from lower-level features through Hadamard product. Our novel skip connections empower features of all the levels with enriched semantic characteristics and intricate details. The improved features are subsequently transmitted to the decoder for further processing and segmentation. Our method can be seamlessly integrated into any Encoder-Decoder network. We evaluate our method on several public medical image segmentation datasets for skin lesion segmentation and polyp segmentation, and the experimental results demonstrate the segmentation accuracy of our new method over state-of-the-art methods, while preserving memory and computational efficiency. Code is available at: https://github.com/yaoppeng/U-Net_v2
1. INTRODUCTION
U-Net-based medical segmentation relies on skip connections, but conventional feature fusion can struggle to combine semantic information and fine details efficiently. U-Net v2 introduces straightforward Hadamard-product skip connections that refine multi-level features and improves segmentation across skin-lesion and polyp tasks while preserving efficiency.
- U-Net skip connections preserve semantic information and fine spatial details for precise medical-image boundaries and small structures.
- Low-level features retain details but lack semantics, whereas high-level features provide semantics but lose precise boundaries after resolution reduction.
- Concatenating features across levels can depend heavily on learning capacity, introduce noise, and increase GPU memory usage and FLOPs, especially with limited medical-imaging data.
- U-Net v2 infuses higher-level semantics and lower-level details into each feature map through Hadamard products before decoder reconstruction and segmentation.
- U-Net v2 consistently outperforms state-of-the-art methods on skin-lesion and polyp segmentation while preserving FLOPs and GPU-memory efficiency.
2. METHOD
U-Net v2 uses an encoder, SDI module, and decoder to refine multi-level features before segmentation. The SDI module aligns, smooths, and combines features so each level receives semantic information and fine details.
- 2.1. Overall Architecture: The architecture comprises an encoder, Semantic and Detail Infusion module, and decoder.The encoder generates hierarchical features, the SDI module refines them, and the decoder reconstructs the segmentation.
- 2.2. Semantics and Detail Infusion (SDI) Module: Spatial and channel attention mechanisms are first applied to hierarchical encoder feature maps.These attentions produce processed feature maps for subsequent refinement.
- 2.2. Semantics and Detail Infusion (SDI) Module: A 1 × 1 convolution reduces each processed feature map to c channels before decoder transmission.The resulting feature map is denoted f^2_i and serves as the target reference at decoder level i.
- 2.2. Semantics and Detail Infusion (SDI) Module: Feature maps from different levels are resized to a common resolution and smoothed with a 3 × 3 convolution.Adaptive average pooling, identity mapping, or bilinear interpolation is used according to the relative levels.
- 2.2. Semantics and Detail Infusion (SDI) Module: The SDI module applies an element-wise Hadamard product across resized feature maps, infusing semantic information and finer details into each level.The refined feature map at each level is then dispatched to the corresponding decoder for resolution reconstruction and segmentation.
3. EXPERIMENTS
U-Net v2 is evaluated on skin-lesion and polyp segmentation datasets, where it improves segmentation metrics and maintains computational efficiency. Ablation and qualitative analyses support the contribution of its feature-infusion skip connections.
- Datasets: The experiments cover ISIC 2017, ISIC 2018, and five polyp datasets using established train/test splits.The polyp datasets are Kvasir-SEG, ClinicDB, ColonDB, Endoscene, and ETIS.
- Quantitative Results: 1.44% and 2.48% DSC improvements, plus 2.36% and 3.90% IoU improvements, are reported on ISIC 2017 and ISIC 2018, respectively.These values are reported relative to state-of-the-art comparison methods.
- Quantitative Results: U-Net v2 improves DSC over Poly-PVT by 1.1%, 0.7%, 0.4%, and 0.3% on Kvasir-SEG, ClinicDB, ColonDB, and ETIS, respectively.The cited comparison does not report a corresponding improvement for Endoscene.
- Ablation Study: The SDI module contributes most to overall performance, while removing it reverts U-Net v2 to a vanilla U-Net with a PVT backbone.The ablation compares U-Net v2, U-Net with PVT, and UNet++ using ISIC 2017 and ColonDB.
- Qualitative Results: Qualitative ISIC 2017 examples indicate that semantic and fine-detail infusion helps the model capture finer object-boundary details.The examples use PVT as the encoder for the compared U-Net and UNet++ models.
- Computational Analysis: U-Net v2 has superior FLOPs and FPS to UNet++, while its FPS reduction relative to U-Net with PVT is limited.Complexity measurements use float32 inputs of size (1, 3, 256, 256) on an NVIDIA P100 GPU.
4. CONCLUSIONS
The conclusion presents U-Net v2 as an efficient U-Net variant with skip connections that integrate semantic information and fine details for medical image segmentation.
- Conclusion: U-Net v2 introduces straightforward skip connections that combine higher-level semantics and lower-level details through a Hadamard product.Experiments on skin-lesion and polyp segmentation validate its effectiveness, while complexity analysis indicates efficiency in FLOPs and GPU memory usage.