Source-linked AI summary
Sharp U-Net: Depthwise Convolutional Network for Biomedical Image Segmentation
Hasib Zunair, A. Ben Hamza
TL;DR
Biomedical segmentation requires accurate pixel-level labeling, but U-Net skip connections can fuse semantically different features, producing blurred maps and segmentation errors. Sharp U-Net inserts depthwise sharpening convolutions before encoder-decoder fusion, and experiments report consistent advantages over baselines without extra learnable parameters. The approach also outperforms baselines with more than three times as many learnable parameters.
Problem
U-Net skip connections can fuse semantically different low- and high-level features, causing blurred feature maps and under- or over-segmented regions.
Method
Sharp U-Net replaces plain skip connections with depthwise convolution of encoder feature maps using a sharpening spatial filter before decoder fusion.
Results
Sharp U-Net consistently outperforms or matches state-of-the-art baselines across binary and multi-class biomedical segmentation benchmarks without adding learnable parameters.
Takeaways & Limitations
The architecture provides a segmentation framework for binary and multi-class medical images across multiple modalities, including EM, endoscopy, dermoscopy, nuclei, and CT.
Takeaways & Limitations
Experiments use no data augmentation or post-processing, including conditional random fields or median filtering.
Abstract
from arXiv · showhide
The U-Net architecture, built upon the fully convolutional network, has proven to be effective in biomedical image segmentation. However, U-Net applies skip connections to merge semantically different low- and high-level convolutional features, resulting in not only blurred feature maps, but also over- and under-segmented target regions. To address these limitations, we propose a simple, yet effective end-to-end depthwise encoder-decoder fully convolutional network architecture, called Sharp U-Net, for binary and multi-class biomedical image segmentation. The key rationale of Sharp U-Net is that instead of applying a plain skip connection, a depthwise convolution of the encoder feature map with a sharpening kernel filter is employed prior to merging the encoder and decoder features, thereby producing a sharpened intermediate feature map of the same size as the encoder map. Using this sharpening filter layer, we are able to not only fuse semantically less dissimilar features, but also to smooth out artifacts throughout the network layers during the early stages of training. Our extensive experiments on six datasets show that the proposed Sharp U-Net model consistently outperforms or matches the recent state-of-the-art baselines in both binary and multi-class segmentation tasks, while adding no extra learnable parameters. Furthermore, Sharp U-Net outperforms baselines that have more than three times the number of learnable parameters.
1 Introduction
Biomedical image segmentation labels pixels to distinguish regions of interest and supports clinical applications. Sharp U-Net addresses semantic mismatch in U-Net feature fusion with sharpening filters and reports strong performance across segmentation settings.
- Motivation: Semantic segmentation classifies each biomedical-image pixel into predefined categories to distinguish regions such as tumors or organs.The task supports clinical applications including COVID-19 detection in CT images.
- U-Net background: U-Net uses an encoder for contextual representation and a decoder for precise localization, connected by feature-fusing skip connections.These connections combine deep, coarse decoder features with shallow, fine-grained encoder features.
- Problem: Semantic mismatch between encoder and decoder features can blur feature maps and contribute to under- or over-segmented regions.The mismatch arises when semantically different low- and high-level features are fused during learning.
- Proposed approach: Sharp U-Net applies depthwise convolution with a sharpening spatial filter to encoder feature maps before fusing them with decoder features.The design targets reduced feature mismatch, improved feature fusion, and artifact smoothing during early training.
2 Related Work
Related U-Net variants improve biomedical segmentation, but the semantic gap between encoder and decoder features remains unresolved. Sharp U-Net replaces ordinary skip connections with sharpening spatial filters to mitigate this mismatch.
- U-Net variants: Recent U-Net variants have pursued uniform network scaling and ImageNet-pretrained CNN encoders to improve biomedical segmentation.Wide UNet increases filters in both encoder and decoder subnetworks.
- U-Net variants: Ki-Net addresses performance drops on smaller anatomical structures with blurred noisy boundaries by projecting data into higher dimensions.When combined with U-Net, it reportedly improves segmentation while using fewer parameters.
- Remaining challenge: Despite improved results, prior U-Net variants leave the large semantic gap between encoder and decoder features as an open challenge.The unresolved gap concerns low- and high-level feature incompatibility.
- Sharp U-Net distinction: Sharp U-Net differs from prior work by replacing skip connections with sharpening spatial filters to mitigate encoder-decoder feature mismatch.The approach is described as applicable to any encoder-decoder network.
3 Proposed Method
Sharp U-Net modifies U-Net’s encoder-decoder connections by sharpening encoder feature maps before fusion, aiming to reduce semantic mismatch and preserve fine details. Its sharp blocks use depthwise spatial filtering to produce same-sized maps for feature fusion in binary and multi-class segmentation.
- Sharp U-Net architecture: The encoder captures context through downsampling, while the decoder performs upsampling for precise localization and outputs a pixel-by-pixel class mask.The encoder uses convolution and max pooling; the decoder uses transposed convolutions and convolutional layers.
- Sharp U-Net architecture: Sharp U-Net replaces plain skip connections with sharp blocks that sharpen encoder features before fusing them with decoder features.The design targets the semantic gap between low-level encoder and high-level decoder representations.
- Sharpening spatial kernel: A sharpening spatial filter emphasizes high-frequency details by responding to intensity transitions through Laplacian-based convolution.The kernel has a positive center and negative off-center values, increasing center brightness relative to neighbors; its elements sum to zero.
- Sharp blocks: Depthwise convolution applies the sharpening kernel independently to each of M encoder feature maps, producing M same-sized maps for fusion.Padding preserves dimensions, yielding an output of size W × H × M.
- Scaling and training: Sharp U-Net can be scaled by increasing convolutional kernels or using deep convolutional encoders, and it is trained end-to-end with categorical cross-entropy for multi-class segmentation.The parameters are optimized across all samples, with Adam and early stopping used during training.
4 Experiments
Experiments across six biomedical-image datasets evaluate Sharp U-Net for binary and multi-class segmentation against U-Net and stronger baselines. Sharp U-Net generally improves performance, converges faster, and produces smoother, less noisy predictions without adding learnable parameters.
- Model validation performance: Sharp U-Net achieves higher validation Jaccard scores than U-Net at the same epoch count across all datasets and converges faster in some cases.The authors attribute this behavior to preventing noise propagation during early training; on CVC-ClinicDB, Sharp U-Net converges faster despite fluctuations in both models.
- Sharp U-Net outperforms U-Net: 2.52% improvement on COVID-19 CT Segmentation demonstrates that Sharp U-Net also benefits multi-class segmentation.Qualitative results show smoother predictions than U-Net.
- Scaling Sharp U-Net improves performance: 2.63% relative Jaccard improvement results when sharp blocks replace skip connections in Wide U-Net on CVC-ClinicDB.Wide Sharp U-Net also has lower standard deviation than Wide U-Net, 1.8024 versus 3.4125, while using 7.8 versus 9.1 million learnable parameters.
- Pretrained and deeper encoders: Sharp blocks reduce variability relative to traditional skip connections, with standard deviation falling from 3.6895 to 0.8090 in one comparison.Comparable performance with lower variability is also reported against VGG Sharp U-Net, which uses three times more parameters than Sharp U-Net.
- Qualitative results: On difficult images with artifacts or similar foreground and background, Sharp U-Net produces less noise and fewer fractured or non-ROI regions than U-Net.The authors also report improved handling of over-segmentation and under-segmentation cases.
- Sharp U-Net outperforms U-Net: Sharp U-Net outperforms U-Net across six datasets, with relative Jaccard improvements from 0.48% to 12.6% and no additional learnable parameters.The largest gains occur on CVC-ClinicDB and ISBI-2012, while performance on other datasets is matched or modestly improved.
5 Conclusion
Sharp U-Net is an encoder-decoder depthwise fully convolutional network for binary and multi-class biomedical image segmentation. It reduces feature mismatch through sharpening before fusion and achieves strong benchmark performance without extra learnable parameters.
- Sharp U-Net targets binary and multi-class biomedical image segmentation with a depthwise fully convolutional encoder-decoder architecture.
- The method sharpens encoder feature maps before fusion, making encoder and decoder features semantically less dissimilar.
- The sharpening operation also smooths artifacts throughout network layers during early training with untrained parameters.
- Sharp U-Net consistently outperforms or matches state-of-the-art baselines across biomedical segmentation benchmarks and modalities.
- The architecture improves performance without extra learnable parameters and can outperform baselines with three times as many learnable parameters.