Source-linked AI summary
U-Net and its variants for medical image segmentation: theory and applications
Nahian Siddique, Paheding Sidike, Colin Elkin, Vijay Devabhaktuni
TL;DR
Medical image analysis needs precise segmentation despite limited labeled data and demanding computational and interpretability constraints. This review surveys U-net architectures, deep-learning innovations, modalities, and applications, finding broad use across medical imaging and continued architectural development. It concludes that U-net-based architectures are valuable and likely to remain a major path in medical image diagnosis despite ongoing challenges.
Problem
Medical image analysis requires precise pixel-level segmentation from scarce labeled data, while deep-learning systems also face computational and black-box challenges.
Method
The review surveys U-net variants, deep-learning methods incorporated into them, and their medical-image modalities and application areas.
Results
U-net is extensively used for medical image segmentation across major modalities and diverse application areas, with recurrent, adversarial, and cascaded variants among the reviewed developments.
Takeaways & Limitations
U-net-based architecture is valuable in medical image analysis and is expected to remain one of the major paths forward despite continuing challenges.
Takeaways & Limitations
Deep-learning medical-image systems remain constrained by training computational power, scarce data, and limited interpretability of black-box internal behavior.
Abstract
from arXiv · showhide
U-net is an image segmentation technique developed primarily for medical image analysis that can precisely segment images using a scarce amount of training data. These traits provide U-net with a very high utility within the medical imaging community and have resulted in extensive adoption of U-net as the primary tool for segmentation tasks in medical imaging. The success of U-net is evident in its widespread use in all major image modalities from CT scans and MRI to X-rays and microscopy. Furthermore, while U-net is largely a segmentation tool, there have been instances of the use of U-net in other applications. As the potential of U-net is still increasing, in this review we look at the various developments that have been made in the U-net architecture and provide observations on recent trends. We examine the various innovations that have been made in deep learning and discuss how these tools facilitate U-net. Furthermore, we look at image modalities and application areas where U-net has been applied.
1. Introduction
U-net is a widely adopted medical-image segmentation architecture addressing the need for pixel-level analysis with limited labeled data. This review surveys its development, applications, and recent use in medical imaging.
- U-net is a neural-network architecture designed primarily for image segmentation in medical image analysis.
- Its contracting and expansion paths combine classification features with localized information and increase output resolution for fully segmented images.
- U-net produces highly detailed segmentation maps from limited labeled samples, using random elastic deformation to learn variations without new annotations.
- A weighted loss function helps separate touching objects of the same class, while context-based learning makes U-net faster to train than most other segmentation models.
- The survey reviews U-net-related medical-image papers published since 2017 and limits its sources to three major publishers.
2. U-net architectures
U-net variants extend the basic encoder–decoder design to volumetric segmentation, attention-guided feature selection, and other architectural adaptations. Across these variants, the review emphasizes how modified operations preserve or refine contextual and localized information.
- 2.1 Base U-net: The base U-net contracts features with convolution and pooling, then upsamples and concatenates cropped encoder maps to produce a higher-resolution segmentation.
- 2.1 Base U-net: The U-shaped architecture propagates contextual information from larger overlapping areas to support object segmentation.
- 2.2 3D U-net: 3D U-net replaces 2D operations with 3D convolutions, pooling, and up-convolutions for volumetric segmentation using few annotated examples.
- 2.3 Attention U-net: Attention U-net places attention gates on expansive-path skip features to trim irrelevant information before concatenation, improving segmentation without much added computational complexity.
- 2.3 Attention U-net: The attention gate combines separately convolved input and gating signals through addition, ReLU, convolution, sigmoid activation, and optional resampling before concatenating the original input.
- 2.3 Attention U-net: Attention U-net has been applied to ocular disease, cancer, abdominal-structure, fetal-development, and brain-tissue segmentation problems.
2.4 Inception U-net
Inception and residual designs adapt U-net to varied feature scales and deeper networks. The reviewed architectures use multi-size or factorized filters and skip connections, with applications across diverse biomedical imaging tasks.
- 2.4 Inception U-net: Inception networks apply filters of multiple sizes within the same layer, concatenate their outputs, and transfer the combined feature maps to the next layer.
- 2.4 Inception U-net: Replacing one 5x5 convolution with two successive 3x3 convolutions makes the operation 2.78 times less computationally expensive while preserving equivalent network function.
- 2.4 Inception U-net: Further factorizing an nxn filter into 1xn and nx1 filters makes a factorized 3x3 filter 33% less expensive.
- 2.4 Inception U-net: Inception U-net configurations have been used for brain tumors, brain tissue, cardiac structures, lung nodules, embryos, and ultrasound nerves.
- 2.5 Residual U-Net: Residual U-net adds skip connections from each block’s input to its second-convolution output before downsampling or upsampling, helping alleviate vanishing gradients.
- 2.5 Residual U-Net: Deep residual U-nets have been applied to nuclei, brain, retinal, cancer, endoscopy, melanoma, bone, and cardiac imaging tasks.
2.6 Recurrent Convolutional Network
Recurrent neural networks add feedback loops that let units update outputs using prior states and contextual information. Recurrent convolutional layers extend this mechanism to U-net feature maps, and a recurrent-residual U-net is reported to outperform related variants.
- Recurrent neural networks use feedback loops so nodes update outputs from previous outputs, creating internal memory for sequential contextual information.
- Recurrent convolutional layers feed filtered feature maps back into their associated layer, allowing updates based on adjoining-unit context.
- The recurrent convolutional output combines feedforward and recurrent inputs through corresponding weights and a bias for each feature map.
- A U-net combining recurrent convolutional layers with residual connections outperformed solely residual and recurrent U-net models and similar-parameter prior methods.
- 2.7 Dense U-net: Dense U-nets replace regular layers with DenseNet blocks, concatenating preceding feature maps to preserve identities and promote gradient propagation.
- 2.8 U-net++: U-net++ inserts intermediary skip-connection blocks between contracting and expansive paths, combining same-level features with upsampled lower-level features.
2.9 Adversarial U-net
Adversarial U-nets combine a U-net generator with a discriminator to learn image transformations against manually transformed examples. The survey describes applications in brain and breast imaging, among others, and contrasts them with broader cascaded and parallel U-net arrangements.
- GANs train a generator and discriminator competitively, with the discriminator distinguishing dataset images from generated outputs and guiding generator updates.
- Adversarial U-nets use a U-net generator that transforms input images, while a discriminator evaluates outputs against manually transformed images.
- After training, the adversarial U-net generator is intended to apply the learned transformation to new images faster than manual physician conversion.
- Adversarial U-nets have been applied to brain quantitative susceptibility mapping, brain-tumor detection, and breast imaging.
- 2.10 Cascaded arrangement: Cascaded U-nets perform successive segmentations, such as isolating the liver before segmenting tumors, and the survey reports better performance than single U-nets.
- 2.11 Parallel arrangements: Parallel U-net arrangements aggregate outputs from multiple networks or encoders, while 2.5D U-net fuses three axis-wise 2D segmentations with reduced computational load versus equivalent 3D networks.
3. Image modalities
U-net-based models are used across major medical imaging modalities for local object delineation, with MRI identified as the most popular modality in the survey. Applications span anatomical structures, tumors, vessels, cells, skin lesions, and pregnancy-related imaging.
- Medical image segmentation outlines and separates local objects, supporting diagnosis that requires careful analysis of regions such as brain tumors.
- MRI: MRI was the most popular U-net segmentation modality in the survey, with extensive applications to brain structures, tumors, cardiovascular anatomy, and other cancers.
- CT: CT applications include liver, lung, bone, and cervical cancers, as well as multiorgan abdominal segmentation.
- Fundus imaging: Fundus imaging uses U-net models to segment retinal blood vessels, supporting analysis of ocular diseases requiring early diagnosis.
- Microscopy: Microscopy applications address overlapping cells and poorly defined boundaries, challenges that U-net has been reported to overcome.
- Dermoscopy and ultrasound: Dermoscopy applications focus mainly on melanoma or skin-cancer analysis, while ultrasound supports noninvasive, real-time, and pregnancy-related imaging tasks.
- X-ray: X-ray applications include bone diseases, pulmonary diseases, coronary imaging, breast tumors, blood vessels, and surgical catheters.
4. Other Canonical Tasks by U-net
Although U-net is primarily a segmentation architecture, the survey reports additional uses in reconstructive image processing, de-aliasing, registration, synthesis, super-resolution, and annotation support.
- U-net has been used to remove imaging artifacts by reconstructing images and to perform de-aliasing.
- Other reported applications include medical-image registration, image synthesis, and image super-resolution.
- U-net has also supported data augmentation intended to make medical-image annotation easier.
5. Discussion
The survey attributes U-net’s high utility to its modular architecture and broad use across medical imaging tasks and modalities. It also identifies computational demands, limited annotations, and interpretability as continuing challenges for deep learning adoption.
- U-net’s modularity allows researchers to incorporate multiple deep learning methods while preserving its high-level design.These adaptations modify the low-level architecture without changing the overall U-net structure.
- The survey found U-net applied across popular and niche application areas, with MR as the most popular modality among reviewed use cases.Other image types remain well represented, and some studies use U-net for alternate tasks beyond segmentation.
- Computational power limits the feasibility of training powerful deep learning algorithms because longer training times increase practical costs.The survey discusses transfer learning and EfficientNet as approaches that may reduce this burden.
- Annotated-data scarcity remains a major training challenge, addressed through random deformation and adversarial models that synthesize additional images.These approaches provide alternatives to collecting new labeled samples.
- Deep learning models remain difficult to interpret because their hidden-layer behavior is not well understood, complicating error correction and reproducibility.The survey links this black-box problem to limited use in large-scale real-world medical trials.
- U-net has also been explored for COVID-19 diagnosis, particularly using chest CT scans.The survey places this work within broader medical-imaging responses to the COVID-19 pandemic.
6. Conclusions
The conclusion presents the survey as a starting point for studying U-net variants and applications across modalities and fields. It characterizes U-net-based architectures as valuable and ground-breaking while acknowledging unresolved deep-learning challenges.
- The survey organizes U-net variants, applications, deep learning methods, image modalities, and application areas for researchers entering the field.Its stated goal is to provide a starting point for exploring U-net in medical image segmentation.
- U-net-based architecture is characterized as valuable in medical image analysis, with growth in papers since 2017 supporting its prominent status.The conclusion also identifies U-net as a major prospective direction despite remaining challenges.
7. Resources
The resources section lists software frameworks, hardware platforms, public benchmarking datasets, and COVID-19 image collections relevant to medical image segmentation research.
- Frameworks and platforms: Listed deep-learning frameworks include TensorFlow, Keras, PyTorch, Caffe, Chainer, and Deeplearning4j.The section also names CNTK, Theano, MXNet, ONNX, Sonnet, PaddlePaddle, and DeepGraphLibrary.
- Frameworks and platforms: NVIDIA CUDA-X AI and Qualcomm mobile platforms are identified as hardware or deployment resources for deep learning.
- Benchmarking datasets: Public segmentation resources cover electron microscopy, retinal fundus images, nuclei microscopy, dermoscopy, brain tumors, MRI, CT, and other medical datasets.Examples include ISBI 2012, DRIVE, Kaggle Data Science Bowl 2018, ISIC, BRATS, Medical Segmentation Decathlon, OASIS, and ABIDE.
- Dataset illustrations: The resource figures illustrate representative samples or ground truths from microscopy, retinal, nuclei, dermoscopy, brain-tumor, chest-X-ray, mammography, and histology datasets.The figures include ISBI 2012, DRIVE, Kaggle Data Science Bowl, ISIC, BRATS, STARE, SCR, DDSM, and PanNuke examples.
- Benchmarking datasets: Additional listed datasets support retinal-vessel, chest-X-ray, mammography, and histology segmentation research.The section names CHASE_DB1, SCR, BCDR, mini-MIAS, and PanNuke.
- Repositories: The listed repositories include the Cancer Imaging Archive and Cardiac Atlas Project for broader cancer and cardiovascular imaging data.
- COVID-19 datasets: Public COVID-19 image resources include COVID-CT, COVID-19 CT, the University of Montreal collection, and the RadiologyAI Consortium.
Conferences:
The section lists conferences spanning artificial intelligence, computer vision, machine learning, robotics, image processing, and medical image computing.
- The listed venues include AAAI, BMVC, CVPR, ECCV, ICCV, ICIP, IROS, ICML, and MICCAI.They represent major publication venues relevant to the survey’s research area.
Journals:
The related-work sources include journals spanning image processing, medical imaging, pattern analysis, computer vision, medical informatics, and medical image analysis.
- The cited journals cover image processing, medical imaging, pattern analysis, computer vision, medical informatics, and medical image analysis.