Source-linked AI summary
A Survey on Deep Learning for Skin Lesion Segmentation
Zahra Mirikharaji, Kumar Abhishek, Alceu Bissoto, Catarina Barata, Sandra Avila, Eduardo Valle, M. Emre Celebi, Ghassan Hamarneh
TL;DR
Skin lesion segmentation is important for computer-aided skin-cancer analysis but is challenged by artifacts, lesion variability, acquisition conditions, and annotation limitations. This survey cross-examines 177 deep-learning studies across data, models, and evaluation, synthesizing seminal and systematic perspectives. It highlights annotation variability and future directions while documenting model, data, and evaluation trade-offs.
Problem
Skin lesion segmentation is important for computer-aided diagnosis but is difficult because of artifacts, intrinsic lesion factors, acquisition variation, and limited reliable annotations.
Method
The survey cross-examines 177 deep-learning segmentation papers across input data, model design, evaluation, and future research directions.
Results
The review finds that segmentation research spans diverse data-generation, architecture, loss, annotation, and evaluation choices, with annotation disagreement limiting attainable agreement with ground truth.
Takeaways & Limitations
The survey provides a structured comparison of existing methods and identifies multi-class segmentation, mobile analysis, and broader evaluation measures as future directions.
Takeaways & Limitations
Synthetic-data methods can increase image diversity but may produce lower-quality images and require plausible paired images and segmentation masks.
Abstract
from arXiv · showhide
Skin cancer is a major public health problem that could benefit from computer-aided diagnosis to reduce the burden of this common disease. Skin lesion segmentation from images is an important step toward achieving this goal. However, the presence of natural and artificial artifacts (e.g., hair and air bubbles), intrinsic factors (e.g., lesion shape and contrast), and variations in image acquisition conditions make skin lesion segmentation a challenging task. Recently, various researchers have explored the applicability of deep learning models to skin lesion segmentation. In this survey, we cross-examine 177 research papers that deal with deep learning-based segmentation of skin lesions. We analyze these works along several dimensions, including input data (datasets, preprocessing, and synthetic data generation), model design (architecture, modules, and losses), and evaluation aspects (data annotation requirements and segmentation performance). We discuss these dimensions both from the viewpoint of select seminal works, and from a systematic viewpoint, examining how those choices have influenced current trends, and how their limitations should be addressed. To facilitate comparisons, we summarize all examined works in a comprehensive table as well as an interactive table available online at https://github.com/sfu-mial/skin-lesion-segmentation-survey.
1. Introduction
Skin lesion segmentation is important for automated diagnosis and related clinical applications, but remains difficult because of image artifacts, lesion variability, acquisition differences, and limited annotation reliability.
- Accurate segmentation supports diagnostic criteria such as asymmetry, border irregularity, and lesion size, while also improving some machine-learning classification workflows.It can regularize attention maps, enable lesion cropping, and support lesion-evolution tracking.
- Segmentation is also useful for wide-field images, radiation therapy, and image-guided surgical or robotic lesion excision.
- Manual lesion delineation is laborious and exhibits substantial inter- and intra-observer variability, motivating fast, reliable, automated algorithms.
- Skin cancer is a major public health problem, with melanoma prognosis strongly dependent on early diagnosis.The United States expected 97,610 new melanoma cases in 2023, and five-year survival falls from over 99% to 32% when diagnosis occurs at an advanced stage.
- Automated segmentation is hindered by illumination and contrast issues, artifacts, lesion variability, ambiguous boundaries, and limited expert ground-truth masks.Natural artifacts include hair and blood vessels; artificial artifacts include marker annotations, lens artifacts, and air bubbles.
- Earlier segmentation methods used hand-crafted features with classical image processing and machine learning, while prior surveys focused on pre-deep-learning algorithms.
Survey of surveys.
Earlier surveys covered classical skin-lesion segmentation methods, while later work surveyed segmentation alongside classification but was much narrower in segmentation coverage.
- Adegun and Viriri surveyed both lesion classification and segmentation, resulting in almost an order of magnitude fewer segmentation papers than this review.
Main contributions.
The survey cross-examines 177 deep-learning papers on automated skin-lesion segmentation across data, model design, and evaluation, combining seminal examples with systematic analysis.
- The review covers 177 research papers on automated skin-lesion segmentation in clinical and dermoscopic images.
- It analyzes datasets, preprocessing, synthetic data generation, architectures, modules, losses, annotations, and evaluation metrics.
- The survey examines these choices through selected seminal works and systematic analysis of their influence on trends and limitations.
- All examined works are summarized in a comprehensive comparison table.
Search strategy.
The review searched DBLP and arXiv-related records for deep-learning skin-segmentation publications from 2014–2022, then organized the survey around the segmentation pipeline and future challenges.
- The search covered peer-reviewed papers, conference and workshop proceedings, and non-peer-reviewed preprints published from 2014 to 2022.
- DBLP queries combined deep-learning terms with skin and segmentation-related terms, while arXiv Sanity Preserver was searched with a similar query.
- The authors removed 31 false positives and retained papers related to skin-lesion segmentation.
- The survey is organized by pipeline components, covering datasets and preprocessing, network architectures and losses, evaluation, and future challenges.
2. Input Data
The survey emphasizes that skin-lesion segmentation depends on limited, unevenly annotated datasets, with strong reliance on ISIC resources and increasing use of synthetic data and annotation-noise methods. Dataset overlap, bias, and image quality constrain how broadly results can be interpreted.
- Datasets: Skin-lesion datasets are often small because copyright, privacy, acquisition, annotation, standardization, and scalability issues restrict data collection.
- Datasets: The ISIC Archive is the largest curated repository, containing more than 240,000 images, over 71,000 publicly available at the time of writing.
- Datasets: ISIC 2017 contains 2,750 images across training, validation, and test splits, whereas ISIC 2019 and ISIC 2020 contain 25,331 and 33,126 training images, respectively.
- Datasets: The surveyed ISIC Challenge datasets contain substantial overlap, including 706 images shared across all three 2016–2018 datasets and duplicates within datasets.
- Synthetic Data Generation: GAN-based synthesis can increase image diversity, but generated images may be lower quality and segmentation requires plausible image–mask pairs.
- Annotation and Preprocessing: Pixel-level annotation is costly, motivating methods that learn from noisy annotations, partial labels, or combinations of unreliable and clean masks.
- Image Preprocessing: Preprocessing generally benefits classical models more than properly trained deep-learning models, but may remain useful for small or noisy datasets.
3. Model Design and Training
The model-design background traces skin-lesion segmentation from shallow pixel classifiers to CNNs that learn hierarchical image features and automate feature engineering. Semantic segmentation remains challenging because it must recognize lesion content while preserving precise localization.
- Neural Architectures: Shallow multilayer perceptrons for pixel classification suffered from excessive parameters, limited invariance, and disregard for image structure.
- Neural Architectures: CNNs address these limitations by extracting progressively abstract features from multidimensional signals and automating feature engineering.
- Semantic Segmentation: Semantic segmentation combines recognizing what appears in an image with determining where it appears, requiring both global and well-localized features.
3.1. Architecture
The surveyed architectures span single and multiple networks, hybrid-feature models, and Transformers, with encoder-decoder designs and shortcut connections recurring throughout. Architectural modules trade accuracy, information flow, receptive field, computational cost, and feature representation.
- Model taxonomy: DL segmentation architectures are classified into single-network, multiple-network, hybrid-feature, and Transformer models.Multiple-network models include ensembles, multi-task methods, and GANs; hybrid-feature models combine deep learning with hand-crafted features.
- Single-network models: Single-network models usually use encoder-decoder FCNs, where encoders extract increasingly abstract features and decoders produce segmentation masks.Earlier works adopted FCN or U-Net, with U-Net using a symmetric decoder and corresponding encoder-decoder skip connections.
- Shortcut connections: Skip connections preserve localization and combine multi-scale features, while residual connections provide direct gradient paths that support very deep networks.Skip connections connect high-resolution encoder features to decoder semantics, producing sharper predicted boundaries; residual connections add inputs to block outputs.
- Module frequency: Skip connections appear in 112 surveyed papers, residual connections in 70, attention mechanisms in 41, and Transformers in 12 papers.Dilated convolutions appear in 35 papers and expand receptive fields without additional parameters; Transformer use began appearing from 2021 onward and is increasing.
- Attention modules: Attention modules re-weight feature maps through channel-wise interdependencies, helping networks focus on lesions and suppress irrelevant background responses.The survey reports experimental evidence that attention modules improve representation power and lesion-focused responses.
- Transformer models: FAT-Net combines a dual CNN-Transformer encoder, feature adaptation, and a memory-efficient squeeze-and-excitation decoder, while being reported as more accurate and faster than several CNN-only models.The model is described as using fewer parameters and less computation than several CNN-only alternatives.
3.2. Loss Functions
Loss functions define the optimization objective and encode prior information that guides segmentation learning. The surveyed approaches use pixel-wise, norm-based, adversarial, and metric-related losses, often combining losses to improve generalization.
- Optimization objective: Training minimizes a loss between estimated and ground-truth segmentation masks after fitting model parameters to image-mask pairs.The segmentation function maps an input image to a binary lesion mask, and training seeks parameters that maximize the likelihood of the observed data.
- Optimization objective: Loss choice is critical because it specifies the optimization objective and prior information that constrains the learning search space.The survey notes that many models combine losses to enhance generalization.
- Norm-based losses: Norm-based losses include mean squared error for p = 2 and mean absolute error for p = 1.These are presented as the simplest loss functions in the survey’s taxonomy.
- Hybrid losses: GAN-based segmenters commonly use hybrid losses containing MSE or MAE to regularize generator outputs.MSE is also used as a regularizer for matching attention maps with ground-truth maps.
- Cross-entropy loss: Cross-entropy treats semantic segmentation as pixel-level classification and is widely used for pixel-wise prediction likelihoods.Its gradient gives greater parameter-update weight to hard-to-predict samples, while weighted variants can emphasize lesion boundaries or address class imbalance.
3.2.3. Dice and Jaccard Loss
Dice and Jaccard losses optimize differentiable approximations of overlap metrics used to evaluate segmentation. Their variants and combinations address overlap quality and class imbalance by changing how errors are weighted.
- Overlap-based losses: Soft Dice and soft Jaccard losses provide differentiable objectives directly related to the Dice and Jaccard evaluation metrics.Both metrics measure overlap between predicted and ground-truth segmentations.
- Jaccard variants: Tanimoto distance loss is a modified Jaccard loss used by some skin lesion segmentation models.It is equivalent to Jaccard loss when both the true and predicted labels are binary.
- Tversky loss: Tversky loss addresses class imbalance by weighting false negatives and false positives differently through α and β, with α + β = 1.The parameters control the relative contributions of the two error types.
- Focal combinations: Combining Tversky and focal losses encourages greater focus on hard-to-predict pixels.The focal-loss parameter γ controls the relative importance assigned to hard-to-predict samples.
3.2.4. Matthews Correlation Coe
MCC loss is a metric-based objective that uses correlation between predicted and ground-truth labels. Unlike overlap-based losses, it penalizes background misclassification to better handle skewed class distributions.
- MCC loss: MCC loss is based on the correlation between predicted and ground-truth labels.It is presented as a metric-based loss function for segmentation.
- Class imbalance: MCC loss penalizes false-negative background labels, making it more effective when class distributions are skewed.This distinguishes it from the overlap-based losses discussed in the survey.
3.2.5. Deep Supervision Loss
Deep supervision applies losses to intermediate hidden layers as well as the final output, guiding feature learning and helping address vanishing gradients. The star-shape loss instead imposes radial spatial coherence using prior knowledge about lesion shape.
- Deep supervision guides intermediate feature learning, addresses vanishing gradients, accelerates convergence, and can improve segmentation performance.
- The star-shape loss preserves lesion structure by penalizing discontinuous decisions in predicted masks.
- Pixels between two same-labeled points and the lesion center are encouraged to receive consistent predictions, producing radial spatial coherence.
3.2.7. End-Point Error Loss
End-point error loss focuses on lesion borders by comparing spatial derivatives rather than raw segmentation values. Adversarial training complements pixel-wise supervision by encouraging predicted masks to resemble ground-truth masks.
- End-point error loss emphasizes lesion borders by using first derivatives of segmentation masks instead of raw values.
- Matching the magnitude and orientation of predicted and ground-truth edges helps mitigate vague boundaries.
- Adversarial training combines a pixel-wise loss with an adversarial loss optimized in a mini-max game between segmentation and discriminator parameters.
- Skin lesion adversarial-learning studies have incorporated cross-entropy, soft Jaccard, end-point error, MSE, and MAE losses.
3.2.9. Rank Loss
Rank loss targets hard-to-predict pixels by ranking prediction errors and selecting the largest errors from lesion and background regions. Narrowband suppression loss similarly focuses on difficult pixels near lesion boundaries.
- Rank loss selects the top K background and lesion pixels with the largest prediction errors to encourage more discriminative learning.
- The selected lesion predictions are constrained to exceed selected background predictions by a margin.
- Narrowband suppression loss selects high-error pixels within a radius-r band around the ground-truth lesion boundary rather than across the whole image.
4. Evaluation
Skin lesion segmentation evaluation depends on expert annotations and metric choices, both of which introduce important uncertainty. Multiple expert ground-truths and complementary measures can improve interpretability, but annotation costs and metric differences complicate comparisons.
- Objective evaluation reuses expert ground-truths, whereas subjective grading is more nuanced but usually too laborious for routine use.
- Synthetic images can have ground-truth masks by construction, while real-patient images require human expert delineation.
- Fully manual computer annotation typically has experts trace lesion borders with a mouse or stylus.
- Ground-truth segmentations are estimators of unobserved true labels and can contain systematic bias and substantial noise.
- 12 dermatologists showed approximately 15% average pairwise XOR dissimilarity, exceeding 28% in 10% of cases; more experienced dermatologists agreed more and outlined tighter borders.
- Because expert annotations vary, evaluation should use multiple ground-truths when possible, although annotation costs often force comparison with a single ground-truth.
- Available multi-annotation strategies include selecting one annotation, reporting annotator-specific results, averaging results, or forming ensemble ground-truths.
- Different evaluation measures capture different performance aspects, so no universally applicable metric exists and multiple metrics complicate algorithm comparisons.
5. Discussion and Future Research
The survey finds rapid growth in deep learning research for skin lesion segmentation, yet the field remains constrained by data, evaluation, reproducibility, and model-design challenges. It identifies priorities including more representative annotations, broader evaluation settings, and improved transferability to clinical and diverse-skin-tone images.
- More than three times as many skin lesion segmentation papers appeared during 2015–2022 than during 1998–2014.
- The ISIC 2018 leaderboard’s best thresholded Jaccard index is 83.6%, indicating that skin lesion segmentation remains an open problem.
- Datasets and annotations: Larger, more diverse datasets with multiple manual segmentations per image are needed to train more accurate and robust models.
- Future research: Future work should address coarse CNN segmentations, hyperparameter-tuning burdens, limited clinical and total-body evaluation, diverse skin tones, multiclass segmentation, and largely supervised annotation requirements.
- Reproducibility: Only 38 of 177 surveyed papers, or 21.47%, provided publicly accessible code, limiting reproducibility.
- Clinical images: Clinical-image research remains limited because few benchmark datasets provide expert pixel-level annotations, leaving tools for non-specialists underdeveloped.