Source-linked AI summary
Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)
David Gutman, Noel C. F. Codella, Emre Celebi, Brian Helba, Michael Marchetti, Nabin Mishra, Allan Halpern
TL;DR
Automated melanoma diagnosis from dermoscopic images needs standardized, comparative evaluation, especially given melanoma’s lethality and limitations of unaided inspection. The paper implements a public ISBI 2016 challenge using an ISIC Archive snapshot across segmentation, feature detection, and classification, with 79 submissions from 38 participants. Results covered all three task areas, with segmentation appearing potentially useful for annotation, feature detection promising but needing improvement, and disease recognition within prior expert-dermatologist ranges.
Problem
Melanoma is a lethal skin cancer, while unaided expert visual inspection has about 60% diagnostic accuracy and large-scale centralized comparative evaluation has not been implemented.
Method
The paper implements a public benchmark challenge using an ISIC Archive image snapshot across lesion segmentation, dermoscopic feature detection, and melanoma classification.
Results
79 submissions from 38 participants produced the field’s largest standardized comparative study, with segmentation potentially useful for annotation, promising feature detection, and disease recognition within prior expert-dermatologist ranges.
Takeaways & Limitations
The challenge provides a publicly accessible, coordinated basis for developing and comparing automated melanoma-analysis algorithms.
Abstract
from arXiv · showhide
In this article, we describe the design and implementation of a publicly accessible dermatology image analysis benchmark challenge. The goal of the challenge is to sup- port research and development of algorithms for automated diagnosis of melanoma, a lethal form of skin cancer, from dermoscopic images. The challenge was divided into sub-challenges for each task involved in image analysis, including lesion segmentation, dermoscopic feature detection within a lesion, and classification of melanoma. Training data included 900 images. A separate test dataset of 379 images was provided to measure resultant performance of systems developed with the training data. Ground truth for both training and test sets was generated by a panel of dermoscopic experts. In total, there were 79 submissions from a group of 38 participants, making this the largest standardized and comparative study for melanoma diagnosis in dermoscopic images to date. While the official challenge duration and ranking of participants has concluded, the datasets remain available for further research and development.
I. INTRODUCTION
Melanoma diagnosis remains difficult and important, motivating automated dermoscopic assessment. The challenge addressed the lack of a large-scale, centralized, comparative effort by using an international ISIC image archive snapshot across three analysis stages.
- Over 5 million skin cancer cases were newly diagnosed annually in the United States, while melanoma caused over 10,000 estimated deaths in 2016.
- Unaided expert visual inspection has about 60% diagnostic accuracy, whereas dermoscopy improves recognition by approximately 50% to 75%-84% absolute accuracy when interpreted by experts.No improvement is demonstrated when clinicians lack expertise.
- Growing image availability and dermatologist shortages increase the need for automated tools supporting triage, screening, and evaluation.
- The ISIC Archive provides a publicly accessible, internationally contributed dataset exceeding 10,000 quality- and privacy-screened images with expert-vetted metadata.Images were collected from leading clinical centers using varied devices, and contributors were international.
- The challenge supplied an ISIC Archive snapshot and covered lesion segmentation, dermoscopic feature detection, and lesion classification.
II. CHALLENGE TASKS & DATASET
The challenge organized lesion analysis into three tasks, with dermoscopic feature detection and disease classification each having two variants, producing five active task parts.
- The challenge comprised lesion segmentation, dermoscopic feature detection, and disease classification, with the latter two components each divided into two variants.Teams could participate in five active task parts.
A. Part 1: Lesion Segmentation Task
The lesion segmentation task asked participants to generate binary lesion masks from dermoscopic images using expert-traced boundaries for training and a separate test set for evaluation.
- 900 training images paired each original dermoscopic image with an expert manual lesion-boundary tracing represented as a binary mask.Pixel value 255 indicated lesion interior and 0 indicated exterior.
- Participants submitted automated lesion segmentations as binary masks.
- The example contrasts an original dermoscopic image with its corresponding binary segmentation mask.
- 379 additional images formed the participant evaluation test set.
B. Part 2: Dermoscopic Feature Classification Task
The dermoscopic feature classification task targeted localization and classification of globules and streaks using superpixel-based expert annotations, with a related variant producing binary feature masks.
- Participants automatically detected the clinically defined dermoscopic features “globules” and “streaks.”
- Pattern detection combined feature localization and classification using SLIC superpixels to reduce annotation variability and dimensionality.
- Feature data paired lesion images and superpixel masks with expert annotations indicating feature presence or absence.
- An additional variant required binary masks for each dermoscopic feature to compare another development and evaluation mechanism.
- The classification-task example shows four melanoma images alongside four benign-nevus images.
D. Part 3: Disease Classification Task
Disease classification asked participants to distinguish benign from malignant lesions using normalized confidence scores, with an additional variant supplying the ground-truth lesion segmentation mask.
- Disease Classification Task: Participants classified dermoscopic images as benign or malignant using scores normalized from 0.0 benign to 1.0 malignant.The classification data included definitive malignancy diagnoses and ground-truth lesion segmentations.
- Disease Classification Task: The task provided 900 training images with ground truth and 379 test images for evaluating participant systems.Approximately 30.3% of the dataset was malignant, according to the supplied passage.
- Disease Classification Task: A second classification variant additionally supplied the ground-truth lesion segmentation mask.
A. Segmentation Tasks
Segmentation submissions were evaluated against pixel-level lesion masks using common overlap and classification metrics, with participants ranked by Jaccard.
- Segmentation Tasks: Segmentation submissions were compared using common segmentation metrics.The supplied equations define accuracy and overlap measures used for evaluation.
- Segmentation Tasks: Accuracy was computed from pixel-level true positives, true negatives, false positives, and false negatives.Pixel values above 128 were treated as positive and values below 128 as negative.
- Segmentation Tasks: Dice measured overlap using 2·TP divided by 2·TP + FN + FP.
- Segmentation Tasks: Participants were ranked according to the Jaccard metric.
B. Classification Tasks
Classification submissions were evaluated with image-level diagnostic metrics, ROC and precision-recall measures, and specificity at high-sensitivity thresholds; the challenge also reported top classification and segmentation results.
- Classification Tasks: Image-level classification submissions were compared using accuracy, sensitivity, specificity, AUC, high-sensitivity specificity, and average precision.Specificity was measured at thresholds yielding 95% and 98% sensitivity.
- Classification Tasks: Average precision ranked systems across sensitivity from 0% to 100% using the area under the precision-recall curve.
- Classification Tasks: AUC was computed from the ROC curve by integrating true positive rate with respect to false positive rate.The scikit-learn Python package was used for AUC computation.
- Classification Tasks: Tables I and II reported top evaluation results for the classification and segmentation tasks, respectively.Reported metrics included classification, ROC, high-sensitivity specificity, Dice, and Jaccard measures.
IV. HOSTING PLATFORM
The Covalic grand challenge platform hosted the datasets and automated submission evaluation, participant ranking, and parsing feedback.
- HOSTING PLATFORM: Covalic enabled realtime evaluation of submissions according to defined criteria.
- HOSTING PLATFORM: The platform automatically ranked participants and informed them whether submissions were properly parsed.
- HOSTING PLATFORM: The data remained available on the site for the foreseeable future.
V. RESULTS
The challenge evaluated melanoma-image analysis submissions using standardized whole-image classification metrics and segmentation metrics, with results reported for multiple task parts.
- Participation: 79 submissions came from 38 participants across five active task parts.Submissions included 24 for Part 1, 4 for Part 2, 4 for Part 2B, 25 for Part 3, and 18 for Part 3B.
- Evaluation metrics: Evaluation included accuracy, sensitivity, specificity, average precision, AUC, specificity at 95% and 98% sensitivity, Dice, and Jaccard.Tables I and II reported the top evaluation results using these metrics.
VI. DISCUSSION
The challenge results indicate practical promise for automated lesion analysis while emphasizing that segmentation, feature detection, and disease-recognition methods still require further validation or improvement.
- Findings: Segmentation performance appeared sufficient to support annotation of additional data, pending comparison with expert variability.Inter-observer and intraobserver variability analyses were identified as necessary before establishing statistical equivalence to expert annotation.
- Findings: Dermoscopic feature-detection results appeared promising but still required further improvement.
- Findings: Disease-recognition performance fell within the range previously reported for expert dermatologists.The authors planned direct comparison between automated test-set results and blinded expert dermatologists.
- Contribution: The challenge yielded the largest standardized and comparative study in dermoscopic melanoma diagnosis to date.It evaluated a wide variety of independently submitted approaches at the 2016 International Symposium on Biomedical Imaging.