Source-linked AI summary
Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)
Noel C. F. Codella, David Gutman, M. Emre Celebi, Brian Helba, Michael A. Marchetti, Stephen W. Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, Allan Halpern
TL;DR
Automated melanoma diagnosis needs stronger shared evidence because prior dermoscopic-image studies often relied on isolated datasets, while melanoma remains difficult to diagnose accurately. The paper presents ISIC’s 2017 public benchmark across segmentation, feature detection, and disease classification using shared datasets and evaluations. The challenge became the largest standardized comparative study in the field, and collaborative fusions outperformed individual systems, although dataset bias and incomplete annotations remained limitations.
Problem
Prior automated dermoscopic-image studies often used isolated datasets unavailable to the broader research community, while melanoma diagnosis remains difficult.
Method
ISIC organized a public benchmark with three tasks—lesion segmentation, dermoscopic feature detection, and disease classification—using shared annotated training, validation, and test datasets.
Results
Collaborative fusions of all participant systems outperformed any single system alone, and the challenge involved 593 registrations, 81 pre-submissions, and 46 finalized submissions.
Takeaways & Limitations
The benchmark established a large standardized comparative study for dermoscopic-image analysis and provides shared data for continued research and development.
Takeaways & Limitations
Dataset bias, incomplete dermoscopic feature annotations, and reliance on single evaluation metrics limited the study.
Abstract
from arXiv · showhide
This article describes the design, implementation, and results of the latest installment of the dermoscopic image analysis benchmark challenge. The goal is to support research and development of algorithms for automated diagnosis of melanoma, the most lethal skin cancer. The challenge was divided into 3 tasks: lesion segmentation, feature detection, and disease classification. Participation involved 593 registrations, 81 pre-submissions, 46 finalized submissions (including a 4-page manuscript), and approximately 50 attendees, making this the largest standardized and comparative study in this field to date. While the official challenge duration and ranking of participants has concluded, the dataset snapshots remain available for further research and development.
1. INTRODUCTION
Melanoma diagnosis remains difficult, while limited public datasets constrain automated dermoscopic-image research. ISIC addressed this gap by creating a large public dataset and a three-task benchmark challenge.
- Over 9,000 people die annually from melanoma, while unaided expert visual inspection achieves only about 60% diagnostic accuracy.
- Dermoscopy raises reported diagnostic accuracy to 75%-84% with proper training, but clinicians often rely on personal experience instead of procedural algorithms.
- Most prior studies used isolated datasets, and an earlier public archive contained only 200 images, limiting representation of the task’s scope.
- ISIC aggregated more than 20,000 publicly accessible dermoscopic images from international clinical centers and multiple devices.
- The benchmark supported automated melanoma-diagnosis research across lesion segmentation, dermoscopic feature detection, and disease classification.
2. DATASET DESCRIPTIONS & TASKS
The 2017 challenge provided shared annotated datasets for three lesion-analysis tasks: segmentation, feature detection, and disease classification. Data were divided into training, validation, and holdout test sets.
- The challenge supplied 2,000 training, 150 validation, and 600 holdout test images for each of three tasks.Validation submissions returned immediate performance and ranking feedback; test feedback arrived after the deadline.
- Lesion Segmentation: Lesion segmentation required binary masks predicting lesion regions from dermoscopic images, using expert boundary tracings as training targets.
- Dermoscopic Feature Classification: Feature classification detected network, negative network, streaks, and milia-like cysts through superpixel-level localization and classification.SLIC superpixels reduced the variability and dimensionality of spatial feature annotations.
- Disease Classification: Disease classification assigned images to melanoma, seborrheic keratosis, or benign nevi, with category scores normalized from 0.0 to 1.0.Approximate age and gender were included when available.
3. EVALUATION METRICS
Evaluation used task-specific metrics and decision thresholds. Classification relied on ROC-based AUC and melanoma specificity at specified sensitivity levels, while segmentation ranking used Jaccard Index.
- Classification confidence above 0.5 counted as positive, while segmentation pixel values above 128 counted as positive.
- Classification performance was evaluated using area under the ROC curve (AUC).
- Melanoma specificity was measured at sensitivities of 82%, 89%, and 95%, representing dermatologist and desired sensitivity levels.
- Segmentation submissions were compared using Jaccard Index, Dice coefficient, and pixel-wise accuracy, with participant ranking based on Jaccard.
4. RESULTS
The 2017 challenge attracted substantial participation across segmentation, feature detection, and disease classification, with results showing strong but uneven performance and benefits from collaborative fusion.
- Challenge participation: 593 registrations produced 81 pre-submissions and 46 finalized submissions, making the challenge the largest standardized comparative study in this field.The associated workshop had approximately 50 attendees.
- Lesion segmentation: 21 final-test and 39 validation submissions were received for lesion segmentation.
- Lesion segmentation: 0.765 average Jaccard Index, 93.4% accuracy, and 0.849 Dice coefficient were achieved by the top segmentation submission.The submission used a variation of a fully convolutional network ensemble.
- Disease classification: Top disease-classification submissions used deep-learning ensembles and additional data, while simpler fusion methods improved overall performance and non-linear SVM fusion decreased it.The top average performer was not best in any single classification category, and collaboration among submissions outperformed any single method alone.
5. DISCUSSION & CONCLUSION
The 2017 challenge was a large standardized benchmark spanning segmentation, feature detection, and disease classification, but its evaluation and dataset design retained important limitations.
- 593 registrations, 81 pre-submissions, and 46 finalized submissions made the challenge the largest standardized and comparative study in this field.
- Average Jaccard Index may not accurately reflect how often automated segmentation falls outside inter-observer variability.
- Poor participation affected dermoscopic feature detection, despite submitted systems achieving reasonable performance.
- Future feature-detection challenges may use segmentation or bounding-box outputs to better align with existing image-detection benchmarks and facilitate participation.
- Ensembles of deep learning approaches and additional data led to the highest disease-classification performance, while collaborative fusions outperformed individual systems.
- Dataset bias, incomplete dermoscopic feature annotations, and reliance on single evaluation metrics limited the study.