Source-linked AI summary
Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)
Noel Codella, Veronica Rotemberg, Philipp Tschandl, M. Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, Harald Kittler, Allan Halpern
TL;DR
The paper examines how public benchmarks can evaluate skin-image algorithms under clinically relevant annotation, prevalence, and domain-shift conditions. It summarizes the 2018 ISIC challenge across three tasks using revised metrics and held-out institutional data. The results show that segmentation failures remain common, evaluation choices alter rankings, and test performance does not fully determine generalization.
Problem
Existing skin-image benchmarks needed evaluation protocols that better represented clinical variability, prevalence imbalance, and performance beyond training institutions.
Method
The 2018 ISIC Challenge evaluated lesion segmentation, attribute detection, and disease classification using revised metrics and dual-partition held-out test data.
Results
Top segmentation submissions achieved Thresholded Jaccard 0.802, while Thresholded Jaccard, balanced accuracy, and multipartition testing changed failure assessment, participant rankings, and generalization comparisons.
Takeaways & Limitations
Future challenges and regulators should consider these criteria when quantifying algorithm performance, robustness, and generalization in clinical scenarios.
Takeaways & Limitations
Ideal segmentation evaluation using multiple annotators and software systems was impractical because it would multiply manual ground-truth labor.
Abstract
from arXiv · showhide
This work summarizes the results of the largest skin image analysis challenge in the world, hosted by the International Skin Imaging Collaboration (ISIC), a global partnership that has organized the world's largest public repository of dermoscopic images of skin. The challenge was hosted in 2018 at the Medical Image Computing and Computer Assisted Intervention (MICCAI) conference in Granada, Spain. The dataset included over 12,500 images across 3 tasks. 900 users registered for data download, 115 submitted to the lesion segmentation task, 25 submitted to the lesion attribute detection task, and 159 submitted to the disease classification task. Novel evaluation protocols were established, including a new test for segmentation algorithm performance, and a test for algorithm ability to generalize. Results show that top segmentation algorithms still fail on over 10% of images on average, and algorithms with equal performance on test data can have different abilities to generalize. This is an important consideration for agencies regulating the growing set of machine learning tools in the healthcare domain, and sets a new standard for future public challenges in healthcare.
1 Introduction
The 2018 ISIC Challenge addressed the need for more accurate, scalable melanoma-detection methods through a larger, clinically oriented benchmark with revised evaluation and generalization testing.
- Melanoma’s 5-year survival can reach 99% with early detection but falls to 23% after delayed diagnosis.
- ISIC’s earlier public benchmarks attracted over 900 registrations and over 350 submissions, establishing large standardized comparisons.
- The 2018 challenge expanded dataset and diagnostic-label sizes while redesigning evaluation to better reflect clinical complexity.
- Its protocol introduced Thresholded Jaccard for segmentation, balanced accuracy for classification, and external institutional test data for generalization.
2 Methods
The challenge used three image-analysis tasks with task-specific datasets and evaluation criteria designed to address annotation variability, attribute sparsity, prevalence imbalance, and generalization.
- Challenge design: The challenge comprised lesion segmentation, attribute detection, and disease classification, with participants allowed to share data or analytics across tasks.
- Part 1: Lesion Segmentation: 2,594 dermoscopic images with segmentation masks supported Part 1 training, with 100 validation and 1,000 test images lacking ground-truth masks.
- Part 1: Lesion Segmentation: Thresholded Jaccard sets scores below threshold T to zero, marking segmentations below the chosen correctness boundary as failures.
- Part 1: Lesion Segmentation: T=0.65 was chosen from prior inter-observer agreement values to increase certainty that a segmentation was considered incorrect.
- Part 2: Lesion Attribute Detection: 2,594 images and 12,970 masks for five attributes supported Part 2 training, while dataset-level TP, FP, and FN calculations handled absent attributes.
- Part 3: Disease Classification: Part 3 trained on 10,015 images with seven classification labels and evaluated held-out data using balanced accuracy across internal and external partitions.
3 Results
Across the three challenge tasks, segmentation exposed failures hidden by aggregate scores, attribute detection remained weak, and classification revealed substantial differences in generalization and ranking depending on evaluation choices.
- Lesion Segmentation: 112 submissions were evaluated for segmentation, with the top Thresholded Jaccard reaching 0.802.Many other leading algorithms also achieved approximately 0.8.
- Lesion Segmentation: Over 10% of images remained segmentation failures for most methods despite average Jaccard values exceeding 0.8.Failures were most common for seborrheic keratoses.
- Lesion Segmentation: Thresholded Jaccard tracked segmentation-failure proportion more closely than Jaccard and changed participant algorithm rankings.Its correlation slope with failure proportion was closer to 1, suggesting better assessment of clinical utility.
- Lesion Attribute Detection: 26 submissions evaluated lesion attributes, but the best average Jaccard was only 0.473.The paper notes that poor inter-observer correlation among expert clinicians may contribute to the low performance.
- Lesion Disease Classification: 141 classification submissions achieved a highest balanced accuracy of 0.885, while internal and external test performance exposed overfitting and differing generalization.Some top-performing approaches did not overfit, and similar whole-test performance could accompany widely different generalization ability.
- Lesion Disease Classification: Balanced accuracy, accuracy, and mean AUC produced different participant rankings, making evaluation-metric choice consequential for classifier selection.The paper identifies balanced accuracy as critical for selecting an unbiased classifier rather than one fitted to arbitrary dataset prevalence.
4 Discussion & Conclusion
The 2018 ISIC Challenge expanded benchmark scale and introduced evaluation criteria designed to better reflect clinical difficulties, robustness, and generalization. Results show that these criteria reveal segmentation failures, ranking changes, and differences in algorithm generalization.
- Over 12,500 training images and over 2,000 test images supported three tasks, with 900 teams registering and 299 submissions received.
- Thresholded Jaccard better captures the proportion of segmentation failures than standard Jaccard.
- Balanced accuracy significantly changes participant rankings relative to metrics more vulnerable to class imbalance or clinically irrelevant ROC regions.
- Multipartition held-out tests containing data absent from training effectively differentiate algorithms by their ability to generalize.
- Poor performance in lesion attribute detection may indicate that dermoscopic attributes require further maturation before additional machine-learning research.
- Future challenges and regulatory agencies should consider these evaluation criteria when quantifying algorithm performance, robustness, and clinical generalization.
5 Supplementary Material
Supplementary figures and tables detail submission distributions, disease-specific performance, ROC behavior, and top-performing entries across segmentation and classification.
- Supplementary Material: Submission histograms for lesion segmentation show performance distributions by disease state, with averages and standard deviations marked.
- Supplementary Material: Table 1 lists the top five lesion-segmentation submissions using overall, disease-specific, failure-rate, Thresholded Jaccard, and Jaccard results.
- Supplementary Material: Table 2 lists the top five lesion-classification submissions across overall, internal, external, accuracy, balanced accuracy, and disease-specific results.
- Supplementary Material: ROC plots display all seven disease states for the twelve highest-performing submissions, with separate whole-test, internal, and external curves.
- Supplementary Material: Disease-specific classification histograms compare AUC distributions across the entire, internal, and external test partitions and their performance differences.