Source-linked AI summary
IMA++: ISIC Archive Multi-Annotator Dermoscopic Skin Lesion Segmentation Dataset
Kumar Abhishek, Jeremy Kawahara, Ghassan Hamarneh
TL;DR
Large public multi-annotator datasets for dermoscopic skin lesion segmentation are lacking, despite the importance of segmentation and the challenges of ambiguous lesion boundaries. The authors introduce IMA++, a dataset of ISIC Archive images with masks and annotator-related metadata. It contains 17,684 masks across 14,967 images, including 2,394 images with multiple segmentations, and provides consensus masks and dataset analyses.
Problem
Public skin lesion segmentation datasets largely lack multiple annotations per dermoscopic image, limiting the available data for studying annotator-specific segmentation tasks.
Source: §I · p. 2Method
The authors collect and organize ISIC Archive images, segmentation masks, and segmentation metadata into IMA++, and include masks from two consensus algorithms.
Source: §I · p. 2, A. Data Collection · p. 3Results
IMA++ contains 17,684 segmentation masks across 14,967 images, including 2,394 images with multiple segmentations, and is the largest publicly available skin lesion segmentation dataset.
Source: §I · p. 2, C. IMA++ versus other datasets · p. 8Takeaways & Limitations
IMA++ supports research on annotator-specific preferences, consensus modeling, segmentation styles, and analysis of tool- and skill-related segmentation variability.
Source: §I · p. 2
Abstract
from arXiv · showhide
Multi-annotator medical image segmentation is an important research problem, but requires annotated datasets that are expensive to collect. Dermoscopic skin lesion imaging allows human experts and AI systems to observe morphological structures otherwise not discernable from regular clinical photographs. However, currently there are no large-scale publicly available multi-annotator skin lesion segmentation (SLS) datasets with annotator-labels for dermoscopic skin lesion imaging. We introduce ISIC MultiAnnot++, a large public multi-annotator skin lesion segmentation dataset for images from the ISIC Archive. The final dataset contains 17,684 segmentation masks spanning 14,967 dermoscopic images, where 2,394 dermoscopic images have 2-5 segmentations per image, making it the largest publicly available SLS dataset. Further, metadata about the segmentation, including the annotators' skill level and segmentation tool, is included, enabling research on topics such as annotator-specific preference modeling for segmentation and annotator metadata analysis. We provide an analysis on the characteristics of this dataset, curated data partitions, and consensus segmentation masks.
I. BACKGROUND
Skin lesion segmentation is important for automated skin-image analysis, but ambiguous boundaries and other image factors make ground-truth annotation challenging. IMA++ addresses the shortage of large multi-annotator dermoscopic datasets with a large collection that reflects varied annotators, tools, and skill levels.
I. BACKGROUND
- Skin lesion segmentation supports automated image analysis, but artifacts, variable lesion appearance, skin tones, contrast, illumination, and ambiguous boundaries complicate annotation.Segmentation serves as an endpoint, an intermediate task, or a benchmark in deep-learning skin-image analysis.Source: §I · pp. 1–2
- Existing public skin lesion segmentation datasets generally provide one mask per image; ISIC 2019-Seg has three masks for each of only 100 images.The limited scale of multi-annotator data constrains work on annotator-specific tasks.Source: §I · p. 2
- IMA++ contains 14,967 images and 17,684 segmentation masks, including multiple annotations for 2,394 images, making it the largest publicly available skin lesion segmentation dataset.The dataset also includes consensus masks produced by two algorithms, bringing the total number of segmentations to 22,472.Source: §I · p. 2
- IMA++ captures segmentation-style variation associated with annotator preferences, tools, and manual reviewers’ skill levels.Tool and skill metadata support exploring how these factors relate to segmentation variability.Source: §I · p. 2
- IMA++ models a realistic incomplete image–annotator graph: each image has at least one annotator, but not every image is labeled by every annotator.Other multi-annotator datasets cited in the paper use a complete bipartite setup, with every image labeled by every annotator.Source: §I · p. 2
- IMA++ supports research on segmentation preferences, consensus modeling, segmentation distributions and styles, inter-annotator agreement, and multimodal or multitask skin-image analysis.Subsets have also been used to study segmentation-style discovery without annotator-mask correspondence and the association between agreement and malignancy.Source: §I · pp. 2–3
II. COLLECTION METHODS AND DESIGN
The authors assembled IMA++ from ISIC Archive images, segmentations, and associated metadata. Because current ISIC access tools do not expose segmentations, they combined previously downloaded masks with metadata obtained from sources, then anonymized and organized the collection.
A. Data Collection
- The authors combined previously downloaded segmentations with metadata obtained from sources, then anonymized and organized the materials to build IMA++.The current ISIC API v2, CLI, and web interface do not provide access to the corresponding segmentations or segmentation metadata.Source: A. Data Collection · p. 3
- The images and raw masks were originally obtained through the now-deprecated ISIC API v1.The authors provide collection, processing, and analysis scripts and curated a dedicated ISIC Archive image collection.Source: A. Data Collection · p. 3
- The authors gathered segmentation metadata from sources because current ISIC access tools do not expose it alongside the images.The scripts used for data collection, processing, and analysis are available on GitHub.Source: A. Data Collection · p. 3
B. Data Filtering
The authors filter unusable masks and images without metadata, then organize the retained annotations by annotator, tool, and reviewer skill level. They report uneven contributions and describe the limits of available reviewer information.
B. Data Filtering
- Quality checks flag empty, full-image, disconnected-region, and border-touching masks, but only empty masks are removed.The checks found 59 empty masks, 3 full-image masks, 0 disconnected-region masks, and 1,129 border-touching masks.Source: B. Data Filtering · p. 3
- After removing empty masks and 59 images without associated metadata, IMA++ retained 14,967 images with 17,684 segmentation masks in total.Of the retained images, 12,573 have one segmentation mask each and the remaining 2,394 have multiple masks.Source: B. Data Filtering · p. 3
C. Data Processing
- Masks are labeled by annotator, drawing tool, and binary reviewer skill level; finer-grained experience information was unavailable.T1 is manual polygon tracing, T2 is semi-automated flood-fill, and T3 is automated segmentation followed by human review.Source: C. Data Processing · pp. 3–4
- The sample figure pairs images with two to five original masks and shows majority-voting and STAPLE consensus masks.Two examples are shown for each segmentation count.Source: Figure 2 · p. 4
- Six annotators contribute approximately 78% of masks, while tools T1 and T3 account for approximately 81%.About 70% of segmentations were manually reviewed by an S1-level reviewer.Source: C. Data Processing · pp. 4, 6
- Agreement is analyzed across annotator, tool, and skill-level factors using mean Dice and 95th-percentile Hausdorff distance.Combinations absent from the dataset are grayed out in the figure.Source: C. Data Processing · p. 4, Figure 4 · p. 5
D. Computing Segmentation Consensus
The authors compute two consensus masks for images with multiple annotations, providing alternatives to the individual masks for analysis and use.
D. Computing Segmentation Consensus
- IMA++ provides majority-voting and STAPLE consensus masks for its 2,394 images with multiple annotations.The consensus masks support comparison with original annotations and use in SLS training and evaluation without a multi-annotator setup.Source: D. Computing Segmentation Consensus · p. 6
- The sample figure shows consensus outputs for images with two, three, four, and five segmentations.Both majority voting and STAPLE outputs are shown alongside the annotation examples.Source: D. Computing Segmentation Consensus · p. 6
E. Data Splits
The authors split the 2,394 multi-annotated images into standardized training, validation, and test partitions, stratifying by annotation count and agreement level.
E. Data Splits
- The training, validation, and test partitions also preserve similar proportions of images with two, three, four, or five masks.The respective counts are (1,493, 144, 37, 1), (214, 19, 6, 1), and (423, 46, 8, 2).Source: E. Data Splits · p. 6
- 1,675 training, 240 validation, and 479 test images form the 70:10:20 split of the multi-annotated subset.The splits are stratified by segmentation count per image and inter-annotator agreement level.Source: E. Data Splits · p. 6
- Images are stratified into low, medium, or high agreement using averaged pairwise Dice ranges of [0, 0.5), [0.5, 0.8], and (0.8, 1.0].The resulting low/medium/high counts are 165/403/1,107 in training, 24/58/158 in validation, and 47/115/317 in testing.Source: E. Data Splits · p. 6
III. VALIDATION AND QUALITY
The authors use an UpSet plot to characterize annotator overlap, finding many distinct intersections in the incomplete annotation setup.
A. Visualizing annotator overlap
- The UpSet plot encodes annotators as rows and their intersection sets as columns, with bars showing intersection counts.Filled dots mark which annotators belong to each set, and connected dots identify multi-annotator intersections.Source: A. Visualizing annotator overlap · p. 7
- Annotator A00 contributes 3,286 masks across the nine sets in which its segmentations appear.The A00-only set contains 2,319 masks.Source: A. Visualizing annotator overlap · p. 7
- 57 distinct annotator intersections occur in IMA++, compared with one possible intersection under complete bipartite annotation.With 16 annotators, 2^16 − 1 non-empty intersections are possible.Source: A. Visualizing annotator overlap · p. 7
B. Analyzing inter- and intra-factor agreement in IMA++
Agreement varies across annotators and tools, including low-agreement pairs and limited agreement between consensus methods. The analysis compares segmentation masks using Dice and HD95.
B. Analyzing inter- and intra-factor agreement in IMA++
- Dice and HD95 distributions are reported for annotator combinations that segmented at least 50 images.The figure covers all qualifying combinations.Source: Figure 6 · p. 7
- Annotators show markedly different agreement patterns: A01 and A03 have lower intra-annotator agreement, while A08 agrees poorly with others and consensus masks.Some pairs have unexpectedly low or high agreement; majority voting and STAPLE also do not agree to a very high degree.Source: B. Analyzing inter- and intra-factor… · p. 8
- Tools T1 and T3 agree more with each other than with themselves or T2, whereas T2 has higher intra-tool than inter-tool agreement.The passage notes a similar pattern for tools and skill levels, without specifying the skill-level results.Source: B. Analyzing inter- and intra-factor… · p. 8
C. IMA++ versus other datasets
IMA++ is larger than the other publicly available skin lesion segmentation datasets compared here, while most of its images and masks are new relative to previous ISIC challenges.
C. IMA++ versus other datasets
- ISIC 2019-Seg has 100 images and 300 masks, while HAM10000 has 10,015 images with one segmentation per image.ISIC 2019-Seg is described as the only other dataset in the comparison with multiple annotations per image.Source: C. IMA++ versus other datasets · p. 8
- IMA++ is the largest publicly available skin lesion segmentation dataset across the dermoscopic and clinical modalities, with 14,967 images and 17,684 segmentations.The cited comparison includes single-annotator and multi-annotator datasets.Source: C. IMA++ versus other datasets · p. 8
- 11,081 of IMA++’s 14,967 images (about 74%) are new, and their masks differ from those in previous ISIC challenge datasets.Some IMA++ images also appear in previously released ISIC datasets.Source: C. IMA++ versus other datasets · p. 8
IV. RECORDS AND STORAGE
The repository provides segmentation masks and rich linked image- and segmentation-level metadata, with completeness varying substantially across image-level fields.
IV. RECORDS AND STORAGE
- The repository contains 22,472 masks, including 17,684 annotator masks and majority-voting and STAPLE consensus masks for multiply segmented images.The associated images are downloadable from the ISIC Archive rather than included in the repository.Source: §IV · p. 8
- Image metadata includes patient and clinical fields, while segmentation metadata records annotator, tool, and skill level; both files can be linked using the ISIC image identifier.Segmentation filenames also encode metadata such as annotator, tool, and skill level.Source: §IV · p. 9
- The linked image- and segmentation-level metadata supports multimodal analyses.The passage identifies leveraging both metadata levels as a use of the shared image identifier.Source: §IV · p. 9
V. INSIGHTS AND NOTES
Agreement is usually high but varies widely, with a substantial subset of images showing poor agreement or zero overlap.
A. Understanding Zero Overlap Scenarios
- 0.866 ± 0.187 is the mean Dice coefficient across images, but 236 images have mean Dice below 0.5 and 23 have zero overlap.The distribution is skewed toward high agreement with a long tail toward low agreement; 43 images have mean Dice below 0.1.Source: A. Understanding Zero Overlap Scenarios · p. 9
B. Inter-annotator variability, malignancy, and ABCD
The section connects segmentation variability with malignancy and asks how it may affect ABCD-based diagnosis, while identifying limits of pairwise agreement measures. It also points to IMA++ as a testbed for groupwise agreement research and provides scripts for reproducing the work.
B. Inter-annotator variability, malignancy, and ABCD
- Lower inter-annotator agreement is statistically associated with malignant lesions, prompting a question about segmentation variability’s effect on ABCD calculations and resulting diagnoses.The section also notes that expert dermatologists favor tighter segmentations and frames clinical decision-making effects as a direction for future research.Source: B. Inter-annotator variability,… · p. 9
C. Beyond Pairwise Multi-Annotator IAA
- Pairwise agreement measures lose information about the full distribution when agreements across N segmentations are aggregated, and IMA++ offers an incomplete annotator graph as a testbed for groupwise measures.A groupwise inter-annotator agreement measure is currently missing from the literature.Source: C. Beyond Pairwise Multi-Annotator IAA · p. 9
VI. SOURCE CODE AND SCRIPTS
- The authors make processing, validation, and analysis scripts publicly available, with a sequential pipeline script, configurable module parameters, and reproduction instructions.The work reports its computational environment and package versions.Source: §VI · p. 10