Source-linked AI summary
Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset
Matthew Groh, Caleb Harris, Luis Soenksen, Felix Lau, Rachel Han, Aerin Kim, Arash Koochek, Omar Badri
TL;DR
Dermatology image datasets underrepresent darker skin and commonly omit Fitzpatrick labels, limiting direct evaluation of model performance across skin color. This paper annotates 16,577 atlas images, trains a 114-condition classifier, and compares human Fitzpatrick labels with ITA. Models trained on two skin types are most accurate on nearby Fitzpatrick types, while ITA differs substantially from human labels in many images.
Problem
Dermatology image data mostly represent light skin, and most public datasets lack skin-type labels needed to evaluate classification accuracy across skin color.
Method
The study annotates 16,577 clinical images with human Fitzpatrick labels, trains a neural network for 114 skin conditions, evaluates accuracy across skin types, and compares labels with ITA.
Results
Models trained on images from two Fitzpatrick types are most accurate on holdout images with the closest skin types to their training data.
Takeaways & Limitations
The dataset exposes underrepresentation of dark skin images and accuracy disparities associated with training on only a subset of skin types.
Takeaways & Limitations
ITA was calculated from entire images or automatic segmentation masks, so future comparisons should use more precise segmentation masks.
Abstract
from arXiv · showhide
How does the accuracy of deep neural network models trained to classify clinical images of skin conditions vary across skin color? While recent studies demonstrate computer vision models can serve as a useful decision support tool in healthcare and provide dermatologist-level classification on a number of specific tasks, darker skin is underrepresented in the data. Most publicly available data sets do not include Fitzpatrick skin type labels. We annotate 16,577 clinical images sourced from two dermatology atlases with Fitzpatrick skin type labels and open-source these annotations. Based on these labels, we find that there are significantly more images of light skin types than dark skin types in this dataset. We train a deep neural network model to classify 114 skin conditions and find that the model is most accurate on skin types similar to those it was trained on. In addition, we evaluate how an algorithmic approach to identifying skin tones, individual typology angle, compares with Fitzpatrick skin type labels annotated by a team of human labelers.
1. Motivation
Deep neural networks offer potential value for dermatology, but their training data disproportionately represent light skin and usually lack skin-type labels. The paper addresses this gap by annotating images, evaluating accuracy across skin types, and comparing human Fitzpatrick labels with ITA estimates.
- Clinical-image classifiers could support dermatology and healthcare, but existing training data mostly depict people with light skin.
- Most publicly available dermatology datasets identified by the Sixth ISIC Workshop lack skin type, skin color, race, or ethnicity labels.
- The Fitzpatrick 17k dataset makes potential bias more transparent by annotating dermatology-atlas images with human Fitzpatrick skin types.
- The study trains a neural network for image-only skin-condition classification and evaluates accuracy across skin types.
- The study also compares human Fitzpatrick labels with individual typology angle, an image-based skin-tone estimation method affected by lighting conditions.
2. Fitzpatrick 17k Dataset
Fitzpatrick 17k is a 16,577-image dataset of dermatology conditions with human-annotated Fitzpatrick skin types. Its images and condition labels reveal substantial imbalance across skin types and conditions, while the annotation process uses multi-annotator consensus and dermatologist-validated gold-standard examples.
- 16,577 clinical images form the Fitzpatrick 17k dataset, each labeled for skin condition and Fitzpatrick skin type.
- 12,672 images come from DermaAmin and 3,905 from Atlas Dermatologico, whose condition labels have been used in dermatology and computer-vision literature.
- Images were selected according to the most common dermatology conditions across the two source atlases, with 22 condition categories excluded.
- The dataset aggregates conditions into three high-level classes and a more granular nine-category taxonomy to support explainability.
- Each image received Fitzpatrick labels from two to five Scale AI annotators through dynamic consensus, using historical accuracy and a 312-image dermatologist-provided gold standard.
- 7,755 images are types 1–2, 6,089 are types 3–4, and 2,168 are types 5–6, with 565 labeled unknown.
- 25 of 114 conditions have no Fitzpatrick type 6 examples, and only 10 conditions have a mean Fitzpatrick type above 3.5.
3. Classifying Skin Conditions with a Deep Neural Network
The study trains a transfer-learning deep neural network to classify 114 skin conditions under multiple holdout designs. Performance varies across task granularity and skin-type similarity, with highest accuracy on skin types closest to those represented in training.
- Model and evaluation: The model uses a VGG-16 transfer-learning architecture, replacing its final layer with a 256-unit layer, ReLU, dropout, category-output, and softmax layers.The model contains 135,335,076 parameters, of which 1,074,532 are trainable.
- Model and evaluation: Five evaluation approaches test dermatologist-verified images, stratified random holdouts, source-specific holdouts, and skin-type-based holdouts.Training uses the remaining non-held-out Fitzpatrick 17k images in each design.
- Results: 20.2% overall accuracy identifies the exact skin condition in the random holdout, compared with 29.0% top-2 and 35.4% top-3 accuracy.Random guessing across 114 conditions would yield 0.9% accuracy.
- Results: 62.4% accuracy is achieved on three high-level categories and 36.1% on nine categories in the random holdout.The corresponding random-guessing baselines are 33.3% and 11.1%; Esteva et al. reported 72.1% and 55.4% using a dataset 7.5 times larger.
- Results: Across 114 conditions, median accuracy is 20.0%, ranging from 0% for 10 conditions to 93.3% for one condition.The minimum covers 433 random-holdout images, while the maximum covers 30 images.
- Skin-type variation: When trained on two Fitzpatrick types and evaluated on the other four, the model is most accurate on skin types closest to its training images.Training on types 1–2 favors types 3–4 over 5–6; training on 3–4 favors 2 and 5 over 1 and 6; training on 5–6 favors 3–4 over 1–2.
4. Evaluating Individual Typology Angle against Fitzpatrick Skin Type Labels
The study estimates skin tone using ITA from full and YCbCr-masked images, then compares converted ITA categories with human-annotated Fitzpatrick labels. The reported discrepancy appears mainly associated with high variance in ITA.
- Methodology: ITA estimates skin tone from image-pixel statistics and is negatively correlated with the melanin index.The study applies ITA to both full images and images masked using YCbCr.
- Methodology: The YCbCr algorithm converts RGBA images and applies masking thresholds intended to segment healthy skin from the rest of each image.The method avoids directly obtaining expensive segmentation masks for non-diseased skin.
- Methodology: ITA is calculated after conversion to CIE-LAB color space using luminance L and yellow component B, with mean values from non-masked pixels within one standard deviation of the actual mean.The study applies the formula to each full and YCbCr-masked image.
- Results: Table 5 compares full-image and YCbCr-masked ITA with Fitzpatrick labels using Kinyanjui and empirical conversion methods.Each entry reports the proportion of ITA scores within plus or minus one of the annotated Fitzpatrick score.
- Results: The empirical conversion uses ITA thresholds learned by minimizing overall error, whereas the Kinyanjui columns use modified published thresholds.Figure 1 shows ITA distributions by Fitzpatrick label, and Figure 3 indicates high ITA variance as the main apparent source of discrepancy.
5. Conclusion
The Fitzpatrick 17k dataset exposes substantial dark-skin underrepresentation and skin-type-dependent accuracy disparities, while also highlighting limitations in current skin-tone evaluation methods. The authors call for future research to measure heterogeneous accuracy across specific skin-condition groups.
- 16,577 images covering 114 skin conditions contain 3.6 times more images from the two lightest than the two darkest Fitzpatrick skin types.
- Models trained on only two Fitzpatrick skin types are most accurate on holdout images from the closest skin types to their training data.This pattern is consistent with dermatological knowledge that skin conditions can appear differently across skin types.
- Which skin conditions show the largest accuracy disparities across skin types remains an open question for future research.The authors point to conditions whose visual features, such as redness in inflammatory disease, may not appear universally across skin types.
- ITA is prone to error even when human labelers agree, with its most accurate scores differing by more than one Fitzpatrick point in about one third of the dataset.The comparison used either entire images or automatic segmentation masks.
- Future research should evaluate classification accuracy across subpopulations where accuracy is suspected to be heterogeneous.