Disease Burden over Skin Tone: Decomposing the Dermatology-AI Generalization Gap

Nirajan Kunwor; Sanjaya Poudel; Quoc-Huy Trinh; Jahidul Arafat; Sunil Kumar Gaire

Source-linked AI summary

Disease Burden over Skin Tone: Decomposing the Dermatology-AI Generalization Gap

Nirajan Kunwor, Sanjaya Poudel, Quoc-Huy Trinh, Jahidul Arafat, Sunil Kumar Gaire

arXiv:2609.02111v1Sep 2, 2026cs.CVcs.AIcs.LG

TL;DR

Dermatology AI models face a generalization gap because deployment populations differ from cancer-focused training data in both skin tone and disease distribution, but those effects are confounded. The paper separates them with public datasets and frozen-feature analyses, finding that disease-distribution shift dominates, reflects representational weakness, and is most recoverable when starting features already encode target conditions.

Problem
Dermatology AI training data underrepresents darker skin tones and unfamiliar non-neoplastic diseases, leaving the relative contributions of skin tone and disease distribution unresolved.
Method
The study compares four frozen encoders across cancer-domain, tone-stratified DDI, and tone-diverse disease-shifted SCIN datasets using probes, label-free representation analysis, and lightweight adaptation.
Results
Disease-distribution shift dominates in the evaluated settings: the cancer baseline falls from 0.62 to 0.21 balanced accuracy, while the DDI tone gap is 0.10-0.18 and inconsistent.
Takeaways & Limitations
Representation quality is the practical bottleneck: dermatology foundation models retain transferable structure, and approximately 10 labeled examples per category recover most attainable performance without retraining.
Takeaways & Limitations
DDI and SCIN are US-sourced proxies rather than RCS-collected cohorts, so conclusions about resource-constrained settings are inferential.

Abstract

from arXiv · show

Dermatology artificial intelligence (AI) models are predominantly trained on light-skinned, cancer-focused image collections, yet they are increasingly proposed for deployment in resource-constrained settings where patients differ from training populations along two confounded axes: skin tone and disease distribution. We investigate whether poor generalization is primarily caused by skin-tone underrepresentation or disease-distribution shift. We evaluate a cancer-trained baseline (ResNet-50 fine-tuned on HAM10000 and ISIC 2019), two dermatology foundation models (DermLIP and MONET), and a general-purpose vision model (DINOv3) as frozen feature extractors. Models are evaluated on a tone-stratified disease-matched dataset (Diverse Dermatology Images, DDI) and a disease-shifted tone-diverse dataset (Skin Condition Image Network, SCIN). Our results show that disease-distribution shift contributes more than skin tone in the evaluated settings. The cancer baseline decreases from 0.62 to 0.21 balanced accuracy when transferred to unfamiliar clinical conditions, while the within-disease skin-tone gap is smaller (0.10-0.18) and inconsistent. Label-free representation analysis shows that this failure reflects a representational limitation rather than only missing output labels: cancer-specialized features poorly cluster unfamiliar conditions (kNN purity lift +0.06 over chance), whereas dermatology-pretrained features retain stronger transferable structure (+0.23). Finally, we show that representation quality predicts recoverable performance under lightweight adaptation. Starting from dermatology foundation models, approximately ten labeled examples per clinical category recover most attainable performance. We release the evaluation protocol and code to support reproducible auditing of dermatology AI generalization.

1 Introduction

Dermatology AI generalization gaps conflate skin tone with disease distribution, leaving unclear which factor drives failures in resource-constrained settings. This study uses public data and reproducible analyses to separate these effects and examine their representational basis and recoverability.

Training archives are dominated by lighter skin tones and cancer-related dermoscopic conditions, unlike resource-constrained clinics serving more pigmented patients with inflammatory and infectious diseases.
Skin tone and disease distribution are confounded in deployment failures, so distinguishing them determines where scarce data-collection and model-development resources should be directed.
Open, reproducible evidence is still needed to decompose the generalization gap under extreme shifts and determine whether failures reflect representational deficits or missing output labels.
The study uses public datasets and free-tier compute to conduct an open, fully reproducible investigation.
DDI isolates tone with matched diagnoses, while SCIN isolates distribution using tone-diverse, non-neoplastic conditions; distribution shift dominates across evaluated models.
A label-free analysis tests whether unfamiliar-condition failures arise before classification, and lightweight adaptation assesses how much performance can be recovered from existing representations.

2 Previous Work

Previous work documents uneven performance across skin tones and suggests condition distribution can drive error disparities. This study extends that evidence through an open, extreme-shift evaluation focused on decomposing causes and testing representation quality.

Dermatology AI has shown lower accuracy on darker Fitzpatrick types, while DDI enables tone-stratified evaluation with biopsy-confirmed labels.
Prior teledermatology evidence found error disparities tracked condition categories more than demographics, but used non-public data and a relatively mild within-system shift.
Open dermatology foundation-model benchmarks evaluate frozen features on pigmented-skin collections, whereas this work uses frozen features to decompose generalization-gap causes rather than rank models.

3 Methodology

The methodology assigns public datasets to in-domain, tone-isolation, and distribution-isolation roles, then compares four frozen encoders under identical downstream evaluation. Patient-level splits, complementary analyses, and constrained compute support reproducible comparison.

3.1 Datasets and Their Roles: DDI contains 656 images across three Fitzpatrick-group ranges for tone isolation, whereas SCIN contains 6,517 cleaned images from 3,061 patients for distribution isolation.
3.1 Datasets and Their Roles: DDI is tone-stratified but SCIN is tone-diverse and unbalanced, so the design observes the axes marginally rather than jointly.
3.2 Models: Four frozen encoders are evaluated: a cancer-trained ResNet-50, DermLIP, MONET, and the general-purpose self-supervised model DINOv3.
3.3 Evaluation Protocol: Linear probes use standardized frozen features and patient-level GroupShuffleSplit, while SCIN conditions are mapped into seven clinically grounded categories.
3.3 Evaluation Protocol: Label-free purity lift measures condition separation beyond random-neighbor chance, and low-compute adaptation compares full probes with 10-examples-per-class probes without retraining.
3.3 Evaluation Protocol: Every downstream stage uses identical splits, preprocessing rules, probe settings, purity metrics, and bootstrap procedures across encoders.
Compute Resources: All experiments run on free-tier cloud compute, with downstream analyses completed on CPUs and inference requiring one frozen-encoder forward pass plus a lightweight probe.

4 Results

Across the evaluated settings, disease-distribution shift causes a sharper generalization failure than skin-tone variation. Label-free analyses and adaptation results indicate that transferable representation quality constrains performance on unfamiliar conditions.

4.1 Distribution Shift Dominates the Generalization Gap: 0.62 in-domain balanced accuracy falls to 0.21 on unfamiliar conditions for the cancer baseline, while the DDI tone gap is 0.10-0.18 and inconsistent.DINOv3 and dermatology foundation models transfer better than the cancer baseline on SCIN.
4.2 Evidence for a Representational, Not Definitional, Gap: +0.42 purity lift in-domain for the cancer baseline falls to +0.06 on SCIN, showing poor clustering of unfamiliar conditions before classifier training.The comparison supports a representational deficit rather than only a missing output head.
4.2 Evidence for a Representational, Not Definitional, Gap: DermLIP retains +0.23 purity lift on unfamiliar conditions, while the general-vision control also exceeds the cancer baseline.The results indicate that cancer specialization improves in-domain structure but reduces transferable structure under disease shift.
4.3 Cheap Adaptation Is Bounded by Latent Structure: r = 0.90 between SCIN purity lift and full-probe balanced accuracy links recoverable performance to label-free representation quality across four models.This relationship is suggestive given n = 4 and is not a significance test.
4.3 Cheap Adaptation Is Bounded by Latent Structure: Ten labeled examples per category nearly match or exceed full-probe performance for DermLIP and MONET, whereas the cancer baseline recovers least.DermLIP reaches 0.41 versus 0.36 for the full probe, and MONET reaches 0.39 versus 0.34.
4.3 Cheap Adaptation Is Bounded by Latent Structure: The adaptation pattern is a potential deployment direction rather than a validated recipe, with scale confirmation left for future work.The authors frame the finding cautiously for resource-constrained settings.

5 Discussion

In the evaluated settings, disease-distribution shift is the larger, representationally grounded contributor to the dermatology-AI generalization gap, while matched-disease skin-tone effects are smaller and inconsistent. The findings support emphasizing starting representation quality, although several dataset, evaluation, and model-comparison limitations constrain interpretation.

Discussion: Disease-distribution shift dominates matched-disease skin-tone effects in the evaluated settings.The tone effect is small and inconsistent in direction, whereas distribution shift is a large, representationally grounded driver.
Discussion: Representation quality appears to matter more than expensive local retraining for recovering performance under resource constraints.The paper links latent structure to cheap recoverability and recommends investing in the starting representation.
Limitations: DDI and SCIN are US-sourced proxies rather than genuinely resource-constrained-setting cohorts, making the RCS conclusions inferential.Validation on genuinely RCS-collected cohorts remains future work.
Limitations: The DDI tone analysis is statistically underpowered, so the study avoids strong directional claims about tone effects.The test split contains 55-77 images per Fitzpatrick group.
Limitations: The conclusion is limited by frozen-feature evaluation and confounding between the cancer baseline's architecture, objective, and pretraining scale.End-to-end or parameter-efficient fine-tuning may alter the observed picture, while specialization cannot be fully isolated from these model differences.
Limitations: The study also notes that accuracy is not chance-normalized across tasks, kNN purity is not dimensionality-controlled, and SCIN categories simplify diagnostic complexity.Fitzpatrick type indexes UV response rather than pigmentation, and stronger dermatology foundation models remain unevaluated.

6 Impact in Resource-Constrained Settings

For dermatology AI in resource-constrained settings, the paper suggests prioritizing model selection and disease-distribution coverage over expensive local retraining. Lightweight adaptation with modest local labels may recover most attainable performance when the starting representation is strong.

Impact in Resource-Constrained Settings: Model selection may matter more than local retraining because recoverable performance is bounded by representation quality.The paper recommends starting with a dermatology-pretrained foundation model rather than a cancer-specialized classifier.
Impact in Resource-Constrained Settings: Approximately ten labeled examples per local category recovered most attainable performance in the experiments.This suggests adaptation can use a small, locally curated label set.
Impact in Resource-Constrained Settings: The adaptation step runs on CPU, so the main hardware demand is one-time feature extraction rather than repeated local training.
Impact in Resource-Constrained Settings: Tone-diverse data remain important for fairness auditing, but closing the deployment gap under a different disease burden requires prioritizing disease-distribution coverage.