Source-linked AI summary
A deep learning system for differential diagnosis of skin diseases
Yuan Liu, Ayush Jain, Clara Eng, David H. Way, Kang Lee, Peggy Bui, Kimberly Kanada, Guilherme de Oliveira Marinho, Jessica Gallegos, Sara Gabriele, Vishakha Gupta, Nalini Singh, Vivek Natarajan, Rainer Hofmann-Wellenhof, Greg S. Corrado, Lily H. Peng, Dale R. Webster, Dennis Ai, Susan Huang, Yun Liu, R. Carter Dunn, David Coz
TL;DR
Limited access to dermatologists and lower diagnostic accuracy among non-specialists motivate tools for skin-disease differential diagnosis. This paper develops and validates a deep learning system for 26 common conditions, finding diagnostic performance non-inferior to dermatologists and higher than PCPs and NPs.
Problem
Non-specialists have limited diagnostic accuracy for skin conditions, contributing to delayed or improper treatment.
Method
The study developed and validated a deep learning system for identifying 26 common skin conditions using adult teledermatology cases from 17 U.S. clinical sites.
Results
The system’s top-1 accuracy was non-inferior to dermatologists and higher than that of PCPs and NPs, while top-3 accuracy was also high.
Takeaways & Limitations
The findings support the potential of the system to provide skin-condition diagnoses at a level comparable to board-certified dermatologists.
Takeaways & Limitations
The study lacked a completely external validation set and pathologic confirmation for its reference standard.
Abstract
from arXiv · showhide
Skin conditions affect an estimated 1.9 billion people worldwide. A shortage of dermatologists causes long wait times and leads patients to seek dermatologic care from general practitioners. However, the diagnostic accuracy of general practitioners has been reported to be only 0.24-0.70 (compared to 0.77-0.96 for dermatologists), resulting in referral errors, delays in care, and errors in diagnosis and treatment. In this paper, we developed a deep learning system (DLS) to provide a differential diagnosis of skin conditions for clinical cases (skin photographs and associated medical histories). The DLS distinguishes between 26 skin conditions that represent roughly 80% of the volume of skin conditions seen in primary care. The DLS was developed and validated using de-identified cases from a teledermatology practice serving 17 clinical sites via a temporal split: the first 14,021 cases for development and the last 3,756 cases for validation. On the validation set, where a panel of three board-certified dermatologists defined the reference standard for every case, the DLS achieved 0.71 and 0.93 top-1 and top-3 accuracies respectively. For a random subset of the validation set (n=963 cases), 18 clinicians reviewed the cases for comparison. On this subset, the DLS achieved a 0.67 top-1 accuracy, non-inferior to board-certified dermatologists (0.63, p<0.001), and higher than primary care physicians (PCPs, 0.45) and nurse practitioners (NPs, 0.41). The top-3 accuracy showed a similar trend: 0.90 DLS, 0.75 dermatologists, 0.60 PCPs, and 0.55 NPs. These results highlight the potential of the DLS to augment general practitioners to accurately diagnose skin conditions by suggesting differential diagnoses that may not have been considered. Future work will be needed to prospectively assess the clinical impact of using this tool in actual clinical workflows.
Introduction
Dermatology shortages place diagnostic responsibility on non-specialists, whose limited accuracy can delay or compromise treatment. This paper develops a deep learning system that provides differential diagnoses across 26 common adult skin conditions using clinical images and medical history.
- Clinical need: 24-70% is the reported diagnostic accuracy of non-specialists, despite access to dermatology textbooks, UpToDate, and online image searches.Dermatologists are in short supply, so triage and diagnosis commonly fall to PCPs, NPs, and physician assistants.
- Clinical need: Low diagnostic accuracy can cause poor patient outcomes, including delayed or improper treatment.The introduction frames improved diagnostic accuracy and specialist access as key motivations for decision-support tools.
- Research gap: Prior work largely addressed lesion recognition or single diagnoses, leaving common non-cancerous conditions and full differential diagnosis insufficiently addressed.Routine dermatology includes inflammatory dermatoses and pigmentary issues, while a differential diagnosis offers a ranked, more comprehensive assessment under diagnostic ambiguity.
- System contribution: 26 common adult skin conditions are identified by the developed DLS, spanning dermatitides, dermatoses, pigmentary conditions, alopecia, and lesions.The system targets cases referred for teledermatology consultation and is intended to aid clinical decision-making.
- System contribution: The DLS uses demographic information and medical history in addition to images, supports a variable number of input images, and evaluates the benefit of multiple images.These inputs reflect data available to dermatologists in a teledermatology service.
Overview of approach · DLS performance · DLS performance compared with clinicians
The DLS combined image and metadata processing and was developed with a temporal teledermatology-data split, achieving strong differential-diagnosis performance across 26 skin conditions. On a clinician-comparison subset, its accuracy and average overlap exceeded or matched the evaluated clinician groups.
- Overview of approach: The DLS combined variable numbers of deep convolutional neural network modules for images with a shallow module processing demographics and medical history.This architecture supported flexible numbers of input images while incorporating patient metadata.
- DLS performance: 0.71 top-1 accuracy and 0.60 average top-1 sensitivity were achieved across the 26 conditions in validation set A.The top-1 accuracy was computed across all cases, while sensitivity was computed for each condition and averaged.
- DLS performance: 0.93 top-3 accuracy and 0.83 average top-3 sensitivity were achieved when the DLS provided three diagnostic possibilities.This setting was intended to mimic a clinical decision support tool suggesting possibilities for clinician consideration.
- DLS performance: 0.69–0.72 top-1 accuracy and 0.91–0.94 top-3 accuracy were observed for Fitzpatrick skin types II–IV.These skin types each comprised at least 5% of the data.
- DLS performance: 0.67 average overlap was obtained overall, with values of 0.66–0.68 across Fitzpatrick skin types.The DLS performance across the 26 conditions was also reported separately.
- DLS performance compared with clinicians: 0.90 top-3 accuracy and 0.63 average overlap were achieved by the DLS, versus 0.75 and 0.58 for dermatologists, 0.60 and 0.46 for PCPs, and 0.55 and 0.43 for NPs.Average top-1 and top-3 sensitivities across the 26 conditions followed the same trend.
Subgroup analysis · Importance of input data: images versus demographics and medical history
Subgroup analyses showed that the DLS often matched or exceeded clinicians in identifying clinically similar skin-condition categories, especially on top-3 sensitivity. Its performance also depended on input composition: a few metadata features mattered, multiple images improved accuracy, and gains plateaued with more images or larger training sets.
- Importance of input data: images versus demographics and medical history: Importance of input data: Among 45 non-image metadata types, self-reported skin problem, psoriasis history, and complaint duration had the greatest impact on accuracy.These metadata features were identified from demographic information and medical history.
- Importance of input data: images versus demographics and medical history: Importance of input data: DLS performance dramatically improved when more than one image was provided and plateaued at least five images.The same image-count trend persisted when non-image metadata were withheld.
- Importance of input data: images versus demographics and medical history: Importance of input data: Training a DLS using only images produced a small improvement compared with withholding metadata from a model developed with metadata.The image-only model avoided relying on metadata.
- Importance of input data: images versus demographics and medical history: Importance of input data: More training data improved top-1 accuracy, with diminishing returns after 10,000 cases; saliency analysis highlighted regions where the skin condition was visible.The saliency findings suggested that the DLS generally focused on the correct region of interest.
Reference Standard Labeling: Validation Set
The validation set’s reference standard was based on aggregated differential diagnoses from three board-certified dermatologists who independently reviewed each case. Cases judged by the majority to contain multiple conditions or be undiagnosable were excluded, and the aggregation improved reproducibility over individual dermatologists.
- Expert review: Three board-certified dermatologists independently reviewed each case’s photographs, demographic information, and medical history.They assessed multiple conditions, diagnosability, and up to three differential diagnoses.
- Exclusion criteria: Cases labeled by the majority as containing multiple skin conditions or as undiagnosable were excluded.Undiagnosability could reflect poor image quality, minimal visible pathology, or limited field of view.
- Reference standard construction: The reference differential aggregated diagnosis votes across the three dermatologists’ ranked differentials.Each diagnosis was mapped to one of 421 conditions, duplicate mapped conditions were removed, and votes were weighted by relative diagnostic position.
- Reproducibility: 0.73 vs 0.62: the voting procedure yielded higher reproducibility for differential diagnoses than individual dermatologists.The comparison was reported as reproducibility for the aggregated procedure versus reproducibility between individual dermatologists.
Reference Standard Labeling: Development Set
The development set was divided into training and tuning subsets, with dermatologists labeling each case to establish the reference-standard differential diagnosis. Cases judged by all graders to involve multiple conditions or be undiagnosable were discarded.
- The development set was split into training and tuning sets to learn neural-network weights and select training hyperparameters.
- 1-29 dermatologists from a cohort of 38 U.S. and 5 Indian board-certified dermatologists labeled each development-set case.
- Cases were discarded when all dermatologists grading them considered them to involve multiple skin conditions or be undiagnosable.
- The reference-standard differential diagnosis for development cases was established using the same method as for the validation set.
Labeling tool and skin condition mapping
The labeling tool used SNOMED-CT search with free-text fallback, while three board-certified dermatologists mapped diagnoses to clinically useful skin-condition labels. The final mapped list contained 421 conditions after excluding invalid or unclear entries and remaining blinded to predictions and case identities.
- Labeling tool: The tool provided search-as-you-type labeling based on SNOMED-CT, which included more than 20,000 cutaneous-disease terms, with free-text fallback when no match existed.This supported standardized terminology while allowing diagnoses absent from the terminology search to be entered manually.
- Skin condition mapping: Three board-certified dermatologists mapped SNOMED-CT terms and free-text diagnoses to a condition list at granularity suited to clinical next steps and provider communication.The list began with common or high-acuity conditions and expanded as needed.
- Skin condition mapping: Diagnoses were invalidated when they were too broad, non-skin entries, represented multiple skin conditions, or were semantically unclear.For example, “alopecia” was considered too broad, whereas “alopecia areata” and “androgenetic alopecia” supported clinical next steps.
- Skin condition mapping: 421 conditions comprised the final list after mappings were performed blinded to DLS predictions and the identities of clinicians or cases.Additional conditions were added based on discussion involving at least two of the three dermatologists.
Selection of the 26 skin conditions
The DLS targeted the 26 most prevalent conditions among 421 candidates to ensure sufficient cases for training and evaluation. All remaining conditions were grouped into an “Other” category, comprising 22% of validation cases.
- Selection criteria: At least 100 cases per condition were required in the development dataset for DLS training.This filtering criterion was intended to ensure sufficient data for development.
- Selection criteria: 25 projected cases per condition were required in the validation set for DLS evaluation.The projection was made during condition selection to support evaluation of each retained condition.
- Residual category: 22% of validation cases were assigned to an “Other” category containing the remaining conditions.Conditions outside the selected 26 were aggregated into this category.
DLS development
The DLS fused image and clinical-metadata networks and learned differential diagnoses from soft reference-standard targets. Training addressed class imbalance, patient-level splitting, augmentation, missing metadata, and ensemble prediction.
- Architecture: The DLS combined an image-processing convolutional network with a shallow clinical-metadata network using late fusion.It processed 1–6 images with shared-weight Inception-v4 modules and joined the components through a fully connected layer.
- Data partitioning: The development set was split into training and tuning sets using stratified sampling and patient-level separation.The tuning set included up to 50 cases per condition, and all cases from sampled patients were moved to that set.
- Regularization and ensemble: Six networks were trained and ensembled after augmentation and metadata dropout to improve generalization and robustness.Augmentation included flipping, rotation, cropping with at least 20% overlap with the pathologic skin region, and color perturbation.
DLS evaluation
The DLS was evaluated against a voting-based reference-standard differential diagnosis using top-k accuracy and sensitivity, with additional agreement analyses based on at least one dermatologist and full differential overlap.
- Evaluation metrics: Top-k accuracy measures how often the top k predictions include the reference standard’s primary diagnosis.The primary diagnosis is the diagnosis ranked first in the reference differential.
- Evaluation metrics: Top-k sensitivity evaluates this capture separately for each of the 26 conditions, while average top-k sensitivity averages performance across conditions.Condition averaging reduces bias toward more common conditions, particularly in validation set A.
- Evaluation metrics: The study used both top-1 and top-3 metrics for DLS evaluation.The DLS and clinicians were compared with the voting-based reference-standard differential diagnoses.
- Additional agreement analyses: Additional analyses evaluated agreement with at least one dermatologist and full differential overlap using average overlap (AO).For AO, DLS predictions were limited to up to three ranked diagnoses and diagnoses below a predicted likelihood of 0.1 were filtered.
Comparison to clinicians
The DLS was compared with three groups of six independent U.S. board-certified clinicians using differential diagnoses from validation set B. Tie-breaking randomly selected among tied candidates, affecting top-1 analyses more often for PCPs than dermatologists or NPs but minimally affecting top-3 analyses.
- Clinician comparison: 18 clinicians comprised three groups of six dermatologists, PCPs, and independently practicing NPs who provided differential diagnoses for validation set B.Each case was graded by two random clinicians from each group, for six clinicians total per case.
- Tie-breaking: 24% of PCP-provided diagnoses required random tie-breaking in top-1 analyses, compared with 14% of NP-provided and 13% of dermatologist-provided diagnoses.Ties were resolved by randomly selecting a diagnosis from the tied candidates.
- Tie-breaking: 0.6% of PCP diagnoses had ties in top-3 analyses, while dermatologists and NPs had no ties.The tie-breaking procedure was intended to avoid bias toward clinicians providing tied differential diagnoses, which indicates uncertainty.
Statistical analysis … DLS (hair loss
The paper used bootstrap and permutation-based statistical procedures for uncertainty estimation and non-inferiority testing, while presenting DLS inputs, dataset structure, clinician comparisons, and representative diagnostic outputs. Case examples show condition-specific predictions across growth, erythematosquamous, and hair-loss presentations.
- Statistical analysis: Confidence intervals used a non-parametric bootstrap with 1,000 samples, while feature-importance intervals used a normal approximation across 20 runs.Each feature-importance run used the entire validation set A.
- FIGURES: The DLS processes 1 to 6 de-identified skin photographs and 45 metadata variables through shared-weight Inception-v4 modules before classification.The architecture averages image features and concatenates them with metadata features.
- FIGURES: The validation materials included clinician performance comparisons, representative challenging cases missed by non-dermatologists, input-importance analyses, and a temporally split dataset from 17 clinical sites.Validation set B was enriched for rarer skin conditions and reviewed by three clinician groups.
- DLS (growth: For growth cases, the DLS assigned BCC a likelihood of 0.84 and classified the malignant-versus-benign distinction as Malignant: 1.0; Benign: 0.0.Other examples listed SCC/SCCIS, scar condition, actinic keratosis, and melanoma among candidate diagnoses.
- DLS (growth: Another growth example assigned SCC/SCCIS a likelihood of 0.74, BCC 0.19, actinic keratosis 0.04, and Malignant: 0.94; Benign: 0.06.The candidate list included Other (skin lesion), SCC/SCCIS, and BCC.
- DLS (erythematosq: For erythematosquamous cases, one prediction assigned Tinea a likelihood of 0.95, with Other: 0.03 and Eczema: 0.02, and Infectious: 0.98; Non-infectious: 0.02.Other examples included generalized granuloma annulare, chronic contact dermatitis, psoriasis, eczema, and granuloma annulare.
- DLS (hair loss: For hair-loss cases, one prediction assigned AA a likelihood of 0.89, with Other: 0.05 and AGA: 0.03.Additional candidate diagnoses included AGA, alopecia localis, AA, and trichotillomania.
Competing interests … DLS (erythematosq
The supplied excerpts describe evaluation materials, clinician onboarding, reference-standard reproducibility, and illustrative DLS differentials for growth and erythematosquamous cases. No competing-interests statement is included in the supplied passages.
- Extended Data: Extended Data Fig. 1 reports DLS and clinician performance across all 26 skin-condition categories, with 95% confidence-interval error bars.The comparisons include dermatologists, primary care physicians, and nurse practitioners.
- Labeler onboarding and certification: All participants completed onboarding for grading tools, and reference-standard dermatologists graded 147 development-set cases to assess grading consistency.
- Reference standard voting procedure and reproducibility: 0.73 agreement was observed between two three-dermatologist panels across the 27 conditions handled by the DLS.The corresponding average overlap was 0.67; across 421 mapped conditions, agreement was 0.71 and average overlap was 0.61.
- DLS (growth: 0.84 probability was assigned to BCC in an illustrative growth case, followed by scar condition at 0.06 and SCC/SCCIS at 0.05.The ranked differential shown was BCC; SCC/SCCIS; scar condition.
- DLS (growth: 0.94 malignant probability was assigned in another illustrative growth case, with SCC/SCCIS at 0.74, BCC at 0.19, and actinic keratosis at 0.04.The displayed differential included other skin lesion, SCC/SCCIS, and BCC.
- DLS (erythematosq: 0.98 infectious probability was assigned in an illustrative erythematosquamous case, with tinea at 0.95, other at 0.03, and eczema at 0.02.Other displayed differentials included eczema, chronic contact dermatitis, psoriasis, and granuloma annulare.
DLS (hair loss
The hair-loss examples show differential predictions involving alopecia areata (AA), androgenetic alopecia (AGA), and Other diagnoses. The system’s clinical metadata includes personal and family history, allergies, medications, pregnancy, nursing, medical problems, prior treatment, and progression information.
- 0.89 AA, 0.05 Other, and 0.03 AGA are reported for one hair-loss example.
- AGA, Other (Alopecia localis), and AA are listed as the differential diagnoses for another example.
- AA is repeatedly listed, with Other specified as trichotillomania, in a further hair-loss example.
- The clinical metadata includes personal and family histories covering skin cancer, melanoma, eczema, and psoriasis, plus six allergy categories.
- Medication use, pregnancy, nursing, medical problems, prior treatment, treatment adherence, treatment response, and progression are also recorded.