PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging Research and Practice: A Systematic Evaluation of Generalist and Dermatology-Specific Models in Clinical Skin Lesion Classification

Emanoel dos Santos, Kelvin Cunha, Rodrigo Mota, Fabio Papais, Thales Bezerra, Natalia Lopes, Erico Medeiros, Shirley Cruz, Jessica Araujo, Paulo Borba, Tsang Ing Ren

arXiv:2610.03193v1cs.CV

TL;DR

Clinical skin-lesion assessment varies with acquisition conditions and patient characteristics, making robustness across heterogeneous data an important open question. Across dermatology datasets, embedding models averaged 88.6% F1, CNN/ViT models 86.6%, and VLMs 65.8%.

  • Problem

    Clinical diagnosis varies with acquisition conditions and patient characteristics, while evidence of model reliability across modalities, datasets, and populations remains limited.

    Source: §1 · p. 2

  • Method

    The study benchmarks embedding models, supervised CNN/ViT architectures, and vision-language models for binary malignancy prediction across heterogeneous dermatology datasets.

    Source: §2.1 · p. 2, §2.2 · p. 3

  • Results

    Embedding-based models averaged 88.6% F1, CNN/ViT models 86.6%, and VLMs 65.8%, with VLMs ranking last across evaluation settings.

    Source: §3.1 · p. 4

  • Takeaways & Limitations

    The findings identify rigorous cross-dataset validation and task-specific representation learning as essential prerequisites for reliable clinical deployment in dermatology.

    Source: §4 · p. 8

    Image-only morphology may not reliably separate benign from malignant lesions in all cases, and clinical risk assessment also uses patient and lesion context absent from images.

    Source: §3.5 · p. 8

Abstract

from arXiv · show

The application of machine learning to dermatology has grown substantially in recent years, moving beyond proof-of-concept studies toward potential applications. However, clinical dermatology remains a challenging and still open problem. Diagnostic assessment is often ambiguous, and skin lesions exhibit high variability, compounded by differences in acquisition modality, device quality, and patient demographics. These factors hinder the development of robust models suitable for safe and equitable clinical use. To support translation into practice, it is essential to systematically evaluate how contemporary models generalize across heterogeneous data sources. In this work, we benchmark a diverse set of architectures on recent dermatology datasets, spanning dermoscopic images and smartphone-based clinical photographs. We assess the robustness of recent general-purpose and medical vision-language models, as well as foundation models, and compare them against task-specific dermatology classifiers, including embedding-based approaches and convolutional neural networks. Our study provides an evaluation of model performance under distribution shifts, modality changes, and demographic variability. By quantifying the gap between current state-of-the-art models and the requirements of clinical deployment, we aim to contribute to the development of reliable, accessible, and clinically applicable AI systems for dermatology.

1 Introduction

Clinical skin-lesion diagnosis is challenged by variability in acquisition conditions, operator expertise, and patient characteristics. The paper motivates systematic evaluation of whether contemporary models remain reliable across heterogeneous datasets and modalities.

  • Variation in acquisition conditions, operator expertise, and patient characteristics makes consistent clinical assessment challenging.Source: §1 · p. 2
  • High performance on curated benchmarks does not establish reliability across modalities, datasets, or patient populations, motivating validation under realistic variability.Source: §1 · p. 2
  • The study benchmarks diverse model families for binary malignancy prediction across modality-specific, merged-dataset, and cross-dataset settings.Source: §1 · p. 2

2 Methodology

The methodology evaluates binary malignancy prediction across heterogeneous dermatology datasets using several model families and inference configurations. It standardizes comparisons through shared splits and preprocessing, and assesses class-imbalance strategies using F1 score.

2.1 Problem Formulation

  • The task is binary malignancy prediction from dermatological images, evaluated on individual datasets and modality-specific or all-dataset merges with shared splits and standardized preprocessing.Source: §2.1 · p. 2

2.2 Model Families

  • The benchmark compares frozen embedding backbones with downstream classifiers, end-to-end supervised CNN/ViT models, and prompt-based vision-language models.Source: §2.2 · p. 3

2.3 VLMs variations

  • VLM inference tests zero-shot, ABCDE-guided, metadata-augmented, and dual-encoder configurations.Source: §2.3 · pp. 3–4

2.4 Datasets

  • Datasets span dermoscopic and clinical photographs, including smartphone images, and include merged clinical, dermoscopic, and all-modality training sets.Source: §2.4 · p. 4

2.5 Class Imbalance and metrics

  • Three class-imbalance strategies are compared, and family-level comparisons use each dataset’s best configuration with F1 score as the focus metric.Source: §2.5 · p. 4

3 Results

Embedding-based and CNN/ViT models outperformed VLMs across evaluation settings. Architecture, imbalance strategy, and available patient context shaped results, while qualitative errors highlight limits of image-only malignancy prediction.

3.1 Overall Performance Across Datasets

  • Embedding-based models averaged 88.6% F1, CNN/ViT models 86.6%, and VLMs 65.8%; the ranking persisted across merged settings.The comparison used each model family's best-performing configuration per dataset.Source: §3.1 · pp. 4–5, Table 1 · p. 5

3.2 Impact of Class Imbalance

  • Unbalanced training achieved the highest scores overall, but undersampling raised DDI performance to 87% versus 82% unbalanced.SMOTE degraded performance on ISIC24 and SD-198, so rebalancing effects varied by dataset.Source: §3.2 · pp. 5–6, Table 2 · p. 6

3.3 Architectures validation

  • Across dermoscopic datasets, EfficientNet, DINOv3, and Derm-Foundation performed comparably; clinical robustness favored Derm-Foundation and EfficientNet, while VLMs remained lower.DINO-based architectures were comparable to dermatology-specific models, and fine-tuning improved transferable representations.Source: §3.3 · pp. 6–7, Table 3 · p. 6

3.4 VLMs Variations

  • Dual-encoder VLMs were more stable than prompt-based models; ABCDE prompts gave inconsistent gains, while metadata helped selected models, especially on PAD.Current VLMs still did not reach specialized-model performance, even with contextual information.Source: §3.4 · pp. 7–8, Table 4 · p. 7

3.5 Qualitative Analysis

  • Benign and malignant lesions overlapped in borders, color, and symmetry, limiting reliable image-only separation under binary malignancy labels.Age, lesion location, evolution, symptoms, and history also inform clinical risk assessment.Source: §3.5 · p. 8

4 Conclusion

Across heterogeneous dermatology datasets, dermatology-aligned embedding models and strong vision backbones outperformed large vision-language models. The results emphasize cross-dataset validation and task-specific representations as prerequisites for reliable clinical deployment.

4 Conclusion

  • Dermatology-aligned embedding models and strong vision backbones consistently outperformed large vision-language models across heterogeneous datasets.Source: §4 · p. 8
  • Increasing architectural scale or adding multimodal pretraining did not improve clinical robustness.Source: §4 · p. 8
  • The findings identify rigorous cross-dataset validation and task-specific representation learning as prerequisites for reliable clinical deployment in dermatology.Source: §4 · p. 8
Loading 2610.03193v1…