Source-linked AI summary
A Comparative Evaluation of Pre-trained Convolutional Neural Networks for Melanoma Detection
Wagner Moreno Schmitz, Marco Antonio de Castro Barbosa, Thiago Magalhães Amaral, Dalcimar Casanova, Jefferson Tales Oliva
TL;DR
Melanoma classification remains difficult because lesion appearance and acquisition conditions vary, while the best pre-trained CNN architecture may depend on imaging modality. This study compares five architectures across three datasets and finds that performance differs across modalities, with ResNet50 showing the strongest reported results and stability.
Problem
Selecting an appropriate pre-trained CNN for melanoma detection across dermatoscopic and histopathological modalities remains an open challenge amid variable image conditions.
Method
The study evaluates ResNet50, VGG16, VGG19, MobileNet, and InceptionV3 as embedding extractors in a supervised metric-learning pipeline across HAM10000, ISIC 2018, and CR-AI4SkIN.
Results
ResNet50 achieved the best performance across the evaluated datasets, reaching 83.8% accuracy and 84.0% macro F1-score on HAM10000, while architecture performance differed across modalities.
Takeaways & Limitations
Architecture comparisons should span multiple datasets and imaging modalities because similar dermatoscopic behavior does not necessarily persist on histopathological data.
Takeaways & Limitations
Future work should expand evaluation to datasets with greater class diversity and acquisition variability and investigate strategies for modality-shift performance drops.
Abstract
from arXiv · showhide
Early diagnosis of melanoma is critical for improving patient survival rates. However, accurately distinguishing melanoma from other skin lesions remains a significant clinical challenge due to the high visual similarity among lesion types and variability in image acquisition conditions. Artificial intelligence, particularly machine learning, has emerged as a promising tool to support dermatological diagnosis by automating feature extraction from medical images. Among the available approaches, convolutional neural networks (CNNs) have demonstrated strong performance in image classification tasks, making them well-suited for analyzing both dermatoscopic and histopathological images, given their ability to capture hierarchical visual patterns relevant to lesion characterization. Nevertheless, despite numerous pre-trained CNN architectures having been proposed, selecting the most appropriate one for a given imaging modality remains an open challenge. In this study, we evaluate pre-trained convolutional neural networks (CNNs) for skin lesion classification using dermatoscopic and histopathological image datasets. Experiments were conducted on the HAM10000, ISIC 2018, and CR-AI4SkIN datasets, evaluating the ResNet50, VGG16, VGG19, MobileNet, and InceptionV3 architectures under the same training protocol. The experimental evaluation showed that the models achieved accuracies ranging from 71% (InceptionV3 on ISIC 2018) to 84% (ResNet50 on HAM10000) on dermatoscopic images. For histopathological images, accuracies ranged from 72% (VGG19) to 83% (ResNet50) on the CR-AI4SkIN dataset. The results demonstrate that model performance differs between dermatoscopic and histopathological image modalities, showing that architectures exhibiting similar performance on dermatoscopic images exhibit different performance on histopathological data.
1. Introduction
Melanoma detection matters because early diagnosis strongly improves survival, yet visual variability and examiner dependence make accurate assessment difficult. This study therefore compares five pre-trained CNN architectures across dermatoscopic and histopathological datasets under controlled conditions.
- Melanoma incidence is projected to rise by more than 50% between 2020 and 2040, exceeding 500,000 new cases and approximately 100,000 annual deaths yearly by 2040.
- 99% of patients diagnosed with early-stage melanoma survive at least five years, whereas naked-eye diagnosis is only around 60% accurate.
- CNNs can automatically extract relevant image patterns, but training from scratch requires large annotated datasets and substantial computational resources.
- The study evaluates five CNN architectures across two dermatoscopic datasets and one histopathological dataset to assess consistency across datasets and modalities.
- Friedman and Nemenyi hypothesis tests are used to assess whether performance differences among architectures are statistically significant.
2. Related Work
The reviewed literature was systematically screened and characterized by datasets, models, features, metrics, and transfer-learning practices. Its recurring limitations motivate standardized comparisons, while prior work also reveals constraints in reproducibility and domain relevance.
- 278 articles were identified, and 27 satisfied the selection protocol for detailed analysis.
- The review classified studies by datasets, image-set size, problem type, data splitting, models, features, evaluation metrics, results, and frameworks.
- The reviewed methods include CNN, hybrid, generative, attention, ensemble, and pruning approaches across skin-cancer datasets and related imaging tasks.
- Public-dataset reliance, inconsistent splits and metrics, class imbalance, and absent external validation hinder direct comparison and limit conclusions about clinical applicability.
- Some reviewed approaches remain constrained by handcrafted features, manual postprocessing, repeated use of small challenge sets, or relevance to different cancer modalities.
- The study adopts standardized metrics, preprocessing, splitting, and class-balancing strategies to support fair and reproducible comparisons.
3.1. Datasets
The study uses three public datasets spanning dermatoscopic and histopathological imaging, with binary melanoma-versus-non-melanoma labels. Their differing visual structures and annotation sources support evaluation of model robustness across modalities.
- HAM10000 and ISIC 2018 provide dermatoscopic images, whereas CR-AI4SkIN provides histopathological tissue images for cross-modality evaluation.
- HAM10000 contains 10,015 dermoscopic images across seven diagnostic classes, including melanoma and six non-melanoma categories.
- The dermatoscopic datasets reorganize their original categories into melanoma versus non-melanoma binary classes.
- CR-AI4SkIN contains patches from 215 patients, with expert dermatopathologist annotations used as the ground truth for training and evaluation.
- CR-AI4SkIN introduces cellular structures, staining patterns, and complex tissue morphology that differ substantially from dermoscopic imagery.
3.2. Data Preprocessing
Preprocessing standardizes image inputs, augments training data, balances the dermatoscopic classes, and separates patients for evaluation. These choices support consistent comparison while preventing patient-level leakage in the histopathological dataset.
- All images were resized to 224×224 pixels, converted to RGB, and normalized for consistent CNN inputs.
- Augmentation used flipping, brightness and contrast changes, rotation, affine transformations, blur, and random resized cropping to address imbalance and increase variability.
- Balancing produced 6,000 HAM10000 images and 3,600 ISIC 2018 images, with melanoma and non-melanoma labels grouped into binary classes.
- HAM10000 and ISIC 2018 used 70% training, 15% validation, and 15% testing, while CR-AI4SkIN followed a patient-level 150/23/43 train-validation-test split.
- CR-AI4SkIN contains substantially different numbers of image patches per patient, ranging from approximately 60 to over 1,000 crops.
3.3. Pre-trained Models
The study compares five ImageNet-pretrained CNN architectures with distinct structural designs for extracting lesion representations. ResNet50 and MobileNet emphasize residual learning and computational efficiency, while VGG variants and InceptionV3 use sequential or multi-scale convolutional designs.
- Five architectures—ResNet50, MobileNet, VGG16, VGG19, and InceptionV3—were selected for their differing architectural strengths in skin-lesion classification.
- ResNet50: ResNet50 uses residual connections to support deeper networks and feature extraction, while its overview includes four residual stages, pooling, and a fully connected output layer.
- MobileNet: MobileNet uses depthwise separable convolutions to reduce computational cost while preserving discriminative capacity, followed by global average pooling and softmax output.
- VGG architectures: VGG16 and VGG19 use sequential stacks of 3 × 3 convolutions, ReLU activations, and periodic max pooling, with VGG19 adding depth and representational capacity.
- InceptionV3: InceptionV3 combines different kernel sizes in parallel branches and uses factorization and dimensionality reduction to capture multi-scale features efficiently.
3.4. Training and Hyperparameter Settings
All architectures were trained under a common transfer-learning and metric-learning protocol. The setup froze pretrained backbones and optimized a shared embedding head using triplet sampling, augmentation, early stopping, and consistent model construction across datasets.
- ImageNet-pretrained convolutional bases used Global Average Pooling and remained frozen, while only the embedding head was optimized.
- The embedding head combined dropout, a 256-neuron ReLU layer, batch normalization, a second dropout layer, and a 128-unit dense layer with L2-normalized outputs.
- Triplet Loss with margin 0.5 and Adam at 1 × 10^-4 trained embeddings using same-class positives and different-class negatives, with optional hard-negative mining.
- Training used geometric augmentation and ran for up to 30 epochs with early stopping after 15 epochs without validation-loss improvement.
- Fifteen models were evaluated: five backbone architectures trained independently on HAM10000, ISIC 2018, and CR-AI4SkIN under a consistent protocol.
3.5. Model Evaluation
Model evaluation combined standard classification metrics with ROC/AUC and complementary analyses of embeddings, training stability, and statistical differences. Accuracy, precision, recall, and F1-score were derived from confusion-matrix outcomes, with class-specific metrics emphasizing malignant lesions.
- Evaluation combined quantitative classification metrics with embedding interpretation, training-stability analysis, and statistical comparisons between models.
- The evaluation protocol included the hyperparameters summarized in Table 2 for the Triplet Loss training experiments.
- Confusion matrices organized predictions into true positives, true negatives, false positives, and false negatives from which subsequent metrics were derived.
- Accuracy measured overall correctness, while precision, recall, and F1-score were computed per class to characterize performance under possible class imbalance.
- ROC curves and AUC assessed discrimination across decision thresholds, with values closer to 1 indicating better separation of malignant and non-malignant lesions.
3.6. Experimental Settings
The experimental pipeline processed image datasets through resizing, balancing, pretrained CNN embeddings, metric learning, and KNN classification, while CR-AI4SkIN used repository-provided embeddings. Evaluation framed classification as binary benign-versus-malignant prediction using accuracy and macro F1-score.
- HAM10000 and ISIC 2018 images were balanced, processed by pretrained CNNs, refined with supervised Triplet Loss, and classified using the resulting embeddings.
- CR-AI4SkIN bypassed image preprocessing and new metric learning because it supplied precomputed embeddings in a PKL file.
- Embeddings were classified with KNN using k=5 and Euclidean distance, directly exploiting similarity in the learned feature space.
- The CR-AI4SkIN classifier used repository-provided embeddings and expert-defined ground-truth labels, excluding majority-vote and individual annotator labels.
- Performance was measured with accuracy and macro F1-score for a binary benign-versus-malignant classification task.
4. Results and Discussion
Across three datasets, ResNet50 achieved the strongest reported performance, while model behavior varied by dataset and imaging modality. Statistical testing confirmed significant global differences, with post-hoc separation strongest on CR-AI4SkIN.
- Overall model performance: 83.8% accuracy and 84.0% macro F1-score on HAM10000, 76.8%/77.0% on ISIC 2018, and 83.0%/84.0% on CR-AI4SkIN made ResNet50 the best-performing architecture across all datasets.These results were reported for accuracy and macro F1-score in Table 3.
- Overall model performance: MobileNet ranked second in most experiments and remained competitive despite being lightweight and computationally efficient, except where VGG16 and VGG19 had higher ISIC 2018 accuracy.The result suggests that less complex architectures can still achieve competitive performance within this evaluation.
- Dataset differences: All models performed approximately 7% to 10% worse on ISIC 2018 than on the other datasets, indicating greater difficulty under that dataset’s conditions.The passage attributes the challenge potentially to lesion variability, acquisition conditions, or class-distribution differences.
- Cross-domain robustness: ResNet50 showed greater stability across datasets, whereas VGG architectures and MobileNet experienced larger drops under domain shifts and heterogeneous clinical scenarios.ResNet50 and VGG16 showed moderate cross-modality variation, while VGG19 and MobileNet showed more pronounced drops.
- Statistical analysis: All datasets showed statistically significant global accuracy differences, with p < 0.05; ISIC 2018 and CR-AI4SkIN had p < 0.0001, while HAM10000 had p = 0.0077.The Friedman test indicates that model performances were not equivalent under the shared experimental conditions.
- Statistical analysis: Nemenyi testing found no significant model pair on HAM10000, one significant pair on ISIC 2018, and significant differences in every pair on CR-AI4SkIN.The significant ISIC 2018 comparison was ResNet50 × InceptionV3 (p=0.017); all listed CR-AI4SkIN pairwise comparisons were significant.
5. Conclusion
ResNet50 achieved the best and most stable performance across all three datasets, while model differences were strongly shaped by imaging modality. The findings support multi-dataset, multimodal validation and continued investigation of methods for improving robustness under modality shifts.
- ResNet50 consistently achieved the best performance across all three datasets and showed the greatest stability across image modalities.
- Model performance patterns differed by modality: HAM10000 showed statistically similar responses, whereas CR-AI4SkIN produced fully differentiated performance groups.
- The results indicate that single-dataset conclusions may not reflect model performance across diverse scenarios, reinforcing multi-dataset evaluation.
- Comprehensive validation should consider accuracy alongside stability, consistency, and adaptability across distinct datasets.
- Future work may investigate attention, transformer-based, augmentation, domain-adaptation, ensemble, and broader-dataset strategies to address modality-shift performance drops.
CRediT authorship contribution statement
The authors assigned contributions across conceptualization, methodology, software, analysis, investigation, writing, validation, and supervision.
- Wagner M. Schmitz contributed conceptualization, formal analysis, methodology, software, investigation, and original-draft writing.
- Marco A. C. Barbosa and Thiago M. Amaral contributed review and editing and methodology.
- D. Casanova contributed review and editing, methodology, and supervision.
- J. T. Oliva contributed review and editing, methodology, investigation, validation, and supervision.
Declaration of generative AI use
The authors used generative AI to assist with scientific-language adaptation, translation, synthesis, and related-literature searches. They state that these tools supported rather than replaced their critical reading and interpretation.
- Generative AI assisted in adapting author-drafted content into scientific language and translating and synthesizing author-developed text.
- The tools also supported searches of related literature without replacing the authors’ critical reading and interpretation.