Source-linked AI summary
On Impact of Loss Function on the Performance of Neural Networks in Melanoma Diagnosis
Morgan May, Pierpaolo Dondio, Simon Caton
TL;DR
Imbalanced melanoma datasets, where malignant cases are scarce, challenge deep-learning classification. This work compares loss functions across neural networks and evaluates performance and calibration; focal loss offers a good AUC-and-ECE combination, particularly for larger models.
Problem
Malignant cases are outnumbered by benign cases in clinical datasets, creating an imbalance challenge for melanoma classification and naive learning.
Source: §1 · p. 1Method
Results
Focal loss gives EfficientNet-B2 and EfficientNetV2-S higher AUC than the other evaluated losses while maintaining relatively low ECE across all three models.
Source: §3 · p. 3Takeaways & Limitations
The authors suggest focal loss is a good choice for evaluated EfficientNet models when seeking good AUC performance alongside ECE calibration in melanoma diagnosis.
Source: §4 · p. 3
Abstract
from arXiv · showhide
Melanoma is the deadliest type of skin cancer, whose early diagnosis is crucial for patients' survival. Image classification using deep learning models has shown promising results for melanoma diagnosis. However, the performance of these models on the melanoma datasets such as SIIM-ISIC melanoma classification dataset is a challenge due to the class imbalance. One of the methods to deal with this challenge is using loss function modifications. In this work, we have investigated the effect of different loss functions on the performance of deep neural networks. We trained these networks using focal loss, logit-adjusted softmax cross-entropy (CE) loss, and weighted softmax CE loss, and we report different metrics for evaluating performance and uncertainty calibration. Our results suggest that focal loss delivers a good combination of performance in terms of AUC and uncertainty calibration in terms of expected calibration error (ECE) simultaneously.
1 Introduction
Melanoma diagnosis remains a critical clinical issue, and deep learning classification faces the challenge of imbalanced datasets. This work investigates loss-function choices and model uncertainty calibration for that task.
1 Introduction
- Deep learning has produced highly accurate melanoma classification models, while early diagnosis and timely treatment remain critical for malignant melanoma.Skin cancer incidence and mortality rates have been increasing yearly.Source: §1 · p. 1
- Malignant cases are scarce relative to benign cases, making melanoma diagnosis an imbalanced classification task that can bias naive learning toward the majority class.Loss-function modification is one algorithmic-level approach among four groups of methods for addressing imbalance.Source: §1 · p. 1
- This study evaluates focal, logit-adjusted softmax CE, and balanced softmax CE losses across EfficientNet-B0, EfficientNet-B2, and EfficientNetV2-S, including uncertainty calibration.The prior comparison used EfficientNet-B3 and included weighted CE and large margin-aware focal loss; it reported higher AUC for LMF.Source: §1 · p. 2
2 Methodology
The study evaluates loss-function effects on SIIM-ISIC 2020 melanoma classification, using pretrained EfficientNet variants and grid search while avoiding model-to-model ranking or ensembling.
2 Methodology
- The study uses 33,126 labeled SIIM-ISIC 2020 images, with an approximately 55 imbalance ratio and an 80/10/10 train/validation/test split.The stated goal is to assess loss-function effects, not compare models or maximize performance; no ensembling is used.Source: §2 · p. 2
- ImageNet-pretrained EfficientNet-B0, EfficientNet-B2, and EfficientNetV2-S are trained for melanoma classification.EfficientNetV2-S is described as designed for better parameter efficiency and faster training than its predecessor.Source: §2 · p. 2
- The three evaluated losses are balanced softmax CE, focal loss, and logit-adjusted softmax CE.Balanced softmax CE has no specific hyperparameters to tune.Source: §2 · p. 2
- Performance is assessed with AUC, F1-score, and recall, while ECE measures confidence calibration; grid searches tune learning rates and loss-specific parameters.Learning rates range from 1e−4 to 3e−4; focal-loss alpha and gamma and logit-adjustment temperature are also searched.Source: §2 · p. 2
3 Results and Discussion
The results show no strong overall trend across losses and models, but focal loss combines higher AUC for the larger models with relatively low ECE, while logit adjustment offers higher recall with an ECE trade-off.
3 Results and Discussion
- Logit-adjusted softmax CE gives EfficientNet-B0 and EfficientNet-B2 higher recall than focal and balanced softmax CE, while sacrificing ECE.The authors note this may result in overconfident predictions, which they consider undesirable for safety-critical melanoma diagnosis.Source: §3 · p. 3
- Focal loss yields higher AUC than logit-adjusted and balanced softmax CE for EfficientNet-B2 and EfficientNetV2-S, while maintaining relatively low ECE across all three models.Table 1 summarizes performance by model and loss function; the text reports no strong trend across all model-loss combinations.Source: §3 · pp. 2–3, Table 1 · p. 3
4 Conclusion
The conclusion favors focal loss for balancing AUC performance and ECE calibration in the evaluated EfficientNet models, and points to broader loss and model evaluations as future work.
4 Conclusion
- The results suggest focal loss is a good choice for EfficientNet and EfficientNetV2 melanoma classification when seeking good AUC performance alongside good ECE calibration.The conclusion identifies melanoma diagnosis as a safety-critical application.Source: §4 · p. 3
- Future work could evaluate more loss functions across a broader range of melanoma-classification models.The authors also propose testing combinations of multiple loss functions as the training objective.Source: §4 · p. 3