Source-linked AI summary
Impact of Data Augmentation on Confidence Calibration in Melanoma Classification
Morgan May, Simon Caton, Pierpaolo Dondio
TL;DR
Reliable confidence calibration matters in melanoma classification, but the effect of partially augmenting training data on calibration in imbalanced melanoma datasets is under-explored. The paper compares augmentation conditions using SIIM-ISIC 2020 and two EfficientNets; all tested augmentations improved ECE, with AUC effects varying by model and method.
Problem
The effect of partially augmenting data on calibration for models trained on imbalanced melanoma datasets is under-explored, despite calibration's importance in medical diagnosis.
Source: §1 · pp. 1–2Method
The study compares with- and without-augmentation training on SIIM-ISIC 2020 using pretrained EfficientNet-B0 and B2 with four partial augmentation methods.
Source: §2 · p. 2Results
All augmentations improved ECE; trivialaugment's highest baseline improvement was 0.0321 for EfficientNet-B0 and 0.0039 for EfficientNet-B2.
Source: §3 · p. 3Takeaways & Limitations
Partial augmentation improved calibration on the studied melanoma dataset, with predicted probabilities described as more similar to actual outcomes and as leading to safer diagnosis.
Source: §4 · p. 3
Abstract
from arXiv · showhide
Accurately quantifying the predictive uncertainty or improving model calibration plays an important role in medical image classification, in particular in melanoma diagnosis, where accurate uncertainty quantification can have significant implications for patient care. One of the methods for calibration improvement is data augmentation. In addition, data augmentation as a method for synthetically increasing the size of the dataset has been proven to improve the performance of models trained on imbalanced datasets. However, the impact of data augmentation, as a transformation of a part of the original data, on calibration of models trained on imbalanced datasets, in particular in melanoma classification is under-explored. We train neural networks on SIIM-ISIC 2020 melanoma classification dataset under two conditions: with and without data augmentation, and compare the differences in AUC and expected calibration error (ECE) in both scenarios. Our results shows improvements in uncertainty calibration using different augmentation methods.
1 Introduction
The introduction motivates reliable confidence calibration for melanoma diagnosis and examines partial data augmentation as a way to improve calibration while maintaining competitive classification performance.
1 Introduction
- Calibration aligns predicted probabilities with observed event frequencies and is especially relevant to safety-sensitive medical diagnosis, including melanoma classification.The introduction also notes melanoma's early-diagnosis stakes: malignant melanoma can be cured when diagnosed in early stages.Source: §1 · p. 1
- Prior work reports calibration benefits from augmentation, while this study tests partial transformations of training data across a wider range of augmentation methods and EfficientNet models.The introduction reports that partial augmentation can improve ECE while maintaining competitive AUC or slightly improving it.Source: §1 · p. 2
- The study focuses on whether augmentation applied to a randomly selected part of training data affects melanoma-classification calibration and performance.Source: §1 · p. 2
2 Methodology
The study compares melanoma classifiers trained with and without partial data augmentation, using two pretrained EfficientNets and a logit-adjusted loss on an imbalanced dataset.
2 Methodology
- The study examines augmentation's effects on calibration rather than comparing the models against each other or optimizing their performance.The authors therefore omit additional steps such as ensembling.Source: §2 · p. 2
- The experiments use 33,126 labeled images from the imbalanced SIIM-ISIC 2020 dataset, split 80/10/10 for training, validation, and testing.The dataset's imbalance ratio is about 55.Source: §2 · p. 2
- The models are ImageNet-pretrained EfficientNet-B0 and EfficientNet-B2.Source: §2 · p. 2
- Four methods partially transform randomly selected training examples: mixup and CutMix, plus trivialaugment and RandAugment.Source: §2 · p. 2
- Training uses logit-adjusted softmax cross-entropy, shifting logits by τ log(π_y), where π_y is the class prior and τ is temperature.Source: §2 · p. 2
- AUC evaluates classification performance, while expected calibration error (ECE) evaluates calibration.Source: §2 · p. 2
3 Results and Discussion
All tested augmentations improved ECE for both models, while their AUC effects varied by model and augmentation method.
3 Results and Discussion
- Baseline ECE was 0.0500 for EfficientNet-B0 and 0.0207 for EfficientNet-B2, indicating better baseline uncertainty quantification for B2.Source: §3 · p. 3
- For EfficientNet-B0, CutMix slightly improved AUC while mixup, RandAugment, and trivialaugment performed slightly worse; for B2, trivialaugment slightly improved AUC while the others performed slightly worse.Source: §3 · p. 3
- All augmentations improved ECE; trivialaugment had the highest baseline improvement: 0.0321 for EfficientNet-B0 and 0.0039 for EfficientNet-B2.These evaluation results are presented in Table 1.Source: Table 1 · p. 3, §3 · p. 3
4 Conclusion
The conclusion reports improved calibration from partial augmentation on the SIIM-ISIC melanoma dataset and connects better-calibrated probabilities with safer diagnosis.
4 Conclusion
- Partial application of different augmentations improved calibration on the SIIM-ISIC melanoma classification dataset.Source: §4 · p. 3
- The authors explain that improved calibration means predicted probabilities are more similar to actual malignant or benign outcomes, which they say leads to safer melanoma diagnosis.Source: §4 · p. 3
- The conclusion identifies testing a wider variety of augmentation methods and a wider range of melanoma-classification models as future work.Source: §4 · p. 3