Source-linked AI summary
Skin Deep Unlearning: Artefact and Instrument Debiasing in the Context of Melanoma Classification
Peter J. Bevan, Amir Atapour-Abarghouei
TL;DR
Surgical markings, rulers, and instrument-related variation can bias melanoma classification and complicate deployment across domains. The paper applies bias-unlearning techniques to reduce artefact effects and remove instrument-associated variation, reporting artefact mitigation and generalisation benefits.
Problem
Surgical markings and rulers can bias lesion classification, while instrument variation can contribute to domain shift and poor deployment performance.
Source: §1 · pp. 1–2Method
The paper applies bias-unlearning techniques to reduce artefact influence and remove instrument-associated variation from melanoma classifiers.
Source: §1 · pp. 1–2Results
Unlearning mitigated artefact bias and improved generalisation; Turning a Blind Eye yielded an 11.6% AUC increase compared to the baseline on the Asan dataset with ResNeXt-101.
Source: §6 · p. 9Takeaways & Limitations
Debiasing models performed on par with the baseline on images without artefacts and better on images with artefacts.
Source: §6 · p. 9Artefact unlearning requires manual labels, and image size cannot be assumed to proxy instrument type across all datasets.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Convolutional Neural Networks have demonstrated dermatologist-level performance in the classification of melanoma from skin lesion images, but prediction irregularities due to biases seen within the training data are an issue that should be addressed before widespread deployment is possible. In this work, we robustly remove bias and spurious variation from an automated melanoma classification pipeline using two leading bias unlearning techniques. We show that the biases introduced by surgical markings and rulers presented in previous studies can be reasonably mitigated using these bias removal methods. We also demonstrate the generalisation benefits of unlearning spurious variation relating to the imaging instrument used to capture lesion images. Our experimental results provide evidence that the effects of each of the aforementioned biases are notably reduced, with different debiasing techniques excelling at different tasks.
1. Introduction
CNNs can match experienced dermatologists on skin-lesion diagnosis, but artefact bias and instrument-related variation raise concerns for deployment. This section motivates unlearning as a way to reduce those effects without changing physicians’ use of aids or preprocessing images.
1. Introduction
- Prior work found that surgical markings and rulers can introduce bias, while removing markings manually or segmenting images has practical drawbacks.Marking removal requires an experienced dermatologist, and segmentation or preprocessing may alter information useful for classification.Source: §1 · p. 1
- Instrument-associated variation is treated as a source of domain shift; unlearning it aims to produce more generalisable, instrument-invariant models.The authors propose removing spurious variation associated with the instrument used to capture each lesion image.Source: §1 · p. 2
- The authors report that many tested architectures exceed experienced dermatologists’ average performance on a benchmark dataset.The comparison is based on classification of a single lesion image, while dermatologists can use additional information during consultation.Source: §1 · p. 2
- Unlearning methods mitigate bias from surgical markings and rulers without requiring artefact removal from images.The paper applies Learning Not to Learn and Turning a Blind Eye to these artefact biases.Source: §1 · pp. 1–2
2. Related work
Related work describes artefact bias in lesion images and domain shift across datasets. It motivates comparing debiasing approaches and exploring unlearning for domain generalisation.
2.1. Artefacts bias
- Earlier studies reported reduced classification performance associated with surgical markings and rulers, while proposed segmentation and removal approaches can discard useful information or require expert effort.The paper focuses on documented performance-degrading biases and compares individual debiasing heads before combining them.Source: §2.1 · p. 2
2.2. Domain generalisation
- Variation in imaging methods and instrument models can create domain bias, and CNNs can discriminate between camera models and overfit to that variation.Domain shift has also been shown to affect skin-lesion classification performance.Source: §2.2 · p. 3
- Domain adaptation requires knowledge of the target distribution, whereas domain generalisation targets unseen domains; this work explores unlearning for instrument-invariant features.The aim is improved performance across possible test data without compromising performance.Source: §2.2 · p. 3
3. Methods
The methods use Learning Not to Learn (LNTL) and Turning a Blind Eye (TABE) to reduce unwanted bias in feature representations. Experiments use ISIC training data and separate test sets for artefact debiasing and domain generalisation.
3.1. Learning Not to Learn
- LNTL combines a regularisation loss with gradient reversal to remove bias from a CNN feature representation during backpropagation.The feature extractor and auxiliary bias head play a minimax game while the primary head performs lesion classification.Source: §3.1 · pp. 3–4
3.2. Turning a Blind Eye
- TABE trains an auxiliary bias classifier, then minimizes the primary classification loss with a confusion loss; CLGR additionally applies gradient reversal.The confusion loss compares predicted bias probabilities with a uniform distribution, while α controls its influence on the overall loss.Source: §3.2 · p. 4
3.3.1. ISIC CHALLENGE TRAINING DATA
- Training uses 35,574 images from the 2017 and 2020 ISIC challenges; artificially skewed versions are used only for surgical-marking and ruler debiasing experiments.The study duplicates and augments marked malignant lesions to induce bias, while original data is used for the other experiments.Source: §3.3.1 · p. 5
3.3.2. HEIDELBERG UNIVERSITY TEST DATA
- The Heidelberg test set pairs images of 130 lesions with and without surgical markings; ruler images are superimposed on plain images for testing.The set includes 23 malignant and 107 benign lesions; the superimposition approach was validated against in-vivo rulers.Source: §3.3.2 · p. 5
3.3.3. MCLASS BENCHMARK TEST DATA
- The MClass benchmark tests domain generalisation on 100 dermoscopic and 100 clinical images, with published average performances from experienced dermatologists.Images overlapping with the dermoscopic test set were removed from ISIC training data to prevent leakage.Source: §3.3.3 · p. 5
3.3.4. INTERACTIVE ATLAS OF DERMOSCOPY AND ASAN DATA
- The Atlas and Asan test sets extend domain-generalisation evaluation across clinical and dermoscopic images, seven lesion classes, and broader ethnic representation.Atlas contains 1,011 lesions with one image of each modality per lesion; Asan contains 852 clinical images.Source: §3.3.4 · p. 5
3.4. Implementation
- Experiments compare EfficientNet-B3, ResNet-101, ResNeXt-101, DenseNet, and Inception-v3, using EfficientNet-B3 for artefact debiasing and ResNeXt-101 for domain generalisation.AUC is the primary metric because test-set class proportions are imbalanced; models use test-time augmentation over eight random flips.Source: §3.4 · pp. 5–6
4. Experimental results
Experiments compare LNTL, TABE, and CLGR for artefact and instrument debiasing. They report reduced but persistent artefact effects and task-dependent domain-generalisation results.
4.1. Artefacts bias removal
- 0.055 AUC gain (6.1% increase) for LNTL on Heid marked; CLGR gave the largest ruler-image gain, 0.127 AUC (15.3% increase).Figure 5 compares artefact-debiasing models with the baseline on artificially skewed ISIC data.Source: Figure 5 · p. 6, §4.1 · p. 6
- Even after unlearning, artefacts still lower performance than plain images; the baseline also fell on same-lesion marked and ruler tests.The authors report six runs with different random seeds and state that the artefact effects were reduced, not eliminated.Source: §4.1 · pp. 6–7
4.2. Domain generalisation
- 8.5% AUC increase on ResNeXt-101/MClass clinical with CLGR moved performance from below to above the dermatologist benchmark; TABE and CLGR beat baselines in most experiments.ResNeXt-101 was selected for focused experiments after achieving the highest score on three of five test sets.Source: §4.2 · p. 7, Table 2 · p. 8
- 11.6% AUC improvement on Asan clinical and 6.6% on MClass dermoscopic for TABE versus baseline with ResNeXt-101.The MClass dermoscopic gain was also observed when both training and test data came from ISIC, which the authors suggest may reflect instrument invariance benefits within a similar distribution.Source: §4.2 · pp. 7–8, Figure 7 · p. 8
- A single TABE instrument head was the most effective overall configuration; LNTL was less successful and sometimes reduced performance, so no one technique suits every task.The reported largest gains were generally on clinical test sets, which the authors associate with greater domain shift from the dermoscopic training data.Source: §4.2 · pp. 7–8
4.3. Ablation studies
- A single instrument head generalized better than combined heads; TABE worked with or without GRL across tasks, while removing GRL from LNTL generally hurt.Table 4 reports the LNTL gradient-reversal ablation for instrument-bias removal with ResNeXt-101.Source: Table 4 · p. 9, §4.3 · p. 9
5. Limitations and future work
The authors identify manual artefact labeling and uncertain instrument proxies as constraints, and outline automation and improved metadata as future directions.
5. Limitations and future work
- Artefact unlearning requires manually labeling artefacts in every training image, though untrained individuals can often identify them quickly and easily.Future work could automate artefact labeling and adapt the model architecture to apply the needed bias-removal heads.Source: §5 · p. 9
- Image size cannot be assumed to identify the imaging instrument across all datasets; the authors recommend recording instrument model as metadata.They also propose instrument identification and further evaluation on diagnostic smartphone apps.Source: §5 · p. 9
- The authors propose investigating why debiasing methods perform differently across tasks.They also suggest studying other artefacts and evaluating whether unlearning mitigates their effects.Source: §5 · p. 9
6. Conclusion
The conclusion reports that unlearning mitigated artefact bias and improved generalisation across test sets, while the appendix supplies background and supplementary material.
6. Conclusion
- Unlearning performed on par with the baseline on artefact-free images and better on images with artefacts; instrument unlearning generally improved generalisation across five test sets.Confusion loss with gradient reversal improved baseline AUC by 15.3% on ruler images, LNTL improved performance by 6.1% with surgical markings, and Turning a Blind Eye induced an 11.6% AUC increase on Asan with a ResNeXt-101 feature extractor.Source: §6 · p. 9
- The authors report that models consistently beat the average dermatologist AUC on the MClass test sets.They suggest unlearning as an alternative to changing dermatologists’ behaviour to avoid artefacts.Source: §6 · p. 9
Appendix
- The appendix describes supplementary material on datasets, metric choices, hyperparameter tuning, additional results, and gradient-saliency interpretation.The general literature review is also presented there.Source: Appendix · p. 13
A. Skin lesion classification
- The background section traces skin-lesion classification from traditional methods to CNNs, noting that modern models often perform best without segmentation.This work focuses on neoplastic lesions and uses benign/malignant as the binary classes.Source: A. Skin lesion classification · p. 13
B. Examples of data
The examples illustrate the training and test image collections, including artefact variants and differences among clinical and dermoscopic sets.
B. Examples of data
- ISIC dermoscopic training examples include surgical markings and rulers, whose labeled class distribution is highly imbalanced.The imbalance motivated weighted loss functions to stabilize training.Source: B. Examples of data · p. 13, Figure 8 · p. 13, Figure 9 · p. 13
- Heidelberg examples include plain lesions, the same lesions with surgical markings, and plain images with electronically superimposed rulers.The markings were either applied in vivo and recaptured or electronically superimposed.Source: B. Examples of data · p. 13, Figure 10 · p. 14, Figure 11 · p. 14
- Interactive Atlas examples show dermoscopic and clinical images from the same set; the dermoscopic images show more skin and lesion detail due to reduced surface shine.Source: B. Examples of data · pp. 13–14
- The Asan clinical test set was collected at a medical centre in Seoul and features predominantly South Korean patients.Source: B. Examples of data · p. 14
C. Metrics
The metrics section motivates AUC for threshold-flexible melanoma classification, while the tuning section describes epoch selection and a constrained grid search.
C. Metrics
- Sensitivity and specificity vary with the classification threshold, and ROC curves visualize that tradeoff across possible thresholds.A clinician is anticipated to select the threshold for the desired sensitivity and specificity balance.Source: C. Metrics · p. 15
- AUC is the primary metric because it evaluates performance across thresholds, unlike accuracy, sensitivity, and specificity, which depend on a selected threshold.The authors also note that benign/malignant imbalance makes accuracy less descriptive of test-set performance.Source: C. Metrics · p. 15
D. Hyperparameter tuning
- Training epochs were selected where validation AUC reached its maximum or plateaued.Table 5 reports the selected epoch counts for each architecture and dataset.Source: D. Hyperparameter tuning · p. 15, Table 5 · p. 15
- Learning rate and momentum were selected by grid search on the baseline ResNeXt-101, and those hyperparameters were reused for debiasing models for comparability.The search used a 33% random validation subset of 3,326 images because cross-validation was computationally infeasible for the grid search.Source: D. Hyperparameter tuning · p. 15
E.1. Artefact bias removal
Debiasing improves robustness to surgical-marking and ruler artefacts: LNTL is strongest for markings, while TABE is strongest for rulers.
E.1. Artefact bias removal
- Colour thresholding partly identifies surgical markings but misses some; rulers require manual labeling because hairs have similar pixel values.Source: §E.1 · p. 15
- With markings, all models lose performance, but debiasing models are more robust, especially LNTL; with rulers, TABE is most robust.Source: §E.1 · pp. 15–16
- Accuracy scores at a 0.5 threshold corroborate the AUC results that debiasing improves robustness to artefact bias.Source: §E.1 · p. 16
E.1.1. SALIENCY MAPS
Vanilla gradient saliency maps suggest that LNTL shifts the model's focus from surgical markings back toward the lesion, though the visualization method has limitations.
E.1.1. SALIENCY MAPS
- Vanilla gradient saliency backpropagates loss gradients to input pixels, producing heatmaps of pixels most relevant to classification.Source: §E.1.1 · p. 16
- With markings, baseline maps highlight fewer lesion pixels and potentially more marked regions, while LNTL maps place salient pixels back around the lesion.Source: §E.1.1 · p. 16
- ReLU can cause saturation in this simple saliency method; the authors suggest GRAD-CAM for potentially better visualizations.Source: §E.1.1 · p. 16
E.2. Domain generalisation
Instrument-bias debiasing is associated with better generalization than the baseline across most test sets. A single instrument-bias head is generally more effective than combinations of debiasing heads.
E.2. Domain generalisation
- A single instrument-bias head is generally more effective than combinations of instrument, surgical-marking, or ruler debiasing heads.Source: §E.2 · p. 16, Table 8 · p. 18
- Across most test sets, TABE and CLGR differ from the baseline, providing evidence of better generalization when removing instrument bias.Source: §E.2 · p. 16, Figure 21 · p. 19