Source-linked AI summary
Skin Lesion Classification Using Ensembles of Multi-Resolution EfficientNets with Meta Data
Nils Gessert, Maximilian Nielsen, Mohsin Shaikh, René Werner, Alexander Schlaefer
TL;DR
The challenge combines severe class imbalance, heterogeneous image resolutions, and an unknown test-set class across image-only and metadata-assisted skin-lesion classification tasks. The paper addresses these issues with balanced training, multi-resolution EfficientNets, external data, metadata fusion, and model ensembling, ranking first in both tasks while identifying unknown-class prediction and metadata use as ongoing challenges.
Problem
The ISIC 2019 challenge requires skin-lesion classification despite severe class imbalance, varying image resolutions, and an unknown class absent from the training data.
Method
The approach combines loss balancing, multi-resolution EfficientNets with varied cropping, external data for the unknown class, metadata fusion through a two-path architecture, and ensembling.
Results
The method ranked first in both ISIC 2019 challenge tasks.
Takeaways & Limitations
EfficientNets with mixed input resolutions and architectural diversity were useful for skin-lesion classification, while unknown-class prediction and metadata use remained challenging.
Takeaways & Limitations
The unknown class lacked metadata in the training images, and the models appeared to overfit to this missing-data characteristic.
Abstract
from arXiv · showhide
In this paper, we describe our method for the ISIC 2019 Skin Lesion Classification Challenge. The challenge comes with two tasks. For task 1, skin lesions have to be classified based on dermoscopic images. For task 2, dermoscopic images and additional patient meta data have to be used. A diverse dataset of 25000 images was provided for training, containing images from eight classes. The final test set contains an additional, unknown class. We address this challenging problem with a simple, data driven approach by including external data with skin lesions types that are not present in the training set. Furthermore, multi-class skin lesion classification comes with the problem of severe class imbalance. We try to overcome this problem by using loss balancing. Also, the dataset contains images with very different resolutions. We take care of this property by considering different model input resolutions and different cropping strategies. To incorporate meta data such as age, anatomical site, and sex, we use an additional dense neural network and fuse its features with the CNN. We aggregate all our models with an ensembling strategy where we search for the optimal subset of models. Our best ensemble achieves a balanced accuracy of 74.2% using five-fold cross-validation. On the official test set our method is ranked first for both tasks with a balanced accuracy of 63.6% for task 1 and 63.4% for task 2.
1 Introduction
The ISIC 2019 challenge extends skin-lesion classification with an unknown test class, severe class imbalance, heterogeneous image resolutions, and two tasks involving images alone or images plus metadata. The paper addresses these issues with external data, preprocessing, balanced loss, multi-resolution models, metadata fusion, and ensembling.
- An unknown test class is absent from the training data, complicating evaluation and prediction.
- Severe class imbalance remains a major challenge in real-world skin-lesion datasets.
- Training images vary in resolution and preprocessing because they come from multiple datasets and protocols.
- The challenge evaluates dermoscopic-image classification alone and classification using images plus patient metadata.
- The proposed procedure combines external data, loss balancing, cropping and color preprocessing, multi-resolution CNNs, metadata fusion, and model ensembling.
2 Materials and Methods
The method combines heterogeneous dermoscopy datasets, preprocessing for uncropped and variably prepared images, balanced EfficientNet training, metadata fusion, and ensemble selection. External images provide a broad ninth class, while evaluation uses lesion-grouped five-fold splits over the eight known classes.
- 2.1 Datasets: The main training set contains 25,331 dermoscopic images from multiple sites, preprocessing protocols, and eight lesion classes.It combines HAM10000, BCN 20000, and MSK data with different image sizes and preparation characteristics.
- 2.1 Datasets: External data include 995 images from the 7-point dataset and 1,339 in-house images, including 353 images for a broad unknown-class pool.The pool contains healthy skin, angiomas, warts, cysts, and other benign alterations.
- 2.1 Datasets: Models predict nine classes during training, but internal evaluation excludes unknown-class images and scores only the eight known classes.All images from the same lesion are kept in the same cross-validation fold.
- 2.2 Preprocessing: Uncropped images are binarized and ellipse-analyzed to derive a bounding box covering the relevant dermoscopy field of view.The method automatically cropped 6,226 training images and 3,864 test images.
- 2.2 Preprocessing: Preprocessing also applies Shades of Gray color constancy with Minkowski norm p = 6 and resizes larger images.
- 2.2 Preprocessing: Task 2 encodes anatomical site and sex as one-hot features and represents missing properties with all-zero feature groups.Age is also encoded for the metadata network, while missing metadata are addressed during training.
- 2.3 Deep Learning Models: The image pathway uses pretrained EfficientNets with extensive augmentation and multiple cropping and resizing strategies.Augmentations include brightness and contrast changes, flipping, rotation, scaling, shear, and CutOut.
- 2.3 Deep Learning Models: CNNs use frequency-weighted cross-entropy, with class weight ni = (N/Ni)^k and k = 1 selected as best.Models are trained with Adam for 100 epochs, and the best and final checkpoints are retained.
3 Results
Cross-validation favored larger EfficientNets and substantially improved performance through ensembling, while metadata generally added 1–2 percentage points. Official-test performance was lower than cross-validation, with especially reduced performance for the unknown class.
- Cross-validation results: Ensembling substantially improved cross-validation performance, and the optimal ensemble selected nine of sixteen configurations.The optimal ensemble used a searched subset rather than all configurations.
- Model and input strategies: Larger EfficientNets tended to perform better, while same-sized and random-resize cropping performed similarly in most cases.The cross-validation comparison covered different configurations and input resolutions.
- Metadata: 1 to 2 percentage points of performance improvement generally came from incorporating patient metadata, especially for smaller models.Larger models showed only minor performance changes from metadata, while the final ensemble improved.
- Official test results: Official-test performance was substantially lower than cross-validation, and Task 2 performed lower than Task 1.The submitted ensembles differed in checkpoint selection across tasks.
- Unknown class: The unknown class performed substantially worse than the other classes across several official-test metrics, with further reduction for Task 2 compared with Task 1.Table 2 reports AUC, AUC-S, sensitivity, and specificity for each class.
4 Discussion
The discussion finds that multi-resolution EfficientNets and architectural diversity support strong skin lesion classification, while metadata offers only slight cross-validation gains and unknown-class performance remains difficult.
- Model resolution and architecture: Large-input EfficientNets performed better, consistent with EfficientNet scaling models’ width and depth with input size while using fewer parameters and less computation.The authors suggest these scaling concepts transfer well to skin lesion classification.
- Metadata: Adding patient metadata slightly improved cross-validation performance, particularly for smaller, lower-performing models.This pattern may indicate that metadata helps models that do not extract all available information from images alone.
- Unknown class: Unknown-class performance was substantially lower than performance on the other classes, which may explain the gap between cross-validation and official-test results.Cross-validation did not include unknown-class images because their representativeness was uncertain.
- Unknown class: Task 2 showed no corresponding improvement for the unknown class, likely because unknown-class training images lacked metadata and models overfit to missingness.Metadata dropout was attempted but did not prevent this apparent overfitting.
- Model resolution and architecture: A mixture of input resolutions helped cover multi-scale context, while the largest EfficientNets performed best in the final ensemble.SENet154 and ResNext models were also selected, indicating architectural variability was useful.
5 Conclusion
The method ranked first in both ISIC 2019 tasks by combining loss balancing, multi-resolution EfficientNets, external data, ensembling, and fused image-metadata modeling. The authors nevertheless identify unknown-class prediction and optimal metadata use as ongoing challenges.
- Conclusion: The method ranked first in both tasks of the ISIC 2019 Skin Lesion Classification Challenge.It combined loss balancing, varied EfficientNet resolutions and crops, healthy-skin external data, and a two-path image-metadata architecture.
- Conclusion: Predicting the unknown class and optimally using metadata remained challenging problems despite the method’s state-of-the-art performance.These challenges are stated as unresolved in the conclusion.