PaperlayerView exact revision on arXiv

Source-linked AI summary

Skin Lesion Classification Using CNNs with Patch-Based Attention and Diagnosis-Guided Loss Weighting

Nils Gessert, Thilo Sentker, Frederic Madesta, Rüdiger Schmitz, Helge Kniep, Ivo Baltruschat, René Werner, Alexander Schlaefer

arXiv:1905.02793v2cs.CV

TL;DR

The paper addresses high-resolution image use and severe class imbalance in multi-class skin lesion classification. It adds patch-based attention to pretrained CNNs and compares sampling and loss-weighting strategies, finding improved mean sensitivity from both attention and diagnosis-guided weighting.

  • Problem

    The paper addresses the difficulty of using high-resolution images with pretrained CNNs and severe class imbalance in multi-class skin lesion classification.

  • Method

    The study integrates patch-based attention into three pretrained architectures and compares oversampling, balanced sampling, class-frequency weighting, and diagnosis-guided loss weighting.

  • Results

    The patch-based attention method improves MC-sensitivity by up to 7 %, while balanced sampling improves it by 5 % to 8 % and attention with five crops is only 0.2 −0.5 percent points below averaging with 16 crops.

  • Takeaways & Limitations

    Patch-based attention enables pretrained architectures to use high-resolution images with global context, while diagnosis-guided weighting improves classification under class imbalance.

Abstract

from arXiv · show

Objective: This work addresses two key problems of skin lesion classification. The first problem is the effective use of high-resolution images with pretrained standard architectures for image classification. The second problem is the high class imbalance encountered in real-world multi-class datasets. Methods: To use high-resolution images, we propose a novel patch-based attention architecture that provides global context between small, high-resolution patches. We modify three pretrained architectures and study the performance of patch-based attention. To counter class imbalance problems, we compare oversampling, balanced batch sampling, and class-specific loss weighting. Additionally, we propose a novel diagnosis-guided loss weighting method which takes the method used for ground-truth annotation into account. Results: Our patch-based attention mechanism outperforms previous methods and improves the mean sensitivity by 7%. Class balancing significantly improves the mean sensitivity and we show that our diagnosis-guided loss weighting method improves the mean sensitivity by 3% over normal loss balancing. Conclusion: The novel patch-based attention mechanism can be integrated into pretrained architectures and provides global context between local patches while outperforming other patch-based methods. Hence, pretrained architectures can be readily used with high-resolution images without downsampling. The new diagnosis-guided loss weighting method outperforms other methods and allows for effective training when facing class imbalance. Significance: The proposed methods improve automatic skin lesion classification. They can be extended to other clinical applications where high-resolution image data and class imbalance are relevant.

I. INTRODUCTION

The paper targets high-resolution image use and severe class imbalance in multi-class skin lesion classification. It proposes patch-based global-context modeling and diagnosis-guided loss weighting to address these challenges.

  • Motivation: Skin lesion classification matters because skin cancer is widespread and expert assessment can be subjective and limited by specialist availability.Dermoscopy improves diagnostic performance, while computer-aided systems aim to reduce interobserver variability and address limited access to trained experts.
  • Motivation: The HAM10000 dataset provides a public benchmark for multi-class lesion diagnosis, improving comparability and reproducibility across studies.It contains 10 015 dermoscopic images and supports classification beyond the binary biopsy-or-no-biopsy decision.
  • Key problems: High-resolution information is difficult to use because CNNs typically downsample images, potentially removing fine-grained features important for medical diagnosis.Existing multi-resolution or segmentation-based approaches can require extensive pooling or risk excluding relevant regions.
  • Key problems: Multi-crop processing enables pretrained standard architectures to use small high-resolution patches, but traditional averaging does not model global context between patches.The proposed approach extends multi-crop processing with patch-based attention while retaining transfer-learning compatibility.
  • Contributions: Diagnosis-guided loss weighting incorporates the ground-truth annotation method, assigning higher weight when costly diagnosis of a benign lesion suggests difficult image classification.The paper compares this strategy with oversampling, balanced sampling, and class-frequency-based weighting.
  • Contributions: The authors make code and training, validation, and test splits publicly available to support reproducible research.

II. METHODS

The study evaluates classification methods on imbalanced dermoscopic datasets, using controlled splits and reporting results on independent test data. It also tests generalization on a second dataset with six matched classes.

  • A. Datasets: HAM contains seven mutually exclusive lesion classes with a real-world distribution containing more benign images and a key class imbalance.The dataset also over-represents malignant cases relative to their real-world prevalence, and similar imbalance appears in other public datasets.
  • A. Datasets: HAM is divided into four equally sized parts, with one held-out test set and three-fold cross-validation for hyperparameter tuning.The split uses 2500 test examples, 5000 training examples, and 2500 validation examples.
  • A. Datasets: HAM, SPC, and ISIC all exhibit highly imbalanced class distributions across lesion types.The listed classes include melanoma, melanocytic nevus, basal cell carcinoma, actinic keratosis, benign keratosis, dermatofibroma, and vascular lesions.
  • A. Datasets: The HAM images receive no additional preprocessing beyond publisher-applied histogram equalization, centering, downsampling, and resizing.The images were resized by the publishers to 600 × 450, and the ISIC 2018 Challenge test set is excluded because public ground truth is unavailable.
  • A. Datasets: The proposed attention mechanism is also evaluated on the SPC dataset after matching its classes to HAM and excluding unmatched images.SPC lacks actinic keratosis, so experiments use six classes with a 395-image test set and 616 training images.

B. Metrics

The study ranks models with multi-class sensitivity because ordinary accuracy favors overrepresented classes. It also reports multi-class specificity and F1-score across three pretrained architectures.

  • B. Metrics: MC-Sensitivity is used to rank approaches because ordinary accuracy can favor correct predictions on overrepresented classes in an imbalanced dataset.
  • B. Metrics: MC-Sensitivity treats all classes equally, making it challenging when the dataset is highly imbalanced.The study additionally reports MC-specificity and F1-score.
  • C. Baseline Models: The experiments use Inception-V3, DenseNet-121, and SE-ResNeXt-50 as pretrained baseline architectures.These models represent Inception, densely connected CNN, and multipath ResNeXt architectures with squeeze-and-excitation recalibration, respectively.

D. Model Input Strategies

The study compares downsampling, single-crop, multi-crop, and ordered-crop strategies for mapping images to 224 × 224 model inputs. Ordered cropping adds patch dropout to reduce overfitting from fixed locations.

  • Input strategies: All input strategies map images to the standard model input size of 224 × 224.
  • Downsampling: Downsampling resizes the whole image to the model input size during both training and evaluation.
  • Single-Crop: Single-crop uses random resized crops during training and a center crop covering 85 % of the image during evaluation.This strategy increases training variation while retaining a fixed evaluation crop.
  • Multi-Crop: Multi-crop extracts non-resized patches and averages predictions from 5, 9, or 16 predefined evaluation locations.Five crops cover the four corners and center; nine and sixteen crops are distributed over the image with overlap.
  • Ordered-Crop: Ordered-crop uses fixed patch locations during training and evaluation, so patch dropout randomly zeros patches to reduce overfitting.The dropout probability pd is varied experimentally.

E. Local and Global Information Aggregation Strategies

The study compares strategies for combining local and global information from high-resolution image patches. Its attention mechanism learns interactions between patches while preserving computational efficiency and compatibility with pretrained architectures.

  • Local and Global Information Aggregation Strategies: Multi-Crop emphasizes local features, but averaging isolated patches provides only weak global context.The approach uses small crops and combines their predictions, leaving relationships between patches weakly represented.
  • Local and Global Information Aggregation Strategies: The proposed patch-based attention mechanism recovers global context between small patches.It reshapes patch features, computes patch weights, and reweights patches through learned interactions.
  • Local and Global Information Aggregation Strategies: Attention weights are transformed with a fully connected layer and sigmoid before reweighting patch feature tensors.The reweighted tensor is reshaped for continued simultaneous processing, and predictions are averaged across crops.
  • Local and Global Information Aggregation Strategies: Attention modules can be inserted at different locations in pretrained architectures because they operate on the architecture-independent batch dimension.The effect of insertion location is not directly comparable across architectures because their internal structures differ.
  • Local and Global Information Aggregation Strategies: The attention approach is compared with recurrent patch combination using a GRU that learns a feature combination from the patches.The recurrent alternative replaces the attention block at the model output.

F. Balancing Strategies

The paper evaluates sampling and loss-weighting strategies for class imbalance, including a diagnosis-guided method that uses the annotation procedure as additional information.

  • Balancing Strategies: Oversampling repeats minority-class samples so every class contributes an equal number of samples to the training pool.Training then samples uniformly and randomly from the expanded pool.
  • Balancing Strategies: Balanced batches construct each batch with exactly the same number of samples from every class.This provides stricter per-batch balance than oversampling over the entire training process.
  • Balancing Strategies: Loss weighting increases the contribution of underrepresented classes using n_i = (N/N_i)^k.The exponent k controls weighting strength, with k values of 0.5 and 1.5 evaluated around a baseline of 1.
  • Balancing Strategies: Diagnosis-guided loss weighting incorporates the method used to establish each lesion’s ground-truth annotation.The annotation categories include expert consensus, serial imaging, confocal microscopy, and histopathology.

G. Training

Training uses online image augmentation, cross-entropy loss, and Adam-based stochastic optimization implemented in PyTorch.

  • Training: Online augmentation randomly flips images along both axes and changes brightness and saturation.The models are trained with cross-entropy loss and stochastic gradient descent using Adam.
  • Training: The models are implemented in PyTorch.Additional implementation details are provided in the authors’ publicly available code.

H. Experiments

The experiments assess patch-based attention, crop counts, attention visualizations, patch dropout, and class-balancing approaches across the study’s datasets and model settings.

  • Patch-Based Attention Experiments: The experiments compare input strategies for local and global processing and evaluate the proposed attention method against alternative approaches.The attention experiments use loss weighting and report results primarily for HAM, with additional evaluation on SPC.
  • Patch-Based Attention Experiments: The study visualizes learned attention weights and compares different numbers of image crops.It also analyzes patch dropout using different values of p_d.
  • Class-Balancing Experiments: The balancing experiments compare alternative approaches and vary the strength parameter k for loss weighting.These experiments use Multi-Crop models with N_C = 9 and include the diagnosis-guided strategy.
  • Patch-Based Attention Experiments: Table II reports results for initial, end, and dual attention configurations alongside other methods.The table distinguishes attention placement at the model input, output, or both locations.
  • Patch-Based Attention Experiments: Table III evaluates different crop counts for models using attention and other crop-based strategies.The table concerns crops taken from the fully sized image, with attention denoting a block at the model beginning.
  • Visualization and Regularization: Figures 4 and 5 examine attention-weight visualizations and the performance impact of patch dropout.Figure 4 uses N_C = 9 and Figure 5 focuses on Densenet121 with initial attention.

III. RESULTS

The patch-based attention model performs best across the evaluated comparisons, while balanced sampling and diagnosis-guided loss weighting improve class-imbalance performance.

  • Patch-based attention: 7%: The attention-based method improves MC-sensitivity by up to 7% over previous methods.The previous recurrent-unit approach to global context modeling is substantially outperformed.
  • Patch-based attention: Attention weights emphasize lesion-containing patches, including lesion borders and smaller lesion parts, independently of lesion location.Brighter borders indicate higher patch weighting.
  • Crop efficiency: Five attention-model crops achieve almost the same performance as 16 Multi-Crop crops while processing one-third as many crops.Increasing attention-model crops from 9 to 16 produces only minor differences.
  • Cross-dataset evaluation: The attention mechanism performs best on the SPC dataset, although overall performance is slightly lower than on HAM.Relative performance differences between methods are similar across the two datasets.
  • Patch dropout: Patch dropout improves performance considerably for all evaluated NC values, with performance declining slightly at larger dropout probabilities.Performance first improves as patch-dropout strength increases.
  • Class balancing: Balanced sampling clearly improves performance, whereas simple oversampling produces similar or worse results.Loss weighting performs slightly better than balanced sampling, and diagnosis-guided weighting achieves the overall best performance.
  • Loss-weighting strength: For loss-weighting exponent k, performance rises below k = 1 and MC-sensitivity quickly saturates above k = 1.At larger k, MC-sensitivity improves or remains unchanged while F1-score deteriorates; MC-specificity varies only slightly.

IV. DISCUSSION

The discussion connects fine-grained lesion classification to high-resolution processing and class imbalance, emphasizing efficient global context, regularization, and diagnosis-aware weighting. Results generalize across architectures and remain comparatively strong on another dataset, though HAM-specific cropping may bias attention.

  • Discussion: Fine-grained lesion differentiation better matches dermatologists’ workflow but introduces additional classification challenges.The discussion contrasts this multi-class setting with earlier biopsy-versus-no-biopsy work.
  • High-resolution processing: Patch-based attention adds global context between local high-resolution patches while remaining compatible with pretrained architectures.This addresses the information loss caused by downsampling without abandoning transfer learning.
  • High-resolution processing: 7%: The attention mechanism improves MC-sensitivity while increasing the number of parameters by approximately 0.001%.The method also maintains similar accuracy with fewer crops, supporting faster prediction.
  • Regularization: Patch dropout regularizes patch-based models by forcing them to handle occasionally missing patches, countering overfitting to fixed patch positions.The fixed crop order can otherwise encourage reliance on particular patch locations.
  • Limitations and generalization: The attention mechanism may be biased by HAM’s particular zoom and manual cropping, although relative improvements persist on SPC.SPC performance is slightly lower, likely because of its smaller training set and image artifacts.
  • Class imbalance: 8%: Balanced sampling improves MC-sensitivity by 5% to 8%, while loss weighting improves it by 10% over no balancing and by 2% to 4% over balanced sampling.Diagnosis-guided weighting adds approximately 3% further improvement, indicating that annotation method contains relevant learning information.
  • Generalization: Key insights are supported by consistent results across multiple state-of-the-art architectures.The experiments were conducted across multiple architectures rather than a single model.

V. CONCLUSION

The paper addresses high-resolution input and severe class imbalance in multi-class dermoscopic classification using pretrained CNNs, patch-based attention, and diagnosis-guided loss weighting.

  • The study targets high-resolution image use and severe class imbalance in multi-class dermoscopic lesion classification.
  • Patch-based attention combines local crops with global context and improves performance across three network architectures and two datasets.
  • The study compares sampling and weighting schemes for addressing class imbalance during training.
  • Diagnosis-guided loss weighting uses the type of ground-truth annotation to improve classification performance.
  • Both methods are relevant to other clinical applications involving high-resolution data and class imbalance.
Loading 1905.02793v2…