PaperlayerView exact revision on arXiv

Source-linked AI summary

SSD-KD: A Self-supervised Diverse Knowledge Distillation Method for Lightweight Skin Lesion Classification Using Dermoscopic Images

Yongwei Wang, Yuheng Wang, Tim K. Lee, Chunyan Miao, Z. Jane Wang

arXiv:2203.11490v2cs.CV

TL;DR

Limited computing resources constrain the use of complex skin-diagnosis models on portable devices. SSD-KD combines relational knowledge distillation with self-supervision and weighted loss; the authors report promising multi-disease classification with low model complexity and storage requirements while improving predictive performance.

  • Problem

    Complex CNNs can require substantial computation and memory, constraining their use on embedded devices and other resource-limited settings.

    Source: §1 · p. 2

  • Method

    SSD-KD transfers intra- and inter-instance knowledge with self-supervised auxiliary training and weighted cross-entropy to a lightweight student model.

    Source: §5 · p. 12

  • Results

    On ISIC 2019, the authors report promising multi-disease classification with low model complexity and storage requirements while improving predictive performance.

    Source: §5 · p. 12

  • Takeaways & Limitations

    The authors conclude that SSD-KD could assist the promotion of deep-learning techniques in clinical practice while retaining low complexity and storage requirements.

    Source: §5 · p. 12

    The study selected two representative teacher and student models, and the authors describe this as a starting point for exploring other architectures.

    Source: §4 · p. 12

Abstract

from arXiv · show

Skin cancer is one of the most common types of malignancy, affecting a large population and causing a heavy economic burden worldwide. Over the last few years, computer-aided diagnosis has been rapidly developed and make great progress in healthcare and medical practices due to the advances in artificial intelligence. However, most studies in skin cancer detection keep pursuing high prediction accuracies without considering the limitation of computing resources on portable devices. In this case, knowledge distillation (KD) has been proven as an efficient tool to help improve the adaptability of lightweight models under limited resources, meanwhile keeping a high-level representation capability. To bridge the gap, this study specifically proposes a novel method, termed SSD-KD, that unifies diverse knowledge into a generic KD framework for skin diseases classification. Our method models an intra-instance relational feature representation and integrates it with existing KD research. A dual relational knowledge distillation architecture is self-supervisedly trained while the weighted softened outputs are also exploited to enable the student model to capture richer knowledge from the teacher model. To demonstrate the effectiveness of our method, we conduct experiments on ISIC 2019, a large-scale open-accessed benchmark of skin diseases dermoscopic images. Experiments show that our distilled lightweight model can achieve an accuracy as high as 85% for the classification tasks of 8 different skin diseases with minimal parameters and computing requirements. Ablation studies confirm the effectiveness of our intra- and inter-instance relational knowledge integration strategy. Compared with state-of-the-art knowledge distillation techniques, the proposed method demonstrates improved performances for multi-diseases classification on the large-scale dermoscopy database.

1. Introduction

The authors report that SSD-KD can help a lightweight student match or exceed its teacher on multi-class skin lesion classification. The method combines intra- and inter-instance relational knowledge with self-supervision and weighted loss to address diagnostic and resource constraints.

1. Introduction

  • Early diagnosis is important because melanoma and keratinocyte carcinomas have a promising prognosis when detected and treated at an early stage.The paper also describes dermoscopy as a way to assess lesion colors and structures that are hard to see with the naked eye, while traditional diagnosis is time-consuming and expensive.Source: §1 · p. 1
  • Sophisticated CNNs can require substantial parameters and computation, limiting their practicality on embedded devices and in resource-constrained settings.The paper gives ResNet50 as an example, with about 25.6 million parameters and 4.11 GFlops.Source: §1 · p. 2
  • Prior skin-diagnosis KD work largely transferred category information and often overlooked intra- and inter-instance relations that may preserve diagnostic knowledge.The authors note limited work on multi-disease classification using large-scale dermoscopy datasets.Source: §1 · p. 3
  • The authors evaluate SSD-KD on ISIC 2019, reporting more than 20,000 dermoscopy images across eight skin-disease types.They also compare it with state-of-the-art methods and conduct ablation studies.Source: §1 · p. 3
  • SSD-KD combines conventional KD knowledge with intra-instance relations, self-supervised auxiliary learning, and weighted cross-entropy for classes with fewer subjects.The paper describes dual relational blocks for intra- and inter-instance information and says auxiliary learning stabilizes their training dynamics.Source: §1 · p. 3
  • The authors report that SSD-KD can help the student achieve comparable or better performance than the teacher in multi-class skin lesion classification.They present the lightweight method as relevant to settings with limited computing resources.Source: §1 · p. 3

2. Methodology

SSD-KD transfers logits, batch-level relations, and within-image channel relations from a pretrained teacher to a student trained from scratch. It combines these signals with self-supervised teacher predictions and weighted classification loss for imbalanced skin-disease data.

2. Methodology

  • The framework distills logits, inter-instance relations, and intra-instance channel relations from a pretrained teacher to a student trained from scratch.The three signals use the teacher’s output logits, batch-level representations, and feature maps within each image.Source: §2 · p. 4

2.1. Logit-based dark knowledge

  • BLKD matches teacher and student softened logits while supervising student predictions with weighted cross-entropy to address disease-class imbalance.Class weights are inversely proportional to class frequency in the training set.Source: §2.1 · pp. 4–5

2.2. Inter-instance relational knowledge

  • DRKD aligns teacher and student distance-wise and angle-wise relations among embeddings of instances within a training batch.The method uses these batch-level relations as additional supervision in response to data scarcity in skin-cancer diagnosis.Source: §2.2 · p. 5

2.3. Intra-instance relational knowledge

  • CRKD matches relations between channel feature maps within each image, using the last convolutional layer and a 1 × 1 student-channel adaptation module.The channel adaptation aligns student and teacher channel counts before relation matching.Source: §2.3 · p. 5

2.4. Diverse knowledge integration

  • D-KD combines BLKD, DRKD, and CRKD losses, and SSD-KD incorporates self-supervised teacher predictions into the training objective.The self-supervised strategy follows a setup that pre-trains the teacher with contrastive loss and transfers its self-supervised predictions to the student.Source: §2.4 · p. 6

3. Experiments and results

Experiments on ISIC 2019 compare SSD-KD with other distillation methods, assess its components, and examine prediction and activation visualizations. SSD-KD leads the compared methods in ACC, BACC, and mAP, and ranks second in AUC.

3.1. Dataset

  • 25,331 ISIC 2019 dermoscopic images cover eight skin diseases and include diagnosis labels, with some images also providing metadata.Figure 2 presents example images and their detailed information.Source: §3.1 · p. 7, Figure 2 · p. 7

3.2. Implementation details

  • ResNet50 teachers distill into MobileNetV2 students with different backbones; images were split 10% for testing and 8:2 for training and validation, resized to 224 × 224, and fine-tuned for 150 epochs.The authors describe this differing-backbone setup as more challenging than distillation studies using the same structure with different numbers of layers.Source: §3.2 · pp. 7–8

3.3. Evaluation metrics

  • The study evaluates multi-class skin-disease predictions using ACC, BACC, mAP, and AUC.Source: §3.3 · p. 8

3.4. Comparison of the teacher and student models

  • 82.0% accuracy for ResNet50 compares with 75.4% for MobileNetV2; the teacher has a 98M model size and 25.6M parameters, over six times the student’s parameter count.The teacher’s inference step takes 58.2 ms, while the student uses half that time; the teacher also has over 10 times the FLOPs.Source: §3.4 · p. 8, Table 1 · p. 9

3.5. Comparison with other KD methods

  • SSD-KD achieves the best ACC, BACC, and mAP, and the second-best AUC among the compared distillation methods.Across the methods, accuracy improves by 5%–9%, balanced accuracy by 4%–8%, and mAP by up to 18% over the student without KD.Source: §3.5 · p. 8, Table 2 · p. 9

3.6. Ablation studies

  • DRKD and CRKD each improve all four metrics over BLKD, while self-supervision further improves ACC and BACC; the combined configuration achieves the highest ACC and BACC.Source: §3.6 · p. 9, Table 3 · p. 9

3.7. Visualizations of class activation maps

  • Without KD, the student misclassified 17 melanomas as benign keratoses, whereas the KD student classified them correctly; its discriminative regions also align more with the teacher’s activation maps.Confusion matrices compare models without KD and with different KDs, while ROC curves show the corresponding comparisons.Source: §3.7 · p. 10, Figure 3 · p. 10, Figure 4 · p. 11

4. Discussion and future works

The discussion frames SSD-KD as a way to improve lightweight skin-lesion classification while retaining the complexity advantage needed for resource-limited use. It reports stable trends, notes limits in model selection and tuning, and identifies transfer after fine-tuning as a possible extension.

4. Discussion and future works

  • 82% accuracy for ResNet50 versus 75% for MobileNetV2 illustrates the performance gap between the larger and lightweight models.The discussion presents improving lightweight models while maintaining their complexity advantage as an important direction.Source: §4 · pp. 11–12
  • Figure 5 compares CAMs from different models on examples correctly predicted by the teacher and student only after the proposed method.Panel (a) is the original dermoscopy image; panels (b)–(d) show CAM outputs.Source: Figure 5 · p. 12
  • Ten repeated experiments had standard deviations below 0.01, while module gains over BLKD were modest under fixed hyperparameters.The authors fixed module hyperparameters for a fair comparison and suggest further tuning may improve combinations of modules.Source: §4 · p. 12
  • The authors selected two representative models and identify other architectures, including multi-teacher KD and teaching assistants, as future directions for generalizability.They describe their results as a starting point for this research line.Source: §4 · p. 12
  • The authors propose applying SSD-KD to other medical problems with similar scenarios after fine-tuning.Source: §4 · p. 12

5. Conclusion

The conclusion presents SSD-KD as a lightweight dermoscopic skin-lesion classification framework that combines intra- and inter-instance knowledge with self-supervised training. On ISIC 2019, the authors report promising multi-disease results and describe low complexity and storage requirements alongside improved predictive performance.

5. Conclusion

  • SSD-KD combines intra- and inter-instance knowledge to guide lightweight models in dermoscopic skin-lesion classification.The framework also uses a self-supervised auxiliary training strategy and weighted cross-entropy loss.Source: §5 · p. 12
  • Experiments on ISIC 2019 compared SSD-KD with existing KD methods, and the authors report promising multi-disease classification results.Source: §5 · p. 12
  • The authors report improved predictive performance with low model complexity and storage requirements, which they connect to promoting deep-learning techniques in clinical practice.Source: §5 · p. 12

CRediT authorship contribution statement

The statement assigns conceptualization, methodology, analysis, software, validation, supervision, project administration, funding, and writing contributions across five authors.

CRediT authorship contribution statement

Loading 2203.11490v2…