PaperlayerView exact revision on arXiv

Source-linked AI summary

Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillation

Mingi Ji, Seungjae Shin, Seunghyun Hwang, Gibeom Park, Il-Chul Moon

arXiv:2103.08273v1cs.CV

TL;DR

Existing self-knowledge distillation avoids pretrained teachers but can lose spatial locality or lack refined feature knowledge. FRSKD uses an auxiliary self-teacher to transfer refined feature maps and soft labels, and reports improvements across classification and semantic segmentation while remaining compatible with other self-distillation methods and augmentation.

  • Problem

    Existing self-knowledge distillation can lose spatial information through augmentation, while auxiliary approaches struggle to provide refined feature knowledge for feature-map distillation.

  • Method

    FRSKD introduces an auxiliary self-teacher network that transfers refined feature maps and soft labels to the classifier, using an efficient top-down and bottom-up structure with adaptive channel dimensions.

  • Results

    FRSKD reports state-of-the-art image-classification performance across various datasets, improves semantic segmentation, and is compatible with other self-distillation methods and data augmentation.

  • Takeaways & Limitations

    FRSKD extends self-knowledge distillation to classification and semantic segmentation while supporting refined feature-map and soft-label distillation.

Abstract

from arXiv · show

Knowledge distillation is a method of transferring the knowledge from a pretrained complex teacher model to a student model, so a smaller network can replace a large teacher network at the deployment stage. To reduce the necessity of training a large teacher model, the recent literatures introduced a self-knowledge distillation, which trains a student network progressively to distill its own knowledge without a pretrained teacher network. While Self-knowledge distillation is largely divided into a data augmentation based approach and an auxiliary network based approach, the data augmentation approach looses its local information in the augmentation process, which hinders its applicability to diverse vision tasks, such as semantic segmentation. Moreover, these knowledge distillation approaches do not receive the refined feature maps, which are prevalent in the object detection and semantic segmentation community. This paper proposes a novel self-knowledge distillation method, Feature Refinement via Self-Knowledge Distillation (FRSKD), which utilizes an auxiliary self-teacher network to transfer a refined knowledge for the classifier network. Our proposed method, FRSKD, can utilize both soft label and feature-map distillations for the self-knowledge distillation. Therefore, FRSKD can be applied to classification, and semantic segmentation, which emphasize preserving the local information. We demonstrate the effectiveness of FRSKD by enumerating its performance improvements in diverse tasks and benchmark datasets. The implemented code is available at https://github.com/MingiJi/FRSKD.

1. Introduction

Knowledge distillation compresses large vision models, but conventional methods require costly pretrained teachers and existing self-distillation approaches struggle to refine local feature information. FRSKD addresses this gap with an auxiliary self-teacher that transfers refined features and soft labels for classification and semantic segmentation.

  • Knowledge distillation transfers knowledge from a pretrained teacher to a smaller student that can replace it at deployment.
  • Pretraining and running a large teacher impose substantial computational burdens, motivating self-knowledge distillation without a pretrained teacher.
  • Existing self-knowledge distillation uses data augmentation or auxiliary branches, but these approaches struggle to generate refined feature or soft-label knowledge.
  • FRSKD introduces an auxiliary self-teacher network that transfers refined feature-map knowledge to the classifier while supporting soft-label distillation.
  • FRSKD applies to classification and semantic segmentation because it supports feature-map distillation while preserving local information.
  • FRSKD achieves state-of-the-art image-classification performance across various datasets, improves semantic segmentation, and remains compatible with other self-distillation methods and augmentation.

2. Related Work

Prior work develops knowledge distillation through logits, intermediate features, and self-distillation, but existing self-distillation structures do not provide sufficiently refined feature knowledge. FRSKD adds an auxiliary self-teacher adapted from multi-scale feature networks to generate refined feature maps and soft labels efficiently.

  • Knowledge distillation transfers logits, feature maps, or penultimate-layer relations from a pretrained complex network to train a simpler student.
  • Self-distillation methods use data augmentation or auxiliary branches, but augmentation can disrupt spatial locality needed for feature-map distillation.
  • Prior auxiliary structures do not make features more complex, limiting their ability to extract refined knowledge for feature-map distillation.
  • FRSKD proposes an auxiliary self-teacher that generates refined feature maps and soft labels from a single instance.
  • The self-teacher adapts BiFPN-style top-down and bottom-up paths for classification, while varying channel dimensions to reduce computation.

3. Method

FRSKD uses an auxiliary self-teacher network to refine classifier feature maps through top-down and bottom-up paths, then distills both refined features and soft labels back to the classifier.

  • Self-Teacher Network: FRSKD modifies BiFPN-style top-down and bottom-up paths to build an auxiliary self-teacher for refining classifier feature maps.The self-teacher receives feature maps from the classifier network and produces refined feature maps and soft labels.
  • Self-Teacher Network: Channel dimensions in lateral convolutions scale with feature-map depth through d_i = w × c_i, reducing lateral-layer computation.Deeper layers receive higher channel dimensions, while the channel-width parameter w controls the scaling.
  • Self-Teacher Network: The self-teacher combines differently sized features through top-down and bottom-up paths, using diagonal connections to link shallow, middle, and deep layers efficiently.The design uses direct lateral inputs for the shallowest and deepest bottom-up layers and adds two diagonal forward connections.
  • Self-Feature Distillation: FRSKD also distills the self-teacher’s soft labels and combines classification, soft-label, and feature-map losses during joint optimization.The classifier and self-teacher are optimized simultaneously, while distillation losses update only the student classifier to prevent collapse.
  • Self-Feature Distillation: FRSKD distills refined feature maps by applying an attention-transfer loss that makes the classifier learn their spatial locality.Channel-wise pooling and L2 normalization abstract spatial information before feature-map comparison.

4. Experiments

Experiments evaluate FRSKD across classification, semantic segmentation, attention-map refinement, self-teacher design, and augmentation compatibility. Across these settings, FRSKD generally outperforms self-knowledge-distillation baselines and extends to tasks requiring local information.

  • Classification: Experiments cover classification on seven datasets, including CIFAR-100, TinyImageNet, FGVR datasets, and ImageNet.The evaluation includes small-scale, fine-grained, and large-scale image-recognition settings.
  • Classification: FRSKD consistently outperforms other self-knowledge-distillation methods on CIFAR-100, TinyImageNet, and FGVR classification.FRSKD without feature distillation also outperforms other baselines, while full FRSKD further improves over that variant.
  • Classification: FRSKD+SLA achieves large performance improvements on most experiments and outperforms other methods on FGVR tasks.These results support compatibility between FRSKD and data-augmentation-based self-knowledge distillation.
  • Semantic Segmentation: Semantic-segmentation experiments use EfficientDet with stacked BiFPN layers and an additional BiFPN-based self-teacher network.The setting combines VOC2007 and VOC2012 trainval for training and uses VOC2007 test data for validation.
  • Further Analyses on FRSKD: Self-teacher attention maps concentrate more appropriately on target objects than classifier maps for CUB200 and Dogs examples.For MIT67, the comparison concerns attention to overall scene context rather than a single object.
  • Further Analyses on FRSKD: A high-channel BiFPNc self-teacher offers performance compatible with BiFPN while requiring less computation.The analysis contrasts self-teacher efficiency with larger BiFPN and duplicated-classifier approaches.

5. Conclusion

FRSKD uses a specialized self-knowledge-distillation structure with top-down and bottom-up paths to refine feature maps and soft labels. It applies to classification and semantic segmentation, with quantitatively demonstrated performance improvements and ablation-based analysis.

  • FRSKD uses top-down and bottom-up paths to provide refined feature maps and soft labels to the classifier network.
  • FRSKD reduces parameters by changing channel dimensions while maintaining feature-map refinement.
  • FRSKD applies self-knowledge distillation to classification and semantic segmentation.
  • The paper reports large quantitative performance improvements and uses ablation studies to examine the method’s working mechanisms.

6. Implementation Details

The implementation configures self-teacher widths and augmentation parameters for classification, while semantic segmentation uses a narrower self-teacher and pretrained backbones with warm-up before distillation.

  • Classification: Classification uses width hyperparameter w=2, producing self-teacher channel dimensions of 64, 128, 256 for WRN-16-2 and 128, 256, 512, 1028 for ResNet18.
  • Classification: Mixup uses beta-distribution parameters 0.2 for CIFAR-100 and TinyImageNet and 0.3 for FGVR, while Cutmix uses 1.0 for all datasets.
  • Semantic segmentation: Semantic segmentation uses width hyperparameter one and attaches the self-teacher network with two repeated BiFPN components.
  • Semantic segmentation: Semantic segmentation uses pretrained EfficientNet-B0 and EfficientNet-B1 backbones, with 40 warm-up epochs before FRSKD annealing because early distillation can hinder backbone training.

7. Sensitivity Analysis

The sensitivity analysis evaluates FRSKD across α and β settings on CIFAR-100 and MIT67. The method is robust to these hyperparameters, although different settings perform best across experiments.

  • Figure 4 reports accuracy averaged over three repeated experiments for WRN-16-2 on CIFAR100 and ResNet18 on MIT67.
  • The analysis varies α over {1, 2, 3} and β over {50, 100, 200, 500} using WRN-16-2 on CIFAR-100 and ResNet18 on MIT67.
  • FRSKD is robust to α and β, although different hyperparameter settings perform well across the evaluated models and datasets.

8. Qualitative Attention Map Comparison

The qualitative attention-map analysis tracks how classifier and self-teacher focus changes during training. Both increasingly concentrate on the main object, with a larger early-training difference between them.

  • As training progresses, both the classifier network and self-teacher network concentrate attention on the main object.
  • The difference in main-object concentration between the classifier and self-teacher is larger during early training.
Loading 2103.08273v1…