Source-linked AI summary
A Comprehensive Overhaul of Feature Distillation
Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, Jin Young Choi
TL;DR
Feature distillation methods must determine how to transform features, where to distill them, and how to measure their distance without losing useful information. The paper proposes a coordinated design using margin ReLU, pre-ReLU distillation, asymmetric transforms, and partial L2 distance, achieving improvements across image classification, object detection, and semantic segmentation. In ImageNet, ResNet50 reaches 21.65% top-1 error, outperforming ResNet152.
Problem
Feature distillation requires design choices that preserve important teacher information while compressing a smaller student network.
Method
The method combines margin ReLU teacher transformation, asymmetric student transformation, pre-ReLU distillation, and partial L2 distance.
Results
The method improves performance across image classification, object detection, and semantic segmentation; ResNet50 achieves 21.65% top-1 error on ImageNet.
Takeaways & Limitations
The proposed feature distillation design substantially outperforms state-of-the-art methods across networks and tasks.
Abstract
from arXiv · showhide
We investigate the design aspects of feature distillation methods achieving network compression and propose a novel feature distillation method in which the distillation loss is designed to make a synergy among various aspects: teacher transform, student transform, distillation feature position and distance function. Our proposed distillation loss includes a feature transform with a newly designed margin ReLU, a new distillation feature position, and a partial L2 distance function to skip redundant information giving adverse effects to the compression of student. In ImageNet, our proposed method achieves 21.65% of top-1 error with ResNet50, which outperforms the performance of the teacher network, ResNet152. Our proposed method is evaluated on various tasks such as image classification, object detection and semantic segmentation and achieves a significant performance improvement in all tasks. The code is available at https://sites.google.com/view/byeongho-heo/overhaul
1. Introduction
The paper proposes a feature-distillation loss that coordinates teacher and student transforms, distillation position, and distance function to improve network compression across tasks.
- Knowledge distillation trains a smaller student network under supervision from a larger teacher while allowing architectural differences.
- Output distillation can be limited because a high-performing teacher’s softmax output differs little from the ground truth.
- Prior feature-distillation methods increase transferred information but may deform teacher features or lose information through transformation.
- The proposed loss combines a margin ReLU transform, pre-ReLU distillation position, and partial L2 distance to transfer useful feature information while skipping adverse information.
- The method is evaluated on image classification, object detection, and semantic segmentation, with significant performance improvements reported across these tasks.
2. Motivation
Feature distillation is designed around four interacting choices: teacher transform, student transform, distillation position, and distance function. The proposed design preserves important feature information by using asymmetric transforms, pre-ReLU features, and a distance function that suppresses harmful negative values.
- General formulation: Feature distillation matches transformed teacher and student features by minimizing a distillation loss.The teacher and student features are transformed to compatible dimensions before their distance is computed.
- Teacher transform: Existing teacher transforms often reduce dimensions, binarize features, or compress them, causing information loss before distillation.The paper identifies teacher transformation as a major source of missing information in prior methods.
- Student transform: The proposed student transform is asymmetric: it uses a 1×1 convolution to match feature dimensions without reducing the teacher feature.This avoids the shared information loss produced by using the same transform for both networks.
- Distillation feature position: Distillation is placed before ReLU so the student can learn the teacher’s activation boundary from both negative and positive pre-ReLU values.The exact target-layer location varies with network architecture and is positioned between the last block and first ReLU.
- Distance function: Because negative pre-ReLU values are blocked by ReLU, the distance function should avoid transferring these adverse values despite using pre-ReLU features.This motivates a distance function tailored to the chosen teacher transform and distillation position.
3. Approach
The method distills features before ReLU using margin ReLU and a partial L2 distance, combining teacher and student transforms with a selected distillation position. It adds continuous distillation to the task loss and considers batch-normalization mode during knowledge distillation.
- Distillation position: The distillation position is after the last block of one spatial size and before ReLU, allowing the student to access preserved teacher information.The method uses three target layers for CIFAR and four for ImageNet.
- Teacher transform: Margin ReLU preserves positive teacher responses while assigning negative responses a channel-specific margin.The margin is less than zero and is computed as the expectation of negative responses for each channel over training images.
- Student transform: The student transform uses a regressor consisting of a 1×1 convolution layer and a batch-normalization layer.
- Distance function: Partial L2 distance penalizes higher-than-target negative student responses while avoiding increases toward negative teacher targets that ReLU would block equally.Positive teacher responses are transferred by their exact values.
- Training objective: The proposed loss is used for continuous distillation, with the final objective combining distillation loss and task loss.The study also investigates whether the teacher uses batch normalization in training or evaluation mode.
4. Experiments
The experiments evaluate the distillation method across classification settings, datasets, network structures, and compression types. They include CIFAR-100 comparisons across varied architectures and ImageNet experiments involving ResNet and MobileNet pairs.
- Evaluation setup: The evaluation covers several domains, beginning with image classification and comparing reproduced implementations of other distillation methods under controlled factors.Experiments were implemented with PyTorch on the NSML platform.
- CIFAR-100: CIFAR-100 experiments compare knowledge-distillation settings across teacher and student architectures, including depth, channel, combined, and different-architecture compression.The listed settings include WideResNet, ResNet, and PyramidNet models with compression rates from 11.9% to 47.2%.
- ImageNet: ImageNet experiments use 1.2 million training images and 50 thousand validation images cropped to 224×224.The dataset provides larger images than CIFAR for evaluating distillation performance.
- ImageNet: The ImageNet study evaluates distillation from ResNet152 to ResNet50 and from ResNet50 to MobileNet.Three recent distillation algorithms are included in the reported comparisons.
- Metrics: CIFAR-100 performance is reported as classification error rate in percent, where lower values are better and Baseline denotes training without distillation.
4.3. Object detection
The proposed feature distillation method improves SSD-based object detection across student architectures and teacher choices, with stronger gains when teacher and student structures are similar.
- ResNet18-T1 improves more than T2, indicating better distillation quality between similar network structures.Both student architectures outperform their baselines.
- The experiments evaluate SSD detection on the PASCAL VOC2007 test set using mAP.Training uses a mixture of VOC2007 and VOC2012 trainval data.
- MobileNet students consistently improve regardless of the teacher type.The results support applying the method across SSD-based detectors.
- The section places object detection alongside semantic segmentation and other evaluated tasks for assessing the method’s broader applicability.The supplied passages report improvements for detection and segmentation, but do not provide the classification results here.
4.5. Analysis
The analysis attributes the method’s gains to stronger teacher–student output similarity, complementary loss components, and careful selection of the teacher’s batch-normalization mode.
- Analysis: The analysis measures output similarity, evaluates components one by one, and studies teacher batch-normalization mode.These analyses use the setting identified as (c) in Table 2.
- Teacher-student similarity: The proposed method produces output most similar to the teacher among the compared continuous distillation methods.It also reduces KL divergence, which the authors associate with improved performance.
- Ablation study: Pre-ReLU distillation provides the greatest ablation improvement by transferring the activation boundary through negative and positive values.The baseline uses L2 loss at the end-of-block position.
- Ablation study: The loss function adds a second improvement by excluding useless and harmful values below a small negative margin.Batch-normalization mode contributes an additional improvement.
- Batch normalization: The proposed method improves substantially when the teacher’s batch-normalization layer uses training mode.The authors conclude that the appropriate mode should be chosen carefully for distillation methods.
5. Conclusion
The paper proposes a feature distillation method that combines pre-ReLU distillation, margin-ReLU teacher transformation, partial L2 distance, and batch-normalization analysis.
- The method uses pre-ReLU features with a margin-ReLU teacher transform and partial L2 distance for feature distillation.The partial L2 function and margin transform are designed to support effective distillation at the pre-ReLU location.
- The proposed method substantially outperforms state-of-the-art feature distillation methods across various networks and tasks.The experiments cover image classification, object detection, and semantic segmentation.
- Teacher batch-normalization mode provides additional performance improvements.
A. margin evaluation
The channel-wise margin represents the expected negative teacher-feature value and can be estimated from training averages or batch-normalization statistics.
- The margin for each channel is the expected negative value of the teacher feature.It can be obtained directly during training or through a batch-normalization layer.
- Without a batch-normalization layer before ReLU, the expectation is estimated by averaging over the training process.
- A preceding batch-normalization layer determines each channel’s feature distribution using a mean and variance.The batch-normalization parameters correspond to these distribution statistics.
- For batch-normalized features, the channel margin is computed analytically by integrating the negative region of a Gaussian distribution.The result is expressed using the normal-distribution CDF Φ(·).
- Equation 10 obtains channel-wise margins without sampling and averaging during training when the relevant batch-normalization condition holds.Otherwise, the method uses training-process averaging.
B. implementation details
The method distills features at selected pre-downsampling positions, with layer-size-aware loss scaling and task-specific distillation weights and training configurations.
- Features are selected immediately before downsampling layers, totaling three layers for CIFAR and four for ImageNet.
- The loss sums values across each entire layer rather than averaging them, then halves the loss as spatial resolution doubles.
- α is set to 10^-3 for CIFAR, 10^-4 for ImageNet and detection, and 10^-5 for segmentation.
- Detection uses an extra backbone layer for distillation and trains for 120k iterations with scheduled learning-rate reductions.
C. additional experiments
Additional experiments examine distillation at the preReLU position and compare single-layer with multi-layer implementations of the proposed method.
- PreReLU evaluation: PreReLU improves the performance of most evaluated distillation algorithms.
- Single-layer evaluation: The proposed method performs similarly in single-layer and multi-layer settings.
- Single-layer evaluation: The single-layer result implies that the proposed method outperforms existing methods across the tested settings.