PaperlayerView exact revision on arXiv

Source-linked AI summary

Like What You Like: Knowledge Distill via Neuron Selectivity Transfer

Zehao Huang, Naiyan Wang

arXiv:1707.01219v2cs.CVcs.LGcs.NE

TL;DR

Large neural networks are costly to store and compute, motivating knowledge transfer to smaller students. The paper proposes matching teacher and student neuron-selectivity distributions with MMD, and reports improved performance across datasets, compatibility with other transfer methods, and benefits on other tasks.

  • Problem

    Large neural networks impose high storage and computational costs, motivating acceleration and compression through smaller student models.

    Source: §1 · p. 1

  • Method

    Neuron Selectivity Transfer casts knowledge transfer as distribution alignment by minimizing MMD between teacher and student neuron-selectivity patterns.

    Source: §1 · p. 1, What is wrong with directly matching… · p. 4, §7 · p. 8

  • Results

    NST improves student performance across several datasets, combines with other transfer methods for new state-of-the-art results, and benefits object detection.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    Neuron selectivity provides a transferable feature representation that supports student-network compression and other computer-vision tasks.

    Source: §1 · pp. 1–2, §7 · p. 8

    The paper leaves applications to regression problems such as superresolution and optical-flow prediction for future work.

    Source: §7 · p. 8

Abstract

from arXiv · show

Despite deep neural networks have demonstrated extraordinary power in various applications, their superior performances are at expense of high storage and computational costs. Consequently, the acceleration and compression of neural networks have attracted much attention recently. Knowledge Transfer (KT), which aims at training a smaller student network by transferring knowledge from a larger teacher model, is one of the popular solutions. In this paper, we propose a novel knowledge transfer method by treating it as a distribution matching problem. Particularly, we match the distributions of neuron selectivity patterns between teacher and student networks. To achieve this goal, we devise a new KT loss function by minimizing the Maximum Mean Discrepancy (MMD) metric between these distributions. Combined with the original loss function, our method can significantly improve the performance of student networks. We validate the effectiveness of our method across several datasets, and further combine it with other KT methods to explore the best possible results. Last but not least, we fine-tune the model to other tasks such as object detection. The results are also encouraging, which confirm the transferability of the learned features.

1. Introduction

The paper addresses the deployment costs of large neural networks by proposing Neuron Selectivity Transfer, which aligns teacher and student activation distributions while retaining label-based training. NST improves student performance across several datasets and can be combined with other transfer methods and downstream tasks.

Motivation

  • Large neural networks improve performance but impose high computational and memory costs that hinder real-time deployment.These costs motivate acceleration and compression methods, including knowledge transfer to smaller student networks.Source: §1 · p. 1

Method

  • NST transfers neuron selectivity by aligning the distribution of intermediate activations between teacher and student networks.The method treats knowledge transfer as distribution matching and uses intermediate activation maps rather than only final predictions.Source: §1 · pp. 1–2, Figure 1 · p. 2

Contribution

  • NST is presented as a novel knowledge-transfer method for network acceleration and compression.The contribution frames neuron selectivity as task-related knowledge that can be transferred from a teacher to a student.Source: §1 · pp. 1–2

Results

  • NST achieves higher student performance across several datasets, including CIFAR-10, CIFAR-100, and ImageNet.The paper reports that NST improves student performance notably and significantly across these evaluations.Source: §1 · p. 2

Extensions

  • NST can be combined with other knowledge-transfer methods to pursue stronger model acceleration and compression results.The paper also reports that transferred features benefit other computer-vision tasks such as object detection.Source: §1 · p. 2

2. Related Works

Prior compression methods reduce model cost through pruning, quantization, or knowledge transfer, but existing transfer methods impose task or representation constraints. This work motivates distribution matching with MMD as a broader way to transfer neuron and feature knowledge.

Compression and transfer

  • Knowledge transfer methods reduce deployment cost by training a smaller student from a larger teacher, avoiding some specialized acceleration requirements of pruning and quantization.Pruning and quantization can require dedicated hardware or implementations, whereas knowledge transfer directly trains a smaller network.Source: §2 · p. 2

Knowledge Distillation

  • Knowledge Distillation transfers softened teacher outputs that encode intra-class variation and inter-class similarities.The student combines softened-softmax supervision with the original softmax objective.Source: §2 · p. 2
  • KD is limited to softmax-based settings and depends on the number of classes, motivating methods that transfer intermediate representations.In binary classification, KD may provide almost no additional supervision.Source: §2 · pp. 2–3

Intermediate representations

  • FitNet transfers full feature maps, while Attention Transfer summarizes activations and FSP represents relationships through Gram matrices.FitNet's strict feature-matching assumption can hurt performance or convergence when teacher and student capacities differ.Source: §2 · p. 3

Distribution matching

  • Maximum Mean Discrepancy compares distributions in a reproducing-kernel Hilbert space and has been used for domain adaptation and feature regularization.The paper extends this distribution-matching perspective to knowledge transfer.Source: §2 · p. 3

3. Background

The paper defines CNN feature-map notation and introduces MMD as a distribution-distance metric for comparing teacher and student activation samples.

3.1. Notations

  • CNN feature maps have C channels and spatial dimensions H × W, with rows representing channel maps and columns representing spatial-position activations.Teacher and student feature maps are assumed to share spatial dimensions, or are interpolated when necessary.Source: §3.1 · p. 3

3.2. Maximum Mean Discrepancy

  • MMD measures the distance between probability distributions using samples and a kernel-induced feature space.The kernel projects sample vectors into a higher or infinite-dimensional feature space.Source: §3.2 · p. 3
  • With a universal RKHS, minimizing MMD is equivalent to minimizing the distance between the compared distributions.The MMD loss is zero if and only if the distributions are equal in this setting.Source: §3.2 · p. 3
  • Figure 2 visualizes neuron activation heat maps for two selected images.The supplied caption identifies the visual but does not state a comparison or outcome.Source: Figure 2 · p. 4

4. Neuron Selectivity Transfer

Neuron Selectivity Transfer treats intermediate neuron activations as distributions of task-related selectivity patterns and matches teacher and student distributions with MMD.

4.1. Motivation

  • Neuron activations reveal which input regions or samples share task-related properties, motivating student neurons to mimic teacher activation patterns.The paper illustrates this intuition with neurons selective for monkey faces and characters.Source: §4.1 · p. 4

What is wrong with directly matching the feature maps?

  • Direct feature-map matching ignores sample density, whereas distribution alignment captures how frequently activation patterns occur in the selectivity space.Flattened filter activation maps are treated as samples describing where and what the CNN emphasizes.Source: What is wrong with directly matching… · p. 4

4.2. Formulation

  • NST transfers neuron selectivity by matching the distribution of normalized intermediate activation patterns between teacher and student networks.Each feature-map row represents one neuron's selectivity knowledge, and minimizing MMD transfers that knowledge to the student.Source: §4.2 · p. 4
  • NST combines the MMD feature-distribution loss with standard cross-entropy supervision using the student probabilities and true labels.The formulation defines H as cross-entropy, ytrue as the true label, and pS as the student's output probability.Source: §4.2 · p. 4
  • The method studies linear, polynomial, and Gaussian kernels, with polynomial settings d = 2 and c = 0 and Gaussian σ2 set by mean pairwise squared distance.The discussion connects kernel choices to relationships with existing methods.Source: §4.2 · p. 4, §4.3 · p. 5
  • Figure 3 compares knowledge-transfer methods on CIFAR10 and CIFAR100 using bold test-error curves and dashed train-error curves.Its caption states that NST improves final accuracy with fast convergence.Source: Figure 3 · p. 5

4.3.1 Linear Kernel

  • For ReLU activation maps, NST's linear-kernel formulation is equivalent to activation-based Attention Transfer apart from normalization, making AT a special case.Both formulations represent where neurons have high responses.Source: §4.3.1 · p. 5

4.3.2 Polynomial Kernel

  • With a second-order polynomial kernel and c = 0, NST can be viewed as matching Gram matrices whose entries represent pairwise region similarities.This supplies task-driven similarity supervision for the student's internal representation.Source: §4.3.2 · p. 5

5. Experiments

Experiments evaluate NST across CIFAR, ImageNet, and object detection, using standard knowledge-transfer comparisons and combinations. NST improves student performance, complements KD, and transfers to detection without test-time modifications or extra computation.

5.1. CIFAR

  • On CIFAR, NST improves over the original student, with polynomial NST best on CIFAR-10 while KD performs best on CIFAR-100.The choice of kernel affects NST performance, and all tested transfer methods outperform the original student on CIFAR-10.Source: §5.1 · p. 6, Table 1 · p. 6
  • Combining transfer methods improves over individual transfers: KD+NST gives the strongest combined results, with absolute gains of about 1.6% and 4.2%.Combined methods also converge faster than the student network, with KD+NST converging fastest.Source: §5.1 · p. 6, Table 2 · p. 6

5.2. ImageNet LSVRC 2012

  • ImageNet experiments use NST with a second-order polynomial kernel and evaluate top-1 and top-5 validation errors under a single-crop setting.The experiments use 1.28M training images, 50K validation images, and report results in Table 3 and Figure 4.Source: §5.2 · p. 6, Table 3 · p. 7, Figure 4 · p. 7
  • KD+NST achieves the best ImageNet accuracy, improving top-1 and top-5 accuracy by 1.4% and 1%, respectively.The result supports NST's complementarity with another knowledge-transfer method in a large-scale classification setting.Source: §5.2 · p. 7, Table 3 · p. 7

5.3. PASCAL VOC 2007 Detection

  • On PASCAL VOC 2007 detection, every knowledge-transfer model exceeds the baseline mAP; NST gains 1.2 and KD+NST gains 1.6.The comparison keeps the detection system fixed while varying pretrained ImageNet classification models, and requires no testing modifications or extra computation.Source: §5.3 · p. 7
  • The detection results indicate that feature-map methods have greater advantages than KD for object detection, consistent with the importance of spatial information.KD ignores spatial information, whereas FitNet, AT, and NST exploit it to some extent.Source: §5.3 · p. 7

6. Discussion

The discussion shows that NST makes student activations more teacher-like, while contrasting the constraints of KD and FitNet and outlining extensions beyond MMD.

6.1. Analysis of Different Transfer Methods

  • MMD loss significantly reduces the discrepancy between teacher and student activation distributions, making the student act more like the teacher.The visualization uses activation patterns from the CIFAR100 experiment with.Source: §6.1 · p. 8
  • KD works best with many classes but is limited by its reliance on softmax loss, restricting broader applications such as regression and ranking.Source: §6.1 · p. 8
  • FitNet’s requirement that students match full teacher activations can make training highly noise-sensitive and difficult to converge.Directly matching samples ignores density in the feature space.Source: §6.1 · p. 8

6.2. Beyond Maximum Mean Discrepancy

  • NST’s distribution-matching view is not limited to MMD: moment matching or GAN-based discrimination could align teacher and student features in other settings.The student can serve as a generator while a discriminator distinguishes student-generated from teacher features.Source: §6.2 · p. 8

7. Conclusions

The paper casts knowledge transfer as neuron-selectivity distribution alignment and reports improved student performance, complementarity with existing methods, and transfer to object detection. Future work targets additional applications, especially regression problems.

7. Conclusions

  • NST matches spatial neuron-activation distributions between teacher and student networks using MMD and improves small student networks across datasets.Source: §7 · p. 8
  • Combining NST with existing knowledge-transfer methods yields new state-of-the-art results, while experiments support transfer to object detection.Source: §7 · p. 8
  • Future work will explore NST in regression problems such as superresolution and optical-flow prediction.Source: §7 · p. 8

Related papers

Loading 1707.01219v2…