PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Student Networks via Feature Embedding

Hanting Chen, Yunhe Wang, Chang Xu, Chao Xu, Dacheng Tao

arXiv:1812.06597v1cs.LGcs.CVstat.ML

TL;DR

Deep networks’ storage and computation demands hinder mobile deployment, and feature-transfer bridges can add substantial parameters. The paper uses locality-preserving feature embedding to transfer teacher relationships into low-dimensional student features, reporting state-of-the-art benchmark performance for portable networks.

  • Problem

    Deep networks are difficult to deploy on mobile devices because of their storage and computation demands, while fully connected feature bridges can require substantial additional parameters.

    Source: §I · p. 1, C. Knowledge distillation · p. 3

  • Method

    The method uses a locality-preserving loss to embed teacher feature relationships in the student’s low-dimensional features.

    Source: §I · pp. 1–2, B. Locality Preserving Loss · p. 4

  • Results

    Benchmark experiments report that the proposed method learns portable networks efficiently with state-of-the-art performance.

    Source: §I · p. 2

  • Takeaways & Limitations

    The paper concludes that the resulting portable network can achieve performance similar to its teacher by embedding teacher-represented neighbor relationships in the student.

    Source: §VI · p. 10

Abstract

from arXiv · show

Deep convolutional neural networks have been widely used in numerous applications, but their demanding storage and computational resource requirements prevent their applications on mobile devices. Knowledge distillation aims to optimize a portable student network by taking the knowledge from a well-trained heavy teacher network. Traditional teacher-student based methods used to rely on additional fully-connected layers to bridge intermediate layers of teacher and student networks, which brings in a large number of auxiliary parameters. In contrast, this paper aims to propagate information from teacher to student without introducing new variables which need to be optimized. We regard the teacher-student paradigm from a new perspective of feature embedding. By introducing the locality preserving loss, the student network is encouraged to generate the low-dimensional features which could inherit intrinsic properties of their corresponding high-dimensional features from teacher network. The resulting portable network thus can naturally maintain the performance as that of the teacher network. Theoretical analysis is provided to justify the lower computation complexity of the proposed method. Experiments on benchmark datasets and well-trained networks suggest that the proposed algorithm is superior to state-of-the-art teacher-student learning methods in terms of computational and storage complexity.

I. INTRODUCTION

The introduction motivates portable students by the storage and computation demands of deep networks and the parameter cost of fully connected teacher–student bridges. It proposes locality-preserving feature embedding to transfer teacher relationships into low-dimensional student features.

I. INTRODUCTION

  • Deep networks’ storage and computation demands make them difficult to use on edge devices such as mobile phones and smart cameras.The passage gives VGGNet as an example of a network with more than 500MB of parameters and more than 1010× floating number multiplications.Source: §I · p. 1
  • Teacher–student learning produces flexible portable networks that run as regular neural networks without special inference support.The introduction contrasts this with pruning-based methods that may require special architectures and implementation tricks.Source: §I · p. 1
  • Fully connected layers used to bridge teacher and student features add substantial auxiliary parameters, reaching about 135MB in the Student 4 example.That layer is larger than the 9MB student network and 35MB teacher network.Source: §I · p. 1
  • Feature embedding lets students with fewer filters generate low-dimensional features that preserve relationships between examples.The approach introduces a locality-preserving loss and provides theoretical analysis for lower computational complexity.Source: §I · pp. 1–2
  • Benchmark experiments are reported to learn portable networks efficiently with state-of-the-art performance.The introduction also states that the method’s theoretical analysis justifies lower computational complexity.Source: §I · p. 2

II. RELATED WORKS

The related-work section groups CNN compression into network trimming, layer decomposition, and knowledge distillation. It contrasts direct compression methods with teacher–student approaches and identifies the auxiliary-parameter burden of feature-matching bridges.

A. Network Trimming

  • Network trimming methods compress CNNs through techniques including quantization, low-rank approximation, binarization, pruning, and frequency-domain transforms.The section notes that such compressed architectures can require special implementations, increasing development costs.Source: A. Network Trimming · p. 2

B. Layer Decomposition

  • Layer decomposition work seeks efficient networks through lightweight or redesigned layers, including factorized and depth-wise separable convolutions.The review also describes bottleneck, group-convolution, channel-shuffle, and filter-reuse approaches.Source: B. Layer Decomposition · p. 3

C. Knowledge distillation

  • Knowledge distillation transfers teacher information through softened outputs, intermediate features, pairwise distances, multiple teachers, attention, distribution matching, or adversarial learning.These approaches aim to train smaller student networks using information from teacher networks.Source: C. Knowledge distillation · p. 3
  • Feature-bridging fully connected layers can impose large space and computation costs, motivating flexible ways to transfer teacher and student features.One cited example requires a 19GB layer to connect intermediate features from ResNet-101 and Inception-BN.Source: C. Knowledge distillation · p. 3

III. STUDENT NETWORK EMBEDDING

This section reviews teacher–student interactions before presenting student-network embedding with a locality-preserving loss. It describes output matching and intermediate-feature transfer, while noting information and cost limitations in existing approaches.

A. Teacher-Student Interactions

  • Knowledge distillation matches softened teacher and student outputs, which convey relationships between classes beyond one-hot outputs.The student objective combines the ground-truth classification objective with a weighted softened-output matching term.Source: A. Teacher-Student Interactions · pp. 3–4
  • FitNet adds an intermediate fully connected layer to match student hint features with teacher guide features, then tunes the student with knowledge distillation.The review says higher-dimensional intermediate hints can transfer more information and yield higher student performance.Source: A. Teacher-Student Interactions · p. 4
  • The reviewed approaches leave two issues: independent pointwise treatment does not investigate sample connections, and the added fully connected layer increases training cost.These issues motivate the section’s student embedding approach with a locality-preserving loss.Source: §III · p. 3, A. Teacher-Student Interactions · p. 4

B. Locality Preserving Loss

The paper frames student learning as embedding teacher features into a lower-dimensional space while preserving local relationships between examples. It combines this locality-preserving objective with classification training and optimizes the student using mini-batches.

B. Locality Preserving Loss

  • Locality preserving projections provide a linear manifold-learning alternative that can be embedded into convolutional-network learning.The passage contrasts LPP with nonlinear manifold-learning methods that are costly in computation and storage for large-scale problems.Source: B. Locality Preserving Loss · p. 4
  • The student is trained to preserve the local relationships among teacher features in its lower-dimensional feature space.The method treats portable-network learning as low-dimensional embedding because student features have lower dimensionality than teacher features.Source: B. Locality Preserving Loss · p. 4
  • Teacher features define the local relationships, with each sample’s neighborhood given by its k nearest teacher-feature neighbors.The passage identifies α_i,j as describing relationships between selected teacher hint-layer features and defines N(i) using the k nearest neighbors.Source: B. Locality Preserving Loss · p. 4
  • To avoid finding neighbors across the entire training set at every iteration, the method discovers k nearest neighbors within each mini-batch.The passage identifies mini-batch neighbor discovery as the reformulation used to address the inefficiency of full-training-set neighbor computation.Source: B. Locality Preserving Loss · p. 5
  • The objective combines classification cross-entropy with a locality-preserving term that embeds teacher-space samples into the student feature space.The classification term maintains student performance, while the embedding term preserves relationships across the two feature spaces.Source: B. Locality Preserving Loss · p. 5

IV. ANALYSIS ON THE COMPLEXITY

The proposed method avoids FitNet’s additional fully connected bridge and has lower stated computation and storage complexity. Its cost grows with batch size and the combined teacher and student feature dimensions.

IV. ANALYSIS ON THE COMPLEXITY

  • O(m^2(dS + dT)) is the proposed method’s computational complexity, compared with O(mdSdT) for traditional fully connected teacher–student bridges.The analysis gives example complexity ratios of approximately 23 and 131 for two network settings.Source: §IV · p. 5
  • The complexity analysis counts teacher-feature distances, nearest-neighbor search, and student-feature distances when computing the locality-preserving loss.The stated terms are O(m^2dT), O(km^2), and O(m^2dS), respectively.Source: §IV · p. 5
  • When feature dimensions greatly exceed k, the O(km^2) neighbor-search term is treated as negligible, yielding O(m^2(dS + dT)).The passage illustrates this condition with k = 5, m = 128, dT = 6912, and dS = 5120.Source: §IV · p. 5
  • The proposed method needs no additional parameters, whereas the fully connected bridge has O(dSdT) space complexity and may require more than 100MB in practice.The passage contrasts this with about 9MB for storing the student network’s parameters.Source: §IV · p. 5

V. EXPERIMENTS

The experiments cover MNIST, CIFAR-10, and CIFAR-100. The supplied figure caption describes a MNIST feature visualization comparing teacher and student-network variants.

V. EXPERIMENTS

  • The experiments evaluate the method on MNIST, CIFAR-10, and CIFAR-100.The passage says the experiments validate effectiveness across these three benchmark datasets.Source: §V · p. 5
  • The MNIST test-set visualization compares features from the teacher, standard-backpropagation student, FitNet student, and proposed-method student.Its panels are labeled (a) through (d), respectively.Source: Figure 2 · p. 6
  • In the feature visualization, each category is marked with the same color across the compared networks.Source: Figure 2 · p. 6

A. Validations on MNIST

On MNIST, the proposed method transfers teacher features to a thinner student while preserving class separation. The experiments report student performance, storage, and sensitivity to hyperparameters.

Visualization of Features

  • The proposed student reached 99.2% test accuracy with 734KB of convolution filters, about one-quarter of the teacher's 2,982KB.The teacher reached 99.3%; the thinner student's conventional back-propagation and FitNet accuracies were 97.5% and about 98.5%.Source: A. Validations on MNIST · p. 6

Compression Results

  • The proposed student outperformed standard back-propagation, knowledge distillation, and FitNet on MNIST; the passage reports 0.48% accuracy.The reported comparison also gives 1.90% error for standard back-propagation, 0.65% misclassification error for distillation, and 0.51% error for FitNet.Source: Table III · p. 7, A. Validations on MNIST · pp. 7–8

Visualization of Features

  • The proposed locality-preserving loss produced separated student features, whereas conventional back-propagation features were distorted relative to the teacher's.The comparison visualizes features supplied to the softmax layer.Source: A. Validations on MNIST · pp. 6–7

Impact of parameters

  • The MNIST validation accuracy peaked at 99.54% with k = 5 and γ = 1, the settings retained for later experiments.Figure 3 presents performance under different parameter settings.Source: Figure 3 · p. 8, A. Validations on MNIST · p. 8

B. Validations on CIFAR-10

On CIFAR-10, the proposed method was evaluated across student architectures with different compression levels. It outperformed state-of-the-art student-teacher approaches in the reported comparisons.

Trade off between compression/speed-up and accuracy

  • The CIFAR-10 teacher had about 9M parameters and 90.21% classification accuracy; the four student networks ranged from 250K to 2.5M parameters.The dataset images were preprocessed with global contrast normalization and ZCA whitening, and the last 10,000 training images were used for validation.Source: B. Validations on CIFAR-10 · p. 8
  • Across four student architectures, the proposed method outperformed FitNet; Students 2–4 exceeded teacher accuracy, while highest-compression Student 1 had lower accuracy.Student parameter counts ranged from 250K to 2.5M; Student 1 had the highest compression and speed-up ratios.Source: B. Validations on CIFAR-10 · p. 8

C. Validations on CIFAR-100

On the more challenging CIFAR-100 benchmark, the proposed method outperformed state-of-the-art student-teacher methods. The student achieved 65.13% accuracy despite the teacher’s accuracy being about 62%.

C. Validations on CIFAR-100

  • CIFAR-100 contains 60,000 32×32 RGB images across 100 categories, with 600 images per class, making it more challenging than CIFAR-10.The teacher used in the experiment was about 62% accurate.Source: C. Validations on CIFAR-100 · p. 9
  • 65.13% accuracy was achieved by the student trained with the proposed locality-preserving loss, which outperformed state-of-the-art methods using the same architecture.The teacher network used for this benchmark was about 62% accurate.Source: C. Validations on CIFAR-100 · p. 9
  • The experiment used Student 4 to compare student-teacher learning methods with the same student architecture.Images were normalized with global contrast normalization and ZCA whitening, with random flipping, cropping, and zero padding for augmentation.Source: C. Validations on CIFAR-100 · p. 9

D. Validations on ImageNet

On ImageNet, the proposed method achieved the highest accuracy among the compared student-teacher methods while adding no parameters. The evaluation used Inception-BN as a 32M-parameter student and ResNet-101 as its teacher.

D. Validations on ImageNet

  • The ImageNet experiment paired a 128M-parameter ResNet-101 teacher with a 32M-parameter Inception-BN student.The dataset has about 1.28M training images and 50K validation images.Source: D. Validations on ImageNet · p. 9
  • Networks were trained for 100 epochs with Nesterov Accelerated Gradient, and the proposed method used the same hyperparameters as in the CIFAR experiments.The learning rate began at 0.1 and was divided by 10 at epochs 30, 60, and 90.Source: D. Validations on ImageNet · p. 9
  • The student trained without teacher information achieved 74.80% top-1 accuracy and 92.18% top-5 accuracy, compared with the teacher’s 77.32% and 93.42%, respectively.All teacher-student methods outperformed standard back-propagation.Source: D. Validations on ImageNet · p. 9
  • 93.13% top-5 accuracy was achieved with the proposed method, about 1% above the original student and the highest accuracy among compared methods.The method adds no parameters, whereas FitNet-based connections require about 5,035M additional parameters.Source: D. Validations on ImageNet · p. 9

E. Intermediate Layer Selection

The paper reports that intermediate-layer selection has only a minor effect in prior results and therefore focuses on comparisons under the same settings. Its conclusion presents locality preservation as a way to embed teacher sample relationships in portable student networks.

E. Intermediate Layer Selection

  • Table VI reports classification results for different networks on ImageNet.Source: Table VI · p. 10
  • Different intermediate-layer selections changed CIFAR-10 accuracy by less than 0.2%, while using more than one guided/hint layer improved it by less than 0.1%.The authors therefore do not study layer selection and instead compare methods under the same experimental settings.Source: E. Intermediate Layer Selection · p. 10

VI. CONCLUSION

  • The conclusion describes locality-preserving loss as embedding teacher-represented neighbor relationships between samples into the student network.The paper reports that benchmark experiments produce portable networks with higher performance than state-of-the-art approaches.Source: §VI · p. 10

Related papers

Loading 1812.06597v1…