PaperlayerView exact revision on arXiv

Source-linked AI summary

FitNets: Hints for Thin Deep Nets

Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, Yoshua Bengio

arXiv:1412.6550v4cs.LGcs.NE

TL;DR

Large, deep networks are costly to run and difficult to train, while existing distillation methods struggle with substantially deeper students. FitNets uses intermediate teacher representations as hints to train thinner, deeper students, which can outperform larger teachers with far fewer parameters.

  • Problem

    Wide, deep networks deliver strong performance but require substantial computation and memory, while distillation does not adequately support substantially deeper student architectures.

  • Method

    FitNets trains thinner, deeper students by guiding an intermediate student layer to predict a corresponding teacher hidden-layer representation.

  • Results

    Deep, low-capacity students achieved comparable or better representations than teachers with up to 10 times more parameters across benchmark experiments.

  • Takeaways & Limitations

    Intermediate hints can help train deeper, thinner networks that generalize better and may run faster than their teachers.

  • Takeaways & Limitations

    Some alternative stage-wise training strategies initialized the network well but failed to learn sufficiently during subsequent training.

Abstract

from arXiv · show

While depth tends to improve network performances, it also makes gradient-based training more difficult since deeper networks tend to be more non-linear. The recently proposed knowledge distillation approach is aimed at obtaining small and fast-to-execute models, and it has shown that a student network could imitate the soft output of a larger teacher network or ensemble of networks. In this paper, we extend this idea to allow the training of a student that is deeper and thinner than the teacher, using not only the outputs but also the intermediate representations learned by the teacher as hints to improve the training process and final performance of the student. Because the student intermediate hidden layer will generally be smaller than the teacher's intermediate hidden layer, additional parameters are introduced to map the student hidden layer to the prediction of the teacher hidden layer. This allows one to train deeper students that can generalize better or run faster, a trade-off that is controlled by the chosen student capacity. For example, on CIFAR-10, a deep student network with almost 10.4 times less parameters outperforms a larger, state-of-the-art teacher network.

1 INTRODUCTION

FitNets addresses the inference cost of wide, deep networks by training thinner and deeper students that exploit the representational benefits of depth. Extending Knowledge Distillation, it uses intermediate teacher representations as hints to guide student training.

  • Wide and deep networks achieve strong computer-vision performance but require many parameters and costly inference involving numerous multiplications.
  • Depth promotes feature reuse and more abstract, invariant higher-layer representations, making it important for representation learning.
  • Deep architectures are difficult to train because successive nonlinearities produce highly non-convex and non-linear functions.
  • FitNets compress wide and shallower networks by training thin and deep student models that take advantage of depth.
  • FitNets extends Knowledge Distillation with intermediate-level hints from teacher hidden layers to guide student training.

2 METHOD

FitNets combine softened-output knowledge distillation with intermediate teacher representations to guide deeper, thinner students. Training proceeds stage-wise, first matching a teacher hint through a regressor and then optimizing the full student with distillation and label losses.

  • Knowledge Distillation: Knowledge distillation trains the student to match the teacher’s softened output alongside the true labels, with temperature τ > 1 providing more informative training signals.The loss combines label cross-entropy with teacher cross-entropy, balanced by tunable parameter λ.
  • Hints: Because deeper student networks remain difficult to optimize with standard distillation, FitNets use a teacher hidden-layer output as a hint for a student guided layer.The guided layer is trained to predict the teacher hint, making hints a form of regularization.
  • Hints: A regressor maps the narrower guided layer to the wider hint-layer dimensions, with matching nonlinearities required for comparable outputs.This allows the student guided representation to predict the teacher hint despite differing widths.
  • Regressor Design: Convolutional regressors reduce parameter and memory costs compared with fully connected regressors by adapting kernel dimensions to the teacher hint’s spatial size.The convolutional weight matrix uses k1 × k2 × Oh × Og parameters, where the kernel area is significantly smaller than the corresponding fully connected spatial-product terms.
  • Stage-Wise Training: Stage-wise training first pre-trains the FitNet through the guided layer with the regressor and hint loss, then trains the complete student using the distillation loss.The procedure starts from a trained teacher and randomly initialized FitNet, transferring the pre-trained guided parameters before full-network optimization.
  • Curriculum Interpretation: Teacher confidence weights the distillation contribution: high-confidence examples exert stronger influence, while λ controls the teacher cross-entropy’s importance.Low-confidence examples produce probabilities closer to the uniform distribution and exert less push on student parameters.

3 RESULTS ON BENCHMARK DATASETS

The experiments evaluate FitNets across CIFAR-10, CIFAR-100, SVHN, MNIST, and AFLW, showing that thinner or deeper students can retain or improve performance with substantially reduced capacity. Results also demonstrate benefits from hint-based training over standard backpropagation and knowledge distillation in selected settings.

  • CIFAR-10: On CIFAR-10, a 17-layer FitNet used roughly 1/3 of the teacher’s parameters while training its 11th layer to mimic the teacher’s 2nd layer.The teacher used maxout convolutional layers, and the student was followed by maxout fully connected and softmax layers.
  • CIFAR-100: On CIFAR-100, the FitNet used the same architecture as on CIFAR-10, outperformed the teacher, and reduced parameters by a factor of 3.The passage also states that the FitNet outperformed state-of-the-art methods, but the supplied text truncates the subsequent comparison.
  • SVHN: 32% of teacher capacity yielded SVHN performance comparable to the teacher and to state-of-the-art Maxout and Network in Network methods.The model was a 13-layer FitNet with 11 maxout convolutional layers, a fully connected layer, and a softmax layer.
  • MNIST: 1.9% misclassification error with standard backpropagation fell to 0.65% with KD on the same deep, thin MNIST FitNet.The FitNet was twice as deep as the teacher and had roughly 8% of its parameters; the supplied passage truncates the HT result.
  • AFLW: 4.21% teacher error contrasted with FitNet 1 errors of 4.58% with KD and 2.55% with HT, while FitNet 2 achieved 1.95% with KD and 1.85% with HT.FitNet 1 used 15 times fewer multiplications and FitNet 2 used 2.5 times fewer multiplications than the teacher.

4 ANALYSIS OF EMPIRICAL RESULTS

On CIFAR-10, Hint-based Training (HT) makes deeper, thinner networks easier to optimize than standard backpropagation or knowledge distillation, while deeper models perform better at fixed capacity. FitNets also achieve favorable speed, compression, and accuracy trade-offs against their teacher.

  • Impact of HT: HT trains deeper FitNets than standard backpropagation and knowledge distillation under fixed computational budgets.At 30M multiplications, standard backpropagation reaches only 5 layers, KD reaches 7 layers, and HT reaches 13 layers.
  • Impact of HT: At fixed capacity, deeper models outperform shallower models on CIFAR-10.The comparison uses networks with 3, 5, 7, and 9 convolutional layers under approximately 30M and 107M operation budgets.
  • Accuracy/Speed Trade-off: 36× less capacity makes FitNet 1 one order of magnitude faster than the teacher, with only a 1.3% performance decrease.The evaluation measures GPU inference time and parameter compression relative to the teacher.
  • Accuracy/Speed Trade-off: 0.9% higher performance lets FitNet 2 outperform the teacher while remaining 4.64× faster.Increasing FitNet capacity and depth further improves the performance gain, reaching 1.6% for FitNets 3 and 4 while they remain faster than the teacher.
  • Accuracy/Speed Trade-off: 10× fewer parameters can still yield performance improvements over the teacher when FitNets exploit depth.Convolutional-layer factorization and quantization methods are described as complementary approaches for further speed or storage reductions.

5 CONCLUSION

The paper proposes compressing wide and deep networks into thinner, deeper ones by using intermediate-level hints from teacher hidden layers to guide student training. These hints enable very deep students with fewer parameters that can generalize better and/or run faster than their teachers.

  • 5 CONCLUSION: The framework compresses wide and deep networks into thin and deeper ones using intermediate-level hints from teacher hidden layers.The hints guide the student’s training process.
  • 5 CONCLUSION: Teacher hidden-state hints enable training very deep student models with fewer parameters.The conclusion reports empirical evidence for this approach.
  • 5 CONCLUSION: Hinted thin and deep students can generalize better and/or run faster than their teachers.The reported benefits depend on the chosen compressed student models.

A SUPPLEMENTARY MATERIAL: NETWORK ARCHITECTURES AND TRAINING PROCEDURES

The supplementary material documents the network architectures and hyperparameters used throughout the paper.

  • The supplement describes all network architectures and hyperparameters used throughout the paper.

A.1 CIFAR-10/CIFAR-100

This section describes the teacher and FitNet architectures and the hyperparameters used in the CIFAR-10 and CIFAR-100 experiments.

  • The section presents the teacher architectures used in the experiments.
  • It also presents the FitNet architectures used in the experiments.
  • The reported hyperparameters apply to both CIFAR-10 and CIFAR-100 experiments.

A.1.1 TEACHERS

The CIFAR-10 and CIFAR-100 teachers are maxout convolutional networks with three convolutional hidden layers, followed by specified maxout and max-pooling operations.

  • Teachers: Both teachers use the same architecture with three convolutional hidden layers containing 96, 192, and 192 units.The networks were taken from Goodfellow et al. (2013b).
  • Teachers: Each convolutional layer applies a two-piece maxout non-linearity and max-pooling with 4x4, 4x4, and 2x2 windows, respectively.All max-pooling units have 2x2-pixel overlap.

A.1.2 FITNETS

FitNets use successive zero-padded 3×3 convolutional layers with maxout nonlinearities and three max-pooling units, and are trained with RMSProp-based stochastic gradient descent. CIFAR-10 training uses a two-stage procedure, while CIFAR-100 training uses the full training set with matched hyperparameters.

  • Architecture: Each FitNet stacks zero-padded 3×3 convolutions with two-piece maxout nonlinearities and includes three non-overlapping 2×2 max-pooling units.The final max-pooling aggregates all remaining spatial dimensions into a 1 × 1 vector representation.
  • Experimental architectures: FitNet 4 is used for the results in Tables 1–3, while separate architectures support the depth and efficiency–performance trade-off experiments.Table 6 covers the depth experiment in Figure 2, and Table 7 covers the efficiency–performance trade-off experiment in Table 5.
  • Optimization: FitNet parameters are initialized uniformly in U(-0.005,0.005) and optimized with stochastic gradient descent using RMSProp, an initial learning rate of 0.005, and mini-batches of 128.λ starts at 4 and decays linearly over 500 epochs to 1, while τ is set to 3.
  • Training procedure: On CIFAR-10, the training set is split into 40K training and 10K validation examples, followed by sequential stage-1 and stage-2 optimization with validation-based early stopping.Each stage allows up to 500 epochs and stops after 100 epochs without validation-error improvement; stage 1 minimizes Eq. (3), and stage 2 minimizes Eq. (2).
  • Training procedure: On CIFAR-100, FitNets are trained directly on the whole training set with stochastic-gradient descent using RMSProp and the same hyperparameters as CIFAR-10.The epoch count is determined by the CIFAR-10 stopping criterion.

A.2 MNIST

The MNIST experiments compare a maxout convolutional teacher with a substantially deeper, thinner FitNet trained using intermediate-layer hints. The FitNet has twice the teacher’s depth and roughly 8% of its parameters, with stage-wise hint training used to guide optimization.

  • A.2 MNIST: The teacher uses three maxout convolutional hidden layers with 48-48-24 units, followed by pooling and a fully connected softmax layer.Each maxout layer has 2 linear pieces, and pooling regions are 4x4-4x4-2x2 pixels.
  • A.2 MNIST: The FitNet is twice as deep as the teacher and has roughly 8% of the parameters.It contains 6 maxout convolutional hidden layers with 16-16-16-16-12-12 units.
  • A.2 MNIST: The FitNet applies max-pooling every second layer and ends with a fully connected softmax layer.Pooling uses 4x4-4x4-2x2 regions with 2x2-pixel overlap.
  • A.2 MNIST: Stage-wise training teaches the FitNet’s 4th layer to mimic the teacher’s 2nd layer by minimizing Eq. (3) with stochastic gradient descent.The mini-batch size was 128 samples, the learning rate was 0.0005, and λ started at 4 before decaying to 1 over 150 epochs.
  • A.2 MNIST: The MNIST training set was split into 50K training samples and 10K validation samples.Training stopped after 100 epochs without validation error improvement.

A.3 SVHN · A.4 AFLW

The SVHN experiments used a maxout teacher and FitNet 4 with CIFAR-10 training hyper-parameters, while AFLW compared two deeper, thinner seven-layer FitNets trained stage-wise against a three-layer teacher.

  • A.3 SVHN: SVHN used a maxout teacher with three convolutional hidden layers of 64-128-128 units, a 400-unit fully connected maxout layer, and a top softmax layer.Teacher training followed Goodfellow et al. (2013b).
  • A.3 SVHN: SVHN used FitNet 4, randomly initialized in U(-0.005,0.005) and trained with the same hyper-parameters and early stopping as CIFAR-10.Unlike CIFAR-10, FitNet was not retrained on the combined training and validation set.
  • A.4 AFLW: AFLW used a teacher with three ReLU convolutional layers of 128-512-512 units followed by a sigmoid layer.A 2 × 2 non-overlapping max-pooling operation followed the first layer, with receptive fields of 3-2-5.
  • A.4 AFLW: AFLW FitNet 1 used seven ReLU convolutional layers with widths 16-32-32-32-32-32-32-32 and a sigmoid output.All layers used 3 × 3 receptive fields, and no max-pooling was applied.
  • A.4 AFLW: AFLW FitNet 2 used seven ReLU convolutional layers with widths 32-64-64-64-64-64-64-64 and a sigmoid output.The architecture also used 3 × 3 receptive fields without max-pooling because of the small image resolution.
  • A.4 AFLW: Both AFLW FitNets were randomly initialized in U(-0.05,0.05), trained stage-wise, and used 90% of the data for training.The first stage matched the FitNet 5th layer to the teacher 3rd layer by minimizing Eq. (3) with stochastic gradient descent.
  • A.4 AFLW: AFLW first-stage optimization used mini-batches of 128 samples and an initial learning rate of 0.001, decayed during the first 100 epochs.The supplied passage truncates the remainder of the learning-rate decay schedule.
Loading 1412.6550v4…