PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Model Compression: Distilling Knowledge from Noisy Teachers

Bharat Bhusan Sau, Vineeth N. Balasubramanian

arXiv:1610.09650v2cs.LG

TL;DR

Deep networks are difficult to deploy on mobile devices because their depth increases storage and runtime demands, while existing compression methods often focus mainly on storage. The paper extends teacher-student compression with noise-based regularization, reports improvements across MNIST, SVHN, and CIFAR-10, and examines related settings on CIFAR-10.

  • Problem

    Deep models’ storage and runtime demands restrict deployment on mobile devices, while most existing compression methods focus primarily on storage complexity.

  • Method

    The paper trains a shallow student from a deep teacher using noise-based regularization that simulates learning from multiple teachers.

  • Results

    The method improves shallow-student performance over the baseline teacher-student algorithm, with experiments on MNIST, SVHN, and CIFAR-10 and the best performance on CIFAR-10.

  • Takeaways & Limitations

    Noise-based regularization helps shallow students approach teacher performance while the teacher-student framework targets storage and runtime complexity.

  • Takeaways & Limitations

    The empirical evaluation uses student networks of depth 4 in the runtime-compression analysis, whereas earlier experiments used depth-3 students.

Abstract

from arXiv · show

The remarkable successes of deep learning models across various applications have resulted in the design of deeper networks that can solve complex problems. However, the increasing depth of such models also results in a higher storage and runtime complexity, which restricts the deployability of such very deep models on mobile and portable devices, which have limited storage and battery capacity. While many methods have been proposed for deep model compression in recent years, almost all of them have focused on reducing storage complexity. In this work, we extend the teacher-student framework for deep model compression, since it has the potential to address runtime and train time complexity too. We propose a simple methodology to include a noise-based regularizer while training the student from the teacher, which provides a healthy improvement in the performance of the student network. Our experiments on the CIFAR-10, SVHN and MNIST datasets show promising improvement, with the best performance on the CIFAR-10 dataset. We also conduct a comprehensive empirical evaluation of the proposed method under related settings on the CIFAR-10 dataset to show the promise of the proposed approach.

1. Introduction

Deep networks improve performance but strain mobile deployment through storage, runtime, and battery costs. The paper therefore extends teacher-student compression with noise-based regularization to improve shallow students while targeting broader complexity reductions.

  • Motivation: Deepening networks have improved performance but widened the gap between model capability and deployment on small mobile devices.Mobile devices have limited memory and computational capability, while large networks increase memory access and battery consumption.
  • Existing approaches: Existing compression methods include parameter sharing, network pruning, teacher-student learning, and matrix decomposition.These approaches represent four broad categories discussed in the paper’s related work.
  • Limitations of prior work: Most existing methods primarily reduce storage complexity, while runtime deployment issues can remain because compressed models must be decompressed at runtime.The paper identifies this as a continuing limitation for mobile deployment.
  • Teacher-student framework: Teacher-student compression trains a shallow student from a deep teacher, but matching the teacher’s accuracy remains challenging when the student must stay shallow.The proposed work addresses this challenge while retaining the student’s shallow architecture.
  • Contributions: The paper proposes noise-based regularization to simulate learning from multiple teachers and evaluates it on MNIST, SVHN, and CIFAR-10.It also performs a comprehensive empirical analysis on CIFAR-10 to examine success and failure cases.

2. Background and Related Work

Deep model compression methods span four main families, but many target storage reduction alone. Teacher-student methods are presented as a promising route toward compression across storage, runtime, and training-time complexity.

  • Compression method families: Deep model compression methods are broadly categorized into parameter sharing, network pruning, teacher-student learning, and matrix decomposition.The paper reviews representative methods from each category.
  • Parameter sharing: Parameter sharing methods reduce parameters by grouping weights into shared buckets or quantizing them.Examples include hash-based sharing and k-means clustering of weights.
  • Network pruning: Network pruning removes small-weight, weakly activated, or redundant components after or during network reduction.The reviewed methods target weights, convolutional filters, or neurons.
  • Teacher-student methods: Teacher-student methods train smaller networks from larger teachers by transferring outputs or other representations.The reviewed literature includes synthetic labels, logit matching, hidden-layer targets, and temperature-based softmax training.
  • Open compression gap: Parameter sharing, matrix decomposition, and pruning largely focus on storage, whereas teacher-student methods show promise across storage, runtime, and training-time complexity.The paper positions its method within this broader compression gap.

3. Proposed Methodology

The methodology trains a shallow student from a pretrained teacher’s logits while selectively perturbing target logits with Gaussian noise. This simulates learning from multiple teachers and acts as a noise-based regularizer during student training.

  • Teacher-Student Learning: Teacher-student compression trains a typically shallow student model from a pretrained deep teacher.The student receives teacher-produced outputs as training targets.
  • Teacher-Student Learning: Ba and Caruana’s baseline trains the student by regressing directly on the teacher’s pre-softmax logit outputs.The training data pairs each input with its corresponding teacher logit vector, and optimization uses an L2 loss.
  • Noisy Teachers: Logit perturbation injects Gaussian noise into selected teacher targets to simulate learning from multiple teachers.The noise vector has zero mean and dimension equal to the number of teacher classes or logits.
  • Noisy Teachers: The perturbation strength is controlled by σ, while α determines the probability that a mini-batch sample’s logits are perturbed.Only selected samples receive perturbed targets; the student then minimizes the resulting L2 loss.
  • Training Procedure: Student parameters are updated with stochastic gradient descent using the loss computed from original or perturbed teacher logits.The procedure iterates over mini-batches, generates noise, selects samples, computes the L2 loss, and updates the student parameters.
  • Equivalence to Noise-Based Regularization: Perturbing teacher logits is equivalent to adding a noise-based regularization term to the student’s loss.The paper contrasts this target-output perturbation with Gaussian noise on inputs and relates it to L2 regularization.

4. Experimental Results

Across MNIST, SVHN, and CIFAR-10, the experiments compare shallow students trained with baseline logit regression against students using noisy teacher logits. Noise improves student performance in the reported settings, though its effect depends on dataset and noise level.

  • Experimental setup: The evaluation uses MNIST, SVHN, and CIFAR-10, with SGD-based training and comparisons against baseline student performance.MNIST covers handwritten digits, SVHN house-number digits, and CIFAR-10 natural images.
  • MNIST: On MNIST, the teacher achieved a 0.0068 error rate, while the baseline student achieved 0.0097.The student uses two fully connected layers with 800 neurons per layer, encoded as FC800-FC800-FC10.
  • MNIST: Applying Gaussian perturbation to MNIST logits consistently improved student performance relative to the baseline.Around 15% of each mini-batch was selected for perturbation, using α = 0.15 and varying noise levels.
  • SVHN: On SVHN, the teacher reached 3.82% error, the baseline student reached 4.6%, and the noisy student reached 4.45% error.Higher noise levels deteriorated performance, indicating that α and σ require appropriate setting.
  • CIFAR-10: On CIFAR-10, the teacher obtained 8.4% error and the baseline student obtained 21.94% error, with the proposed method producing a healthy improvement over baseline.The authors selected α = 0.5 because of the large teacher–student performance gap and reported that substantial perturbation helped.
  • Overall findings: The authors conclude that the results support the proposed methodology and proceed to analyze related settings, including runtime complexity.The reported experiments therefore examine both student performance and broader compression-related behavior.

5. Discussions and Analysis

The analysis examines how noisy-teacher training behaves under different perturbation settings and compression levels. Across CIFAR-10 experiments, noise improves shallow-student performance over baseline logit regression while preserving substantial runtime compression.

  • Varying Number of Noisy Logits: Higher noisy-logit selection probabilities improved CIFAR-10 student performance, reaching the best result at α = 0.8.With σ = 0.6 fixed, around 80% of mini-batch samples were selected for perturbation.
  • Varying Number of Noisy Logits: The helpful noise level depends on the baseline student–teacher performance gap.Larger gaps require more noise, whereas smaller gaps in MNIST and SVHN benefit from fewer noisy logits.
  • Noise in Teacher versus Student: Noisy-teacher perturbation improved student performance over baseline logit regression, whereas directly regularizing the student was evaluated as a distinct setting.The distinction is that teacher outputs remain fixed during student training, while noisy-student outputs change as the student learns.
  • Comparison with DropOut: The proposed noisy-teacher method outperformed DropOut in the CIFAR-10 comparison.The baseline error rate without regularization was 21.94%, while the best proposed-method student performance was 18.68%.
  • More Results: Randomizing σ across mini-batches reduced error by 2.86% over baseline, below the best fixed-σ decrease of 3.26%.Random σ was sampled uniformly from [0.01, 1], while the best fixed-noise result used σ = 0.9.
  • More Results: Combining two teachers yielded a 20.44% student error rate, better than using Teacher1 alone at 21.94% or Teacher2 alone at 22.62%.The two teachers had baseline error rates of 8.4% and 13.99%, and their logits were combined using a geometric mean.
  • Runtime Compression of Shallow Students: Runtime compression increased as shallow student computations decreased, while the proposed method remained superior to baseline logit regression.The three students achieved runtime compression ratios of 3.66, 19.91, and 33.28 relative to the teacher, whose forward pass required 223 million multiplications.

6. Conclusions

The paper presents a teacher-student compression method targeting storage and runtime complexity. Noise-based regularization improves shallow students over the baseline and can simulate learning from multiple teachers.

  • The proposed method applies noise-based regularization within teacher-student learning for deep model compression.It considers both storage and runtime complexities.
  • The method helped shallow student models perform significantly better than the baseline teacher-student algorithm.The authors also view it as simulating learning from multiple teachers, helping students approach teacher performance.
Loading 1610.09650v2…