PaperlayerView exact revision on arXiv

Source-linked AI summary

Variational Information Distillation for Knowledge Transfer

Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D. Lawrence, Zhenwen Dai

arXiv:1904.05835v1cs.CVcs.AIcs.LG

TL;DR

Training deep networks is difficult when labeled data are limited, and existing knowledge-transfer methods primarily match teacher and student activations or engineered features. VID instead maximizes a variational lower bound on their mutual information, and it empirically outperforms existing benchmarks across distillation and transfer-learning tasks, including heterogeneous CNN-to-MLP transfer. The method’s scope is bounded by its Gaussian observation models and the paper’s stated assumption that teacher and student layers represent attributes shared across tasks.

  • Problem

    Limited labeled data motivate knowledge transfer, while existing approaches lack a commonly agreed theory and typically match teacher-student activations or hand-crafted features.

  • Method

    VID formulates knowledge transfer as maximizing mutual information between teacher and student networks and optimizes a variational lower bound because exact mutual information is intractable.

  • Results

    VID empirically outperforms existing knowledge-transfer benchmarks across distillation and transfer-learning tasks, including heterogeneous CNN-to-MLP transfer on CIFAR-10.

  • Takeaways & Limitations

    VID provides a general framework that encompasses several existing knowledge-transfer methods and supports transfer between heterogeneous network architectures.

  • Takeaways & Limitations

    The implementation uses Gaussian observation models, while more flexible recognition models and alternative mutual-information estimators remain future work.

Abstract

from arXiv · show

Transferring knowledge from a teacher neural network pretrained on the same or a similar task to a student neural network can significantly improve the performance of the student neural network. Existing knowledge transfer approaches match the activations or the corresponding hand-crafted features of the teacher and the student networks. We propose an information-theoretic framework for knowledge transfer which formulates knowledge transfer as maximizing the mutual information between the teacher and the student networks. We compare our method with existing knowledge transfer methods on both knowledge distillation and transfer learning tasks and show that our method consistently outperforms existing methods. We further demonstrate the strength of our method on knowledge transfer across heterogeneous network architectures by transferring knowledge from a convolutional neural network (CNN) to a multi-layer perceptron (MLP) on CIFAR-10. The resulting MLP significantly outperforms the-state-of-the-art methods and it achieves similar performance to the CNN with a single convolutional layer.

1. Introduction

Knowledge transfer addresses limited labeled data by using information from pretrained teacher networks, but existing methods lack a common theory. The paper proposes VID, which formulates transfer through mutual information and reports broad empirical gains, including heterogeneous CNN-to-MLP transfer.

  • Motivation: Limited labeled data can make training deep networks impractical, motivating transfer learning from source tasks to target tasks.The paper notes medical imaging as an example where data availability is constrained by patient counts.
  • Prior approaches: Existing teacher-student methods transfer knowledge by matching final outputs, intermediate layers, or hand-crafted intermediate features.These approaches differ in which teacher and student representations they regularize.
  • Research gap: Knowledge transfer lacks a commonly agreed theory, making empirical results difficult to understand and new methods harder to develop systematically.VID is introduced as an attempt to provide a more principled framework.
  • Proposed method: VID formulates knowledge transfer as maximizing mutual information between teacher and student networks using variational information maximization.The framework replaces intractable mutual-information computation with a variational lower bound.
  • Empirical findings: VID generalizes several existing knowledge-transfer methods and empirically outperforms state-of-the-art methods across varied transfer experiments.The experiments include heterogeneous networks and transfers on the same or different datasets.
  • Empirical findings: On CIFAR-10, VID enables knowledge transfer from a CNN to an MLP, whose student model significantly outperforms the best-reported MLPs.This demonstrates transfer across heterogeneous network architectures.

2. Variational information distillation (VID)

VID formulates knowledge transfer as maximizing mutual information between selected teacher and student layers while training the student on its target task. It uses a variational distribution to make this objective tractable and supports flexible layer mappings across architectures.

  • Information-theoretic formulation: VID selects corresponding teacher and student layers and encourages high mutual information between their activations while optimizing the student for its target task.The framework applies to same-task settings such as model compression or distillation and to related source and target tasks.
  • Variational objective: The method replaces intractable mutual-information computation with a variational lower bound based on q(t|s), an approximation to the teacher activation conditional distribution.The bound follows from the non-negativity of the Kullback-Leibler divergence and yields the VID minimization objective.
  • Optimization: The objective is jointly minimized over the student parameters and the variational distribution, equivalently fitting teacher activations through conditional likelihood while solving the target task.This gives the student compressed knowledge for recovering selected teacher activations.
  • Variational parameterization: VID commonly models q(t|s) with a Gaussian whose mean depends on the student activation and whose variance is parameterized separately from that mean.For intermediate teacher layers, the implementation uses convolutional parameterizations and softplus-enforced positive variance, with spatial matching handled by convolution or transposed convolution when needed.
  • Variational parameterization: For teacher logits, VID pairs the teacher logit layer with the student penultimate layer and uses a Gaussian parameterization to match their hierarchy without restricting the student output.The paper also notes that a simple linear transformation can parameterize this case.
  • Implementation boundary: Heteroscedastic variance was unstable and produced negligible improvements, motivating the paper’s use of simpler variance choices in its implementations.The authors leave heavy-tailed distributions and mixture density networks for future exploration.
  • Visualization: Figure 2 visualizes variational log-likelihood over spatial dimensions during training; VID raises likelihood relative to a student trained without knowledge transfer, indicating higher teacher-student mutual information.The plots compare input images, normalized interpolated likelihood maps, a no-transfer variational distribution, and averaged teacher-layer magnitude.
  • Connections to existing works: Compared with unit-variance Gaussian matching, VID allows nonlinear mean functions and dimension-specific variances, reducing pressure to fit teacher activations irrelevant to the student task.The paper frames unit variance as potentially restrictive because it can waste student capacity on irrelevant teacher units.

3. Experiments

Experiments evaluate VID across knowledge distillation, transfer learning, and heterogeneous CNN-to-MLP transfer. VID generally outperforms competing methods, including under data scarcity and architecture changes.

  • Knowledge distillation: VID-I consistently outperforms other knowledge-transfer methods across reduced-data knowledge-distillation regimes.With 100 data points per class, VID-I’s accuracy drop is 10.26%, compared with 31.88% without transfer and 15.52% for KD + AT.
  • Knowledge distillation: VID-I generally outperforms FitNet, AT, NST, and KD across student-network sizes, except when teacher and student architectures are identical.For identical WRN-40-2 networks, combining VID-I and KD yields the best performance.
  • Transfer learning: Transfer learning uses an ImageNet-pretrained ResNet-34 teacher for data-scarce MIT-67 and CUB-200-2011 classification tasks.Students use either ResNet-18 or the heterogeneous VGG-9 architecture, with multiple training-set sizes evaluated.
  • Transfer learning: VID-I and VID-LP outperform competing transfer methods across comparisons, while ResNet-34-to-VGG-9 transfer performs similarly to ResNet-34-to-ResNet-18 transfer.On MIT-67, the VID methods also outperform fine-tuning, which is not directly comparable as a knowledge-transfer method.
  • Heterogeneous transfer: The experiments investigate whether knowledge transfer remains useful between completely different architectures, motivated by CNN-to-MLP transfer on CIFAR-10.The MLP student receives information from intermediate CNN representations through a transfer setup that avoids spatial-structure-dependent methods.
  • Heterogeneous transfer: VID-I improves CNN-to-MLP transfer performance relative to directly training the student’s intermediate layers.The CIFAR-10 comparison uses a WRN-40-2 convolutional teacher and a fully connected MLP student.

4. Conclusion

The paper concludes that VID provides effective knowledge transfer by maximizing a variational lower bound on mutual information. Its Gaussian-observation implementation outperforms benchmark methods in distillation and transfer learning, while more flexible mutual-information estimators remain future work.

  • VID maximizes a variational lower bound of mutual information between two neural networks for knowledge transfer.The framework is presented as an effective approach to knowledge transfer.
  • The Gaussian observation-model implementation empirically outperforms other benchmarks in knowledge distillation and transfer learning.
  • More flexible recognition models and alternative mutual-information estimators are identified as future research directions.

Supplementary Material:

The supplementary-material passage is the paper title rather than substantive supplementary content.

  • The paper is titled Variational Information Distillation for Knowledge Transfer.

A.1. Network architectures

The supplementary architecture section describes network and VID parameterizations for distillation and transfer learning, including mappings between MLP and CNN representations.

  • The experiments use established WRN and ResNet architectures, with a VGG-9 variant for transfer learning.The VGG-9 network modifies VGG-11 while following the VGG design philosophy.
  • For knowledge distillation, VID-I parameterizes the mean function with three 1 × 1 convolutional layers using batch normalization and ReLU.Hidden channel sizes are twice the output channel size.
  • For MLP-to-CNN intermediate-layer transfer, the mean function maps a one-dimensional student representation into a three-dimensional teacher representation.The mapping begins by reshaping the input to unit spatial dimensions and uses transposed convolutions.

A.3. Loss function and training scheme

VID combines the target-task loss with variational regularization, using tunable weights and layer dimensions to control knowledge transfer. Training applies gradient clipping and task-specific optimization schedules.

  • Loss function: The regularization term is an expected log-likelihood, Et(k),s(k)[log q(t(k)|s(k))], evaluated between teacher and student representations.This term appears in the VID loss used in the experiments.
  • Loss function: VID balances the target-task loss and regularization terms using positive hyperparameters λ1 and λ2.The selected layer contributes Nk dimensions, with Nk = CkHkWk for convolutional representations.
  • Hyperparameter selection: λ1 and λ2 are selected from {0.1, 1} and {10, 100}, respectively, using validation-set performance.The competing methods use separately tuned regularization scales, and KD uses temperature T = 4.
  • Training scheme: Gradient norms are clipped at 100, and the homoscedastic variational-distribution variance is initialized to 5.0.These settings are used throughout the experiments.
  • Training scheme: Transfer-learning experiments for ResNet34 and ResNet-18 use SGD for 250 epochs with batch size 128 and weight decay 0.0005.The initial learning rate is 0.05 and is multiplied by 0.2 at epochs 150 and 200.

B. Additional Experiments

Additional experiments examine learned variance parameters, transfer learning under limited data, and comparison with adversarial network compression. The reported results describe diverse variances and further performance advantages for VID.

  • Learned parameters: Learned variance parameters σn are diverse, especially across layers, supporting layer-dependent variance modeling in VID.The paper connects this diversity to obtaining a tighter mutual-information lower bound.
  • Transfer learning: VID outperforms other methods when transferring knowledge from grayscale-SVHN to MNIST with 200 data points per class.The experiment uses LeNet-like teacher and student architectures.
  • Comparison with ANC: VID methods outperform Adversarial Network Compression by a small margin in CIFAR-10 distillation between ResNet-164 and ResNet-20.The comparison reports validation accuracy.

B.4. Experimental results with standard deviation

The appendix provides full experimental results corresponding to earlier tables, adding standard deviations from three repeated runs.

  • Additional reporting: Tables 7–12 reproduce the full results for Tables 1–4 with standard deviations from three repeated runs.The additional results cover the corresponding knowledge-distillation and transfer-learning experiments.

B.5. Additional heat maps for VID training

The appendix adds heat-map visualizations for VID training and full result tables spanning distillation, transfer learning, and heterogeneous architectures. The visualizations show how variational distributions are evaluated across spatial dimensions.

  • Heat-map visualizations: Figure 4 provides additional VID knowledge-transfer visualizations using the same plotting procedure as Figure 2.The figure focuses on heat maps associated with variational distributions in intermediate teacher-network layers.
  • Distillation results: Tables 7 and 8 report CIFAR-10 and CIFAR-100 distillation test accuracy across student-data, depth, and width settings.The CIFAR-10 experiment transfers from WRN-40-2 to WRN-16-1, while CIFAR-100 varies student depth d and width w.
  • Transfer-learning results: Tables 9–11 report ResNet-34 transfer-learning test accuracy for ResNet-18 and VGG-9 students on MIT-67 and CUB-200.The tables vary the number of data points per class M.
  • Heterogeneous architectures: Table 12 reports CIFAR-10 distillation test accuracy from convolutional WRN-40-2 to fully connected MLP-h students across hidden dimension sizes h.This table targets knowledge transfer across heterogeneous network architectures.
  • Heat-map visualizations: The heat maps include normalized and interpolated log-likelihoods, an untreated student-network distribution, and spatially averaged teacher-layer magnitudes.Red pixels indicate high probability in the normalized log-likelihood maps.
Loading 1904.05835v1…