PaperlayerView exact revision on arXiv

Source-linked AI summary

Variational Information Distillation for Knowledge Transfer

Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D. Lawrence, Zhenwen Dai

arXiv:1904.05835v1cs.CVcs.AIcs.LG

TL;DR

Knowledge transfer lacks a commonly agreed theory, making empirical results difficult to understand and principled methods harder to develop. VID frames transfer as maximizing teacher–student mutual information and empirically outperforms existing methods across knowledge-distillation and transfer-learning experiments.

  • Problem

    Knowledge transfer lacks a commonly agreed theory, complicating understanding empirical results and development of more principled methods.

    Source: §1 · p. 2

  • Method

    VID formulates knowledge transfer as maximizing mutual information between teacher and student networks using a variational lower bound.

    Source: §1 · p. 2, §2 · pp. 2–3

  • Results

    VID empirically outperforms state-of-the-art knowledge-transfer methods across distillation and transfer-learning experiments, and its CIFAR-10 student MLP significantly outperforms best-reported MLPs [17].

    Source: §1 · p. 2, §4 · p. 8

  • Takeaways & Limitations

    VID provides an actionable objective for knowledge transfer and a way to quantify how much information transfers from teacher to student.

    Source: §1 · p. 2

    The transfer setup assumes that teacher-layer attributes learned on the source task also exist in the target task.

    Source: §2 · p. 2

Abstract

from arXiv · show

Transferring knowledge from a teacher neural network pretrained on the same or a similar task to a student neural network can significantly improve the performance of the student neural network. Existing knowledge transfer approaches match the activations or the corresponding hand-crafted features of the teacher and the student networks. We propose an information-theoretic framework for knowledge transfer which formulates knowledge transfer as maximizing the mutual information between the teacher and the student networks. We compare our method with existing knowledge transfer methods on both knowledge distillation and transfer learning tasks and show that our method consistently outperforms existing methods. We further demonstrate the strength of our method on knowledge transfer across heterogeneous network architectures by transferring knowledge from a convolutional neural network (CNN) to a multi-layer perceptron (MLP) on CIFAR-10. The resulting MLP significantly outperforms the-state-of-the-art methods and it achieves similar performance to the CNN with a single convolutional layer.

1. Introduction

The paper introduces variational information distillation (VID), a knowledge-transfer framework that maximizes mutual information between teacher and student networks, and reports strong results across transfer settings, including CNN-to-MLP transfer on CIFAR-10.

1. Introduction

  • Because labeled data can be scarce, teacher-student transfer uses a source-task teacher to regularize target-task learning, while prior methods match outputs, intermediate layers, or designed features.The introduction notes that knowledge transfer lacks a commonly agreed theory; recent work also extends the approach to different source and target datasets [31].Source: §1 · pp. 1–2
  • VID empirically outperforms existing knowledge-transfer methods across experiments on the same or different datasets, with especially large gaps for small data and heterogeneous architectures.The authors also report that VID generalizes several existing transfer methods.Source: §1 · p. 2
  • VID formulates knowledge transfer as maximizing teacher-student mutual information, using a variational lower bound because mutual information is computationally intractable.The framework provides an actionable objective, quantifies transferred information, and derives existing methods [16] [22] as specific implementations.Source: §1 · p. 2
  • On CIFAR-10, VID enables heterogeneous CNN-to-MLP knowledge transfer and yields a student MLP that significantly outperforms the best-reported MLPs [17].Source: §1 · p. 2

2. Variational information distillation (VID)

VID frames teacher–student knowledge transfer as preserving mutual information between selected teacher and student layer activations while training the student for its target task. It approximates that objective with a variational distribution over teacher activations given student activations.

2. Variational information distillation (VID)

  • VID maximizes mutual information between paired teacher and student layer activations, assuming teacher layers encode attributes relevant to both source and target tasks.The source and target tasks may be the same, as in model compression or knowledge distillation.Source: §2 · p. 2
  • The objective combines the target-task loss with layerwise mutual-information regularization, replacing intractable exact computation with a variational lower bound based on q(t|s).The student and q(t|s) are jointly optimized; the resulting objective can be viewed as fitting teacher activations conditionally on student activations.Source: §2 · p. 3

2.1. Algorithm formulation

  • For spatial intermediate layers, VID uses a Gaussian q(t|s) with a learned mean and channel-specific positive variance, typically mapping similarly shaped student layers with 1 × 1 convolutions.Larger or transposed convolutions can match differing spatial dimensions; softplus enforces positive variance, with a minimum variance for numerical stability.Source: §2.1 · p. 3
  • For teacher logits, VID pairs them with the student’s penultimate layer and uses a linear mean transformation, avoiding a direct constraint on student logits.The logit-layer variance is also parameterized with softplus to enforce positivity.Source: §2.1 · p. 3
  • Heteroscedastic variance performed unstably with negligible improvement; heavy-tailed or mixture-density distributions are proposed as future alternatives.The reported implementation otherwise performed satisfactorily during experiments.Source: §2.1 · pp. 3–4
  • During VID training, the variational distribution increasingly estimates teacher intermediate-layer activations, whereas its version for a student trained without transfer fails to achieve high log-likelihood.The authors note that background regions often achieve higher magnitudes because they correspond largely to zero activations and are easier to fit.Source: §2.1 · p. 4, Figure 2 · p. 4

2.2. Connections to existing works

  • VID relates to infomax by maximizing retained teacher-layer information, while handcrafted feature-matching methods [13] [31] risk discarding useful information when their features are unsuitable.The infomax formulation reduces to traditional semi-supervised infomax when the teacher representation t(k) is set to the input x.Source: §2.2 · pp. 4–5
  • VID generalizes MSE transfer: logit matching by Ba and Caruana, cross-task logit matching by Li and Hoiem [16], and intermediate-layer matching by Romero et al. [22] arise from particular layer and mean choices.Unlike those fixed-variance formulations, VID allows nonlinear means and dimension-specific variances, which can reduce over-regularization from task-irrelevant teacher activations.Source: §2.2 · p. 5

3. Experiments

The experiments compare variational information distillation (VID) with existing knowledge-transfer methods across knowledge distillation, transfer learning, and CNN-to-MLP transfer, using intermediate-layer or logit-to-penultimate-layer matching.

3.1. Knowledge distillation

  • 10.26% accuracy drop with VID-I at 100 data points per class, versus 31.88% without transfer and 15.52% with the best baseline, KD + AT; VID-I leads across data regimes.The experiment uses CIFAR-10 and transfers from WRN-40-2 to WRN-16-1.Source: Table 1 · p. 6, §3.1 · p. 6
  • With student size varied on CIFAR-100, VID-I beats FitNet, AT, NST, and KD except for the WRN-40-2 student, where combining methods yields the best performance; students can outperform teachers.The exception is when student and teacher architectures are identical; the student matching the teacher’s size outperforms the teacher with all transfer methods, as also noted by Furlanello et al..Source: §3.1 · pp. 6–7

3.2. Transfer learning

  • Across data-scarce MIT-67 and CUB-200-2011 tasks, VID methods outperform other transfer methods, remain robust across ResNet-18 and VGG-9 students, and beat fine-tuning on MIT-67.The teacher is ImageNet-pretrained ResNet-34; data are subsampled to 50, 25, or 10 examples per MIT-67 class and 20, 10, or 5 per CUB-200-2011 class.Source: §3.2 · p. 7, Table 3 · p. 8

3.3. Knowledge transfer from CNN to MLP

  • Both FitNet and VID-I improve performance over directly training the intermediate layers of an MLP student distilled from a convolutional teacher.The CIFAR-10 experiment transfers from WRN-40-2 to a fully connected MLP; AT and NST are not applicable to MLPs because they rely on spatial structures.Source: §3.3 · pp. 7–8, Table 4 · p. 8

4. Conclusion

VID transfers knowledge by maximizing a variational lower bound on mutual information between neural networks, and its Gaussian-observation implementation outperforms other benchmarks in distillation and transfer learning.

4. Conclusion

  • VID empirically outperforms other benchmarks in both distillation and transfer learning tasks.The reported implementation uses Gaussian observation models.Source: §4 · p. 8
  • The VID framework performs knowledge transfer by maximizing a variational lower bound on mutual information between two neural networks.Its algorithm is based on Gaussian observation models.Source: §4 · p. 8
  • Future work includes using more flexible recognition models to maximize mutual information accurately and exploring alternative mutual-information estimation [4].Source: §4 · p. 8

Supplementary Material: … A.2. Parameterization of VID

The supplementary material specifies the network architectures used and gives experiment-specific parameterizations for VID across distillation, transfer learning, and CNN-to-MLP transfer.

A.1. Network architectures

  • WRNs and ResNets follow the architectures originally described by Zagoruyko et al. [32] and He et al. [11], respectively.Source: §A.1 · p. 10
  • VGG-9 uses eight convolutional layers with batch normalization and ReLU, max-pooling after selected layers, then global average pooling and a linear prediction layer.It modifies VGG-11 [25] while retaining the VGG design philosophy.Source: §A.1 · p. 10
  • The CNN-to-MLP student stacks an input linear layer, three bottleneck linear layers, and an output layer, with dropout, batch normalization, and ReLU between layers.Source: §A.1 · p. 10
  • Each MLP bottleneck comprises two linear layers without nonlinearity and reduces parameter count to speed learning.Source: §A.1 · p. 10

A.2. Parameterization of VID

  • For knowledge distillation, VID-I uses three 1 × 1 convolutional layers with batch normalization and ReLU between layers; hidden channels are twice the output channels.Source: §A.2 · p. 10
  • For transfer learning, VID-I uses two 1 × 1 convolutional layers with hidden channels half the output size, while VID-LP uses a single linear mean function.Source: §A.2 · p. 10
  • For CNN-to-MLP transfer, VID-I maps the student’s one-dimensional intermediate representation to the teacher’s three-dimensional representation using transposed convolutions.The first layer uses a 4 × 4 kernel, unit stride, and zero padding; subsequent layers use 4 × 4 kernels, stride two, and unit padding, with layer count varied to match teacher spatial dimensions.Source: §A.2 · p. 10

A.3. Loss function and training scheme

The section specifies VID’s loss and validation-based hyperparameter selection, then describes gradient handling and experiment-specific SGD schedules.

A.3. Loss function and training scheme

  • VID combines the task-specific loss with a weighted expected log q(t^(k)|s^(k)) regularization term, with λ1 and λ2 selected on validation data.The selected values come from {0.1, 1} for λ1 and {10, 100} for λ2; KD uses temperature T = 4, and baseline regularization scales are also validation-selected.Source: §A.3 · p. 10
  • Across experiments, gradient norms were clipped at 100 and the homoscedastic variational variance was initialized to 5.0.For full-dataset knowledge distillation, SGD ran for 200 epochs with batch size 128 and weight decay 0.0005; the initial learning rate 0.1 was decayed by 0.2 at epochs 60, 120, and 160.Source: §A.3 · p. 10
  • Transfer learning used SGD for 250 epochs, while CNN-to-MLP knowledge transfer used 700 epochs, with schedules and batch settings varying by model and task.For ResNet34 and ResNet-18, batch size was 128, weight decay 0.0005, and learning rate 0.05 decayed by 0.2 at epochs 150 and 200; VGG-9 used batch size 12, no weight decay, and learning rate 0.01 with the same decay schedule. CNN-to-MLP used batch size 128, weight decay 0.0005, and learning rate 0.001 decayed by 0.2 at epochs 500 and 600; subset-dataset schedules were scaled to match parameter updates.Source: §A.3 · p. 11

B. Additional Experiments … B.3. Comparison with adversarial network compression

The additional experiments examine learned variance behavior, transfer learning from grayscale SVHN to MNIST, and distillation against adversarial network compression.

B.1. Visualization of learned parameters

  • VID’s learned variance parameters σn vary across channels, especially between layers, as visualized in Figure 3.The passage describes channel-wise values for different layers.Source: §B.1 · p. 11
  • The diversity of σn across layers makes modeling homoscedastic variance necessary for a tighter lower bound on mutual information.This conclusion follows the observed learned-variance pattern shown in Figure 3.Source: §B.1 · p. 11

B.2. Transfer learning from SVHN to MNIST

  • VID outperforms other methods when transferring from grayscale SVHN to MNIST using LeNet-like teacher and student networks.The teacher is trained on full grayscale-converted SVHN, and the student on 200 MNIST data points per class.Source: §B.2 · p. 11, Table 5 · p. 11
  • The SVHN-to-MNIST experiment trains the teacher on full grayscale-converted SVHN and the LeNet-like student on 200 MNIST samples per class.Table 5 identifies the reported measure as test accuracy.Source: §B.2 · p. 11, Table 5 · p. 11

B.3. Comparison with adversarial network compression

  • VID outperforms Adversarial Network Compression by a small margin in CIFAR-10 distillation from ResNet-164 to ResNet-20.The comparison repeats the knowledge-distillation experiment presented by [3].Source: §B.3 · p. 11, Table 6 · p. 11
  • The ANC comparison uses CIFAR-10 knowledge distillation between ResNet-164 and ResNet-20, following the experiment presented by [3].Table 6 reports validation accuracy and identifies underlined values as results reported by Belagiannis et al. [3].Source: §B.3 · p. 11, Table 6 · p. 11

B.4. Experimental results with standard deviation

This section reports standard deviations for the full experimental results corresponding to Tables 1, 2, 3a, 3b, 3d, and 4.

B.4. Experimental results with standard deviation

  • Tables 7–12 provide the corresponding full experimental results with standard deviations across three repeated runs.They correspond respectively to Tables 1, 2, 3a, 3b, 3d, and 4.Source: §B.4 · p. 11

B.5. Additional heat maps for VID training

Figure 4 adds heat-map visualizations of VID-based knowledge transfer, plotted in the same way as Figure 2.

B.5. Additional heat maps for VID training

  • Figure 4 visualizes the variational distribution evaluated across spatial dimensions of an intermediate teacher-network layer.The caption identifies the plotted quantity as log q(t_h,w|s).Source: Figure 4 · p. 14
  • The plots include input images, normalized and interpolated log-likelihood maps where red indicates high probability, and student-network comparisons and layer-magnitude maps.The caption describes a variational distribution optimized for a student trained without knowledge transfer and the magnitude of layer t averaged across spatial dimensions.Source: §B.5 · p. 14

Related papers

Loading 1904.05835v1…