PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Student-Friendly Teacher Networks for Knowledge Distillation

Dae Young Park, Moon-Hyun Cha, Changwook Jeong, Dae Sin Kim, Bohyung Han

arXiv:2102.07650v4cs.LG

TL;DR

Knowledge distillation can be suboptimal when teacher representations are not well matched to students. This paper trains teachers alongside student branches to make their representations easier to transfer, then reports consistent accuracy gains across varied architectures and distillation algorithms. It also notes limits in evaluation settings, data demands, computational cost, and privacy.

  • Problem

    Knowledge distillation can be suboptimal when student capacity is limited, algorithms are suboptimal, or teacher representations are poorly matched to students.

    Source: §1 · p. 1

  • Method

    SFTN trains teacher networks with student branches to learn student-friendly representations before teacher-to-student distillation.

    Source: §1 · pp. 1–2

  • Results

    SFTN achieves consistent performance gains over standard teacher networks across architectural and algorithmic variations.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The approach applies to diverse teacher and student architectures and can be incorporated into various distillation algorithms.

    Source: §1 · p. 2

    The approach is not sufficiently verified for out-of-distribution inputs, domain shifts, or few training examples, and retains high computational cost and potential privacy concerns.

    Source: §5 · p. 10

Abstract

from arXiv · show

We propose a novel knowledge distillation approach to facilitate the transfer of dark knowledge from a teacher to a student. Contrary to most of the existing methods that rely on effective training of student models given pretrained teachers, we aim to learn the teacher models that are friendly to students and, consequently, more appropriate for knowledge transfer. In other words, at the time of optimizing a teacher model, the proposed algorithm learns the student branches jointly to obtain student-friendly representations. Since the main goal of our approach lies in training teacher models and the subsequent knowledge distillation procedure is straightforward, most of the existing knowledge distillation methods can adopt this technique to improve the performance of diverse student models in terms of accuracy and convergence speed. The proposed algorithm demonstrates outstanding accuracy in several well-known knowledge distillation techniques with various combinations of teacher and student models even in the case that their architectures are heterogeneous and there is no prior knowledge about student models at the time of training teacher networks.

1 Introduction

Knowledge distillation transfers a larger teacher’s representations to a smaller student, but transfer can be suboptimal when teacher and student features are inconsistent. SFTN trains teachers with student branches to improve transfer, and the paper reports consistent accuracy gains across varied algorithms and models.

1 Introduction

  • Knowledge distillation can fall short when student capacity is limited, algorithms are suboptimal, or teacher representations are poorly matched to students.The authors identify teacher–student feature consistency as critical to knowledge transfer.Source: §1 · p. 1
  • SFTN trains teachers alongside student branches to shape representations that are easier to transfer, then distills from teacher to student.This student-aware teacher training precedes a separate, one-directional distillation stage.Source: §1 · pp. 1–2
  • The approach is designed for diverse teacher and student architectures and can be incorporated into various knowledge-distillation algorithms.The paper presents SFTN as general for training models and transferring knowledge.Source: §1 · p. 2
  • SFTN integration improves accuracy consistently with substantial margins across varied baseline algorithms and models.The introduction states this result without specifying benchmark values.Source: §1 · p. 2

2 Related Work

Prior work studies both what knowledge to transfer and how to organize distillation. The approaches reviewed include intermediate and relational representations, alternative sources of knowledge, and strategies addressing model capacity or joint learning.

2 Related Work

  • Knowledge distillation is used to build efficient models across computer vision, NLP, and cross-modal tasks.This section surveys methods intended to improve model performance through distillation.Source: §2 · p. 2

2.1 What to distill

  • Methods distill intermediate features (FitNets [18]), attention [20], relational information [22], or mutual information (CRD [23]); others use student ensembles [24], deeper layers [25], or self-supervised features [26].The surveyed alternatives broaden the knowledge sources beyond temperature-scaled softmax outputs [1].Source: §2.1 · pp. 2–3

2.2 How to distill

  • Distillation strategies include sequential training, teacher-trajectory curricula [28], and joint collaborative learning [4] [6], while other work addresses teacher–student capacity gaps [2] [29].The paper positions student-friendly teacher learning as a distinct way to facilitate distillation.Source: §2.2 · p. 3

3 Student-Friendly Knowledge Distillation

SFTN jointly trains student branches with the teacher to shape transferable representations, then uses existing knowledge-distillation methods to transfer the learned teacher’s knowledge.

3.1 Overview

  • SFTN jointly trains student branches with the teacher to align their outputs and fit labels, rather than optimizing the teacher only for ground-truth prediction.The approach aims to make teacher representations more suitable for distillation; the authors expect students distilled from SFTN to perform better than those using conventional teachers.Source: §3.1 · pp. 3–4

3.2 Network Architecture

  • Each student branch uses a teacher feature-transform layer to map features into the next student block’s dimensions, using 3×3 or 1×1 convolutions as needed.The transformation may increase or decrease feature-map size; 1×1 convolutions are used when size need not change.Source: §3.1 · pp. 3–4
  • Teacher and student networks are divided into blocks, with no student branch attached to the teacher’s final block.The described modularization uses blocks based on layer depth and feature-map sizes; distillation is often performed every three or four blocks.Source: §3.1 · p. 3, §3.2 · p. 4

3.3 Loss Functions

  • SFTN’s loss includes teacher cross-entropy and KL divergence between student-branch and teacher outputs, with the KL term comparing softened distributions.The KL loss considers the N−1 branches associated with teacher blocks before the final block.Source: §3.3 · p. 5
  • Student-branch cross-entropy averages over branches; SFTN combines it with teacher and KL losses using separate weights, with τ=1 for cross-entropy.The branch cross-entropy fits the ground-truth labels.Source: §3.3 · p. 5

4 Experiments

The experiments evaluate SFTN across distillation algorithms, architectures, and datasets. SFTN improves student results over standard teachers on CIFAR-100 and ImageNet, with further analyses of comparisons, hyperparameters, versatility, training cost, and representation similarity.

4.1 Experiment Setting

  • Experiments span ImageNet and CIFAR-100, multiple backbone families, and 12 distillation methods, with comparisons to collaborative and curriculum-learning approaches.CIFAR-100 testing includes eight teacher–student pairs; ImageNet testing uses five distillation methods with ResNet50 teachers and ResNet34 students.Source: §4.1 · p. 5, §4.2.1 · p. 6, §4.2.2 · p. 7

4.2.1 CIFAR-100

  • 1.58 percentage points was the average SFTN–standard-teacher difference on CIFAR-100, and the best SFTN student averaged 1.10 points above the standard teacher.Tables 1 and 2 cover homogeneous and heterogeneous teacher–student pairs; Figure 3 also shows four best SFTN students outperforming their standard teachers.Source: Table 1 · p. 6, Table 2 · p. 6, §4.2.1 · pp. 6–7

4.2.2 ImageNet

  • Approximately 0.5 percentage points separated the best SFTN student’s top-1 accuracy from the standard teacher on ImageNet; SFTN outperformed the standard teacher in all settings.The ImageNet evaluation used ResNet50 teachers, ResNet34 students, and five distillation methods.Source: Table 3 · p. 7, §4.2.2 · p. 7

4.3 Comparison with Collaborative and Curriculum Learning Methods

  • On CIFAR-100, SFTN outperformed DML, KDCL, and RCO; its accuracy also improved with more iterations in the SFTN-4 results.RCO-EEI-4 and SFTN-4 were each trained for 240 × 4 epochs, while the authors note that choosing optimal trajectory-based checkpoints can be challenging.Source: §4.3 · pp. 7–8, Table 4 · p. 8

4.4 Effect of Hyperparameters

  • Student accuracy stayed fairly stable across student-aware softmax temperatures despite teacher-accuracy fluctuations, while increasing λKL reduced the teacher–student accuracy gap.The reported observations also indicate that student and teacher accuracies can be inversely correlated, so maximizing teacher accuracy is not necessarily ideal for distillation.Source: §4.4 · p. 8

4.5 Versatility of SFTN

  • SFTN can transfer to students with substantially different architectures, with benefits preserved when their capacity is similar to the student branch used in training.Larger student branches often help distillation, whereas smaller branches are not always helpful.Source: §4.5 · p. 9, Table 7 · p. 9

4.6 Use of Pretrained Teachers

  • Fine-tuning a pretrained teacher reduced convergence training from 240 to 60 epochs and achieved almost the same accuracy as full SFTN in six distillation algorithms.The student branch was trained for the first 30 epochs, followed by 30 epochs of whole-network fine-tuning.Source: §4.6 · p. 9, Table 8 · p. 10, Table 9 · p. 10

4.7 Similarity between Teacher and Student Representations

  • On CIFAR-100, SFTN reduced teacher–student KL-divergence by more than 50% on average and raised average CKA by 7 percentage points versus standard teachers.The comparison uses representations from ResNet32×4 and ShuffleV2 models; lower KL-divergence and higher CKA indicate greater similarity.Source: Table 10 · p. 10, §4.7 · p. 10

5 Conclusion

SFTN trains teacher networks alongside student branches before distillation, and the paper reports consistent performance gains across architectural and algorithmic variations. The authors also identify unverified distribution shifts and continuing data, compute, and privacy concerns.

5 Conclusion

  • SFTN trains teacher networks alongside student branches, then distills from teachers to students.The authors present this as a shift toward focusing on teacher training for knowledge distillation.Source: §5 · p. 10
  • The approach shows consistent performance gains over standard teacher networks across architectural and algorithmic variations.The authors report higher accuracy with reduced model sizes.Source: §5 · p. 10
  • The approach is not sufficiently verified for out-of-distribution inputs, domain shifts, or few training examples, and still entails high computational cost and potential privacy issues.The authors also note that it relies on a large number of training examples.Source: §5 · p. 10

Learning Student-Friendly Teacher Networks for Knowledge Distillation Supplementary Document

The supplementary analyses report higher student accuracy under SFTN on CIFAR-100-C and in transfer tests, and examine teacher–student accuracy relationships. They also describe the conditions and trends behind these comparisons.

A.1 Robustness of Data Distribution Shift

  • SFTN students by KD achieve higher average accuracy than standard-KD students across 19 CIFAR-100-C distortions, though margins diminish as corruption intensity increases.The authors suggest highly corrupted data can make predictions random and distillation prone to incorrect predictions without additional techniques.Source: Table 11 · p. 13, §A.1 · p. 13

A.2 Transferability

  • 4.71 percentage points on STL10 and 6.07 percentage points on TinyImageNet are the average student-accuracy advantages for SFTN over the standard teacher.The transfer experiments use ResNet32×4 and ShuffleV2, with student features learned on CIFAR-100 and fitted to target datasets using linear classifiers.Source: Table 12 · p. 13, §A.2 · pp. 13–14

A.3 Relationship between Teacher and Student Accuracies

  • SFTN students consistently outperform standard-method counterparts even when SFTN teacher accuracy is lower; higher teacher accuracy alone does not guarantee a better student.The authors offer adaptive temperatures for individual logits as a possible explanation; Table 13 reports higher entropy for SFTN teachers.Source: Figure 4 · p. 14, §A.3 · p. 14, Table 13 · p. 14

A.4 Training and Testing Curves

On CIFAR-100 with ResNet32×4 teachers and ShuffleV2 students, the supplementary curves compare SFTN with standard teacher models. They report faster KL-divergence convergence and higher test accuracy for SFTN.

A.4 Training and Testing Curves

  • SFTN's KL-divergence loss for distillation converges faster than that of the standard teacher network.The authors attribute this probably to student-aware training through student branches, which they say learns more transferable knowledge.Source: §A.4 · p. 14
  • SFTN student models have higher test accuracy than those distilled from standard teacher models.Figure 5(b) shows the student results by KD.Source: §A.4 · pp. 14–15, Figure 5 · p. 15
  • ResNet32×4 teachers and ShuffleV2 students are used to visualize CIFAR-100 training and testing curves.Source: Figure 5 · p. 15

A.5 Additional study of hyperparameters

Additional analyses report that multiple student branches and varied loss-weight settings retain accuracy gains, and that SFTN improves teacher–student similarity measures over standard teachers.

A.5 Additional study of hyperparameters

  • KD students using two branches outperform those using a single branch, and single-branch students outperform standard KD students on CIFAR-100.The two single-branch variants are Branch1 and Branch2.Source: §A.5 · p. 15
  • Student accuracies remain consistent across variations in λCE_R and λT, with reported values of 76.83±0.10 and 76.55±0.43, respectively, versus 74.60 for standard distillation.Source: §A.5 · p. 15, Table 16 · p. 16

A.6 Additional study of similarity

  • Compared with the standard teacher, SFTN yields 50% lower average KL-divergence, a 7% point gain in average CKA, and 5% higher average top-1 agreement.The reported similarity results cover additional teacher–student pairs.Source: §A.6 · p. 16

A.7 CIFAR-100 Results with Error Bars

On CIFAR-100, SFTN yields more similar teacher–student outputs and representations, higher top-1 agreement, and error-bar variance similar to standard teachers.

  • The average difference between SFTN and standard-teacher error bars is 0.01 percentage points, indicating similar variance.Source: §A.7 · p. 16
  • SFTN consistently produces more similar teacher–student output distributions, representations, and top-1 prediction agreement than standard approaches.Source: Table 17 · p. 17, §A.7 · p. 17
  • Tables 18 and 19 report three-run CIFAR-100 error-bar comparisons for homogeneous and heterogeneous teacher–student architectures.Source: Table 18 · p. 18, Table 19 · p. 18

B Implementation Details

Implementation details are provided to support reproduction, with separate training protocols for CIFAR-100 and ImageNet.

B Implementation Details

B.1 CIFAR-100

  • CIFAR-100 models train for 240 epochs with batch size 64, scheduled learning-rate reductions, random crops, horizontal flips, and channel normalization.The learning rate drops at epochs 150, 180, and 210; crops are 32×32 with 4-pixel padding, and flipping probability is 0.5.Source: §B.1 · p. 18

B.2 ImageNet

  • ImageNet models train for 100 epochs with batch size 256 and learning-rate reductions at epochs 30, 60, and 90.Training uses random resized crops and horizontal flips; validation images are resized to 256×256 and center-cropped to 224×224.Source: §B.2 · p. 18

C Architecture Details

The architecture example pairs a VGG13 teacher with a VGG8 student and adds student branches to teacher blocks to form SFTN.

C Architecture Details

Related papers

Loading 2102.07650v4…