PaperlayerView exact revision on arXiv

Source-linked AI summary

Learn From the Past: Experience Ensemble Knowledge Distillation

Chaofei Wang, Shaowei Zhang, Shiji Song, Gao Huang

arXiv:2202.12488v1cs.CVcs.LG

TL;DR

Traditional knowledge distillation overlooks knowledge generated during teacher training, termed teacher experience. EEKD preserves and ensembles intermediate teacher models, adaptively weights them with self-attention, and transfers the result to a student. On CIFAR-100 and ImageNet, it outperforms mainstream distillation methods and even standard ensemble distillation while saving training cost.

Abstract

from arXiv · show

Traditional knowledge distillation transfers "dark knowledge" of a pre-trained teacher network to a student network, and ignores the knowledge in the training process of the teacher, which we call teacher's experience. However, in realistic educational scenarios, learning experience is often more important than learning results. In this work, we propose a novel knowledge distillation method by integrating the teacher's experience for knowledge transfer, named experience ensemble knowledge distillation (EEKD). We save a moderate number of intermediate models from the training process of the teacher model uniformly, and then integrate the knowledge of these intermediate models by ensemble technique. A self-attention module is used to adaptively assign weights to different intermediate models in the process of knowledge transfer. Three principles of constructing EEKD on the quality, weights and number of intermediate models are explored. A surprising conclusion is found that strong ensemble teachers do not necessarily produce strong students. The experimental results on CIFAR-100 and ImageNet show that EEKD outperforms the mainstream knowledge distillation methods and achieves the state-of-the-art. In particular, EEKD even surpasses the standard ensemble distillation on the premise of saving training cost.

I. INTRODUCTION

EEKD addresses the limitation that conventional distillation transfers only knowledge formed after teacher training by incorporating the teacher’s intermediate training experience. It ensembles uniformly saved intermediate models, uses self-attention to weight them, and outperforms established distillation methods while challenging the assumption that stronger ensemble teachers produce stronger students.

Motivation

  • EEKD targets deployment of smaller networks that reduce computation and memory costs while narrowing the performance gap with large networks.The broader motivation is deployment on resource-limited embedded systems.Source: §I · p. 1
  • Conventional knowledge distillation transfers post-training teacher knowledge but ignores valuable knowledge generated during the teacher’s training process.The paper frames this omitted training-process knowledge as the teacher’s experience.Source: §I · p. 1

Contribution

  • EEKD transfers teacher experience by preserving intermediate training models, integrating their knowledge, and distilling it into a student network.A self-attention module adaptively assigns weights to the intermediate models.Source: §I · pp. 1–2, Figure 1 · p. 1

Results

  • EEKD significantly outperforms state-of-the-art knowledge distillation methods on CIFAR-100 and ImageNet.The introduction reports that it also exceeds standard ensemble distillation.Source: §I · p. 2

Findings

  • A high-performance ensemble teacher does not necessarily produce a strong student, prompting reconsideration of ensemble distillation.The paper identifies this as a surprising conclusion from its experiments.Source: §I · p. 2

II. RELATED WORK

Prior knowledge-distillation methods transfer response, feature, or relation knowledge from pretrained teachers, while EEKD adds teacher experience from the training process. The related ensemble literature motivates combining ensemble learning with distillation but leaves training-cost and adaptive-weighting gaps that EEKD addresses.

A. Knowledge Distillation

  • Existing distillation methods transfer response-, feature-, or relation-based knowledge from pretrained teacher networks, but omit knowledge generated during teacher training.EEKD treats this omitted knowledge as teacher experience and transfers it through ensemble learning.Source: A. Knowledge Distillation · p. 2

B. Ensemble Learning

  • Ensemble distillation reduces testing cost by distilling multiple teachers into one student, but existing multi-teacher approaches do not effectively reduce training cost.EEKD instead uses intermediate models from one teacher and learns adaptive weights with self-attention.Source: A. Knowledge Distillation · p. 2

B. Ensemble Learning

The paper positions EEKD as a lower-cost ensemble-distillation approach that reuses intermediate models from one teacher and adaptively weights them with self-attention. This method combines ensemble learning with knowledge distillation while targeting stronger student transfer than standard ensemble distillation.

B. Ensemble Learning

  • Unlike standard ensemble distillation, EEKD reuses intermediate models from one teacher to reduce training cost while distilling ensemble knowledge.Standard ensemble distillation independently trains multiple full teacher models.Source: B. Ensemble Learning · p. 2

C. Self-attention

  • EEKD uses self-attention to learn adaptive weights for different intermediate models rather than relying on fixed averaging.The weighting strategy is intended to select a distribution beneficial to student learning.Source: C. Self-attention · p. 2

III. METHOD

  • The method section presents EEKD as a framework that reviews teacher-student distillation, constructs experience ensembles, and analyzes factors limiting performance.Its stated scope includes principles and techniques for improving EEKD.Source: §III · pp. 2–3

A. Teacher-student Optimization

Teacher-student optimization trains a compact student to imitate a pre-trained teacher’s softened outputs, but conventional distillation leaves the teacher’s training-process knowledge unused.

A. Teacher-student Optimization

  • Training uses minibatches from the dataset to estimate prediction errors against ground-truth labels before optimizing the network parameters.The network maps input images to outputs through learnable parameters, and conventional optimization updates them using sampled subsets.Source: A. Teacher-student Optimization · p. 3
  • The student learns from the pre-trained teacher by minimizing an additional KL-divergence loss between their softened outputs.The temperature factor softens predictions, while the ratio parameter balances this term with the standard loss.Source: A. Teacher-student Optimization · p. 3
  • Conventional distillation transfers only the pre-trained teacher’s knowledge and ignores valuable knowledge formed during the teacher’s training process.This underuses the teacher’s knowledge and motivates EEKD’s focus on experiential knowledge.Source: A. Teacher-student Optimization · p. 3

B. Experience Ensemble Knowledge Distillation

EEKD builds a virtual teacher by ensembling uniformly saved intermediate teacher models, then transfers its knowledge to the student. Its design must balance intermediate-model quality, weighting, and quantity because the best ensemble teacher is not necessarily the best distillation teacher.

B. Experience Ensemble Knowledge Distillation

  • EEKD divides teacher training into M uniform stages, ensembles the resulting intermediate models, and uses the virtual teacher for teacher-student optimization.Each intermediate model captures the teacher’s perception of the data at a particular training stage, while the final intermediate model is the full teacher.Source: B. Experience Ensemble Knowledge… · p. 3
  • Each intermediate model is assigned a weight coefficient wi that satisfies the condition 0 < wi < 1.Source: B. Experience Ensemble Knowledge… · p. 3
  • Selecting a suitable virtual ensemble teacher is nontrivial because the teacher yielding the strongest ensemble need not produce the strongest distilled student.Knowledge-transfer performance can depend on the teacher–student capacity gap, motivating student-friendly teacher designs.Source: B. Experience Ensemble Knowledge… · p. 3

C. Principles of Experience Ensemble Knowledge Distillation

EEKD constructs a virtual ensemble teacher from intermediate training models and studies how their quality, weights, and number affect distillation. The analyses show that stronger ensemble teachers do not necessarily yield stronger students, while attention weighting and model-count trade-offs matter.

Method

  • EEKD replaces a single teacher with a virtual ensemble of uniformly saved intermediate models, whose knowledge is then distilled to the student.The design process uses a moderate number of intermediate models and an attention-based weighting strategy.Source: C. Principles of Experience Ensemble… · pp. 3, 5

Principle 1: The Quality of Intermediate Models

  • Multiple intermediate models provide significantly better distillation than a single teacher, but the stronger Cycle Ensemble Teacher does not outperform NoCycle Ensemble Teacher for student distillation.The results attribute this mismatch partly to excessive diversity causing cognitive conflicts among teachers.Source: C. Principles of Experience Ensemble… · p. 4, Table I · p. 4

Principle 2: The Weights of Intermediate Models

  • Attention-based weights dynamically adapt to the student and preserve diversity across inputs, unlike fixed weighting strategies.The comparative experiment fixes M = 5 and evaluates attention, mean, linear-increase, and linear-decrease weighting.Source: C. Principles of Experience Ensemble… · p. 4
  • Attention-based weights achieve the best distillation performance even when they do not produce the best ensemble performance.Mean weighting is simple and effective, while assigning higher weights to later, higher-performing models is not necessarily beneficial.Source: C. Principles of Experience Ensemble… · pp. 4–5, Table II · p. 4

Principle 3: The Number of Intermediate Models

IV. EXPERIMENTS

The experiments evaluate EEKD against state-of-the-art knowledge distillation methods and standard ensemble distillation after defining datasets and experimental settings.

Experimental comparisons

  • The evaluation compares EEKD with state-of-the-art knowledge distillation methods and standard ensemble distillation.These comparisons are conducted in separate experimental subsections following the dataset and setting descriptions.Source: §IV · p. 5

A. Datasets and Experimental Settings

EEKD is evaluated on CIFAR-100 and ImageNet-1K using standard image-classification datasets, augmentations, and teacher-student training settings. The experiments include structurally similar and different teacher-student pairs.

Datasets

  • Experiments use CIFAR-100 and ImageNet-1K, containing 50,000 and 1.2 million training images respectively.CIFAR-100 has 100 classes, while ImageNet-1K has 1,000 categories.Source: A. Datasets and Experimental Settings · p. 5

Experimental settings

  • Teacher-student pairs include both similar structures, such as ResNet-110/ResNet-20, and different structures, such as ResNet-110/MobileNetV2.CIFAR-100 experiments run for 200 epochs, while ImageNet-1K experiments run for 90 epochs.Source: A. Datasets and Experimental Settings · p. 5
  • Unless otherwise specified, EEKD uses α = 0.5, M = 5, τ = 5, and attention-based weighting.CIFAR-100 uses SGD with cosine learning-rate scheduling, while ImageNet-1K uses batch size 256 and temperature 4 for all methods.Source: A. Datasets and Experimental Settings · p. 5

B. Comparison with the State-of-the-art

EEKD outperforms state-of-the-art distillation baselines on CIFAR-100 and ImageNet-1K, improving student performance and narrowing teacher–student gaps.

CIFAR-100

  • 4.86% average student-accuracy improvement on CIFAR-100 lets EEKD exceed all baseline methods and sometimes surpass the teacher.These results hold across multiple teacher–student architecture pairings, including ResNet-110/ResNet-32 and WRN-40-2/WRN-16-2.Source: B. Comparison with the State-of-the-art · p. 5

ImageNet-1K

Evaluation scope

C. Comparison with Standard Ensemble Distillation

Compared with standard ensemble distillation, EEKD achieves better CIFAR-100 distillation performance while requiring less training cost.

Comparison with SED

Experimental setup

Interpretation

V. CONCLUSION

EEKD transfers experience knowledge from intermediate teacher models through attention-weighted ensemble integration, achieving state-of-the-art results on CIFAR-100 and ImageNet.

Method

  • EEKD extracts knowledge from intermediate teacher models, adaptively integrates it with attention-based weights, and transfers it to the student.Source: §V · p. 6

Results

  • EEKD achieves the new state-of-the-art on CIFAR-100 and ImageNet while surpassing standard ensemble distillation with lower training cost.Source: §V · p. 6

Implications

  • The findings motivate further study of teacher experience and suggest combining EEKD’s ensemble teacher with other methods.Source: §V · p. 6

Related papers

Loading 2202.12488v1…