Source-linked AI summary
Curriculum Temperature for Knowledge Distillation
Zheng Li, Xiang Li, Lingfeng Yang, Borui Zhao, Renjie Song, Lei Luo, Jun Li, Jian Yang
TL;DR
Fixed temperatures leave distillation difficulty unchanged even as students progress, while meta-learned alternatives require extra validation data and have narrower training conditions. CTKD learns temperature dynamically with an easy-to-hard adversarial curriculum, and the paper reports broad improvements as a low-cost plug-in across existing distillation frameworks.
Problem
Most distillation methods fix temperature by inefficient grid search, although constant difficulty is sub-optimal for a growing student.
Method
CTKD adversarially learns a dynamic temperature that organizes distillation from easy to hard during student training.
Results
CTKD consistently improves existing state-of-the-art distillation approaches on CIFAR-100 and ImageNet with negligible additional computation cost.
Takeaways & Limitations
CTKD is presented as an easy-to-use plug-in for existing knowledge distillation frameworks.
Takeaways & Limitations
MKD comparisons are unavailable because its official code and detailed training settings were not released, and reimplementation did not reproduce reported accuracy.
Abstract
from arXiv · showhide
Most existing distillation methods ignore the flexible role of the temperature in the loss function and fix it as a hyper-parameter that can be decided by an inefficient grid search. In general, the temperature controls the discrepancy between two distributions and can faithfully determine the difficulty level of the distillation task. Keeping a constant temperature, i.e., a fixed level of task difficulty, is usually sub-optimal for a growing student during its progressive learning stages. In this paper, we propose a simple curriculum-based technique, termed Curriculum Temperature for Knowledge Distillation (CTKD), which controls the task difficulty level during the student's learning career through a dynamic and learnable temperature. Specifically, following an easy-to-hard curriculum, we gradually increase the distillation loss w.r.t. the temperature, leading to increased distillation difficulty in an adversarial manner. As an easy-to-use plug-in technique, CTKD can be seamlessly integrated into existing knowledge distillation frameworks and brings general improvements at a negligible additional computation cost. Extensive experiments on CIFAR-100, ImageNet-2012, and MS-COCO demonstrate the effectiveness of our method. Our code is available at https://github.com/zhengli97/CTKD.
Introduction
CTKD addresses the sub-optimality of fixed distillation temperatures by learning task difficulty dynamically through an easy-to-hard curriculum. Its adversarial temperature learning is designed as a plug-in that improves existing distillation methods with negligible computation.
- Introduction: CTKD dynamically and adversarially learns temperature during student training to progressively increase distillation difficulty.A reversed gradient maximizes the distillation loss between teacher and student.
- Introduction: CTKD includes global and instance-wise temperature variants called Global-T and Instance-T.Both variants are presented as learnable temperature choices within the curriculum-based approach.
- Introduction: CTKD integrates into most existing state-of-the-art knowledge distillation frameworks as an easy-to-use plug-in with negligible additional computation cost.The paper describes the resulting improvements as comprehensive and consistent across existing approaches.
- Introduction: The method organizes distillation from easy to hard through a dynamic and learnable temperature curriculum.This curriculum is explicitly designed to match the student's progressive learning stages.
- Introduction: Experiments report that CTKD consistently improves existing state-of-the-art distillation approaches on CIFAR-100 and ImageNet.The reported improvements are described as substantial in margin.
Related Work
Prior curriculum distillation methods organize samples or teacher states by difficulty, while conventional distillation fixes temperature despite its role in controlling task difficulty. Meta-learned temperature methods require extra validation data and may be mismatched with common augmentation settings.
- Curriculum Learning: Curriculum learning incrementally increases task difficulty, and prior distillation methods apply it through ordered samples or teacher intermediate states.LFME orders samples from easy to hard, while RCO uses the sequence of teacher intermediate states.
- Temperature in Distillation: Temperature affects distillation difficulty: lower values emphasize maximal teacher logits, whereas higher values flatten the distribution.Most methods fix temperature through inefficient grid search rather than adapting it during student learning.
- Temperature in Distillation: A constant temperature imposes fixed distillation difficulty and is sub-optimal for a growing student during progressive learning stages.This motivates changing temperature as the student develops.
- Learnable Temperature: MKD learns temperature through meta-learning on an extra validation set, mainly with ViT backbones and strong data augmentation.Most existing KD methods instead use normal augmentation.
- Learnable Temperature: Applying MKD directly to other distillation methods may weaken distillation, while CTKD is described as more efficient because it avoids preserving an extra validation split.The comparison is framed around training requirements and compatibility with common KD settings.
Method
CTKD replaces fixed-temperature distillation with a dynamic, learnable temperature whose difficulty increases through an easy-to-hard curriculum. It uses adversarial optimization, global or instance-wise temperature modules, and a gradient reversal implementation.
- Background: Temperature controls distillation difficulty by changing the smoothness and discrepancy between teacher and student probability distributions.Lower temperature sharpens distributions and emphasizes maximal teacher logits, whereas higher temperature flattens distributions and considers whole logits.
- Adversarial Distillation: CTKD learns a dynamic temperature during student training and adversarially maximizes distillation loss through a reversed gradient.The temperature module and student form a two-player mini-max optimization, implemented with a gradient reversal layer.
- Curriculum Temperature: The curriculum starts with λ=0 and increases λ so the student progresses from unconstrained task learning toward more difficult distillation.A cosine schedule increases λ from 0 to 1 during the first 10 epochs, then keeps it at 1.
- Learnable Temperature Module: Global-T predicts one temperature for all instances using a single learnable parameter, avoiding additional computational cost.The global module is used as the default scheme in subsequent experiments.
- Learnable Temperature Module: Instance-T predicts a separate temperature for each instance using a two-layer MLP that takes teacher and student predictions as input.Instance-T has stronger representation ability and better distillation performance than Global-T, but requires more computation.
Experiments
Experiments show that CTKD improves distillation across classification and detection settings while adding negligible computation, and that its curriculum and adversarial components are beneficial.
- CIFAR-100 classification: CTKD improves student accuracy across eleven CIFAR-100 teacher-student pairs.The evaluation uses diverse VGG, ResNet, Wide ResNet, ShuffleNet, and MobileNet architectures.
- Adversarial temperature: CTKD produces higher distillation loss than vanilla KD while maintaining an overall downward loss trend during training.The temperature module maximizes distillation loss adversarially, whereas the student minimizes it.
- Dynamic temperature: CTKD yields more separable feature representations and better results than fixed-temperature distillation through its dynamic temperature mechanism.The comparison is reported using t-SNE visualizations and temperature learning curves.
- Applied to existing distillation works: CTKD consistently improves six state-of-the-art distillation methods across seven teacher-student pairs without additional computational cost.The method uses a lightweight learnable temperature module and a non-parameterized GRL.
- Cross-dataset evaluation: CTKD remains effective on ImageNet-2012 and further boosts detection performance on MS-COCO.ImageNet evaluation applies CTKD to four existing distillation works, while MS-COCO evaluation follows DKD with Faster-RCNN-FPN.
- Curriculum strategy: Smoothly increasing task difficulty benefits students, whereas directly using fixed high difficulty, especially λ > 4, reduces performance.Increasing Eloops further improves performance, and the cosine curriculum strategy performs best among the compared strategies.
Conclusion
CTKD organizes knowledge distillation from easy to hard by dynamically learning temperature during student training. It uses adversarial temperature learning to increase distillation difficulty while remaining a lightweight plug-in.
- CTKD organizes the distillation task from easy to hard through a dynamic and learnable temperature.
- A reversed gradient learns the temperature to maximize the distillation loss between teacher and student, increasing learning difficulty adversarially.
- CTKD integrates into existing state-of-the-art knowledge-distillation frameworks and brings general improvements at negligible additional computation cost.
Supplementary Materials
The supplementary material describes CTKD's training settings, temperature curriculum, instance-wise module, benchmark evaluation, and comparisons with fixed-temperature and related curriculum methods. It also reports practical boundaries for difficulty scheduling and notes limitations in reproducing MKD.
- Fixed-temperature comparison: The optimal fixed temperature varies widely across teacher-student pairs, making the default value of 4 suboptimal for existing works.The comparison is reported against fixed-temperature distillation and grid-search results.
- Instance-wise temperature: Larger intermediate channels, such as Cinter ≥256, work better for the instance-wise method on CIFAR-100.The comparison uses ResNet-56 as teacher and ResNet-20 as student.