PaperlayerView exact revision on arXiv

Source-linked AI summary

Temperature-Adaptive Transformed Teacher Matching

Hiroaki Aizawa, Yoshikazu Hayashi

arXiv:2608.29099v1cs.LGcs.CV

TL;DR

Temperature scaling’s role in distillation remains unclear, and TTM does not specify how its fixed teacher-side temperature should vary across samples. This paper introduces a sample-wise inverse-temperature update based on local KL minimization and closed-form curvature information, and reports generally improved TTM and WTTM performance across image-classification distillation benchmarks.

  • Problem

    TTM clarifies teacher-side temperature scaling but still relies on a fixed temperature and does not specify how it should vary across samples.

  • Method

    The method locally minimizes per-sample KL divergence by adapting TTM’s teacher-side inverse temperature with closed-form variance-covariance derivatives and a curvature-aware update.

  • Results

    TA-TTM and TA-WTTM generally improve reproduced TTM and WTTM baselines across CIFAR100 settings and outperform the compared temperature-adaptive baseline on ImageNet1k.

  • Takeaways & Limitations

    Sample-wise temperature adaptation preserves the TTM framework while adapting the teacher target and its implicit Rényi entropy regularization at the sample level.

Abstract

from arXiv · show

Temperature scaling is a core component of knowledge distillation, yet its role and effect are still not fully understood. Transformed Teacher Matching (TTM) clarifies the role of temperature scaling by applying it only to the teacher distribution and interpreting the resulting objective as standard distillation with an implicit Rényi entropy regularization on the student. However, TTM still relies on a fixed temperature and does not specify how the teacher-side temperature should be adapted for individual samples. In this paper, we introduce a sample-wise inverse-temperature update for TTM by locally minimizing the Kullback-Leibler divergence between the temperature-scaled teacher distribution and the student's prediction. We derive closed-form first and second derivatives with respect to the inverse temperature, and show that they can be expressed using variance and covariance statistics of centered teacher and student logits under the transformed teacher weighting. This yields an efficient curvature-aware update that requires one softmax evaluation and a constant number of class-wise weighted sums. Experiments on standard image classification distillation benchmarks show that our temperature adaptation generally improves TTM and WTTM, while remaining competitive with or outperforming prior temperature-adaptive distillation baselines.

1 Introduction

Knowledge distillation benefits from softened teacher predictions, but the role and selection of temperature remain unclear. This paper adapts TTM’s teacher-side temperature per sample and reports generally improved TTM and WTTM performance.

  • Motivation: Knowledge distillation transfers teacher knowledge to smaller students without increasing inference cost, but its design choices are sensitive to the teacher, student, and dataset.Temperature scaling is one such choice and is often tuned empirically.
  • Motivation: Temperature scaling exposes non-maximum class relations by smoothing predictive distributions, yet its role and placement in distillation have remained ambiguous.Standard KD commonly applies temperature to both teacher and student, whereas the effect of alternative placements is less clear.
  • Prior work: TTM applies temperature scaling only to the teacher distribution and interprets the resulting objective as distribution matching plus implicit Rényi entropy regularization on the student.This reframes the teacher-side transformation through power transforms.
  • Research gap: TTM still uses a fixed temperature and does not specify how teacher-side temperature should vary across samples.The paper identifies sample-wise adaptation as the remaining limitation.
  • Contribution: The proposed method locally minimizes the teacher-student KL divergence to update each sample’s inverse temperature, using closed-form derivatives for an efficient curvature-aware update.The update can be integrated into TTM and WTTM with negligible overhead.
  • Results: Experiments on standard image-classification distillation benchmarks show that temperature adaptation generally improves TTM and WTTM while remaining competitive with or outperforming prior adaptive baselines.The reported comparison is broad rather than tied to a single dataset or metric.

2 Related Work

Prior work adapts distillation targets and temperatures through several schedules, normalization schemes, entropy signals, and teacher-difficulty estimates. TTM differs by applying temperature only to the teacher distribution and connecting the objective to Rényi entropy regularization.

  • Knowledge distillation: Knowledge distillation research includes feature-based, relation-based, contrastive, logit-based, and self-distillation approaches.These methods vary the transferred signal or the loss used to match teacher and student behavior.
  • Temperature adaptation: Recent temperature-adaptive methods use curriculum schedules, logit standardization, entropy-based adaptation, or precomputed temperatures based on pretrained-teacher difficulty.These approaches represent several distinct strategies for choosing temperature.
  • Transformed Teacher Matching: TTM applies temperature scaling only to the teacher distribution and shows that the resulting objective includes an implicit Rényi entropy regularization term.This distinguishes TTM from methods that adapt temperature through schedules or sample-difficulty estimates.

3 Preliminaries

The paper formalizes distillation for K-class classification using teacher and student logits, temperature-scaled distributions, and a loss combining supervision with distribution matching. TTM views teacher temperature scaling as a power transform and makes its implicit Rényi entropy regularization explicit.

  • Setting and notation: The setting uses fixed teacher parameters and optimized student parameters for K-class classification with teacher and student logits.The student is trained from both ground-truth labels and the teacher’s predictive distribution.
  • Setting and notation: A temperature T > 0 controls predictive-distribution sharpness or smoothness, with temperature-scaled teacher and student distributions used for distillation.Temperature scaling changes how concentrated the predictive distributions are.
  • Knowledge distillation: Standard KD combines cross-entropy with a KL-divergence term matching temperature-scaled teacher and student predictions, weighted by λ.λ balances supervised learning and distillation.
  • Transformed Teacher Matching: TTM defines the teacher-side temperature transformation as a power transform of the temperature-free teacher prediction.The inverse-temperature correspondence is γ = 1/T, and the transformed distribution can be computed directly from logits.
  • Transformed Teacher Matching: With γ = 1/T and the corresponding β relation, TTM rewrites the distillation objective to expose an implicit Rényi entropy regularization on the student prediction.The objective therefore combines distribution matching with entropy regularization.
  • Weighted TTM: WTTM assigns each sample a distillation weight based on teacher-target sharpness, increasing the weight when the target is smoother and dark knowledge is more informative.The weighting changes the distillation coefficient while leaving the teacher-side temperature unchanged.

Methodology

The method adapts TTM’s teacher-side inverse temperature per sample by locally minimizing the teacher–student KL discrepancy. Closed-form sensitivity and curvature statistics enable efficient updates that extend TTM and WTTM without changing their original loss definitions.

  • Per-sample adaptation: The method optimizes each sample’s inverse temperature γ by locally minimizing the KL discrepancy between transformed teacher distribution p(γ) and student prediction q.The update treats γ as a sample-wise auxiliary variable while holding current teacher and student logits fixed.
  • Per-sample adaptation: A second-order Taylor model converts local sensitivity and curvature of the TTM discrepancy into a closed-form curvature-aware update.The update uses a step size, numerical stabilization, and clipping to a valid inverse-temperature range.
  • Temperature-adaptive extensions: The adaptation is implemented as TA-TTM and TA-WTTM by replacing fixed inverse temperature with a maintained per-sample state while retaining the original loss definitions.TA-WTTM uses a decoupled temperature update rather than minimizing the full weighted objective, because the latter underperforms in the reported ablation.
  • Efficient implementation: The derivative computation requires only low-order moments and class-wise weighted sums under p(γ), making the update computationally efficient.The algorithm uses one softmax over K classes and a constant number of O(K) weighted sums per scheduled update.
  • Closed-form derivatives: The first and second derivatives use centered teacher and student logits under transformed-teacher weighting, expressed through variance, covariance, and higher-order teacher statistics.The first derivative balances teacher dispersion against teacher–student alignment, while curvature determines update aggressiveness.
  • Interpretation: Updating γ also modulates the implicit Rényi entropy regularization and can act as a curriculum by softening targets for mismatched samples and sharpening them for aligned samples.The interpretation follows from γ controlling the transformed teacher distribution’s sharpness.

5 Evaluation

Across CIFAR100 and ImageNet1k evaluations, sample-wise temperature adaptation generally improves TTM and WTTM, including settings without cross-entropy and with transformer teachers. Diagnostics and ablations indicate that sample-level Newton updates provide useful, curvature-aware adaptation with modest gains over strong baselines.

  • CIFAR100 results: TA-TTM generally matches or outperforms TTM, while TA-WTTM consistently improves WTTM across same- and cross-architecture CIFAR100 settings.The gains are described as modest refinements because TTM and WTTM already use carefully tuned temperature configurations.
  • ImageNet1k results: TA-TTM improves reproduced TTM on ImageNet1k, and TA-WTTM further improves WTTM while remaining competitive with EA-KD.The result uses a ResNet-34 teacher and ResNet-18 student, with reproduced TTM and WTTM results obtained in the reported ImageNet1k comparison.
  • Additional settings: Both adaptive variants retain their advantage when the supervised cross-entropy term is removed, indicating gains beyond interaction with that term.The evaluation concerns representative CIFAR100 same-architecture pairs and compares TTM/WTTM variants without cross-entropy.
  • Additional settings: Adaptive variants further improve TTM and WTTM when transferring knowledge from transformer teachers to CNN students, where the teacher-student gap is pronounced.The evaluated teachers include ViT-S, Swin-T, and Mixer-B/16, with ResNet18 as the student.
  • Diagnostics and ablations: The mean inverse temperature decreases before stabilizing, harder samples receive smaller γ, and sample-wise Newton updates perform best in the reported ablations.Smaller γ corresponds to larger effective temperature and softer teacher targets; sample-wise adaptation also outperforms global and class-wise alternatives.

6 Conclusion

The paper introduces per-sample teacher-side inverse-temperature adaptation for TTM and WTTM, using curvature-aware updates that generally improve reproduced baselines across CIFAR100 settings and outperform a compared baseline on ImageNet1k.

  • Contribution: TA-TTM and TA-WTTM adapt the teacher-side inverse temperature per sample within the TTM framework.The update uses closed-form first and second derivatives expressed through variance-covariance statistics of centered logits.
  • Optimization: The curvature-aware update has negligible overhead because its derivatives use variance-covariance statistics under transformed teacher weighting.
  • Results: TA-TTM and TA-WTTM generally improve reproduced TTM and WTTM baselines across multiple CIFAR100 settings.
  • Results: TA-TTM and TA-WTTM outperform the compared temperature-adaptive baseline in the ImageNet1k setting.
  • Analysis: The analysis finds a negative correlation between γ and per-sample difficulty, suggesting softer teacher targets for harder samples.The paper connects this pattern to curriculum-like modulation of implicit Rényi entropy regularization in TTM.

A.1 Sample-wise optimization of the teacher temperature

The appendix formulates sample-wise temperature adaptation by optimizing the teacher-side inverse temperature γ for a fixed teacher-student pair, with logits held constant during differentiation and derivatives expressed through weighted expectations and covariance statistics.

  • Parameterization: The teacher temperature is reparameterized as γ = 1/T and optimized separately for each sample.
  • Objective: For a fixed teacher-student logit pair, the method minimizes the TTM discrepancy with respect to the inverse temperature.
  • Differentiation: The γ-update treats teacher and student logits as constants, corresponding in implementation to stop-gradient on logits.
  • Derivative identities: Centered teacher logits and weighted class-wise expectations provide the statistics used to derive temperature derivatives.The appendix defines expectations under the transformed teacher weighting and uses covariance with teacher logits.
  • Derivative identities: Student log-probabilities can replace student logits inside the covariance because their normalization term is class-independent.
  • Comparison: In the ATD comparison, TA-TTM and TA-WTTM outperform ATD by 0.59 and 0.74 percentage points, respectively, on CIFAR100 WRN-40-2→WRN-16-2.The comparison uses ATD’s CNN-RIS-AT setting without Mixup.

C.1 Cross-architecture temperature diagnostics

Cross-architecture diagnostics for WRN-40-2→ShuffleNetV1 show temperature dynamics, inverse-temperature distributions, and difficulty correlations for TA-TTM and TA-WTTM.

  • Observed pattern: For WRN-40-2→ShuffleNetV1, inverse-temperature distributions broaden during training and larger per-sample cross-entropy losses tend to associate with smaller γ.
  • Diagnostic layout: The diagnostics compare TA-TTM and TA-WTTM across temperature dynamics, stage-wise inverse-temperature distributions, and correlations with per-sample cross-entropy loss.

C.2 Curvature and clipping

Curvature diagnostics examine the estimated update curvature across all 50,000 training samples at three epochs, finding that negative curvature is rare and largely disappears later in training.

  • Evaluation: 50,000 training samples are evaluated at three update epochs for the estimated temperature-update curvature.
  • Negative curvature: 0.164% of samples show negative curvature at epoch 100 for either method.
  • Negative curvature: Negative curvature disappears for TA-TTM at epochs 200 and 240, while TA-WTTM falls to 0.004% and then 0%.

C.3 Correlation between inverse temperature and sample difficulty

The analysis reports consistently negative associations between learned inverse temperature and per-sample cross-entropy loss, alongside the hyperparameter settings used for the proposed method.

  • Correlation analysis: Pearson correlations range from −0.713 to −0.645, while Spearman correlations range from −0.744 to −0.681 across epochs 100, 200, and 240.These correlations compare learned inverse temperature γ with per-sample cross-entropy loss for WRN-40-2→WRN-16-2 using 1,000 samples per epoch.
  • Correlation analysis: All Pearson and Spearman coefficients are negative at epochs 100, 200, and 240 for both methods.
  • Hyperparameters: For CIFAR100, the method uses original TTM and WTTM configurations for γinit and β, while η is grid-searched over {0.001,0.002,...,0.005}.
  • Hyperparameters: For ImageNet1k, the method uses the TTM configuration and grid-searches η over {0.001,0.002,...,0.005}.
Loading 2608.29099v1…