PaperlayerView exact revision on arXiv

Source-linked AI summary

Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model

Karsten Roth, Lukas Thede, Almut Sophia Koepke, Oriol Vinyals, Olivier Hénaff, Zeynep Akata

arXiv:2310.17653v2cs.LGcs.CV

TL;DR

Pretrained models can hold complementary data context, but transferring it between arbitrary already-trained models without external rankings must avoid degrading the student. The paper uses continual-learning-inspired data partitioning, raising transfer success from 32.5% to 92.5% and supporting transfer from weaker teachers.

  • Problem

    The paper asks whether complementary knowledge can be transferred between arbitrary already-trained models without external rankings or performance degradation, despite the need to retain the student’s existing knowledge.

    Source: §1 · p. 2

  • Method

    The approach treats transfer as continual learning and partitions data to select teacher-learning samples and samples where student behavior should be retained.

    Source: §1 · p. 2

  • Results

    Data partitioning raised the success rate of teacher-student pairings with non-zero student gains from 32.5% with normal distillation to 92.5%, including positive transfer from weaker teachers.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The approach can use freely available model collections for gains, including transferring context from weaker, lower-resource models without explicitly requiring additional data or supervision.

    Source: §1 · p. 2

    Weight interpolation constrains updates equally across samples and can struggle to exploit teachers’ relative areas of expertise.

    Source: §4.2 · p. 5

Abstract

from arXiv · show

Training deep networks requires various design decisions regarding for instance their architecture, data augmentation, or optimization. In this work, we find these training variations to result in networks learning unique feature sets from the data. Using public model libraries comprising thousands of models trained on canonical datasets like ImageNet, we observe that for arbitrary pairings of pretrained models, one model extracts significant data context unavailable in the other -- independent of overall performance. Given any arbitrary pairing of pretrained models and no external rankings (such as separate test sets, e.g. due to data privacy), we investigate if it is possible to transfer such "complementary" knowledge from one model to another without performance degradation -- a task made particularly difficult as additional knowledge can be contained in stronger, equiperformant or weaker models. Yet facilitating robust transfer in scenarios agnostic to pretrained model pairings would unlock auxiliary gains and knowledge fusion from any model repository without restrictions on model and problem specifics - including from weaker, lower-performance models. This work therefore provides an initial, in-depth exploration on the viability of such general-purpose knowledge transfer. Across large-scale experiments, we first reveal the shortcomings of standard knowledge distillation techniques, and then propose a much more general extension through data partitioning for successful transfer between nearly all pretrained models, which we show can also be done unsupervised. Finally, we assess both the scalability and impact of fundamental model properties on successful model-agnostic knowledge transfer.

1 INTRODUCTION

Different training choices give pretrained models distinct data context, and arbitrary model pairs can contain complementary knowledge regardless of relative performance. The paper examines general transfer without external rankings, finding standard distillation often degrades pretrained students and proposing data partitioning to improve transfer.

1 INTRODUCTION

  • Arbitrary pretrained model pairs can encode complementary data context, even across model families and when the teacher is weaker or similarly performing.The paper frames this as a consequence of models learning unique feature sets under different training decisions.Source: §1 · pp. 1–2
  • Transferring that context between already-trained models without external ranking measures is difficult because the student’s existing knowledge must also be retained.The aim is to allow transfer without performance degradation, including from weaker, lower-resource teachers.Source: §1 · p. 2
  • Standard distillation is model- and hyperparameter-dependent and causes performance drops for most students, especially with weaker or equiperformant teachers.The paper attributes these drops to catastrophic forgetting outweighing the benefits of transferred complementary knowledge.Source: §1 · p. 2
  • The proposed approach treats transfer as continual learning and partitions data to select where to learn from the teacher and where to retain student behavior.The partitioning reduces constraints on model weights and can be performed without supervision.Source: §1 · p. 2
  • 32.5% with normal distillation versus 92.5% with data partitioning: the success rate of teacher-student pairings with non-zero student gains.The approach supports positive transfer from weaker teachers, needs no pairing-specific hyperparameters, and can scale to sequential transfer from multiple models.Source: §1 · p. 2

2 RELATED WORK

Prior distillation work spans soft-target matching, feature alignment, and multiple-teacher methods, while many approaches assume untrained students or impose architecture constraints. This paper situates arbitrary transfer between already-trained models alongside continual-learning strategies.

2 RELATED WORK

  • Early distillation matched teacher soft targets, while later methods added temperature scaling, structural feature context, and contrastive feature-space alignment.Some feature-matching approaches require teacher and student architectures to match.Source: §2 · pp. 2–3
  • Multi-teacher distillation extends these objectives through methods such as teacher-output reweighting, but the paper targets arbitrary already-trained models without performance or architecture restrictions.Unlike standard distillation, the task also requires retaining the student’s initial knowledge.Source: §2 · p. 3
  • Continual-learning approaches include weight regularization, replay through data memory, and network restructuring; interpolation has also been studied for continual adaptation from pretrained models.The paper includes interpolation in its transfer study.Source: §2 · p. 3

3 COMPLEMENTARY KNOWLEDGE BETWEEN PRETRAINED MODELS

Across pretrained model pairs, teacher-correct/student-incorrect predictions reveal complementary knowledge even when the teacher performs worse. That context is concentrated in semantically related class groups, motivating general-purpose transfer.

3 COMPLEMENTARY KNOWLEDGE BETWEEN PRETRAINED MODELS

  • Complementary knowledge is measured as the proportion of samples correctly classified by the teacher but incorrectly by the student, using pretrained ImageNet classifiers.The study samples model pairs from timm and compares their validation-set predictions.Source: §3 · p. 3
  • Classes carrying the most complementary knowledge are semantically related, with their similarity nearly double the all-class average.Context is more concentrated in a few classes for weaker teachers; the paper calls class groups with significant complementary knowledge relative areas of expertise.Source: Figure 2 · p. 4, §3 · p. 4
  • Up to 20% lower teacher accuracy still yields positive flips; their share approaches 2%, versus 0.1% random chance (randomized ResNet50 teachers: about 0.03%).The analysis covers 466 pairs; the complementary share remains substantial even for weaker teachers.Source: §3 · p. 4

4 GENERAL KNOWLEDGE TRANSFER METHODOLOGY

The methodology frames transfer between pretrained models as balancing knowledge gain with retention, then develops distillation regularizers at the weight and data levels and extensions to multiple teachers.

4.1 KNOWLEDGE TRANSFER THROUGH KNOWLEDGE DISTILLATION

  • The base objective matches teacher and student soft targets through KL divergence, optionally combined with auxiliary cross-entropy as XE-KL distillation.Source: §4.1 · pp. 4–5
  • Standard distillation struggles to transfer knowledge without performance drops for most pretrained teacher–student pairs; transfer delta measures the change in student top-1 accuracy.The transfer delta is post-transfer student accuracy minus initial student accuracy.Source: §4.1 · p. 5

4.2 KNOWLEDGE TRANSFER THROUGH CONTINUAL KNOWLEDGE DISTILLATION

  • Momentum weight interpolation retains a slow student-weight copy, but constrains updates equally across samples and can underuse teachers’ areas of expertise.The update frequency and momentum tune the plasticity–stability trade-off.Source: §4.2 · p. 5
  • Data partitioning distills selected samples from the teacher and others from the frozen initial student, with confidence-based label-free assignment matching the supervised setting in practice.The masks are derived from model confidence and the resulting student is trained with a KL-divergence objective.Source: §4.2 · pp. 5–6, Figure 3 · p. 5

4.3 MULTI-TEACHER KNOWLEDGE TRANSFER

  • For multiple teachers, the paper proposes parallel or sequential transfer and studies weight-interpolated soups of student variants distilled independently from each teacher.Source: §4.3 · p. 6

5 EXPERIMENTAL STUDY ON EFFECTIVE KNOWLEDGE TRANSFER

Across 400 teacher–student pairs, KL-Dist+DP raises positive-transfer success to 92.5%, versus 32.5% for normal distillation, and enables gains from weaker and stronger teachers. The experiments also find that sequential multi-teacher transfer can improve on the best single-teacher result, while transfer depends on complementary class knowledge and student properties.

5.1 EVALUATION OF DIFFERENT APPROACHES FOR KNOWLEDGE TRANSFER

  • For general-knowledge transfer, KL+DP raises success rates to 92.5%, while XE-KL+MCL reaches 80% median success; standard KL distillation yields positive transfer for fewer than 40% of teachers for most students.KL+DP shows positive-transfer deltas even for much weaker teachers and barely limits gains from much stronger teachers.Source: §5.1 · pp. 6–7, Figure 4 · p. 7
  • On full ImageNet, KL+DP transfers more reliably than normal distillation, including gains from weaker teachers and about +0.3% for strong PiT-B and ConvNeXt students.The cited examples include PiT-B at 82.44% top-1 and ConvNeXt at 84.57%, with a nearly 5% performance differential.Source: §5.1 · p. 7, Table 1 · p. 7, Table 2 · p. 8
  • Unsupervised KL-Dist+DP performs comparably to supervised transfer and sometimes better; for PiT-B, the reported transfer deltas are 0.55 and 0.95, respectively.Confidence-based partitioning assigns 72% of positive flips and 9% of negative flips to the teacher, so the unsupervised selection is imperfect but aligns with areas of expertise.Source: §5.1 · p. 8, Table 3 · p. 8
  • KL+DP preferentially transfers complementary knowledge from a teacher’s stronger expertise areas, with transfer rates dropping for less-represented classes.This pattern suggests that selecting a teacher with a suitable expertise area can guide which context a student learns.Source: §5.1 · p. 8, Figure 5 · p. 8
  • Greater student capacity is associated with greater reception of complementary knowledge, while CNN-style models more often overwrite existing knowledge and have lower distillation deltas.The capacity relationship appears across architectural families, including after controlling for initial accuracy in additional experiments.Source: Figure 5 · p. 8, §5.1 · p. 9

5.2 KNOWLEDGE TRANSFER FROM MULTIPLE PRETRAINED MODELS

  • Sequential transfer from three teachers averages 59% more gain than the best single-teacher delta (Twins: 0.26 to 0.65; PiT-B: 0.86 to 1.04).Students consistently gain from each teacher, though returns diminish as they improve.Source: §5.2 · p. 9, Table 4 · p. 9
  • Vanilla KL distillation fails in the multi-teacher setting, while parallel transfer does not beat the best single teacher and student weight averaging performs worst.The authors attribute the parallel result to insufficient retention and the averaging result to limited interpolatability and reduced knowledge retention.Source: §5.2 · p. 9, Table 4 · p. 9
  • Sequential transfer gains are consistent across teacher orders, but putting the strongest teacher first can cause disproportionately greater forgetting.For the large-scale transfer evaluation, the authors use a 10% stratified ImageNet subset and a 20-epoch budget, then validate main claims on full ImageNet.Source: §5 · p. 6, §5.2 · p. 9

6 CONCLUSION

The paper concludes that models trained on the same data with different protocols contain complementary knowledge, and reports a first general transfer approach based on continual learning and confidence-based data partitioning. Experiments show positive transfer from weaker and stronger teachers, including unsupervised and multi-model settings, while identifying model capacity and visual inductive bias as relevant properties.

6 CONCLUSION

  • Models trained on the same dataset with different protocols exhibit complementary knowledge: data context encoded in one model but not another.The conclusion says this holds across arbitrary pairings, including differences in model family or performance.Source: §6 · p. 9
  • The proposed general transfer mechanism combines a continual-learning perspective with confidence-based, hyperparameter-free data partitioning and avoids external ranking measures.The paper presents this as an initial exploration of transfer between arbitrary model pairs without performance degradation.Source: §6 · p. 9
  • Transfer success rises from under 40% to over 92%, with positive transfer from both stronger and weaker teachers.The conclusion also reports that transfer can be unsupervised and showcases transfer from multiple models.Source: §6 · p. 9
  • The reported model properties associated with transfer include greater capacity and reduced visual inductive bias.The conclusion describes the experiments as a proof of concept and a first step toward general-purpose transfer tools.Source: §6 · p. 9

A.1 IMPLEMENTATION OF DISTILLATION-BASED KNOWLEDGE TRANSFER VARIANTS

The appendix reports that ordinary distillation’s transfer behavior is sensitive to teacher–student dynamics and tuning, motivating alternatives for pretrained experts. It also describes contrastive feature-matching variants and their explored settings.

A.1 IMPLEMENTATION OF DISTILLATION-BASED KNOWLEDGE TRANSFER VARIANTS

  • Across 33 randomly selected pairs, learning rate 1e-4 worked best on average for KL distillation, but average transfer delta remained negative regardless of learning rate.The search used four learning rates from 1e-2 to 1e-5.Source: §A.1 · p. 16
  • The preferred classification-loss weight depends on teacher–student direction: larger distillation weight suits a stronger teacher, while smaller weight better suits a weaker teacher.For a weaker teacher, retaining student knowledge is especially important because student knowledge can be overwritten.Source: §A.1 · p. 16
  • Temperature 0.1 limits student performance decreases with weaker teachers but also restricts transfer, and larger temperatures did not improve transfer performance.The authors report that the small temperature also limits gains even from stronger teachers, unlike the larger temperature of 4 proposed by Hinton et al. (2015).Source: §A.1 · p. 16
  • Temperature and loss weighting govern distillation aggressiveness, which varies across arbitrary model pairs and makes standard distillation poorly suited as a general transfer tool.Adding a classification loss does not remove that pair-dependent trade-off.Source: §A.1 · p. 16

A.2 IMPLEMENTATION OF CONTRASTIVE DISTILLATION KNOWLEDGE TRANSFER

  • The appendix explores contrastive transfer by matching teacher–student feature relations, using either pairwise similarity matching or CRD’s positive- and negative-pair alignment.The similarity-based approach matches distances between image-pair representations; CRD brings same-image augmentations together and separates different-class images.Source: §A.2 · p. 17
  • For contrastive distillation, equal weighting of contrastive and classification losses with learning rate 1e-4 worked best on average for CD, while CRD used Tian et al. (2020)’s settings.The CD setting still showed a trade-off similar to the one described for ordinary distillation.Source: §A.2 · p. 17

A.3 IMPLEMENTATION OF CONTINUAL LEARNING BASED TRANSFER APPROACHES

The appendix documents the parameter searches for continual-learning transfer approaches and describes the model sets used in the ImageNet studies. KL+DP adds no hyperparameters beyond standard KL distillation, and the selected setup omits the auxiliary classification objective.

A.3 IMPLEMENTATION OF CONTINUAL LEARNING BASED TRANSFER APPROACHES

  • XE-KL+MCL’s selected setting used momentum 0.9999, interpolation every other iteration, learning rate 0.01, and classification weight 0.7.The search varied learning rate, momentum, interpolation frequency, and classification-objective weighting.Source: §A.3 · pp. 17–18
  • KL+DP adds no hyperparameters over standard KL distillation, and the selected setting uses learning rate 1e-4, temperature 1, and no auxiliary classification loss.The best tested performance consistently used λ = 1; the authors note that deeper searches could improve average success rate, while treating current results as proof of concept.Source: §A.3 · p. 18

A.4 MODEL LISTS: LARGE-SCALE STUDIES ON STRATIFIED IMAGENET SUBSETS

  • The 10% ImageNet-subset evaluation used diverse model families, sizes, and performance levels; the complementary-context exploration covered 466 pairs from 301 timm models.The subset model selection is described in Table 5.Source: §A.4 · p. 18, Table 5 · p. 18

A.5 MODELS EVALUATED ON FULL IMAGENET

  • The full-ImageNet model set likewise spans architecture types, sizes, and performance levels, and multi-teacher experiments reused its teachers with a subset of its students.The model selection for full ImageNet is described in Table 6.Source: §A.5 · p. 18, Table 6 · p. 19

B.1 ADDITIONAL EXPERIMENTS ON VARIANTS OF DISTILLATION-BASED KNOWLEDGE TRANSFER

The tested distillation variants offer limited improvements for transferring knowledge between pretrained models. Contrastive distillation often overwrites student knowledge, while restricting KL divergence to likely classes yields a small average gain in a small sample.

Using a cross-entropy plus distillation transfer objective

  • Cross-entropy loss produced a 1.2 percentage-point lower average transfer delta than KL divergence across 60 teacher-student pairs; linear-layer warmup brought no improvement.The experiments found no advantage for cross-entropy over KL divergence.Source: §B.1 · p. 19

Restricting the set of classes for computing the distillation-based transfer loss

  • Top-10 divergence increased average transfer delta by 0.20 percentage points across six pairs, with gains remaining limited.Low-probability classes can influence KL divergence, and the improvement magnitude aligned with differences between top-10 and total KL divergence.Source: §B.1 · p. 19

Contrastive distillation for knowledge transfer between arbitrary models

  • Contrastive distillation rarely improves pretrained students and appears to overwrite more of their existing knowledge than it transfers.The comparison reports similar levels of transfer but stronger overall overwriting for contrastive approaches.Source: Figure 6 · p. 19, §B.1 · p. 20

B.2 EXTENDED RESULTS ON KNOWLEDGE TRANSFER BETWEEN PRETRAINED MODELS

Extended results examine whether knowledge transfer adds to, rather than replaces, a pretrained student's knowledge. KL+DP shifts most teacher-student pairs toward positive transfer and reduces positive prediction flips while largely preserving negative flips.

B.2 EXTENDED RESULTS ON KNOWLEDGE TRANSFER BETWEEN PRETRAINED MODELS

  • KL+DP moves most teacher-student pairs toward positive knowledge transfer, while vanilla KL distillation generally overwrites rather than augments student knowledge.In the scatterplot, points above the red diagonal indicate positive transfer; KL+DP pushes a vast majority of points in that direction, including transfers from weaker teachers.Source: §B.2 · p. 20
  • The supplementary table reports individual transfer deltas for teacher-student pairs from Tables 1 and 3.It provides additional pair-level transfer results.Source: §B.2 · p. 21, Table 7 · p. 21
  • KL+DP reduces positive prediction flips for all student models, while negative prediction flips change only slightly.Positive flips are cases where the teacher is correct and the student is incorrect; negative flips are the reverse.Source: §B.2 · p. 21, Table 8 · p. 21

B.3 EXTENDED RESULTS ON THE IMPACT OF DIFFERENT STUDENT MODEL PROPERTIES ON KNOWLEDGE TRANSFER

These extended experiments examine student properties, additional datasets, and domain shifts. They report that student capacity relates to transferability, KL+DP transfers across several datasets, and it improves performance when the teacher and student are trained on different domains.

B.3 EXTENDED RESULTS ON THE IMPACT OF DIFFERENT STUDENT MODEL PROPERTIES ON KNOWLEDGE TRANSFER

  • Student capacity is strongly correlated with receiving new knowledge without overwriting previous knowledge, even when controlling for initial accuracy.This pattern is especially pronounced for CNNs; increased size can yield improved transferability in CNN families.Source: §B.3 · pp. 21–22, Figure 8 · p. 22

B.4 EXTENDED RESULTS ON ADDITIONAL DATASETS

  • Across CUB200, Caltech256, and Stanford Cars, KL-Dist.+DP transfers knowledge from weaker teachers without compromising student performance and transfers substantial knowledge when teachers are much stronger.The figure reports mean transfer deltas for the top 25% of pairs in bins of teacher-student performance difference.Source: Figure 9 · p. 22, §B.4 · p. 22

B.5 EXTENDED RESULTS ON KNOWLEDGE TRANSFER UNDER DOMAIN SHIFTS

  • With an Infograph-trained teacher and an ImageNet-pretrained student, KL-Dist.+DP improves student performance, whereas conventional KL-Dist. substantially decreases accuracy, particularly with a weaker teacher.The transfer uses ImageNet data and evaluates nine students with four teachers.Source: Figure 10 · p. 23, §B.5 · p. 23

B.6 EXTENDED RESULTS ON THE TRANSFER FROM MULTIPLE TEACHERS

Sequential transfer from multiple teachers can add to a pretrained student's gains, but the results show diminishing returns as forgetting increases. Vanilla KL transfer does not outperform the best single teacher and loses improvements from earlier transfers.

B.6 EXTENDED RESULTS ON THE TRANSFER FROM MULTIPLE TEACHERS

  • Sequential KL-Dist.+DP transfer can yield positive transfer deltas from successive teachers, but returns diminish as knowledge loss increases.For the PiT-B visualization, teachers are ordered by ascending performance and receive 20 transfer epochs each; transfer delta exceeds that of the strongest single student by 1.04.Source: §B.6 · p. 23, Figure 11 · p. 23
  • Sequential vanilla KL-Dist. gains fall in the range of the three teachers' average transfer delta and do not exceed the best teacher alone.The comparison table covers sequential and soup-based vanilla KL transfer alongside single-teacher results.Source: Table 9 · p. 24, §B.6 · p. 24
  • Vanilla KL-Dist. overwrites earlier improvements during later teacher transfers and cannot be applied directly to parallel multi-teacher transfer.The reported sequential gains reflect average transfer across the teachers rather than accumulation of complementary knowledge.Source: §B.6 · p. 24
Loading 2310.17653v2…