PaperlayerView exact revision on arXiv

Source-linked AI summary

When Less Data Favors Smaller Teachers: Rethinking Teacher Capacity and Data Selection for Knowledge Distillation

Minjae Park, Taesun Yeom, Jaeho Lee

arXiv:2609.34489v1cs.LG

TL;DR

Data pruning can reduce KD training cost, while the preferred teacher can change with the data budget. The paper analyzes teacher supervision and subset properties, then proposes DVA, a training-dynamics-free selection method. DVA consistently outperforms training-dynamics-free baselines and remains competitive with training-dynamics-based methods; the framework assumes a suitable low-capacity proxy teacher and predefined difficulty mapping.

  • Problem

    Data pruning reduces KD training cost, but teacher preference varies with the available data budget, raising questions about what drives the shift and which samples are useful.

    Source: §1 · p. 1

  • Method

    DVA uses a small-teacher proxy for budget-aware difficulty filtering and class-conditional selection of subsets with diverse relational signals.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    DVA consistently outperforms training-dynamics-free baselines and remains competitive with training-dynamics-based methods, surpassing them in several settings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The analyses identify budget-appropriate sample difficulty and relational diversity as properties of effective pruned subsets for KD.

    Source: §1 · p. 2

    DVA assumes access to a suitable low-capacity proxy teacher and a predefined mapping from pruning rate to preferred difficulty region.

    Source: A LIMITATION AND FUTURE WORK · p. 14

Abstract

from arXiv · show

Data pruning reduces the training cost of knowledge distillation (KD). However, the preferred teacher capacity changes with the data budget: smaller teachers can outperform larger ones when limited training data are available. Understanding what drives this shift is important not only for teacher choice but also for identifying which samples are useful for distillation. We analyze teacher supervision by decomposing it into relational ordering---the ranking of classes---and score geometry---the magnitudes and margins of class probabilities---and show that the small-teacher advantage in the low-data regime arises not only from score geometry but also from relational ordering. Beyond understanding teacher capacity, our analysis reveals two properties of effective subsets: samples should match the difficulty appropriate for the available budget, and their relational signals should be diverse rather than redundant. Based on these findings, we propose DVA (Difficulty- and Volume-Aware data selection for KD), a training-dynamics-free method, which uses a small teacher as a proxy for budget-aware difficulty filtering and class-conditional relational volume maximization. Despite requiring no training dynamics statistics, our method remains competitive with training-dynamics-based methods while consistently outperforming training-dynamics-free baselines.

1 INTRODUCTION

Data pruning lowers KD training cost, but teacher preference can shift with the available data. The paper analyzes that shift, identifies subset properties, and introduces DVA as a training-dynamics-free selection method.

Motivation

  • KD transfers knowledge from a larger teacher to improve a smaller student, while inference on large datasets adds training cost.Data pruning selects a smaller training subset to reduce that cost while preserving student performance.Source: §1 · p. 1

Motivation and research questions

  • Teacher preference varies with the data budget, challenging capacity-gap-only accounts of teacher quality in KD.The paper asks why teacher capacity preference changes, what pruned subsets should preserve, and how to turn those findings into a data-selection method.Source: §1 · pp. 1–2

Teacher supervision

  • The analysis finds that both relational ordering and score geometry contribute to the small-teacher advantage in the low-data regime.Relational ordering ranks classes; score geometry captures class-probability magnitudes and margins.Source: §1 · p. 2, Figure 1 · p. 2

Pruned subsets

  • Effective subsets pair difficulty suited to the available budget with diverse relational signals rather than redundant teacher outputs.At the sample level, preferred examples become easier as the budget decreases; at the set level, diverse relationships outperform redundant outputs among samples of comparable difficulty.Source: §1 · p. 2

DVA

  • DVA uses a small-teacher proxy for budget-aware filtering and class-conditional relational-volume selection, without student-training statistics.Across ResNet and ViT families on CIFAR-100, Tiny-ImageNet, and ImageNet, it consistently outperforms training-dynamics-free baselines and remains competitive with training-dynamics-based methods.Source: §1 · p. 2

2 UNDERSTANDING BUDGET-DEPENDENT TEACHER PREFERENCE

Controlled vanilla logit-based KD experiments show a capacity–budget crossover and test how teacher supervision components relate to it. The analyses also characterize how large and small teachers distribute non-target probability.

2 UNDERSTANDING BUDGET-DEPENDENT TEACHER PREFERENCE

  • The section studies vanilla logit-based KD, establishing the crossover with controlled experiments and then examining the supervision components involved.The decomposition swaps relational ordering and score geometry between large and small teachers to isolate their effects.Source: §2 · p. 2, §2.2 · p. 3

2.1 CAPACITY–BUDGET CROSSOVER IN KNOWLEDGE DISTILLATION

  • At 90% random pruning, the smaller teacher improves student accuracy by 15.56 pp despite lower test accuracy and approximately 63.3× fewer parameters.Larger teachers become more effective as the data budget increases, with preference crossing at an intermediate point; the reported test accuracies are 67.44% versus 79.71%.Source: §2.1 · p. 3
  • For a representative CIFAR-100 sample, the small teacher assigns more non-target probability to semantically related classes, while the large teacher spreads it more uniformly.The authors describe this output contrast as intuitive context, not evidence identifying which supervision components drive transferability.Source: §2.1 · p. 3, Figure 2 · p. 3

2.2 DECOMPOSING CAPACITY-DEPENDENT TEACHER SUPERVISION

  • At 90% pruning, replacing large-teacher score geometry yields an 8.18 pp accuracy gain, while replacing its relational ordering yields 3.82 pp.Replacing both components yields a 15.56 pp gain; the table decomposes supervision for CIFAR-100 width-64 and width-8 ResNet-34 teachers.Source: Table 1 · p. 3, §2.2 · p. 3

3 WHAT SHOULD A PRUNED DATASET PRESERVE?

The preferred sample difficulty shifts toward easier examples as the data budget shrinks, while diverse teacher-output patterns help avoid redundant supervision in low-data subsets.

Sample-level suitability

  • As the data budget decreases, the preferred difficulty shifts from hard samples toward medium-difficulty samples and then easy samples.Figure 3a partitions samples into easy, medium, and hard regions and marks the best-performing region at each drop rate.Source: §3 · p. 4, Figure 3 · p. 4
  • At low budgets, difficult examples can displace basic class representativeness, making easier samples relatively more useful.The proposed explanation contrasts this with high-data settings, where representative examples are already covered and difficult examples can refine the decision boundary.Source: §3 · p. 4

Set-level supervision complementarity

  • In the low-data regime, cosine farthest-first improves KD accuracy by up to +1.2 points, while closest-first degrades it by down to −2.1 points.At a 50% drop rate, both rules are within 0.02 points of random selection; at lower drop rates, farthest-first's advantage vanishes.Source: §3 · p. 4, Figure 3 · p. 4
  • Among samples of comparable difficulty, diverse relational signals outperform subsets dominated by redundant teacher outputs.The controlled comparison uses the same difficulty-filtered candidate pool; repeatedly selecting similar teacher-output patterns can be harmful in the low-data regime.Source: §3 · p. 4

Section synthesis

  • Effective selection therefore combines budget-appropriate individual sample difficulty with set-level avoidance of redundant teacher supervision.The pairwise cosine rules probe diversity, but do not characterize the selected set as a whole.Source: §3 · p. 5

4 DVA: DIFFICULTY- AND VOLUME-AWARE DATA SELECTION

DVA uses a small teacher as a selection proxy, first filtering samples by budget-appropriate difficulty and then selecting class-wise subsets with diverse relational signals.

DVA overview

  • DVA uses the small model as a selection proxy, while retaining the given teacher for final distillation.The proxy's predictions ease subset search because directly evaluating candidate subsets requires training the student for each one.Source: §4 · p. 5
  • DVA has three stages: budget-aware candidate filtering, class-conditional relational representation, and subset selection that maximizes volume in proxy-output space.The selected examples are intended to cover important relational directions while avoiding redundant proxy outputs.Source: §4 · p. 5

Stage 1: Budget-aware difficulty filtering

  • Budget-aware filtering defines difficulty by the small teacher's probability for the ground-truth class and retains samples in a budget-dependent difficulty interval.The candidate pool is constrained to contain at least the class-wise retained count, supporting a uniform drop rate across classes.Source: §4 · p. 5

Stage 2: Relational projection

  • DVA represents each candidate by its deviations from the class mean along principal directions, weighted by the variation along those directions.The dominant directions correspond to recurring confusions with competing classes.Source: §4 · pp. 5–6

Stage 3: Volume maximization

  • A log-determinant objective selects subsets that cover important relational directions and account for variation magnitude, with diminishing gains for already-covered directions.The objective is regularized with δ = 10^-3; the matrix determinant lemma and Sherman–Morrison updates reduce the cost of greedy selection.Source: §4 · p. 6

5 EXPERIMENT

The experiments compare DVA with pruning baselines, isolate its selection stages, test proxy-teacher capacity, and evaluate generalization across datasets and architectures. DVA leads training-dynamics-free baselines, with stronger gains under tighter data budgets.

5.2 MAIN EXPERIMENT

  • At a 0.9 drop rate, DVA reaches 53.76%, outperforming the strongest training-dynamics-free baseline, IF-Beta, by 4.77 pp.It achieves the best accuracy among training-dynamics-free approaches across all drop rates and remains competitive with strong training-dynamics-based methods.Source: Table 2 · p. 7, §5.2 · p. 7

5.1 EXPERIMENTAL SETUP

  • Experiments use CIFAR-100 primarily, with ResNet-34 teachers and ResNet-18 students; comparisons include training-dynamics-free and training-dynamics-based pruning methods.ViTs on CIFAR-100 and Tiny-ImageNet and ImageNet extend evaluation beyond the default setting.Source: §5 · p. 6, §5.1 · p. 7

5.3 EFFECT OF DIFFICULTY FILTERING AND VOLUME MAXIMIZATION

  • Difficulty filtering improves KD accuracy by 1.02, 2.05, and 6.09 points at drop rates 0.7, 0.8, and 0.9; relational selection adds 1.00, 1.05, and 0.74 points.Filtered random controls for the difficulty-filtered candidate pool, isolating the additional contribution of volume maximization.Source: §5.3 · pp. 7–8, Table 3 · p. 8

5.4 EFFECT OF SMALL TEACHER AS PROXY

  • At a 0.9 drop rate, accuracy falls from 53.76% with a width-8 proxy to 50.30%, 42.05%, and 35.96% with width-16, width-32, and width-64 proxies.At a 0.1 drop rate, all proxy widths achieve comparable performance around 79%.Source: §5.4 · p. 8, Table 4 · p. 8

5.5 BROADER EXPERIMENTS

  • Across ViT CIFAR-100 and ResNet Tiny-ImageNet settings, DVA generally gains more over random selection in the low-data regime, including under multi-teacher supervision.The reported settings include single- and scheduled multi-teacher supervision; Figure 4 plots accuracy differences from random selection by drop rate.Source: §5.5 · p. 8, Figure 4 · p. 9
  • On ImageNet, DVA improves over random selection and is best among training-dynamics-free methods at five of six pruning rates, with gains rising from 0.28 pp at drop rate 0.1 to 1.72 pp at 0.9.This evaluation uses publicly available pretrained teachers scaled by depth, without retraining or adapting them for selection.Source: §5.5 · p. 9

6 CONCLUSION

The study concludes that both relational ordering and score geometry contribute to smaller teachers’ low-data advantage, motivating DVA’s training-dynamics-free selection strategy. It reports consistent gains across settings and direct use with pretrained teachers.

6 CONCLUSION

  • The study finds that both relational ordering and score geometry contribute to smaller teachers’ more effective supervision in low-data regimes, and identifies budget-appropriate difficulty and relational diversity as subset properties.DVA uses a small-teacher proxy and consistently improves KD performance across pruning rates and experimental settings.Source: §6 · p. 9
  • DVA can be applied to pretrained teachers without additional teacher training or adaptation.The ImageNet evaluation reports KD accuracy for a ResNet-18 student distilled from a ResNet-34 teacher, averaged over three seeds.Source: §6 · p. 9, Table 5 · p. 9

AI USE STATEMENT

  • The authors report using generative AI tools for research and manuscript tasks, while stating that they reviewed AI-assisted work and take responsibility for the final content.They state that AI tools were not used to generate synthetic datasets or prove mathematical claims.Source: AI USE STATEMENT · p. 10

A LIMITATION AND FUTURE WORK

DVA’s framework depends on a suitable low-capacity proxy and a predefined mapping from pruning rate to preferred difficulty. The paper positions its contribution against prior work on teacher capacity, pruning, and subset diversity, and identifies larger-scale distillation as future work.

A LIMITATION AND FUTURE WORK

  • The framework assumes a suitable low-capacity proxy and a predefined pruning-rate-to-difficulty mapping; automatically choosing either remains open.The authors identify automatic proxy selection and adapting the difficulty regime from teacher-output distributions as future directions.Source: A LIMITATION AND FUTURE WORK · p. 14
  • The paper asks whether its capacity–budget relationship and relational-selection principles extend to language models and foundation vision encoders.These larger-scale distillation settings are proposed for future examination.Source: A LIMITATION AND FUTURE WORK · p. 14

B RELATED WORK

  • Prior work connects teacher capacity to transferability and pruning, while this paper uses relational ordering and score geometry to analyze budget-dependent teacher preference.The cited related work includes Cho & Hariharan (2019), Mirzadeh et al. (2020), and Ben-Baruch et al. (2025).Source: B RELATED WORK · p. 14
  • DVA differs from prior pruning approaches by using class-conditional proxy-output relations, budget-aware difficulty filtering, and volume maximization without student training dynamics.The related-work discussion contrasts this design with KD-specific selection methods including MDSLR and IF-Beta.Source: B RELATED WORK · p. 14

C POSITIONING OF DVA

  • DVA can select data before student training once the teacher and proxy are available, combining budget-dependent subset construction with set-level interactions and teacher-output relational geometry.The paper presents these properties as distinguishing its KD-specific framework from representative pruning methods.Source: C POSITIONING OF DVA · p. 15, Table 6 · p. 15

D.1 STUDENT TRAINING

The section documents the student-training and data-selection setups, then examines how teacher supervision and preferred teacher capacity vary with the data budget.

D.1 STUDENT TRAINING

  • CIFAR-100 students use a width-64 ResNet-18 trained for 240 epochs with SGD, temperature 4, and equally weighted cross-entropy and distillation losses.The ImageNet protocol instead uses temperature 1; ViT students use AdamW and transformer-oriented augmentation.Source: §D.1 · pp. 15–16

D.2 DATA SELECTION BASELINES

  • The selection baselines span difficulty, forgetting, gradient or prediction-error, influence, coverage, and neighborhood-diversity signals, with class-balanced random sampling as a training-dynamics-free reference.D2 Pruning uses forward message passing and suppresses scores near already-selected examples; IF-Beta estimates influence using a validation set and a surrogate student.Source: §D.2 · pp. 16–17

D.3 TEACHER MODELS AND CHECKPOINTS

  • Controlled CIFAR-100 ResNet-34 teachers vary width while holding depth and training recipe fixed; the width-8 selection proxy requires about 1/63.3 of width-64 FLOPs for training and inference.ImageNet experiments use public pretrained ResNet checkpoints rather than teacher variants trained specifically for the method.Source: §D.3 · pp. 17–18

D.4 TEACHER TRAINING

  • Teacher variants are independently trained for 300 epochs with SGD, while the from-scratch ViT experiments use a 12-layer scaled ViT-Small trained for 600 epochs with AdamW.The pretrained ViT experiments instead fine-tune ImageNet-21k weights on CIFAR-100.Source: §D.4 · p. 18

D.5 BUDGET-AWARE DIFFICULTY FILTERING

  • Difficulty filtering retains a budget-dependent contiguous interval of each class’s ranking, with CIFAR-100 and Tiny-ImageNet sharing a schedule and ImageNet filtering toward harder samples.No difficulty filtering is applied at a 0.1 drop rate.Source: §D.5 · p. 18, Table 7 · p. 18

F.1 TEACHER SUPERVISION DECOMPOSITION

  • At a 0.9 drop rate, ordering and geometry swaps gain 8.68 and 8.79 pp with width-4, 3.82 and 8.18 pp with width-8, and 0.50 and 3.45 pp with width-16; the joint configuration wins on Tiny-ImageNet.The relative contribution shifts toward score geometry as teacher capacity increases; on Tiny-ImageNet, removing relational ordering from the best joint configuration reduces accuracy.Source: §F.1 · pp. 20–21, Table 8 · p. 20, Table 9 · p. 21

F.2 KD PERFORMANCE ACROSS TEACHER CAPACITIES

  • As data budgets shrink, lower-capacity ResNet teachers tend to become preferable across CIFAR-100, Tiny-ImageNet, and ImageNet.On ImageNet, ResNet-34 leads at drop rates 0.1–0.5, while ResNet-18 and ResNet-10 become preferable at 0.7 and 0.8–0.9, respectively.Source: §F.2 · pp. 21–22, Table 11 · p. 21, Table 12 · p. 22
  • At a 0.9 drop rate, the ViT with embedding dimension 48 reaches 43.68% accuracy, versus 41.66% for the pretrained dimension-384 teacher and 37.79% for its from-scratch counterpart.The capacity-performance trend is roughly U-shaped, not monotonically decreasing.Source: §F.2 · p. 22, Table 14 · p. 22

F.3 TEMPERATURE SCALING ANALYSIS

Temperature tuning improves larger teachers but leaves their low-data gap, while principal directions explain how relational diversity can guide selection. Volume maximization helps beyond difficulty filtering, and DVA is fairly stable for small regularization values.

F.3 TEMPERATURE SCALING ANALYSIS

  • After temperature tuning, width-64 reaches 47.29% at T = 8 while width-8 reaches 63.69% at its best temperature, leaving a 16.40 pp gap.Temperature changes score geometry while preserving class ordering, so the remaining gap is consistent with relational ordering contributing additional useful supervision.Source: §F.3 · p. 23, Table 15 · p. 23

G.1 INTERPRETATION OF PRINCIPAL RELATIONAL DIRECTIONS

  • The leading principal direction is dominated by the ground-truth class, while later directions capture recurring confusion with semantically related non-target classes.Projecting onto these directions emphasizes non-target relations, allowing volume-based selection to favor complementary rather than repetitive teacher outputs.Source: §G.1 · p. 23, Figure 5 · p. 24

G.2 EFFECT OF VOLUME MAXIMIZATION

  • At drop rates 0.7, 0.8, and 0.9, volume maximization improves accuracy over filtered random by 1.00, 1.05, and 0.74 pp, while reverse greedy falls short by 2.95, 3.01, and 2.44 pp.All strategies select from the same difficulty-filtered pool, isolating the effect of volume-based selection.Source: §G.2 · p. 24, Table 16 · p. 25

G.3 SENSITIVITY TO THE REGULARIZATION PARAMETER

  • For δ values from 10^-6 to 10^-2, performance varies modestly; at a 0.9 drop rate, it falls to 52.72% at δ = 1 and 51.62% at δ = 10.The reported accuracy range for small δ values is 53.47–54.19% at that drop rate.Source: §G.3 · p. 25, Table 17 · p. 25

G.4 PROXY TEACHER ABLATION

DVA’s proxy choice matters most when forming the difficulty-filtered candidate pool, while volume maximization is robust to proxy choice once that pool is suitable. Early checkpoints improve a large proxy under aggressive pruning, but the fully trained small proxy performs strongly across budgets without checkpoint search.

G.4 PROXY TEACHER ABLATION

  • At a 0.9 drop rate, the best early width-64 checkpoint reaches 53.47% accuracy, compared with 35.96% for the fully trained width-64 proxy and 53.76% for the width-8 proxy.Early checkpoints recover much of the large proxy’s degradation under aggressive pruning.Source: §G.4 · p. 25, Table 18 · p. 26
  • The fully trained width-8 proxy matches or outperforms the best early width-64 checkpoint at five of six pruning rates, needs no checkpoint search, and has approximately 63.3× fewer parameters.No single early checkpoint performs best across all budgets, requiring checkpoint selection for each pruning rate.Source: §G.4 · p. 25
  • At a 0.9 drop rate, small-teacher difficulty filtering yields 53.76% accuracy versus 39.94% with large-teacher filtering, while volume maximization gives 53.76% versus 53.74%.Volume maximization also beats random selection within the same filtered pool by 1.05 points at a 0.8 drop rate (64.45% vs. 63.40%).Source: §G.4 · p. 26, Table 19 · p. 26

H.1 GENERALIZATION ACROSS ARCHITECTURES AND DATASETS

DVA generalizes across architectures and datasets, and continues to add selection gains when supervision shifts from a large teacher toward a small teacher as the data budget decreases. The scheduled supervision benefit is most evident in the low-data regime.

H.1 GENERALIZATION ACROSS ARCHITECTURES AND DATASETS

  • On CIFAR-100 with ViT, DVA is the best training-dynamics-free method at five of six pruning rates; on Tiny-ImageNet, it beats random selection at every rate, with largest gains under aggressive pruning.These results indicate generalization beyond the ResNet/CIFAR-100 setting used for the primary controlled experiments.Source: §H.1 · p. 26, Table 20 · p. 27, Table 21 · p. 27

H.2 MULTI-TEACHER KD

  • The scheduled baseline linearly shifts KD-loss weight from the large teacher to the small teacher as pruning increases, recovering single-teacher supervision at both endpoints.The schedule reflects the observed budget-dependent teacher preference.Source: §H.2 · p. 27
  • Scheduled multi-teacher supervision benefits most in the low-data regime, where it places greater weight on the smaller teacher and can mitigate performance degradation under moderate to aggressive pruning.Data selection still provides additional gains under this schedule.Source: §H.2 · pp. 27–28
  • With scheduled multi-teacher KD, DVA beats random selection by +1.15/+2.55 pp on CIFAR-100 ViT and +1.24/+1.79 pp on Tiny-ImageNet at drop rates 0.8/0.9.On CIFAR-100 with a ResNet student, DVA improves over random selection at every drop rate and is best among training-dynamics-free methods at five of six.Source: §H.2 · p. 28, Table 22 · p. 28, Table 23 · p. 29

H.3 CROSS-ARCHITECTURE TRANSFER

Subsets selected with the ResNet-based setup remain effective when reused for VGG-11 and MobileNetV2 students, without rerunning selection for those architectures.

H.3 CROSS-ARCHITECTURE TRANSFER

  • The evaluation reuses subsets selected under the original ResNet-based setup, changing only the student architecture to VGG-11 or MobileNetV2.This tests whether the selection benefit transfers across student architectures rather than depending on a particular student.Source: §H.3 · p. 29
  • With the ResNet-selected subsets, DVA is best among training-dynamics-free methods at all six pruning rates for VGG-11 and at four of six for MobileNetV2, improving over random by 5.88 and 5.34 points, respectively, at a 0.9 drop rate.At that rate, VGG-11 accuracy is 49.28% versus 43.40% for random selection; MobileNetV2 accuracy is 29.73% versus 24.39%.Source: §H.3 · p. 29, Table 25 · p. 30, Table 26 · p. 30

I COMPUTATIONAL COST AND SCALABILITY

On ImageNet at a 0.9 drop rate, DVA’s fixed-output selection is substantially cheaper than IF-Beta in both computational cost and wall-clock time.

I COMPUTATIONAL COST AND SCALABILITY

  • On ImageNet at a 0.9 drop rate, DVA uses 2.81 PFLOPs versus 71.41 PFLOPs for IF-Beta, a 25.4× reduction, and takes 94 s versus 5,370 s, a 57.1× reduction.DVA uses an off-the-shelf pretrained proxy without additional training or adaptation, while IF-Beta requires validation adaptation, influence estimation, and surrogate student optimization.Source: I COMPUTATIONAL COST AND SCALABILITY · p. 30, Table 27 · p. 30
Loading 2609.34489v1…