PaperlayerView exact revision on arXiv

Source-linked AI summary

Too much of a good thing -- when knowledge distillation promotes overfitting, and how to avoid it

Irene Trigueros-Lorca, Leonardo Concepción, Christian Wagner, Isaac Triguero, Daniel Molina

arXiv:2608.23752v1cs.CVcs.AI

TL;DR

Large CNNs are costly, and conventional distillation mainly supervises only the final output, leaving intermediate transfer insufficiently studied. The paper uses a homogeneous student mirroring the teacher’s blocks and evaluates block-wise supervision across data and dataset conditions. It finds that intermediate supervision is especially beneficial in low-data, fine-grained settings, with later-block-focused schemes particularly effective.

  • Problem

    Intermediate-layer knowledge distillation is less explored than final-output distillation, despite the possibility that additional internal supervision could help when data are scarce and classes are fine-grained.

  • Method

    The paper trains a simpler homogeneous student that mirrors the teacher’s block structure, matching corresponding block feature maps and analyzing supervision with explainability methods and fine-tuning strategies.

  • Results

    Across classic and fine-grained, data-scarce datasets, fine-tuning the teacher alone generally performs best, while later-block-focused supervision can match or exceed broader schemes and performs especially well when only the student is fine-tuned.

  • Takeaways & Limitations

    Intermediate-layer distillation is an effective choice for data-constrained settings, and selecting where knowledge is transferred matters alongside how much supervision is applied.

  • Takeaways & Limitations

    The analysis assumes performance differences arise from which blocks receive KD while network complexity remains unchanged, and Grad-CAM provides coarse localization maps.

Abstract

from arXiv · show

The growing size of Convolutional Neural Networks has led to increasingly large and costly models. Knowledge Distillation (KD) addresses this by transferring knowledge from a large network (teacher) to a small one (student), also reducing the training data required. KD is traditionally applied only at the network's final output. However, its behaviour when applied at intermediate network layers has received little attention. This raises the question of whether intermediate block-wise KD, which provides supervision throughout the network, could offer an advantage under specific conditions, such as few instances per class, which is common in fine-grained datasets. This work proposes a student design based on simple, homogeneous blocks mirroring those of the teacher, distilling knowledge between corresponding blocks. Across eleven datasets, we show that on classic datasets, distilling only the last block is sufficient -- and often best--, whereas fine-grained, data-scarce settings benefit substantially from intermediate supervision, with even a single additional distillation point narrowing the gap considerably. We further study how this supervision should be guided, exploring configurations of varying granularity and informed by an explainability analysis based on attention maps, Centered Kernel Alignment, and Grad-CAM, alongside the impact of teacher and student fine-tuning strategies. This work shows that intermediate block-wise distillation, guided appropriately, is key to building compact data-efficient models without sacrificing accuracy.

1 Introduction

CNNs deliver strong task performance but are costly to train and deploy, motivating compact alternatives such as knowledge distillation. This work investigates whether supervising intermediate blocks improves distillation, especially under data scarcity and fine-grained classification.

  • Motivation: CNNs can be adapted to new domains by retaining their backbone and changing only the final layers, but domain-specific fine-tuning requires substantial computational power.
  • Knowledge Distillation: Knowledge distillation transfers knowledge from a large teacher to a smaller student, supporting competitive results and more efficient learning with scarce data.
  • Intermediate Distillation: Feature-based distillation provides supervision through internal representations, potentially capturing how the teacher progressively builds its decision rather than only its final output.
  • Research Gap: Intermediate distillation remains underexplored, with no consensus on which blocks to select, whether intermediate points help, or how many points to use.
  • Approach: The proposed student mirrors the teacher’s number of blocks with simpler homogeneous structures and matches feature maps between corresponding blocks.
  • Research Questions: The study evaluates fine-tuning strategies, intermediate-block placement and number, data scarcity, fine-grained complexity, and explainability-guided transfer through multiple research questions.

2 Related Work

Related work distinguishes logit-, feature-, and similarity-based distillation, while emphasizing that intermediate transfer and its interpretation remain open design questions. This paper positions block-wise distillation as a systematic feature-based approach for data-scarce and fine-grained settings.

  • Knowledge Distillation: Logit-based methods match softened teacher output distributions, whereas feature-based methods transfer internal representations and similarity-based methods preserve relationships among samples.
  • Feature-Based Distillation: Feature-based methods differ in where supervision is applied, ranging from a single intermediate layer to multiple depths or grouped representations.
  • Block-Wise Distillation: The paper’s block-wise method distils across multiple blocks of a fixed homogeneous student architecture mirroring the teacher, enabling systematic study of distillation-point number and location.
  • Explainability: Explainability methods such as saliency maps and CAMs are used both as distillation signals and as post-hoc tools for examining transferred knowledge beyond accuracy.
  • Explainability: The study uses attention maps, CKA, and Grad-CAM diagnostically to examine knowledge transfer across intermediate blocks rather than to guide training itself.
  • Data-Efficient and Fine-Grained Settings: Data scarcity is widespread in real-world applications, while fine-grained dataset performance has received comparatively little attention in feature distillation.
  • Data-Efficient and Fine-Grained Settings: Controlled augmentation and data-reduction experiments directly evaluate block-wise distillation on fine-grained, data-scarce datasets.

3 Block-wise Knowledge Distillation Using Homogeneous Blocks

The proposed framework uses a homogeneous student with corresponding teacher blocks and flexibly applies knowledge distillation at selected block outputs. It also evaluates how intermediate representations align through attention maps, CKA, and Grad-CAM.

  • Architecture: The student mirrors the teacher’s number of blocks while using simpler homogeneous blocks for corresponding block-wise distillation.For EfficientNet-B0, the teacher is divided into six blocks; each student block contains two Inverted Residual modules.
  • Training Process: Student blocks are trained sequentially to match corresponding teacher feature maps using mean squared error distillation.Each block receives the preceding teacher feature map as input during its training and is trained for 20 epochs.
  • Training Process: After block-wise KD, classification layers are trained either with the student blocks frozen or with the entire student unfrozen for joint fine-tuning.Training uses early stopping and retains the checkpoint with the best validation performance.
  • KD Configurations: The framework supports all-block, selected-subset, and final-block-only KD configurations, changing which blocks receive direct supervision.When consecutive blocks are grouped, only the group’s final block receives direct KD while the others learn through backpropagation.
  • Explainability Analysis: Attention maps, CKA, and Grad-CAM are applied at every block output to assess recovered teacher information, including blocks without direct distillation.Attention maps compare spatial activation patterns, whereas CKA compares representation structure across potentially different dimensionalities.
  • Explainability Analysis: Grad-CAM localization becomes less reliable in earlier blocks because their smaller receptive fields capture less semantic information.This limitation matters when interpreting Grad-CAM across all student blocks.

4 Experimental Framework

The experiments evaluate the framework across eleven image-classification datasets, contrasting classic and data-scarce fine-grained settings. They use controlled training, augmentation, reduction, and student-architecture choices to study KD configurations.

  • Training Setup: EfficientNet-B0 is used as the ImageNet-pre-trained teacher and fine-tuned per dataset with SGD for up to 300 epochs.Training uses learning rate 0.001, momentum 0.9, and early stopping with patience 15.
  • Datasets: Experiments use eleven datasets: seven classic datasets and four data-scarce fine-grained datasets.The dataset summary covers examples, classes, and examples per class; class imbalance makes some ratios intervals or uninformative.
  • Training Setup: The training data is split into 80% actual training and 20% validation subsets for early stopping.Batch sizes vary by dataset group, and the final training stage uses a maximum of 300 epochs with patience 15.
  • Student Design: A manual comparison selects two Inverted Residual modules per student block as a simpler alternative with results similar to a searched student.The choice also reduces the time required to define and train the student.
  • Experimental Studies: The studies compare fine-tuning strategies, data availability, intermediate KD granularity, and explainability-guided configurations.Results tables highlight the best accuracy value for each dataset.
  • Data Conditions: Some experiments augment each training image into four variations, while data reduction retains 5%, 10%, or 80% of the original training data.Augmentation aims to preserve class-distinctive features while adding meaningful variability; reduction uses stratified sampling.

5 Analysis of results

The results analysis compares fine-tuning strategies, data-availability effects, intermediate KD granularity, and explainability-guided supervision. These studies examine how training choices and block-selection patterns relate to performance.

  • Analysis Goals: The analysis compares teacher-only, student-only, and joint fine-tuning strategies for their impact on performance.It also considers whether teacher fine-tuning is practical when access or computational resources are limited.
  • Analysis Goals: The experiments assess how augmenting fine-grained datasets and reducing classic datasets affects intermediate knowledge distillation.This directly varies data availability across the two dataset settings.
  • Analysis Goals: The study tests whether evenly distributed intermediate KD configurations provide a favorable trade-off between all-block and end-block schemes.The comparison spans the two extreme configurations and intermediate block subsets.
  • Analysis Goals: Explainability techniques are used to analyze how information is recovered across blocks and to guide intermediate KD configurations.The analysis examines block representations beyond the blocks receiving direct distillation.

5.1 Influence of Fine-tuning in teacher and student models

Fine-tuning the teacher generally provides the strongest accuracy, while fine-tuning only the student offers a lower-cost alternative. End-block distillation performs best on classic datasets but degrades on fine-grained ones, where all-block distillation is more robust.

  • Fine-tuning strategies: FT matches or outperforms FTS across both extreme configurations, while student fine-tuning adds limited benefit relative to its computational cost.FT also clearly outperforms FS, although FS remains preferable when resources are limited.
  • End-block versus all-block distillation: EBD (FT) outperforms the baseline on 6 of 7 classic datasets but shows clear degradation on fine-grained datasets.It uses fewer parameters at the cost of 1 additional hour of training.
  • End-block versus all-block distillation: ABD (FT) remains competitive with the baseline across all 11 datasets and shows no degradation on fine-grained datasets.The baseline is only marginally superior on 7 of 11 datasets.
  • End-block versus all-block distillation: On classic datasets, EBD achieves the best performance, with ABD typically only 1-3% behind in absolute accuracy.This comparison concerns the two extreme block-wise configurations.
  • End-block versus all-block distillation: On fine-grained datasets, ABD substantially outperforms EBD and narrows the gap with the baseline, while EBD remains far below it.Training EBD requires approximately one hour less than ABD on average.

5.2 Effects of Data Quantity

Data quantity changes which distillation strategy is favored. Intermediate supervision becomes more valuable as instances per class decrease, while augmentation can reduce the gap but substantially increases processing time.

  • Data augmentation: Data augmentation narrows the performance gap between EBD and ABD on fine-grained datasets.This suggests that ABD’s advantage is related to the limited number of instances per class.
  • Data augmentation: Applying augmentation also improves ABD, although less than EBD, consistent with ABD already using intermediate blocks to compensate for limited data.Augmentation increases processing time by a factor of four in the best case.
  • Data reduction: As available data decreases, ABD models, particularly under FS, increasingly outperform EBD models.As data increases, EBD models, particularly under FT, become increasingly competitive.
  • Domain gap: Teacher performance also depends on the target domain’s similarity to its pre-training domain.A small domain gap preserves the teacher’s advantage, whereas a large gap can allow a model trained directly on the target domain to overtake it.
  • Data reduction: For 10% or less of the data, EBD degrades significantly relative to the teacher, whereas ABD remains highly competitive and improves the teacher’s results on several datasets.The results support the value of intermediate knowledge when only a small fraction of data is available.

5.3 Granularity Study

The number and placement of distillation points matter differently across dataset regimes. One additional point can yield a large gain on fine-grained data, while three selected points can match all-block distillation without supervising every block.

  • Statistical comparison: The study compares EBD, ABD, Blocks36, and Blocks246 across classic and fine-grained datasets under FT and FS.Critical difference diagrams use lower ranks for better average accuracy and connect configurations without statistically significant differences.
  • Intermediate configurations: On fine-grained datasets, adding one point in Blocks36 changes accuracy far more than adding one further point in Blocks246.The effect of additional distillation points is therefore non-linear.
  • Intermediate configurations: On classic datasets, accuracy generally decreases as distillation points increase from EBD to ABD, although CIFAR100 is an exception.Both intermediate configurations outperform EBD and match the baseline on CIFAR100.
  • Intermediate configurations: Blocks246 matches or outperforms ABD on most fine-grained datasets despite distilling at only three of six blocks.This indicates that some blocks may provide redundant or noisy supervision.
  • Statistical comparison: On classic datasets, the Friedman test finds an overall ordering, but pairwise comparisons find no significant differences between configurations.The statistical pattern differs from the fine-grained setting.
  • Statistical comparison: On fine-grained datasets, Blocks246 has the best average rank, followed by ABD, Blocks36, and EBD under both FT and FS.Under FT all pairwise differences are significant; under FS, Blocks246 and ABD are not significantly different from each other.

5.4 Explainability

The explainability analysis compares teacher–student alignment across blocks using attention maps, CKA, and Grad-CAM. It finds that later-block alignment tracks classification performance more closely, especially when data are scarce.

  • Attention maps: Attention-map similarity varies across blocks, with fine-tuning strategy affecting its overall level and stability.EBD and ABD were compared under FT and FS across datasets, excluding MNIST.
  • Attention maps: ABD_FT maintains higher and more stable attention-map similarity than ABD_FS, particularly on fine-grained datasets.ABD_FS can drop sharply at the final block, plausibly because student representations drift when the student is unfrozen without a fine-tuned teacher anchor.
  • CKA: On fine-grained datasets, ABD_FT preserves broad CKA alignment, whereas EBD_FT remains low across nearly the entire block-pair grid.The low EBD_FT CKA includes its directly supervised final block, showing that activation-level alignment need not imply structural alignment.
  • Grad-CAM: On CIFAR10, EBD_FT reaches Grad-CAM similarity comparable to or higher than ABD_FT at later blocks despite lacking earlier distillation.With sufficient data, the student can recover class-discriminative behaviour close to the teacher largely independently.
  • Grad-CAM: On OxfordPet, ABD_FT outperforms EBD_FT across almost all blocks, while ABD_FS still exceeds EBD in similarity.This pattern mirrors the accuracy gap and supports the value of intermediate knowledge under data scarcity.
  • Cross-method interpretation: Across CKA and Grad-CAM, accuracy and similarity tend to move together at the last three blocks, while the first three vary little.The analysis therefore identifies later blocks as carrying more classification-relevant information than early blocks.

5.5 Explainability-Guided Student

The explainability findings motivate grouping early blocks while distilling later blocks individually. This targeted design preserves strong performance with fewer distillation points, particularly when the teacher is not fine-tuned.

  • Design: Blocks3456 groups the first three blocks and distils Blocks 4, 5, and 6 individually, while Blocks1236 reverses this grouping.The comparison tests whether performance depends on selected blocks rather than merely the number of distillation points.
  • Classic datasets: On classic datasets, Blocks1236 tracks EBD, whereas Blocks3456 matches ABD, consistent with early-block information diluting regardless of supervision density.Individually supervising early blocks adds little beyond grouping them.
  • Fine-tuning strategy: Blocks3456_FS matches or slightly exceeds other FS models on classic datasets and clearly outperforms them on fine-grained datasets.Distilling a subset of later blocks can provide a favourable time–accuracy trade-off when no fine-tuned teacher is available.

6 Conclusions

The paper studies how block-wise distillation and fine-tuning interact with data availability and dataset granularity. It concludes that later-focused intermediate supervision is most useful in low-data settings, while end-block distillation remains a strong default for data-abundant datasets.

  • Fine-tuning: Fine-tuning the teacher alone generally performs best, student-only fine-tuning is a competitive lower-cost alternative, and jointly fine-tuning both adds no benefit despite higher cost.The teacher’s pre-training domain can also limit transfer quality under data scarcity.
  • Data availability: On data-abundant classic datasets, no configuration consistently surpasses EBD, which remains a sound lower-cost default.Under data scarcity, intermediate supervision narrows the gap, though gains are nonlinear and do not always exceed the teacher.
  • Explainability-guided design: Explainability results indicate that early blocks encode general representations, whereas later blocks carry more class-discriminative information.Later-focused students such as Blocks3456 can match or exceed ABD with fewer distillation points.
  • Practical implication: Intermediate-layer distillation is presented as a practical choice for compact, data-efficient models in low-data regimes.The paper proposes selecting the distillation scheme according to the application scenario rather than only the amount of transferred knowledge.
Loading 2608.23752v1…