Explaining Knowledge Distillation by Quantifying the Knowledge

1. Introduction

• A visual concept is an image region whose information is relatively retained and mainly used by the DNN; concepts are distinguished as task-relevant or task-irrelevant.For classification, foreground concepts are treated as task-relevant and background concepts as task-irrelevant.Source: §1 · p. 1
• Knowledge distillation is examined through three hypotheses: it learns more visual concepts, learns them more simultaneously, and yields more stable optimization directions than raw-data learning.The paper calls the sequential modeling and later discarding of non-discriminative concepts “detours.”Source: §1 · p. 2
• The paper proposes three metric types to measure visual-concept quantity, learning speed, and optimization-direction stability in distilled and raw-data networks.The raw-data baseline is constructed with the same architecture as the student network.Source: §1 · p. 2
• The contribution includes a method for quantifying dark-matter concepts in intermediate DNN layers and using visual-concept metrics to interpret distillation’s superior performance.The paper describes its analysis as testing three hypotheses about knowledge distillation.Source: §1 · p. 2

2. Related Work

• Semantic interpretation includes feature visualization, gradient-based and inversion-based methods, pixel-wise attribution, and human-annotated or user-defined concepts.The cited examples include gradient-based methods [47] [37] [46] [27], human-annotated concepts [1], and TCAV [19].Source: §2 · p. 2
• The paper quantifies visual concepts using input conditional entropy, contrasting this with heuristic importance or saliency measures and reliance on massive human annotations.The authors state that metric coherency supports fair comparisons across DNN layers and across networks learned in different epochs.Source: §2 · p. 3
• Mathematical representation-capacity work uses measures including mutual information [36], stiffness [10], CLEVER [40], Fourier analysis [44], and CCA [21].Other cited work examines instance-wise feature selection via mutual information [3] and knowledge consistency [23].Source: §2 · p. 3
• The paper aims to bridge mathematical and semantic explanations by measuring concept counts, foreground and background concepts, concept-learning order, and optimization-direction stability.The authors describe mathematical explanations for knowledge distillation as rare and present their work as quantifying and comparing concepts in intermediate layers.Source: §2 · p. 3

3. Algorithm

• The student is distilled to approximate a teacher’s intermediate-layer features, then compared with a same-structure baseline trained from raw data.The described setup is limited to object classification.Source: §3 · p. 3

3.1. Preliminaries: Quantification of Information Discarding

• Conditional entropy measures how much input information is discarded given an intermediate-layer feature, and Gaussian perturbations allow image entropy to be decomposed into pixel-level entropies.The pixel-wise information-discarding measure is based on Hi.Source: §3.1 · pp. 3–4

3.2. Quantification of visual concepts

• Dark-matter concepts are quantified from low-entropy image regions, with foreground and background counts used to assess feature discriminativeness and comparisons across layers and networks.The approach does not rely on human-labeled concepts; Table 1 reports coherent entropy-based results across layers and networks.Source: §3.2 · p. 4

3.3. Learning simultaneously or sequentially

• Two metrics assess whether foreground concepts emerge quickly for individual images and simultaneously across images, using mean and variation in weight distance at the richest-concept epoch.The epoch ˆm is a rough estimate of the division between learning phases.Source: §3.3 · p. 5

3.4. Learning with Less Detours

• The detours metric compares final foreground concepts with the union of concepts learned across epochs; a larger ρ indicates fewer discarded concepts and less detour.The paper contrasts teacher-guided learning of target concepts with raw-data learning that models and later discards non-discriminative concepts.Source: §3.4 · p. 5, Figure 4 · p. 6

4.1. Implementation Details

• Experiments used seven DNN architectures across ILSVRC-2013 DET, CUB200-2011, and Pascal VOC 2012, with teacher, student, and baseline architectures matched.For ILSVRC-2013 DET and Pascal VOC 2012, the authors selected FC1 or FC2 as target layers; all three FC layers were used for CUB200-2011.Source: §4.1 · p. 6

4.2. Quantification of Visual Concepts in the Teacher Network, the Student Network and the Baseline Network

• Teachers exceeded students in foreground concept count and λ, while students exceeded baselines on those measures; one teacher had fewer foreground concepts than its student because its average background entropy was higher.The comparison used VGG-16 on ILSVRC-2013 DET and CUB200-2011.Source: Table 2 · p. 7, §4.2 · p. 7

4.3. Verification of Hypothesis 1

• Across most tested datasets and layers, distilled students encoded more foreground and fewer background concepts than baselines, yielding larger λ; pre-trained teachers sometimes supplied more categories than needed.Very few students encoded more background concepts, which the authors attribute to ImageNet teachers’ 1,000 categories being more than necessary.Source: §4.3 · p. 7

4.4. Verification of Hypothesis 2

• Students had smaller Dmean and Dstd than baselines in most cases, supporting simultaneous concept learning, but shallow AlexNet and VGG-11 layers were exceptions.The authors attribute these failures to shallow architectures, which can learn more concepts from raw data and avoid overfitting.Source: §4.4 · p. 7

4.5. Verification of Hypothesis 3

• In most cases, student ρ exceeded baseline ρ, indicating more stable optimization directions; AlexNet and VGG-11 were exceptions and took more detours.The authors associate the exceptions with these networks’ shallow architectures.Source: §4.5 · p. 7

• Knowledge distillation yields more task-relevant and fewer task-irrelevant concepts, faster learning, and fewer optimization detours than learning from raw data.Source: §5 · p. 7
• The study focuses on classification, and its entropy-based analysis cannot precisely separate learning from discarding because both may occur within an epoch.Source: §5 · pp. 7–8
• The target epoch is only a rough estimate of the division between learning and discarding phases.Source: §5 · p. 8