PaperlayerView exact revision on arXiv

Source-linked AI summary

Label-Guided Knowledge Distillation for 3D-CNNs in Action Recognition

Yanjiang Shi, Peng Zhao, Nan Qi, Guiqin Wang

arXiv:2609.13024v1cs.CVcs.AI

TL;DR

Feature distillation for 3D-CNN video models often overlooks the differing importance of frames. LGKD uses labels to guide temporal weighting and class-prototype alignment, and reports improved student accuracy over compared methods on UCF101 and HMDB51.

  • Problem

    Feature distillation methods often adapt image-based techniques and apply equal weights, overlooking different frame-importance contributions in video.

    Source: §I · p. 1, A. Vanilla Feature Knowledge… · p. 3

  • Method

    LGKD combines label- and teacher-probability-guided temporal feature weighting with class-wise prototype distillation to transfer within-category relations.

    Source: §I · p. 2

  • Results

    LGKD reports higher student accuracy than compared methods on both datasets; on HMDB51, Top-1 accuracy improves by 2.42% and Top-5 accuracy by 1.44%.

    Source: C. Main Results · p. 7

  • Takeaways & Limitations

    The two distillation levels help lightweight 3D-CNN students selectively learn teacher features and within-class relations, compacting category representations and improving generalization.

    Source: §V · p. 9

Abstract

from arXiv · show

As a key model compression technique, knowledge distillation aims to transfer knowledge from a high-capacity teacher model to a lightweight student model for enhancing the latter's performance. In this work, we reviewed the feature knowledge distillation for 3D-CNNs and observed that most feature distillation methods in video analysis are simple adaptations of those used in image analysis, often neglecting the differences of video features in the temporal dimension. To address this issue, we proposed Label-Guided Knowledge Distillation (LGKD) to guide the distillation of student model features using ground truth labels. Our method entails two components: sample-wise distillation and class-wise distillation, enabling the student model to learn feature representation of the teacher model at two levels. Sample-wise distillation utilizes label information and the teacher's probability distribution to guide the learning of features that significantly impact temporal accuracy while mitigating noise. Meanwhile, class-wise feature distillation employs a prototype network to further capture the relational knowledge among samples within the same category, enhancing the student's ability to learn higher-dimensional semantic information and improving model generalization. To demonstrate the effectiveness and superiority of our method, we conducted comprehensive experiments on two benchmark action recognition datasets, UCF101 and HMDB51, achieving competitive results.

I. INTRODUCTION

The introduction identifies a mismatch between image-oriented feature distillation and video’s temporal dimension, then proposes label-guided distillation at sample and class levels. Experiments on UCF101 and HMDB51 are reported to improve student feature representation and outperform compared methods.

I. INTRODUCTION

  • Existing feature distillation methods often adapt 2D-CNN techniques and treat teacher features equally, overlooking differences in frame importance for video inference.The introduction notes that features from frames with greater classification impact matter more to the final outcome.Source: §I · p. 1
  • On UCF101 validation, distilling only on teacher-correctly classified samples improves accuracy by approximately 0.4% for both response-based and feature-based distillation.The passage identifies Table I as the location of these validation results.Source: §I · pp. 1–2, Table I · p. 2
  • LGKD uses teacher probabilities and labels to weight temporally important features, and class prototypes to transfer within-category relations and higher-level semantics.The class-wise approach is intended to mitigate feature divergence from sample-wise distillation.Source: §I · p. 2
  • Experiments on UCF101 and HMDB51 report improved student feature representation and performance surpassing compared methods.The paper also explores generalization across different teacher–student model pairs.Source: §I · p. 2

II. RELATED WORKS

The related-work discussion covers action-recognition architectures, knowledge-distillation approaches, and prototype networks. It identifies the limited treatment of temporal feature importance in video distillation and transitions to vanilla feature matching as the methodological baseline.

A. Action Recognition

  • Partially 3D-CNN architectures reduce resource demands but can suffer a noticeable accuracy decline relative to fully 3D-CNNs.The passage names Top-I3D, Bottom-I3D, and S3D as partial 3D-CNN examples.Source: A. Action Recognition · p. 2

B. Knowledge Distillation

  • Feature-based distillation transfers feature-map representations, while video methods often adapt 2D-CNN techniques and overlook temporal differences in feature importance.The related-work passage says video research has also emphasized cross-modal transfer from optical-flow branches to RGB branches.Source: B. Knowledge Distillation · p. 3

C. Prototype Network

  • Prototype networks have been used across tasks; here they extract within-category relational knowledge to guide student learning of teacher representations.Earlier examples include few-shot classification, transfer learning, zero-shot learning, and knowledge distillation for named-entity recognition.Source: C. Prototype Network · p. 3

III. METHODOLOGY

  • The methodology section first reviews feature knowledge distillation before introducing label-guided distillation.This is the stated structure of the methodology section.Source: §III · p. 3

A. Vanilla Feature Knowledge Distillation

  • Vanilla feature distillation uses L2 distance between teacher and student feature maps, with a projection aligning their dimensions.The student is constrained to mimic the teacher’s feature-map distribution, but equal weighting overlooks varying frame importance.Source: A. Vanilla Feature Knowledge… · p. 3

B. Label-guided Knowledge Distillation

LGKD adds label-guided sample-wise weighting and class-wise prototype alignment to feature distillation. Sample-wise weights derive from the teacher’s ground-truth-category probabilities over time, with direct and masked normalization options.

B. Label-guided Knowledge Distillation

  • LGKD weights teacher–student feature-map losses by importance scores and aligns student sample features with teacher-computed category prototypes.The framework pairs a sample-wise weight-adjustment module with a class-wise prototype network.Source: B. Label-guided Knowledge Distillation · p. 3, Figure 2 · p. 4

1) Sample-wise Knowledge Distillation:

  • For each feature map, sample-wise distillation uses ground-truth labels and the teacher’s probability distribution to assign a weight to its feature-loss term.The resulting weighted loss is used as the student’s final sample-wise distillation loss.Source: 1) Sample-wise Knowledge Distillation · p. 4
  • Direct normalization derives feature-importance scores from the teacher’s ground-truth-class distribution across time, then normalizes the scores.The teacher prediction before Softmax and its ground-truth-class probability are described in the implementation passages.Source: 1) Sample-wise Knowledge Distillation · p. 4
  • Masked normalization suppresses low-importance features by masking negative results and retaining positive scores for the distillation loss.The mask is applied using element-wise multiplication and an indicator function.Source: 1) Sample-wise Knowledge Distillation · p. 4

2) Class-wise Knowledge Distillation:

Class-wise distillation aligns category-level prototypes to convey the teacher’s semantic relationships among same-category samples. Prototypes use correctly inferred teacher features, with a warm-up period before class-wise distillation begins.

2) Class-wise Knowledge Distillation:

  • Class-wise distillation aligns student and teacher features at the category level, helping the student mimic higher-level semantics and compact same-category representations.The method uses a prototype network to represent category features.Source: 2) Class-wise Knowledge Distillation · p. 5
  • Category prototypes are formed from teacher features of correctly inferred samples selected using ground-truth labels.This selection is intended to reduce the influence of teacher errors and noisy individual features, supporting student generalization.Source: 2) Class-wise Knowledge Distillation · p. 5
  • Because all same-category samples cannot be held in memory together during training, prototypes are updated batch by batch using a moving average.The batch update accounts for the number of previously computed samples and the number in the current batch.Source: 2) Class-wise Knowledge Distillation · p. 5
  • Class-wise distillation begins only after a warm-up period in which the teacher updates category prototypes without performing that distillation.The warm-up allows prototypes to stabilize before they guide student learning.Source: 2) Class-wise Knowledge Distillation · p. 5

C. Training and Inference

Training freezes the pretrained and finetuned teacher while updating the student, combining task and distillation losses. At inference, only the student predicts the category.

C. Training and Inference

  • The teacher is frozen after pretraining and finetuning, while training updates only the student’s parameters.The task loss is cross-entropy, and the sample-wise and class-wise distillation losses can be used independently or together with it.Source: C. Training and Inference · p. 5
  • The overall training objective adds weighted sample-wise and class-wise distillation losses to cross-entropy.The hyperparameters α and β balance the two distillation losses.Source: C. Training and Inference · p. 5
  • At inference, category prediction uses only the student model.Source: C. Training and Inference · p. 5

IV. EXPERIMENTS

The experiments evaluate the proposed method through weight-adjustment analyses, main results, module ablations, generalization tests, and hyperparameter sensitivity tests.

IV. EXPERIMENTS

  • The experiments begin by comparing two implementations of the weight-adjustment module.Source: §IV · p. 5
  • The section then presents the proposed method’s main experimental results.Source: §IV · p. 5
  • Further experiments examine module ablations, generalization, and hyperparameter sensitivity.Source: §IV · p. 5

A. Experiment Setup

The study uses UCF101 and HMDB51 with Top-1 and Top-5 accuracy, then compares sample-wise weight-adjustment strategies on UCF101. The label-guided approach reaches 70.45% accuracy and is reported to outperform the compared alternatives.

A. Experiment Setup

  • Experiments use UCF101 and HMDB51, with Top-1 and Top-5 accuracy as evaluation metrics.The selected evaluation scheme uses the first official split for each dataset.Source: A. Experiment Setup · p. 5

B. Sample-wise Distillation weight adjustment module

  • Approach 1 uses the teacher’s ground-truth-class probability distribution over time to assign feature-map importance scores.The label-free probability aggregation method instead combines outputs across action categories.Source: B. Sample-wise Distillation weight… · p. 6
  • The masking approach removes negative teacher outputs, but the comparison reports that discarding negative-score segments too aggressively can also lose useful background knowledge.Source: B. Sample-wise Distillation weight… · p. 6
  • 70.45% final accuracy is reported for Approach 1, which also exceeds vanilla feature distillation at 69.92% and the baseline student at 63.36%.The comparison is on UCF101; Table II covers weight-adjustment implementations.Source: B. Sample-wise Distillation weight… · p. 6, Table II · p. 6

C. Main Results

The method improves student accuracy over compared distillation methods on UCF101 and HMDB51. Its sample-wise and class-wise components guide feature learning, while the student retains lower computational and parameter costs than the teacher.

C. Main Results

  • LGKD improves Top-1 by 10.15% and Top-5 by 8.22% on UCF101, and Top-1 by 2.42% and Top-5 by 1.44% on HMDB51.The comparison uses I3D as teacher and Top-I3D as student under the same training configuration.Source: Table III · p. 7, C. Main Results · p. 7
  • LGKD outperforms compared logits-based and feature-based distillation methods, including MGD and SimKD, using the same teacher-student pair.The authors report superiority over recent state-of-the-art logits-based methods and recent feature-based methods.Source: C. Main Results · p. 7
  • Sample-wise feature weighting uses teacher inference and labels to prioritize features contributing to correct inference, while prototype-based class-wise distillation aligns category representations.The authors associate these components with improved high-level semantic feature extraction.Source: C. Main Results · p. 7
  • The student reduces computational cost by 59.1% and parameters by 17.3% compared with the teacher.The authors state that these reductions, combined with LGKD, make deployment on resource-constrained edge devices feasible.Source: C. Main Results · p. 7

D. Ablation Study

The ablation study examines the effects of the proposed submodules and the method’s generalization across backbone networks.

D. Ablation Study

  • The study systematically analyzes each submodule’s effect and the method’s generalization across backbone networks.The section introduces this analysis as an investigation of the method’s validity.Source: D. Ablation Study · p. 7
  • Table IV reports the impact of each submodule on model accuracy on UCF101.Source: Table IV · p. 7

1) Contribution of Each Submodule:

On UCF101, combining sample-wise and class-wise distillation gives the strongest reported submodule result. The section also reports a gain over vanilla feature distillation and improvements across three student architectures, including a pure 2D student.

1) Contribution of Each Submodule:

  • On UCF101, sample-wise distillation raises student accuracy by 9.28% over baseline, while class-wise feature distillation raises it by 7.11%; combining both raises Top-1 by 10.15% and Top-5 by 8.22%.Source: 1) Contribution of Each Submodule · pp. 7–8
  • Combining sample-wise and class-wise distillation raises Top-1 by 10.15% and Top-5 by 8.22% over baseline, reaching 73.51% and 90.30%.Table IV summarizes the four ablation settings on UCF101.Source: 1) Contribution of Each Submodule · pp. 7–8

2) Effectiveness of Weight Adjustment Module:

E. Generalization of Our Method

F. Hyperparameter Sensitivity Study

On UCF101, α = 4 and β = 2 give the strongest reported sensitivity-study improvement; the conclusion describes LGKD as combining selective sample-wise transfer with class-wise relational learning. The authors also identify finer-grained label guidance as future work.

F. Hyperparameter Sensitivity Study

  • α and β weight the sample-wise and class-wise loss functions, respectively; α = 4 was retained while testing β.The authors report α = 4 and β = 2 as the most significant accuracy-improvement setting and use those values throughout experiments.Source: F. Hyperparameter Sensitivity Study · p. 8
  • 73.51% Top-1 accuracy and 90.30% Top-5 accuracy are reported for α = 4 and β = 2 on UCF101, the setting retained for experiments.The passage says accuracy improves over the baseline across tested α and β values; the supplied table captions do not provide other cell values.Source: F. Hyperparameter Sensitivity Study · p. 8, Table VII · p. 8, Table VIII · p. 8

V. CONCLUSION

  • LGKD combines selective learning of teacher features affecting classification with class-wise relational learning of high-level semantics to compact representations and improve generalization.The conclusion describes these as sample-wise and class-wise distillation, respectively.Source: §V · pp. 8–9
  • The authors identify finer-grained label information to guide feature distillation as a promising direction for future work.Source: §V · p. 9

Related papers

Loading 2609.13024v1…