PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-modal knowledge distillation for action recognition

Fida Mohammad Thoker, Juergen Gall

arXiv:1910.04641v1cs.CV

TL;DR

Action recognition often requires annotated data, but the annotated training modality may differ from the target modality. This paper transfers knowledge from an RGB teacher to skeleton students using unannotated paired sequences and mutual learning. The approach nearly reaches fully supervised accuracy and improves over KL-loss distillation.

  • Problem

    Action-recognition models require large annotated datasets, while applications may use a modality different from the modality of available annotated data.

    Source: §1 · p. 1

  • Method

    The approach trains skeleton students from an RGB teacher using unannotated paired sequences, cross-entropy supervision, and peer learning among students.

    Source: §1 · p. 1, §3.2 · p. 2

  • Results

    The approach nearly achieves fully supervised accuracy; with two students, accuracy rises from 73.6% with KL-loss to 77.83% with cross-entropy.

    Source: §5 · p. 4, §4 · p. 4

  • Takeaways & Limitations

    Paired sequences from both modalities suffice for knowledge transfer without additional annotations, and the reported accuracy is very close to fully supervised learning.

    Source: §5 · p. 4

    The approach assumes that paired videos from both modalities are available, though the authors say acquiring them is straightforward.

    Source: §1 · p. 1

Abstract

from arXiv · show

In this work, we address the problem how a network for action recognition that has been trained on a modality like RGB videos can be adapted to recognize actions for another modality like sequences of 3D human poses. To this end, we extract the knowledge of the trained teacher network for the source modality and transfer it to a small ensemble of student networks for the target modality. For the cross-modal knowledge distillation, we do not require any annotated data. Instead we use pairs of sequences of both modalities as supervision, which are straightforward to acquire. In contrast to previous works for knowledge distillation that use a KL-loss, we show that the cross-entropy loss together with mutual learning of a small ensemble of student networks performs better. In fact, the proposed approach for cross-modal knowledge distillation nearly achieves the accuracy of a student network trained with full supervision.

1. INTRODUCTION

The paper addresses transferring action-recognition knowledge when the annotated training modality differs from the target modality, avoiding the need to annotate new data. It proposes cross-entropy and mutual learning for RGB-to-skeleton transfer, which nearly reaches fully supervised accuracy.

1. INTRODUCTION

  • Action-recognition models often require large annotated datasets, making transfer desirable when the application’s modality differs from the annotated data.Source: §1 · p. 1
  • The transfer assumes paired, unannotated sequences from both modalities, which the authors describe as straightforward to acquire.Source: §1 · p. 1
  • The proposed transfer uses cross-entropy and mutual learning among multiple students for RGB-video to 3D-skeleton recognition.Source: §1 · p. 1
  • The approach nearly achieves the accuracy of a fully supervised student while outperforming KL-loss distillation on NTU RGB+D.Source: §1 · p. 1

2. RELATED WORKS

Prior work covers skeleton-based action recognition, knowledge distillation, and multimodal training. This paper instead focuses on transferring knowledge when the annotated training modality differs from the test modality.

2. RELATED WORKS

  • Skeleton action recognition has used recurrent models, convolutional models, and spatio-temporal graph convolution to learn sequence features.Source: §2 · p. 1
  • Knowledge distillation was developed to compress ensembles into smaller networks, with softened predictions conveying information termed dark knowledge.Source: §2 · p. 1
  • The paper’s overview depicts teacher supervision for skeleton students from unlabeled RGB–skeleton pairs, with optional peer learning among multiple students.Source: Figure 1 · p. 2
  • Unlike multimodal training with full supervision on additional modalities, this work addresses a mismatch between the annotated training modality and the test modality.Source: §2 · p. 2

3. CROSS-MODAL ACTION RECOGNITION

The approach transfers action-recognition knowledge from a trained RGB teacher to skeleton students using unannotated paired sequences. It combines teacher supervision with cross-entropy and peer learning among students.

3. CROSS-MODAL ACTION RECOGNITION

  • Training uses paired RGB videos and 3D pose sequences that are unannotated and were not used to train the RGB teacher.Source: §3 · p. 2

3.1. Teacher-Student Network

  • The student is optimized to match the teacher’s class probabilities for the corresponding source-modality training pair.Source: §3.1 · p. 2

3.1.1. Loss Function

  • Cross-entropy replaces KL divergence for cross-modal transfer because selecting an optimal temperature is difficult and depends strongly on the student network.Source: §3.1.1 · p. 2

3.2. Mutual Learning

  • An ensemble of students learns both from the teacher and from peer probability estimates, using softened KL loss between students of the same modality.Source: §3.2 · p. 2
  • The mutual-learning formulation extends to K students, with each student’s loss incorporating its relation to the other students.Source: §3.2 · p. 3

4. EXPERIMENTS

Experiments on NTU RGB+D evaluate cross-modal transfer from RGB videos to skeleton data using an unlabeled student-training split. The tests compare distillation losses, student ensembles, and supervised skeleton-recognition methods.

4. EXPERIMENTS

  • The NTU RGB+D cross-subject split uses 40,320 training and 16,560 test samples; teacher accuracy on Student-Train is 86%, implying about 14% incorrect transfer labels.Source: §4 · p. 3
  • 78.5% is the perfect-teacher transfer upper bound; with 5% wrong labels accuracy falls to 73.2%, while 14% teacher errors imply 71.51%.The noisy-label experiment trains ST-GCN with randomly incorrect labels as ground truth.Source: §4 · p. 3
  • With KL τ=10, three students give the best accuracy; averaging beats max, and mutual learning improves on a teacher-student setup by +3%.Half the improvement is attributed to ensembling and half to mutual learning.Source: §4 · pp. 3–4, Table 3 · p. 3
  • 74.91% vs 71.17% for one student and 77.83% vs 73.6% for two: cross-entropy raises accuracy over KL and nearly matches full-supervision accuracy.The proposed approach also improves accuracy by +6.66% over.Source: §4 · p. 4, Table 4 · p. 4
  • For HCN, KL performs best at τ=2 and larger temperatures decrease accuracy, making temperature selection architecture-dependent.Source: §4 · p. 4, Table 5 · p. 4
  • The approach outperforms many fully supervised skeleton methods, though Table 6 comparisons are not directly comparable because training data differ.Source: §4 · p. 4, Table 6 · p. 4

5. CONCLUSION

The approach transfers knowledge for cross-modal action recognition without additional annotations, using paired sequences instead. On a large-scale multimodal dataset, it achieves accuracy very close to fully supervised learning with both student networks.

5. CONCLUSION

  • With both student networks, cross-modal transfer achieves action-recognition accuracy very close to fully supervised learning.Source: §5 · p. 4
  • The approach transfers knowledge between modalities without requiring additional annotations.Pairs of sequences from both modalities are sufficient for knowledge transfer.Source: §5 · p. 4
  • The approach was evaluated on a large-scale multimodal dataset using two different student networks.Source: §5 · p. 4

Related papers

Loading 1910.04641v1…