Graph Distillation for Action Detection with Privileged Modalities

1 Introduction

• Multimodal action learning is constrained by costly video annotation and by privacy or runtime limits that can restrict test-time access to modalities.In healthcare examples, RGB may violate privacy and optical flow may be too slow for real-time use, leaving depth or thermal video available.Source: §1 · p. 2
• The setting uses a large action-classification dataset as the source to assist target-domain action detection, assuming the domains are similar.A privileged modality is available in the source but not the target domain, and during training but not testing.Source: §1 · p. 2
• Modality types and quantities differ across domains, while the appropriate distillation directions and weights can vary by action or example.For instance, optical flow may suit some actions better than skeleton features, so fixed distillation choices are inadequate.Source: §1 · p. 2
• 9.0% on PKU-MMD at 0.5 tIoU and 6.6% on NTU RGB+D: the method improves the state of the art for detection and classification, respectively.The paper identifies PKU-MMD as the detection benchmark and NTU RGB+D as the classification benchmark.Source: §1 · p. 3
• Graph distillation dynamically learns example-specific transfer across multiple modalities and can be attached to existing models and trained end to end.It is intended to use complementary information from privileged modalities to improve performance when only one modality is available at test time.Source: §1 · pp. 2–3

Multimodal Action Classification and Detection

• Prior multimodal action work includes RGB-D and skeleton tasks, while this paper leverages privileged modalities from a source dataset with abundant examples for small-target training and single-modality testing.The authors state that this setting benefits action detection.Source: §2 · p. 3

Video Understanding Under Limited Data

• Transfer from action classification to action detection is a common approach to limited-data video understanding.The related-work discussion also notes transfer learning and noisy web data as approaches to limited data.Source: §2 · p. 4
• Transfer learning commonly requires matching source and target modalities, leaving richer source modalities difficult to use when the target has only depth, for example.The paper positions its approach as a practical way to use that richer source information.Source: §2 · p. 4

Learning Using Privileged Information; Knowledge Distillation

• Earlier privileged-information and distillation approaches use additional training-time information, but the paper differentiates graph distillation through multiple modalities and dynamically learned directions and weights.The cited prior work includes knowledge distillation, generalized distillation, and cross-modality transfer and.Source: §2 · p. 4

3.1 Knowledge Distillation with Privileged Modalities

• The target-domain model uses source-trained visual encoders, transfers encoders for available target modalities, and fine-tunes them with the target model.The target training modalities are a subset of source modalities.Source: §3.1 · p. 5, Figure 2 · p. 6
• Source-domain training combines classification loss with imitation loss, whose distillation directions and weights are learned dynamically across heterogeneous modalities.This addresses the difficulty of pre-specifying distillation directions and weights for multiple privileged modalities.Source: §3.1 · pp. 5–6

3.2 Graph Distillation

• For multiple modalities, a directed distillation graph represents modality-to-modality message strengths and is learned jointly with the network.Messages contribute to each modality’s imitation loss, and the graph layer is trained end-to-end.Source: §3.2 · p. 7
• The method combines an example-specific graph with a prior favoring accurate modalities, using a constant bias rather than message passing on two separate graphs.The prior is set according to each modality’s cross-validation performance.Source: §3.2 · p. 8

4.1 Network Architecture

• Action detection encodes clips within a video window, processes their features with a sequence encoder, and predicts class distributions including a background class.Classification instead predicts a distribution from an encoded short clip.Source: §4.1 · pp. 8–9
• Image modalities use a modified ResNet-18, while vector modalities use a three-layer GRU to encode each clip.The ResNet-18 omits its final fully connected layer; the vector encoder averages the highest GRU layer’s outputs across time.Source: §4.1 · p. 9

4.2 Training and Testing

• Classification training samples one clip and averages predictions from multiple test clips; detection samples training and test windows, then post-processes clip distributions into detections.Detection training uses inverse-frequency class weights, and its post-processing uses an activity-threshold hyperparameter.Source: §4.2 · p. 9

5.1 Datasets and Setups

• NTU RGB+D supplies source-domain pretraining and PKU-MMD is the target; models are generally tested using one modality despite training with multiple modalities.The datasets provide RGB, depth, 3D joints, and infrared; experiments also use optical flow and three skeleton features.Source: §5.1 · p. 10

5.2 Comparison with State-of-the-Art

• 4.5% over the previous best with depth and 6.6% with RGB on NTU RGB+D; on PKU-MMD, graph distillation outperforms existing methods across all modalities.The experiments also report state-of-the-art performance on both benchmarks.Source: §5 · p. 10, Table 1 · p. 11, Table 2 · p. 11

5.3 Ablation Studies on Limited Training Data

• With 5% of training data, graph distillation outperforms representative privileged-information baselines, and the learned graph performs best among the tested graph choices on both mini datasets.The comparison includes empty, uniform, prior, and learned graphs.Source: Table 3 · p. 13, §5.3 · p. 13
• On mini-PKU-MMD, source-domain privileged-information pretraining, target-domain graph distillation, and adding privileged modalities each improve depth-test performance.The compared privileged modalities are RGB and skeleton feature JJD; examples show flow weighted more for falling and skeleton more for brushing teeth.Source: Table 4 · p. 13, §5.3 · p. 14, Figure 4 · p. 14

5.4 Graph Distillation on UCF-101

• On UCF-101, graph edge distillation improves an RGB model trained from scratch and outperforms fine-tuning with ImageNet-pretrained weights.This setting uses two modalities, RGB and optical flow, with knowledge distilled from the optical-flow stream to the RGB stream.Source: §5.4 · p. 14, Table 5 · p. 15

• The model achieves state-of-the-art action classification on NTU RGB+D and action detection on PKU-MMD.Source: §6 · p. 15
• Graph distillation dynamically leverages privileged modalities to assist model training.Source: §6 · p. 15
• The paper addresses action classification and detection in multimodal video with limited training data and partially observed modalities.Source: §6 · p. 15
• Future work could combine graph distillation with advanced transfer learning and domain adaptation techniques.Source: §6 · p. 15