Source-linked AI summary
Modality Distillation with Multiple Stream Networks for Action Recognition
Nuno Garcia, Pietro Morerio, Vittorio Murino
TL;DR
The paper asks how multimodal training data can improve representations when some modalities are unavailable at test time. It trains an RGB-input hallucination network to distill depth information within generalized distillation, and reports state-of-the-art action-recognition results on NTU RGB+D.
Problem
The central problem is learning robust representations from multimodal data when deployment may have noisy or missing modalities, including RGB-only test settings.
Method
The method trains an RGB-input hallucination network to mimic depth features using generalized distillation, soft and hard labels, feature-map distances, and multiplicative connections.
Results
The authors report state-of-the-art video action-classification results on the NTU RGB+D dataset.
Takeaways & Limitations
The approach demonstrates that depth information can be used during training while the deployed model relies on RGB data only.
Takeaways & Limitations
The demonstrated setting uses RGB and depth, with extending the approach to skeleton joints and infrared identified as future work.
Abstract
from arXiv · showhide
Diverse input data modalities can provide complementary cues for several tasks, usually leading to more robust algorithms and better performance. However, while a (training) dataset could be accurately designed to include a variety of sensory inputs, it is often the case that not all modalities could be available in real life (testing) scenarios, where a model has to be deployed. This raises the challenge of how to learn robust representations leveraging multimodal data in the training stage, while considering limitations at test time, such as noisy or missing modalities. This paper presents a new approach for multimodal video action recognition, developed within the unified frameworks of distillation and privileged information, named generalized distillation. Particularly, we consider the case of learning representations from depth and RGB videos, while relying on RGB data only at test time. We propose a new approach to train an hallucination network that learns to distill depth features through multiplicative connections of spatiotemporal representations, leveraging soft labels and hard labels, as well as distance between feature maps. We report state-of-the-art results on video action classification on the largest multimodal dataset available for this task, the NTU RGB+D. Code available at https://github.com/ncgarcia/modality-distillation .
1 Introduction
The paper addresses how to exploit multimodal training data when test-time deployment may provide only a subset of modalities. It proposes a generalized-distillation framework in which an RGB-driven hallucination network mimics depth information, with staged learning and inter-stream connections.
- Missing modalities at test time motivate learning robust representations from additional modalities available during training.
- The proposed multimodal stream framework learns from multiple modalities during training while permitting deployment and testing on a subset.
- The method teaches an hallucination network receiving RGB input to mimic the depth stream at test time.
- The learning paradigm extends an hallucination-network idea with a more general loss function and inter-stream connections.
- The authors report state-of-the-art results on NTU RGB+D, described as the largest multimodal dataset for video action recognition.
2 Related Work
Related work spans privileged information, network distillation, multimodal action recognition, and connected two-stream architectures. The paper distinguishes its approach by distilling depth into an RGB-input hallucination stream through a generalized loss and cross-stream connections, while testing with RGB only.
- Privileged information and network distillation are presented as instances of the broader generalized-distillation theory.
- The model treats depth as training-only privileged information and distills depth-network supervision into an RGB-driven hallucination network.
- Compared with the cited hallucination model, the proposed loss combines feature-map Euclidean distance with depth-network soft predictions and uses cross-stream connections.
- The architecture builds on two-stream networks and uses multiplicative cross-stream connections rather than only late prediction fusion.
- The related literature includes multimodal NTU RGB+D methods using complementary RGB and depth information for action recognition.
- Unlike prior multimodal methods that use complete modalities at testing, this model exploits RGB and depth during training but uses exclusively RGB at test time.
3 Generalized Distillation with Multiple Stream Networks
The model uses RGB and depth streams during training, then learns an RGB-input hallucination stream to imitate depth representations for RGB-only testing. Generalized distillation combines feature-map matching, hard and soft prediction supervision, cross-stream multiplicative connections, and staged training.
- Model: RGB and depth streams are used during training, while only RGB is available to the model at test time.The architecture is designed for multimodal training with RGB-only deployment.
- Model: Cross-stream connections multiply appearance and depth-related feature maps element-wise, requiring spatial alignment between modalities.The model also adds 1D temporal convolutions in the second residual unit of each ResNet layer.
- Hallucination stream: The hallucination network receives RGB input and is trained to imitate the depth stream at feature and prediction layers.It has the same architecture as the appearance and depth stream models.
- Hallucination stream: Generalized distillation transfers depth-teacher information through hard labels, soft predictions, and a loss combining prediction-based distillation with Euclidean feature-map distance.The temperature parameter smooths teacher probabilities, while λ balances hard and soft labels and α balances the two loss terms.
- Training Paradigm: The training paradigm separates teacher learning from hallucination learning, then fine-tunes the RGB and hallucination streams jointly using RGB-only input.The staged procedure trains the teacher with RGB and depth, learns the hallucination stream with the depth network frozen, and reproduces the test-time setup during final tuning.
4 Experiments
Experiments on NTU RGB+D evaluate the proposed multimodal and hallucination-network designs through benchmark comparisons, ablations, and noisy-depth tests. The results show that staged training, combined losses, multiplicative cross-stream connections, and fine-tuning improve RGB-only deployment performance while remaining robust to missing or corrupted depth.
- Experimental setup: NTU RGB+D contains 56,880 videos across RGB, depth, infrared, and 3D skeleton modalities for evaluating multimodal action recognition.The dataset includes 80 viewpoints, 40 subjects, and 60 actions.
- Ablation study: The proposed loss functions outperform Hoffman et al.’s method, with the total loss improving accuracy by about 6%.Rows #3–#5 compare the original loss with the proposed generalized-distillation and total losses.
- Ablation study: Using both RGB and depth at train and test time reaches about 78–79% cross-subject and 81–82% cross-view accuracy, establishing an upper bound.These results substantially exceed the separate RGB and depth stream performances.
- Ablation study: Fine-tuning increases RGB-only performance by about 1.5%, producing the best accuracy with only RGB available at test time.This corresponds to the final row #15 of Table 2.
- Ablation study: Multiplicative cross-stream connections improve hallucination learning, minimizing the Euclidean loss more effectively after approximately half as many iterations.Rows #13 and #14 start from the same point and use the same loss.
- Inference with noisy depth: The hallucination model outperforms naive RGB-only training and direct use of corrupted depth, while noisy depth accuracy decreases significantly even at low noise variance.Speckle noise is simulated with multiplicative Gaussian noise; the hallucination model’s no-depth accuracy is 77.21%.
5 Conclusions and Future Work
The paper addresses video action recognition with privileged information by teaching an RGB-input hallucination network to mimic depth. Future work would extend the approach to additional training-time modalities.
- The proposed learning paradigm teaches an hallucination network to mimic the depth stream while receiving RGB input.
- The model outperforms many recently evaluated supervised methods and a prior hallucination model on NTU RGB+D.
- An extensive ablation study evaluates how the learning paradigm’s components contribute to model performance.
- Future work will extend the approach to additional modalities available during training, including skeleton joints and infrared sequences.
arXiv:1806.07110v2 [cs.CV] 29 Oct 2018
The supplementary material provides additional details about the model, including an inverted cross-stream experiment in which the depth stream receives RGB information.
- The supplementary document examines how the model behaves when the cross-stream connection is inverted.
- In the inverted setting, the depth stream network receives the RGB signal, making RGB the missing modality.
- The supplementary material reports lower overall performance in this inverted configuration.
1 Inverting the data modalities: RGB distillation
The inverted-modality study shows that the architecture is directional rather than symmetric: RGB is fed a signal from depth, not vice versa. The reported inverted configuration performs worse.
- The architecture is not symmetric when depth and RGB modalities are swapped.The cross-stream connection is engineered from the depth stream to the RGB stream.
- The chosen direction reflects the assumption that depth provides cleaner, more representative foreground data.
- Prior work associated injecting signal into the optical-flow stream with decreased performance, potentially because RGB becomes dominant during training.
- The inverted teacher accuracy is only marginally higher than the final RGB-only model, making it a poor upper bound relative to the original teacher.
- The inverted configuration produces a worse hallucination network and a worse distilled model.
2 Implementation details
Implementation requires spatially aligned RGB and depth frames because the model uses element-wise multiplicative cross-stream connections. The preprocessing aligns, crops, and resizes both modalities before input.
- Pre-processing & alignment: Multiplicative cross-stream connections require RGB and depth frames to be spatially aligned.The operations act element-wise on feature maps.
- Pre-processing & alignment: The preprocessing scales RGB frames, overlays and aligns both skeletons, crops their intersection, and resizes the result to 224x224.
- Sampling and augmentation: Five frames are sampled evenly in time for each video during training and testing.Training additionally applies horizontal flipping with probability P = 0.5.
- Hyperparameters and validation set: Validation selects α = 0.5, λ = 0.5, and T = 10.The supplementary material also specifies validation-set construction for the cross-subject and cross-view protocols.