PaperlayerView exact revision on arXiv

Source-linked AI summary

Depth Guided Adaptive Meta-Fusion Network for Few-shot Video Recognition

Yuqian Fu, Li Zhang, Junke Wang, Yanwei Fu, Yu-Gang Jiang

arXiv:2010.09982v1cs.CV

TL;DR

Few-shot action recognition seeks to learn new actions from very few labeled videos, unlike standard video recognition's reliance on large labeled datasets. AMeFu-Net combines depth-guided RGB-depth fusion with asynchronous clip sampling and establishes state-of-the-art results on Kinetics, UCF101, and HMDB51.

  • Problem

    Few-shot action recognition aims to learn new actions from very few labeled videos, while standard video recognition relies on large amounts of labeled data.

    Source: §1 · p. 2

  • Method

    AMeFu-Net uses depth-guided adaptive fusion of RGB and depth features and augments training with randomly sampled, non-strictly corresponding RGB-depth clips.

    Source: §1 · p. 2

  • Results

    AMeFu-Net establishes state-of-the-art performance across Kinetics, UCF101, and HMDB51, including 74.1% on Kinetics 1-shot.

    Source: §4.3 · p. 6, §5 · p. 8

  • Takeaways & Limitations

    The reported results support depth-guided AMeFu-Net as an effective approach within the evaluated few-shot action-recognition benchmarks.

    Source: §5 · p. 8

Abstract

from arXiv · show

Humans can easily recognize actions with only a few examples given, while the existing video recognition models still heavily rely on the large-scale labeled data inputs. This observation has motivated an increasing interest in few-shot video action recognition, which aims at learning new actions with only very few labeled samples. In this paper, we propose a depth guided Adaptive Meta-Fusion Network for few-shot video recognition which is termed as AMeFu-Net. Concretely, we tackle the few-shot recognition problem from three aspects: firstly, we alleviate this extremely data-scarce problem by introducing depth information as a carrier of the scene, which will bring extra visual information to our model; secondly, we fuse the representation of original RGB clips with multiple non-strictly corresponding depth clips sampled by our temporal asynchronization augmentation mechanism, which synthesizes new instances at feature-level; thirdly, a novel Depth Guided Adaptive Instance Normalization (DGAdaIN) fusion module is proposed to fuse the two-stream modalities efficiently. Additionally, to better mimic the few-shot recognition process, our model is trained in the meta-learning way. Extensive experiments on several action recognition benchmarks demonstrate the effectiveness of our model.

1 INTRODUCTION

Few-shot video recognition is difficult because models need to generalize to new actions from very few labeled videos, while video data labeling at scale is impractical. AMeFu-Net addresses this with depth-guided fusion and temporal asynchronization augmentation, and the paper reports state-of-the-art results on three datasets.

1 INTRODUCTION

  • Few-shot video recognition remains challenging because deep models can overfit scarce examples, and temporal video representations are harder to learn than image representations.Earlier few-shot work focused more on images than videos.Source: §1 · p. 2
  • Depth is proposed as a scene carrier whose geometric relations between people and environments can help recognize actions despite limited labeled instances.The paper gives mountain biking in a mountain scene as an example of useful person-scene geometry.Source: §1 · p. 2
  • Temporal asynchronization augmentation randomly pairs unpaired RGB and depth clips, motivated by actions remaining recognizable across scene shifts.For example, the paper describes recognizing the same action when a scene shifts from mountain to roadside.Source: §1 · p. 2
  • AMeFu-Net predicts depth frames and uses DGAdaIN to learn affine parameters from depth features and deform RGB features for fusion.The model is trained with meta-learning to acquire cross-task knowledge and quickly learn task-specific knowledge at inference.Source: §1 · p. 2
  • AMeFu-Net establishes new state-of-the-art results on Kinetics, UCF101, and HMDB51.The paper reports this as an overall contribution.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses few-shot recognition more extensively for images than videos, while video methods have explored memory, temporal alignment, web data, and attention. This paper distinguishes its approach through depth as contextual information, adaptive RGB-depth fusion, and temporal shifting to sample asynchronized clips.

Few-shot learning

  • Few-shot learning remains less explored for video than images, with prior video work using memory, temporal alignment, web data, or attention.The paper positions its work as exploring extra modality information to address limited samples.Source: §2 · pp. 2–3

Context for video understanding

  • Depth is presented as a middle-level context representation that can encode object, scene, and potential motion detail while being robust to noise.The paper contrasts it with scene, object, pose, and skeleton cues, which it describes as expensive or too abstract.Source: §2 · p. 3
  • Unlike common feature averaging, concatenation, recurrent networks, or fully connected fusion, DGAdaIN learns scale and shift parameters from depth to deform RGB features.This design draws on AdaIN's alignment of content-feature mean and variance to a style feature.Source: §2 · p. 3

Temporal shift mechanism

  • Here, temporal shifting samples asynchronized RGB and depth clips to synthesize feature-space instances, rather than shifting features to fuse temporal information.The paper characterizes this as sampling unpaired data for more abundant video representations.Source: §2 · p. 3

3 METHODOLOGY

AMeFu-Net trains on episodic few-shot tasks over disjoint base and novel action classes, combining RGB and depth features for video classification. Its pipeline uses depth-guided fusion, temporal asynchronization augmentation, and a prototype-based classifier.

3 METHODOLOGY

  • Episodes sample N classes with K support videos each; meta-training uses base classes, while novel-class meta-testing uses one strictly corresponding RGB-depth pair per video.The classifier predicts query actions from support videos in each episode.Source: §3 · p. 3

3.1 Backbone with depth

  • The two-stream backbone combines generic RGB features with depth features as supplementary scene context, using ResNet-50 extractors and depth maps predicted by Monodepth2.The depth predictor is pre-trained on KITTI and used off the shelf.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2 Depth guided adaptive fusion module

  • DGAdaIN normalizes RGB features and uses depth features to learn adaptive scale and shift parameters that deform the RGB representation for fusion.The scale and shift are generated by jointly optimized fully connected layers.Source: §3.2 · pp. 4–5

3.3 Temporal asynchronization augmentation

  • Temporal asynchronization augmentation samples matched and non-strictly matched depth clips for RGB clips, creating 1 + numaug training pairs per video.The sampled pairs are iteratively used during meta-training.Source: §3.3 · p. 5, Figure 3 · p. 5

3.4 Few-shot classifier

  • The ProtoNet-derived classifier averages support-video features into class prototypes, then predicts query labels with softmax over distance similarities.The authors report cosine distance performs better than Euclidean distance under their setting.Source: §3.4 · p. 5

4 EXPERIMENTS

Experiments evaluate few-shot recognition on Kinetics, UCF101, and HMDB51, reporting strong results for AMeFu-Net and testing its fusion and temporal-shifting components.

4.1 Datasets

  • Kinetics contains 400 action classes and 306,245 videos; the experiments use 100 classes split into 64 training, 12 validation, and 24 testing classes.UCF101 and HMDB51 experiments use disjoint training, validation, and testing action classes.Source: §4.1 · pp. 5–6

4.2 Implementation details

  • Training first fine-tunes RGB and depth sub-models separately, then meta-trains DGAdaIN for six epochs with frozen sub-model parameters; testing averages 10,000 episodes.The sub-models are fine-tuned for 6 RGB epochs and 60 depth epochs; DGAdaIN training uses 5-way 1-shot episodes.Source: §4.2 · p. 6

4.3 Results on Kinetics

  • 74.1% on 1-shot, 84.3% on 3-shot, and 86.8% on 5-shot Kinetics establish new state-of-the-art; temporal asynchronization improves performance consistently across shots.The reported advantage is greatest when labeled data are extremely scarce.Source: §4.3 · p. 6, Table 1 · p. 7
  • CAMs compare RGB-only, depth-only, and fused AMeFu-Net attention maps; the fused model retains important regions and reduces irrelevant-background noise.The figure shows four Kinetics examples, with RGB and depth frames preceding the three CAM columns.Source: §4.3 · p. 7, Figure 4 · p. 8

4.4 Results on UCF101 and HMDB51

  • 85.1% on UCF101 1-shot and 95.5% on UCF101 5-shot beat ARN by 18.8% and 12.4%, respectively; HMDB51 reaches 60.2%, 71.5%, and 75.5%.The HMDB51 values are for 1-shot, 3-shot, and 5-shot, respectively.Source: §4.4 · p. 7, Table 2 · p. 8

4.5 Ablation study

  • DGAdaIN outperforms direct feature concatenation and the explored alternative fusion strategies; the design keeps RGB context while using depth as style guidance.The ablations evaluate 5-way few-shot classification on Kinetics.Source: §4.5 · p. 7, Table 3 · p. 8
  • Two temporal shifts usually perform best; more shifts can add misleading unmatched RGB-depth pairs and unwanted noise, while performance eventually converges as shifts increase.The final model uses num_aug = 2.Source: §4.5 · p. 8, Table 3 · p. 8

5 CONCLUSION

The paper uses depth information to address data scarcity in few-shot video action recognition, combining temporal asynchronization augmentation with DGAdaIN fusion. It reports state-of-the-art results on Kinetics, UCF101, and HMDB51.

5 CONCLUSION

  • Depth information is explored as a way to alleviate the extremely data-scarce problem in few-shot video action recognition.The paper frames depth as additional information for this setting.Source: §5 · p. 8
  • Temporal asynchronization augmentation is proposed to augment source video representations.Source: §5 · p. 8
  • DGAdaIN adaptively fuses features from the RGB and depth streams.Source: §5 · p. 8
  • The proposed method establishes new state-of-the-art results on Kinetics, UCF101, and HMDB51.Source: §5 · p. 8

Related papers

Loading 2010.09982v1…