PaperlayerView exact revision on arXiv

Source-linked AI summary

Hybrid Relation Guided Set Matching for Few-shot Action Recognition

Xiang Wang, Shiwei Zhang, Zhiwu Qing, Mingqian Tang, Zhengrong Zuo, Changxin Gao, Rong Jin, Nong Sang

arXiv:2204.13423v1cs.CV

TL;DR

Few-shot action recognition aims to classify unseen actions from very few labeled videos, but existing methods can miss episode-specific relations and struggle with misalignment. HyRSM combines task-specific relational embeddings with bidirectional set matching, and reports state-of-the-art performance on six standard benchmarks.

  • Problem

    Existing few-shot action recognition methods may overlook discriminative cross-video clues within an episode and fail when action substeps are misaligned.

    Source: §1 · pp. 1–2

  • Method

    HyRSM learns task-specific features from within- and cross-video relations, then matches query and support videos as frame sets with a bidirectional Mean Hausdorff Metric.

    Source: §1 · p. 2

  • Results

    HyRSM achieves state-of-the-art performance on six standard benchmarks: Kinetics, SSv2-Full, SSv2-Small, HMDB51, UCF101, and Epic-kitchens.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Its non-parametric metric avoids numerous non-parallel calculations, improving computing efficiency and transfer ability compared with complex alignment classifiers.

    Source: §3.2 · p. 5

    The hybrid relation module adds parameters, increasing GPU memory and computational consumption, although inference is faster than OTAM and TRX.

    Source: §4.6 · p. 8

Abstract

from arXiv · show

Current few-shot action recognition methods reach impressive performance by learning discriminative features for each video via episodic training and designing various temporal alignment strategies. Nevertheless, they are limited in that (a) learning individual features without considering the entire task may lose the most relevant information in the current episode, and (b) these alignment strategies may fail in misaligned instances. To overcome the two limitations, we propose a novel Hybrid Relation guided Set Matching (HyRSM) approach that incorporates two key components: hybrid relation module and set matching metric. The purpose of the hybrid relation module is to learn task-specific embeddings by fully exploiting associated relations within and cross videos in an episode. Built upon the task-specific features, we reformulate distance measure between query and support videos as a set matching problem and further design a bidirectional Mean Hausdorff Metric to improve the resilience to misaligned instances. By this means, the proposed HyRSM can be highly informative and flexible to predict query categories under the few-shot settings. We evaluate HyRSM on six challenging benchmarks, and the experimental results show its superiority over the state-of-the-art methods by a convincing margin. Project page: https://hyrsm-cvpr2022.github.io/.

1. Introduction

Few-shot action recognition seeks to classify unseen action classes from very few labeled videos, but existing methods can miss episode-specific cross-video clues and struggle with misaligned actions. HyRSM combines task-specific relational embeddings with flexible set matching, achieving superior performance over state-of-the-art methods on six challenging benchmarks.

1. Introduction

  • Few-shot action recognition aims to classify unseen action classes using extremely few annotated examples.Collecting large numbers of manually labeled examples is labor-intensive and time-consuming, limiting further applications of action recognition.Source: §1 · p. 1
  • Existing methods can overlook discriminative cross-video clues and fail to match actions with differing subaction orders or offsets.Independent video representations may miss episode-specific information, while strict temporal alignment can fail when subactions occur in different orders.Source: §1 · p. 2
  • HyRSM’s hybrid relation module combines within-video temporal dependencies and cross-video semantic information to produce task-specific embeddings.The inter-relation function emphasizes features relevant to query prediction.Source: §1 · p. 2
  • Its bidirectional Mean Hausdorff Metric treats videos as frame sets, relaxing strict ordering to find better query-support correspondences.This set-matching approach is designed to improve resilience to misaligned instances.Source: §1 · p. 2
  • HyRSM achieves superior performance over state-of-the-art methods on six challenging datasets.Its hybrid relation module learns task-specific embeddings, while bidirectional Mean Hausdorff matching treats each video as a frame set to handle misalignment.Source: §1 · p. 2

2. Related Work

Related work spans few-shot image classification, set matching, and few-shot action recognition. HyRSM draws on task-specific metric learning and set matching while addressing the temporal complexity of video.

Few-shot Image Classification

  • Few-shot image classification methods include data augmentation, optimization-based learning, and metric-based comparison in a learned feature space.HyRSM is related to metric-based methods that learn task-specific features, but targets video’s diverse spatiotemporal dependencies.Source: §2 · p. 2

Set Matching

  • Set matching measures similarity between sets; Hausdorff distance is one approach used in image matching and other vision tasks.The paper introduces set matching to few-shot action recognition.Source: §2 · p. 2

Few-shot Action Recognition

  • Few-shot action recognition methods include video-pair metrics, attention, saliency, depth, temporal alignment, permutation invariance, and tuple matching.Unlike approaches that embed videos independently, HyRSM learns intra- and inter-relational patterns to improve transfer to unseen classes.Source: §2 · p. 3

3. Method

Few-shot action recognition classifies query videos into new action classes using a small labeled support set. HyRSM builds task-specific video features from within- and cross-video relations, then matches support and query videos as frame sets.

3.1. Problem formulation

  • Few-shot action recognition classifies query videos into N action classes using K labeled support videos per class in an episodic task.The support set contains N × K videos; the objective is generalization to new classes with few labeled examples.Source: §3.1 · p. 3

3.2. HyRSM

  • HyRSM embeds support and query videos, enhances their features with the hybrid relation module, and feeds them to the set-matching metric for scores.Training uses negative class distances as logits with cross-entropy and a real-action-class regularization loss.Source: §3.2 · pp. 3, 5
  • The hybrid relation module replaces isolated, task-agnostic features that may overfit irrelevant representations and transfer poorly to unseen classes.It is described as a plug-and-play unit with multiple possible configurations.Source: §3.2 · p. 4
  • HyRSM learns task-specific features by combining long-range within-video temporal relations with semantic relations across videos in the episode.The hybrid relation module aggregates information from other video representations; its intra- and inter-relation functions target within-video structure and cross-video enhancement, respectively.Source: §3.2 · p. 4
  • Bi-MHM treats each video as a set of frames and averages nearest-frame distances in both directions, supporting matching of aligned or misaligned instances.The proposed metric is symmetric and non-parametric; it selects the closest support class at inference.Source: §3.2 · pp. 4–5

4. Experiments

HyRSM is evaluated on six few-shot action datasets and outperforms state-of-the-art methods, including a 1-shot SSv2-Full improvement from 49.2% to 54.3%. Ablations examine its components, matching strategy, scalability, visual behavior, and computational costs.

4.1. Datasets and experimental setups

  • Evaluations cover six datasets, using 10,000 randomly sampled meta-testing episodes and mean accuracy for few-shot recognition.The datasets are Kinetics, SSv2-Full, SSv2-Small, UCF101, HMDB51, and Epic-kitchens; experiments use an ImageNet-pretrained ResNet-50 and sample eight frames per video.Source: §4.1 · p. 5

4.2. Comparison with state-of-the-art

  • 54.3% versus 49.2%: HyRSM improves 1-shot SSv2-Full performance and outperforms state-of-the-art methods on most evaluated settings.It is slightly behind TRX in 5-shot UCF101 and SSv2-Small; the authors attribute this to TRX's ensemble design for multiple shots.Source: §4.2 · p. 6

4.3. Ablation study

  • Compared with baseline, intra-relation modeling gains 6% and 9.7% on 1-shot and 5-shot; inter-relation modeling gains 8.5% and 9.9%, and set matching improves 9.4% and 10.7%, respectively.Stacking modules can further improve performance, indicating complementarity between components.Source: §4.3 · p. 7
  • Adding the hybrid relation module to OTAM yields gains of 8.9% and 11.6% in 1-shot and 5-shot settings.This experiment tests whether the module generalizes to another method.Source: §4.3 · p. 7
  • As N increases, performance decreases, but HyRSM remains ahead of OTAM and TRX in N-way 1-shot evaluation.The study also reports that deeper backbones improve performance; increasing frame count helps until gains saturate beyond eight frames, and performance with more attention heads saturates beyond a point.Source: §4.3 · p. 7

4.4. Comparison with other matching approaches

  • Bi-MHM outperforms temporal alignment alternatives and is more stable to noise than Hausdorff distance; its bidirectional form performs better than a single-direction metric.The comparison replaces OTAM with Bi-MHM while keeping other settings unchanged.Source: §4.4 · p. 8

4.5. Visualization results

  • Visualizations show that the hybrid relation module improves feature discrimination, while Bi-MHM handles alignment and misalignment flexibly.The matching visualization depicts query videos against support videos.Source: §4.5 · p. 8, Figure 7 · p. 8, Figure 8 · p. 8

4.6. Limitations

  • HyRSM uses more GPU memory and computation because of its hybrid relation module, but inference is faster than OTAM and TRX.Reducing complexity without performance loss is left for future work.Source: §4.6 · p. 8

5. Conclusion

HyRSM combines task-specific video representations from within- and cross-video relations with set matching designed to handle misalignment. It achieves state-of-the-art performance on six standard benchmarks.

5. Conclusion

  • HyRSM models semantic relevance within videos and across videos in an episode to generate task-specific features.Source: §5 · p. 8
  • Its set matching metric uses task-specific features to match videos accurately and be resilient to misalignment.Source: §5 · p. 8
  • HyRSM achieves state-of-the-art performance on Kinetics, SSv2-Full, SSv2-Small, HMDB51, UCF101, and Epic-kitchens.Source: §5 · p. 8

A. Splits of Epic-kitchens

The Epic-kitchens experiment divides the dataset by action verbs, with no overlap between meta-training and meta-testing verbs. The accompanying materials also identify other dataset analyses.

A. Splits of Epic-kitchens

B. Other relation modeling forms

The discussion contrasts task-specific relation modeling with earlier approaches and reports that including query videos improves performance over support-only relation modeling. A visualization encodes query-support matches before and after HyRSM’s hybrid relation module.

B. Other relation modeling forms

  • Earlier few-shot image-classification methods report promising results [35] [69], but many use complex, fixed operations and only support-set information for task-specific features.Source: B. Other relation modeling forms · p. 12
  • Figure 10 places query videos in rows and support videos in columns, with colored boxes distinguishing correct from incorrect matches before and after the hybrid relation module.Source: Figure 10 · p. 12
  • Removing the query video from inter-relation modeling reduces SSv2-Full 1-shot and 5-shot performance by 2.2% and 1.8%, respectively; Kinetics shows similar conclusions.The comparison is between the full relation pool and the Support-only variant.Source: B. Other relation modeling forms · p. 12

C. Class improvement

On SSv2-Full, category-level gains vary across actions, and HyRSM’s gains are more evident than those of set matching alone. The accompanying Grad-CAM figures compare OTAM and HyRSM on SSv2-Full and Kinetics.

C. Class improvement

  • HyRSM improves more evidently across SSv2-Full action categories than the set matching metric alone.Source: C. Class improvement · p. 12
  • Set matching alone improves some SSv2-Full categories by more than 20%, while gains are limited for others.The passage gives “tipping something over” and “showing something next to something” as examples of larger gains, and two other actions as limited-gain examples.Source: C. Class improvement · p. 12
  • For “pulling something from left to right” and “pushing something from right to left,” HyRSM increases performance by more than 25%.Source: C. Class improvement · p. 12
  • The Grad-CAM figures show original RGB images alongside OTAM and HyRSM activation maps on SSv2-Full and Kinetics.Source: Figure 11 · p. 13, Figure 12 · p. 14

D. Visualization analysis

The visualization analysis reports improved feature similarities after hybrid relation refinement and contrasts HyRSM’s target-focused activations with OTAM’s inclusion of non-target objects.

D. Visualization analysis

  • HyRSM’s class activation maps focus on different target objects, whereas OTAM’s often include non-target objects.The passage attributes this contrast to HyRSM’s adaptive relation modeling and OTAM’s lack of task-specific embedding adaptation.Source: D. Visualization analysis · p. 15
  • Figure 10’s similarity maps indicate that features improve significantly after refinement by the hybrid relation module.Source: D. Visualization analysis · p. 15
  • Figures 11 and 12 compare HyRSM’s class activation maps with those of OTAM [4].Source: D. Visualization analysis · p. 15

E. Relation modeling operations

The section reviews multi-head self-attention, Transformer, Bi-LSTM, and Bi-GRU as relation-modeling operations. It describes their attention, temporal modeling, and gating mechanisms.

E. Relation modeling operations

  • Multi-head self-attention computes scaled dot-product attention over query, key, and value, then concatenates multiple heads and projects their representations.In the described self-attention use, query, key, and value are the same feature; a residual connection produces the aggregated representation.Source: E. Relation modeling operations · p. 15
  • Transformer combines multi-head self-attention with a feed-forward network whose two MLP layers use GELU nonlinearity [25].The section describes Transformer as having contextual modeling ability.Source: E. Relation modeling operations · p. 15
  • Bi-LSTM applies LSTMs in both sequence directions to learn long-term temporal dependencies, with forget, input, and output gates regulating memory.Source: E. Relation modeling operations · p. 15
  • Bi-GRU uses update and reset gates: the update gate controls retaining prior information or adding new information, while the reset gate controls forgetting past information.The displayed formulation combines the previous hidden state with a candidate hidden state to produce the new hidden state.Source: E. Relation modeling operations · p. 15

Related papers

Loading 2204.13423v1…