PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatio-temporal Relation Modeling for Few-shot Action Recognition

Anirudh Thatipelli, Sanath Narayan, Salman Khan, Rao Muhammad Anwer, Fahad Shahbaz Khan, Bernard Ghanem

arXiv:2112.05132v2cs.CV

TL;DR

Few-shot action recognition must match an unlabeled query video to limited support examples despite variation in action speed, timing, and spatial context. STRM enriches local and global features and models query-support temporal relations; it reports state-of-the-art results on four benchmarks, including a 3.5% absolute gain over TRX on SSv2.

  • Problem

    Few-shot action recognition classifies an unlabeled query video using limited labeled examples, while temporal variation and spatial context complicate matching query and support actions.

    Source: §1 · p. 1

  • Method

    STRM combines local patch-level and global frame-level context enrichment with temporal relationship modeling and an intermediate query-class similarity classifier.

    Source: §1 · p. 2, §3.2.2 · p. 5

  • Results

    STRM outperforms existing methods on all four benchmarks and reaches 68.1% classification accuracy on SSv2 with ResNet-50, a 3.5% absolute gain over TRX.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Ablations support the benefits of STRM's contributions, which enable higher-order temporal relation learning at lower cardinalities.

    Source: §6 · p. 8

    TRX struggles with spatial-context variation and requires separate CrossTransformer branches and manual cardinality search.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

We propose a novel few-shot action recognition framework, STRM, which enhances class-specific feature discriminability while simultaneously learning higher-order temporal representations. The focus of our approach is a novel spatio-temporal enrichment module that aggregates spatial and temporal contexts with dedicated local patch-level and global frame-level feature enrichment sub-modules. Local patch-level enrichment captures the appearance-based characteristics of actions. On the other hand, global frame-level enrichment explicitly encodes the broad temporal context, thereby capturing the relevant object features over time. The resulting spatio-temporally enriched representations are then utilized to learn the relational matching between query and support action sub-sequences. We further introduce a query-class similarity classifier on the patch-level enriched features to enhance class-specific feature discriminability by reinforcing the feature learning at different stages in the proposed framework. Experiments are performed on four few-shot action recognition benchmarks: Kinetics, SSv2, HMDB51 and UCF101. Our extensive ablation study reveals the benefits of the proposed contributions. Furthermore, our approach sets a new state-of-the-art on all four benchmarks. On the challenging SSv2 benchmark, our approach achieves an absolute gain of $3.5\%$ in classification accuracy, as compared to the best existing method in the literature. Our code and models are available at https://github.com/Anirudh257/strm.

1. Introduction

Few-shot action recognition must classify a query video using limited labeled support examples, while temporal, spatial, and appearance variation complicate matching. STRM enriches spatial and temporal context for relational matching and reports state-of-the-art results on four benchmarks.

1. Introduction

  • Existing approaches often use frame-level representations without explicitly modeling video sub-sequences, while TRX can struggle with object and background variation.TRX models higher-order temporal relations with tuples but uses fixed representations and separate branches for different cardinalities.Source: §1 · p. 1
  • Temporal matching is difficult because actions occur at different speeds and temporal offsets, and fine-grained recognition requires spatial and temporal context.The representations should capture relevant information from multiple sub-actions to improve query-support matching.Source: §1 · p. 1
  • The framework is motivated by combining local patch and global frame features to focus on relevant objects and their motion.Feature representations at different stages are also intended to reinforce class-separable features.Source: §1 · pp. 1–2
  • STRM combines local patch-level and global frame-level context enrichment with temporal relationship modeling and an intermediate query-class similarity classifier.Patch enrichment aggregates spatial context; frame enrichment aggregates temporal context, while the classifier reinforces discriminability at an intermediate stage.Source: §1 · p. 2
  • 68.1% classification accuracy on SSv2 with ResNet-50, an absolute gain of 3.5% over TRX; STRM also outperforms existing methods on all four benchmarks.The four benchmarks are Kinetics, SSv2, HMDB51, and UCF101.Source: §1 · p. 2

2. Preliminaries

Few-shot action recognition classifies a query video among unseen action classes represented by a limited labeled support set. The section describes episodic training and the TRX baseline, which matches query and support sub-sequences using attention-based prototypes and tuple representations.

2. Preliminaries

  • Few-shot tasks classify an unlabeled query video among unseen support-set action classes, with K labeled videos per class and uniformly sampled video frames.Training episodes randomly sample C-way K-shot tasks from the training set.Source: §2 · pp. 2–3

2.1. Baseline FS Action Recognition Framework

  • TRX matches query and support sub-sequences by aggregating support value embeddings into query-specific class prototypes using cross-attention.The averaged distances between query subsequences and their corresponding prototypes determine the query-to-class distance.Source: §2.1 · p. 3
  • TRX represents sub-sequences as tuples with different cardinalities to capture higher-order temporal relationships, but this requires separate branches and manual cardinality search.The authors identify spatial-context variation as another challenge for TRX's temporal relationship modeling.Source: §2.1 · p. 3

3. Proposed STRM Approach

STRM enriches local patch and global frame representations before matching query and support video subsequences. An intermediate query-class similarity classifier further reinforces class-discriminative learning.

3.1. Overall Architecture

  • STRM enriches local patch features spatially and global frame features temporally before modeling query–support temporal relationships.Its temporal relationship module uses a single cardinality, Ω = {2}, rather than multiple hand-crafted cardinalities.Source: §3 · p. 3, §3.1 · p. 4, §3.2.2 · p. 5
  • The architecture passes spatially enriched frame features through global temporal enrichment, then matches the resulting representations of query and support subsequences.An additional query-class similarity classifier operates on intermediate representations H.Source: §3.1 · p. 4

3.2. Spatio-temporal Enrichment

  • The spatio-temporal enrichment module combines local spatial context and global temporal context to improve class-specific feature discriminability before relational matching.The approach targets both patch features within individual frames and frame features across a video.Source: §3.2 · p. 4

3.2.1 Enriching Local Patch Features

  • Patch-level enrichment uses self-attention to aggregate context across patches in each frame, producing spatially enriched features.The query and key vectors score patch pairs, and the resulting normalized scores reweight value embeddings.Source: §3.2.1 · p. 4

3.2.2 Enriching Global Frame Features

  • Frame-level enrichment mixes frame tokens across the video and refines them, encoding higher-order temporal relationships with a global receptive field.The enriched representations feed the temporal relationship module, which uses a single cardinality.Source: §3.2.2 · p. 5

3.3. Query-class Similarity

  • The query-class similarity classifier predicts classes from intermediate representations, reinforcing class-specific feature learning at multiple pipeline stages.Its similarity scores are converted to class probabilities and trained with cross-entropy.Source: §3.3 · p. 6

4. Experiments

STRM is evaluated on four few-shot action recognition benchmarks and compared with prior methods under a standard 5-way 5-shot setting. It outperforms existing methods across the benchmarks, while ablations examine its feature-enrichment modules, temporal cardinalities, tuple retention, and support-set size.

4. Experiments

  • The standard evaluation is 5-way 5-shot, reporting average accuracy over 10,000 random test tasks across four benchmarks.The experiments use dataset-specific class splits; video features come from eight uniformly sampled frames using an ImageNet-pretrained ResNet-50.Source: §4 · p. 6

4.1. State-of-the-art Comparison

  • 3.5% absolute gain over TRX on SSv2; STRM sets the state of the art across four benchmarks, including 86.5% accuracy on Kinetics.Source: Table 1 · p. 7, §4.1 · p. 7

4.2. Ablation Study

  • 68.1% final STRM accuracy on SSv2 is a 6.0% absolute gain over baseline; joint PLE and FLE reach 66.8%.The baseline is TRM with cardinality Ω={2}; adding PLE alone yields 63.7%.Source: §4.2 · p. 7, Figure 6 · p. 7
  • STRM performs best on Kinetics and SSv2 with Ω={2}; on SSv2, TRX's Ω={2,3} reaches 64.6%, while STRM requires only ∼4% additional FLOPs.The comparison concerns cardinalities used to model temporal relationships.Source: §4.2 · pp. 7–8, Table 2 · p. 8
  • With only 20% of tuples retained, STRM at Ω={2} reaches 65.4%, exceeding TRX's 64.6% with all tuples at Ω={2,3}.The authors report a marginal performance drop as retained tuples decrease and note that STRM does not require dedicated TRM branches for different cardinalities.Source: Figure 6 · p. 7, §4.2 · p. 8
  • STRM consistently improves on TRM and TRX across the reported K-shot settings, including one-shot and 10-shot evaluation on SSv2.The authors report that STRM effectively leverages larger support sets in the 10-shot setting.Source: §4.2 · p. 8

5. Relation to Prior Art

Prior few-shot action recognition work includes key-frame memory, variable-length alignment, monotonic temporal ordering, and fixed higher-order temporal representations. STRM differs from TRX by enriching local patch- and global frame-level features, and it is compared with both TRM and TRX across support-set sizes on SSv2.

5. Relation to Prior Art

  • Earlier few-shot action methods use key-frame memory, align variable-length videos, or enforce monotonic temporal ordering; TRX models temporal relations with fixed higher-order representations.Source: §5 · p. 8
  • On SSv2, STRM outperforms both TRM and TRX across the reported support-sample settings, including one-shot evaluation.The comparison also includes 10-shot evaluation, where STRM effectively leverages the larger support set.Source: Figure 7 · p. 8
  • Unlike TRX, STRM enriches local patch features with self-attention and global frame features with an MLP-mixer, and supports higher-order temporal modeling at lower cardinalities.STRM also introduces a query-class classifier for intermediate-layer representations.Source: §5 · p. 8

6. Discussion

The discussion presents STRM as combining spatio-temporal enrichment, temporal relationship modeling, and a query-class similarity classifier. It reports state-of-the-art results on all benchmarks and identifies cross-domain generalization as future work.

6. Discussion

  • STRM combines spatio-temporal enrichment and temporal relationship modeling with a query-class similarity classifier.Source: §6 · p. 8
  • The authors report that STRM learns higher-order temporal relations even with lower-cardinality representations.Source: §6 · p. 8
  • A stated future direction is broadening few-shot action recognition to generalize across varying domains.Source: §6 · p. 8

Appendix

The appendix presents additional quantitative and qualitative results for STRM, the proposed few-shot action recognition framework.

Appendix

  • The appendix contains additional quantitative results for STRM.Source: Appendix · p. 9
  • The appendix also contains additional qualitative results for STRM.Source: Appendix · p. 9
  • Both kinds of additional results concern the proposed few-shot action recognition framework.Source: Appendix · p. 9

A. Additional Quantitative Results

On SSv2, STRM’s best enrichment pairing improves over the baseline, while hierarchical local-then-global enrichment uses less computation than joint spatio-temporal attention. The auxiliary classification loss works best at low weight, and gains extend across most test classes.

A. Additional Quantitative Results

B. Additional Qualitative Results

Qualitative examples show STRM matching query tuples to more representative support tuples and encoding longer temporal variations than Baseline TRM. Its attention maps emphasize action-relevant objects, though subtle fine-grained motion differences remain harder to classify.

B. Additional Qualitative Results

  • Baseline TRM’s best support matches can miss the representative features in query tuples, whereas STRM’s matches are more representative and encode longer temporal variations.The examples compare query tuples of cardinality two with their best matches in support videos.Source: B. Additional Qualitative Results · p. 10
  • STRM’s attention maps emphasize action-relevant objects across frames, such as the remote or the bag and table during a dropping action.For the bag example, attention shifts from the table early to the bag’s position in later frames.Source: B. Additional Qualitative Results · p. 10
  • Subtle 2D motion differences between fine-grained novel actions remain harder to classify, including pretending to put something behind versus underneath something.Source: B. Additional Qualitative Results · p. 10
  • The paper’s quantitative and qualitative results together emphasize the enrichment module’s benefits for feature discriminability and model flexibility in few-shot action recognition.Source: B. Additional Qualitative Results · p. 10

C. Additional Implementation Details

Training and evaluation use uniformly sampled video frames and different crop strategies.

C. Additional Implementation Details

  • Videos are resized to height 256 and sampled at 8 uniform frames; training uses random 224 × 224 crops, while evaluation uses a centre crop.Training runs on four NVIDIA 2080Ti GPUs, with gradients accumulated over 16 iterations because only one few-shot task fits in memory.Source: C. Additional Implementation Details · p. 10
Loading 2112.05132v2…