PaperlayerView exact revision on arXiv

Source-linked AI summary

Few-Shot Video Recognition via Hierarchical Metric Learning

Jiaxin Zhang, Haoran Gao, Xizhan Gao, Zihao Dong, Tingwei Wang, Sijie Niu

arXiv:2609.05242v1cs.CV

TL;DR

FSAR must recognize unseen actions from few labeled videos, while existing approaches underuse cross-frame spatial information and lack progressive supervision across the feature pipeline. HML-FSAR combines spatial-enhanced processing with hierarchical metrics from frame representations to class prototypes. It is evaluated on five standard benchmarks, where the authors report competitive or state-of-the-art performance.

  • Problem

    Existing FSAR methods underuse cross-frame global spatial information, and multi-level schemes apply parallel rather than progressive constraints across the full feature pipeline.

    Source: §I · pp. 1–2

  • Method

    HML-FSAR combines SE, TMHA, HA, STFF and DL modules with center, alignment, contrastive, dictionary and prototype metrics for progressive feature supervision.

    Source: §I · p. 2, §V · p. 10

  • Results

    HML-FSAR achieves competitive or state-of-the-art performance on five standard benchmarks.

    Source: §I · p. 2, §V · p. 10

  • Takeaways & Limitations

    The framework jointly targets feature compactness, spatial-temporal alignment, inter-class discriminability and anti-noise robustness in few-shot action recognition.

    Source: §I · p. 2, §V · p. 10

Abstract

from arXiv · show

Few-shot action recognition (FSAR) aims to recognize unseen action categories with only a small number of annotated video samples. Recent works typically apply single-prototype supervision at the network output and fail to sufficiently exploit rich cross-frame global spatial information in videos. Even existing multi-level metric schemes only impose parallel prototype constraints on intermediate layers, without progressive supervision along the full feature pipeline, which results in limited generalization ability of the learned class prototypes. Inspired by this, we present a novel method, hierarchical metric learning for few-shot action recognition (HML-FSAR). First, a spatial-enhanced module is developed to capture cross-frame global spatial representations. Combined with temporal MHA, heterogeneous alignment, spatial-temporal feature fusion and dictionary learning modules, it constructs the complete feature processing pipeline. Second, a hierarchical metric learning (HML) strategy is embedded into HML-FSAR. Composed of center metric, alignment metric, contrastive metric, dictionary metric and prototype metric, HML imposes progressive multi-stage complementary constraints from frame-level representations to final class prototypes, so as to jointly optimize feature compactness, heterogeneous spatial-temporal alignment, inter-class discriminability and anti-noise robustness. The proposed HML-FSAR method is validated on five widely-used FSAR datasets, and experimental results fully demonstrate its effectiveness.

I. INTRODUCTION

FSAR addresses action recognition when labeled videos are scarce, but existing methods underuse cross-frame spatial information and rely on insufficiently progressive metric supervision. HML-FSAR combines spatial-enhanced feature learning with hierarchical constraints across the feature pipeline.

  • FSAR is important because action recognition requires modeling both spatial appearance and temporal dynamics, while collecting labeled videos is expensive and time-consuming.Source: §I · p. 1
  • Existing FSAR methods often restrict spatial representations to local information within individual frames, ignoring cross-frame global spatial features.Source: §I · p. 2
  • Single-output prototype metrics leave intermediate features without explicit constraints, while parallel multi-layer prototype metrics lack progressive complementary supervision across the pipeline.Source: §I · p. 2
  • HML-FSAR uses a spatial-enhanced module together with TMHA, HA, STFF and DL modules to construct its feature-processing architecture.Source: §I · p. 2
  • Its HML strategy progressively applies center, alignment, contrastive, dictionary and prototype metrics from frame-level representations to final class prototypes.These constraints target compactness, heterogeneous alignment, discriminability and robustness.Source: §I · p. 2
  • The end-to-end model is evaluated on five standard benchmarks, with experiments demonstrating overall effectiveness.Source: §I · p. 2

B. Few-shot Action Recognition

Few-shot action recognition classifies novel action videos from few annotated support samples using episodic metric-based learning. Its central objective is a feature space with compact intra-class representations and separable inter-class boundaries.

  • FSAR operates on complex 3-dimensional videos rather than 2-dimensional images, requiring spatial-temporal feature learning before prototype construction.Source: B. Few-shot Action Recognition · p. 3
  • Most FSAR methods use metric-based meta-learning to embed support and query videos into a shared space and classify them through feature similarity.Source: B. Few-shot Action Recognition · p. 3
  • OTAM addresses temporal misalignment with a temporal alignment loss, while MML-FSAR uses stage-specific similarity metrics to improve generalization of high-level features.Source: B. Few-shot Action Recognition · p. 3
  • HML-FSAR is motivated by remaining limitations in spatial-temporal feature modeling and feature metric learning.Source: B. Few-shot Action Recognition · p. 3
  • Standard episodes divide action classes into disjoint meta-training and meta-testing sets and sample N-way K-shot support sets plus query sets.Source: §III · p. 3
  • The learning objective is a generalizable feature space where intra-class samples cluster tightly while inter-class boundaries remain clear and separable.Source: §III · p. 3

A. Overall Framework of HML-FSAR

HML-FSAR processes support and query videos through frame extraction, parallel spatial and temporal modeling, heterogeneous alignment, feature fusion and dictionary refinement before prototype generation. Hierarchical metrics supervise this pipeline progressively, beginning with frame-level compactness and extending to feature alignment and prototype learning.

Overall Framework of HML-FSAR

  • The framework contains FE, SE, TMHA, HA, STFF and DL modules for extracting, aligning, fusing and refining video representations.Source: A. Overall Framework of HML-FSAR · p. 3
  • SE and TMHA operate in parallel to learn video-level global spatial features and preliminary spatial-temporal features, respectively.Source: Figure 2 · p. 4
  • After alignment and self-attention-based fusion, dictionary learning refines fused features before subsequent prototype generation.Source: A. Overall Framework of HML-FSAR · p. 4

1) Center Metric

  • Frame-level features are extracted by a pre-trained MAE feature extractor, with the [cls] token representing each frame.Source: 1) Center Metric · p. 4

Overall Framework of HML-FSAR

  • Support and query samples use identical parameters and computation pipelines throughout the subsequent modules and metric constraints.Source: 1) Center Metric · p. 4

1) Center Metric

  • The center metric encourages frame-level features within each video to cluster closely, indirectly enlarging distances between different videos.Source: 1) Center Metric · p. 5

2) Alignment Metric:

The alignment metric addresses heterogeneous spatial and temporal representations by projecting them into a shared Euclidean space and aligning corresponding instances and batch-level distributions.

Alignment Metric:

  • The SE module mines cross-frame video-level spatial information to complement temporal features limited to frame-local spatial representations.It constructs covariance-based spatial representations and projects them through a two-layer SPDNet.Source: 2) Alignment Metric · p. 5
  • TMHA processes frame-level features with a learnable sequence token to produce preliminary spatial-temporal features.The resulting PST feature is obtained from the output corresponding to the sequence token.Source: 2) Alignment Metric · p. 5
  • HA projects enhanced-spatial and PST features from heterogeneous spaces into a shared Euclidean space Rd4 for fusion.The two inputs originate from a Riemannian-manifold representation and a Euclidean representation, respectively.Source: 2) Alignment Metric · p. 5
  • The alignment metric combines instance-level and distribution-level terms to narrow spatial-temporal representation gaps.The instance term brings corresponding features closer, while the distribution term aligns global mini-batch distributions.Source: 2) Alignment Metric · p. 5

3) Contrastive Metric:

The contrastive metric makes fused spatial-temporal representations more discriminative by reducing distances within action classes and enlarging distances across classes.

Contrastive Metric:

  • Aligned enhanced-spatial and PST features are concatenated with a learnable fusion token and processed by STFF to produce fused features.The sequence is [rfusion, ¯ci, ¯vi], where all elements lie in the shared latent space Rd4.Source: 3) Contrastive Metric · p. 5
  • The contrastive metric reduces distances between samples of the same action and enlarges distances between samples from different actions.This is intended to improve the discriminative performance of fused representations under scarce training samples.Source: 3) Contrastive Metric · p. 5
  • The metric uses yij as a binary same-category indicator and h as a predefined distance threshold.yij = 1 when videos i and j share a category, and yij = 0 otherwise.Source: 3) Contrastive Metric · pp. 5–6

4) Dictionary Metric:

The dictionary metric purifies fused video features before prototype learning, combining reconstruction and discriminative constraints to reduce noise and prototype bias.

Dictionary Metric:

  • Few-shot samples, background interference and viewpoint variation can introduce redundant noise that biases averaged class prototypes.Such bias may cause prototype drift and degrade metric matching accuracy.Source: 4) Dictionary Metric · p. 6
  • The DL module uses analysis and synthesis dictionaries to compute dictionary codes and purified reconstructed features from fused video-level features.The dictionaries are parameterized as bias-free linear layers, and ReLU is used in the reconstruction process.Source: 4) Dictionary Metric · p. 6
  • The dictionary metric balances reconstruction quality with discriminative dictionary-code constraints.Its loss contains three terms weighted by hyperparameters λ1 and λ2.Source: 4) Dictionary Metric · p. 6
  • Sample reconstruction filters noisy feature components to provide more robust features for subsequent prototype learning.Stop-gradient stabilizes dictionary parameter learning by routing gradients only through the reconstructed feature.Source: 4) Dictionary Metric · p. 6
  • The dictionary formulation defines class-wise mean dictionary codes using the category set and sample indices.The notation includes N total categories and μc as the mean dictionary code for class c.Source: 4) Dictionary Metric · p. 6
  • A dictionary orthogonality constraint uses the identity matrix and Frobenius norm to regularize the learned dictionaries.The cited formulation denotes the identity matrix as ID and the matrix norm as ∥·∥F.Source: 4) Dictionary Metric · p. 6

5) Prototype Metric:

The prototype metric averages purified support features to form class prototypes and compares purified query features against them for few-shot classification.

Prototype Metric:

  • Class prototypes gc are generated by averaging purified support embeddings for each action class.The support subset Sc contains the support samples belonging to class c, with K support videos per class.Source: 5) Prototype Metric · p. 6
  • Each query video passes through the feature extraction and dictionary-learning pipeline to obtain a purified reconstructed feature.The resulting query representation is used in the prototype metric.Source: 5) Prototype Metric · p. 6
  • The prototype metric performs episodic query classification using a learnable temperature scaling parameter τ.M denotes the total number of query samples in the episode.Source: 5) Prototype Metric · p. 6

C. Network Optimization

The model is trained end-to-end with a multi-objective loss and evaluated on five few-shot action-recognition benchmarks spanning scene-level and temporally demanding settings.

Network optimization

  • The total loss combines prototype, center, alignment, contrastive, and dictionary losses, with β1–β4 balancing auxiliary constraints.These hyperparameters control trade-offs among the auxiliary loss terms.Source: C. Network Optimization · p. 6

Datasets

  • Evaluation covers HMDB51, UCF101, Kinetics, SSv2-Full, and SSv2-Small, with SSv2 emphasizing temporal modeling.HMDB51, UCF101, and Kinetics primarily emphasize scene-level semantics, whereas SSv2 relies more on motion cues.Source: A. Experimental Settings · pp. 6–7

Dataset protocols

  • The benchmark protocols use class splits of 31/10/10 for HMDB51, 70/10/21 for UCF101, and 64/12/24 for Kinetics and both SSv2 datasets.The ratios denote training, validation, and testing classes.Source: A. Experimental Settings · p. 7

Implementation details

  • Training uses 10,000 episodes for HMDB51, UCF101, and Kinetics, and 60,000 episodes for both SSv2 benchmarks.All experiments use a learning rate of 1×10^-5 and 10,000 testing episodes per dataset.Source: A. Experimental Settings · p. 7

B. Comparison with Various Methods

HML-FSAR outperforms prior methods across five benchmarks, including temporally challenging SSv2 datasets, while ablations show complementary gains from hierarchical metrics and network modules.

Benchmark comparison

  • 71.2%/86.1% on HMDB51, 93.5%/99.2% on UCF101, and 86.0%/96.5% on Kinetics are achieved under 1-shot/5-shot settings.The method improves over previous state-of-the-art approaches in most evaluated cases.Source: B. Comparison with Various Methods · p. 7
  • 52.0%/66.7% on SSv2-Small and 58.3%/73.2% on SSv2-Full are obtained under 1-shot/5-shot settings.The reported gains include 4.8% and 3.6% over TEAM on SSv2-Small and 9.1% over MVP-Shot at 5-shot on SSv2-Full.Source: B. Comparison with Various Methods · p. 7

Comparison with vision-language methods

  • The method retains advantages over CLIP-FSAR and MVP-Shot at 5-shot across all five datasets and remains competitive at 1-shot.The paper attributes this result to more discriminative spatial-temporal features from hierarchical metric learning.Source: B. Comparison with Various Methods · p. 7

Ablation design

Metric ablation

  • The hierarchical metrics progressively improve performance by tightening frame compactness, heterogeneous alignment, inter-class margins, and few-shot feature robustness.The full multi-metric scheme performs best after stacking complementary constraints layer by layer.Source: 1) Influence of Different Metric… · pp. 7–8

2) Ablation on Key Components:

Component and setting ablations show that spatial-temporal modules, dictionary learning, richer frame sampling, and additional support data improve recognition, while harder tasks reduce accuracy.

Component ablation

  • Adding SE raises accuracy from 80.0% to 81.8% on Kinetics and from 43.1% to 45.3% on SSv2-Small.The result supports the contribution of cross-frame global spatial features beyond local frame-level representations.Source: 2) Ablation on Key Components · p. 8
  • TMHA alone reaches 83.6% on Kinetics and 48.9% on SSv2-Small, while SE, TMHA, HA, and STFF together reach 83.9% and 50.3%.These results indicate complementary contributions from temporal, spatial, and fusion modules.Source: 2) Ablation on Key Components · p. 8
  • Adding DL to the complete architecture increases accuracy to 86.0% on Kinetics and 52.0% on SSv2-Small.DL alone provides limited improvement, but contributes substantially when integrated with the other modules.Source: 2) Ablation on Key Components · p. 8

Frame sampling

Few-shot settings

5) Hyperparameter Sensitivity Analysis:

The analysis finds stable performance across broad hyperparameter ranges, with balanced loss weighting performing best. Feature visualizations show progressively tighter intra-class clusters and clearer inter-class separation through the proposed modules, while the full method achieves competitive or state-of-the-art benchmark performance.

Hyperparameter Sensitivity Analysis

  • Recognition accuracy remains stable as λ increases, indicating insensitivity to this hyper-parameter under both 1-shot and 5-shot settings.Accuracy fluctuates only slightly across the tested values.Source: 5) Hyperparameter Sensitivity Analysis · p. 9
  • The highest recognition accuracy is consistently achieved when all loss weights are set to 0.1.This setting balances center, alignment, contrastive, and dictionary losses during optimization.Source: 5) Hyperparameter Sensitivity Analysis · p. 9

Feature Visualization

  • After the SE module, same-class samples become more compact and inter-class boundaries clearer by capturing cross-frame global spatial information.The initial MAE backbone features show substantial overlap and dispersion across categories.Source: D. Visualization · pp. 9–10
  • TMHA further separates actions with similar appearance but different motion patterns, although slight overlap remains between neighboring classes.Temporal features improve category separation while not eliminating all overlap.Source: D. Visualization · p. 10
  • After alignment and fusion, representations show improved intra-class compactness and inter-class separability, while the complete model forms compact clusters with larger margins and minimal overlap.The complete model’s visualization indicates complementary module effects and more discriminative representations.Source: D. Visualization · p. 10

Conclusion

  • Experimental results demonstrate competitive or state-of-the-art performance on five standard benchmarks.The conclusion attributes the framework’s feature optimization to complementary multi-stage metric constraints.Source: §V · p. 10

Related papers

Loading 2609.05242v1…