PaperlayerView exact revision on arXiv

Source-linked AI summary

Few-Shot Video Recognition via Hierarchical Metric Learning

Jiaxin Zhang, Haoran Gao, Xizhan Gao, Zihao Dong, Tingwei Wang, Sijie Niu

arXiv:2609.05242v1cs.CV

TL;DR

FSAR must recognize unseen actions from few labeled videos, while existing approaches underuse cross-frame spatial information and lack progressive supervision across the feature pipeline. HML-FSAR combines spatial-enhanced processing with hierarchical metrics from frame representations to class prototypes. It is evaluated on five standard benchmarks, where the authors report competitive or state-of-the-art performance.

  • Problem

    Existing FSAR methods underuse cross-frame global spatial information, and multi-level schemes apply parallel rather than progressive constraints across the full feature pipeline.

  • Method

    HML-FSAR combines SE, TMHA, HA, STFF and DL modules with center, alignment, contrastive, dictionary and prototype metrics for progressive feature supervision.

  • Results

    HML-FSAR achieves competitive or state-of-the-art performance on five standard benchmarks.

  • Takeaways & Limitations

    The framework jointly targets feature compactness, spatial-temporal alignment, inter-class discriminability and anti-noise robustness in few-shot action recognition.

Abstract

from arXiv · show

Few-shot action recognition (FSAR) aims to recognize unseen action categories with only a small number of annotated video samples. Recent works typically apply single-prototype supervision at the network output and fail to sufficiently exploit rich cross-frame global spatial information in videos. Even existing multi-level metric schemes only impose parallel prototype constraints on intermediate layers, without progressive supervision along the full feature pipeline, which results in limited generalization ability of the learned class prototypes. Inspired by this, we present a novel method, hierarchical metric learning for few-shot action recognition (HML-FSAR). First, a spatial-enhanced module is developed to capture cross-frame global spatial representations. Combined with temporal MHA, heterogeneous alignment, spatial-temporal feature fusion and dictionary learning modules, it constructs the complete feature processing pipeline. Second, a hierarchical metric learning (HML) strategy is embedded into HML-FSAR. Composed of center metric, alignment metric, contrastive metric, dictionary metric and prototype metric, HML imposes progressive multi-stage complementary constraints from frame-level representations to final class prototypes, so as to jointly optimize feature compactness, heterogeneous spatial-temporal alignment, inter-class discriminability and anti-noise robustness. The proposed HML-FSAR method is validated on five widely-used FSAR datasets, and experimental results fully demonstrate its effectiveness.

I. INTRODUCTION

FSAR addresses action recognition when labeled videos are scarce, but existing methods underuse cross-frame spatial information and rely on insufficiently progressive metric supervision. HML-FSAR combines spatial-enhanced feature learning with hierarchical constraints across the feature pipeline.

  • FSAR is important because action recognition requires modeling both spatial appearance and temporal dynamics, while collecting labeled videos is expensive and time-consuming.
  • Existing FSAR methods often restrict spatial representations to local information within individual frames, ignoring cross-frame global spatial features.
  • Single-output prototype metrics leave intermediate features without explicit constraints, while parallel multi-layer prototype metrics lack progressive complementary supervision across the pipeline.
  • HML-FSAR uses a spatial-enhanced module together with TMHA, HA, STFF and DL modules to construct its feature-processing architecture.
  • Its HML strategy progressively applies center, alignment, contrastive, dictionary and prototype metrics from frame-level representations to final class prototypes.These constraints target compactness, heterogeneous alignment, discriminability and robustness.
  • The end-to-end model is evaluated on five standard benchmarks, with experiments demonstrating overall effectiveness.

B. Few-shot Action Recognition

Few-shot action recognition classifies novel action videos from few annotated support samples using episodic metric-based learning. Its central objective is a feature space with compact intra-class representations and separable inter-class boundaries.

  • FSAR operates on complex 3-dimensional videos rather than 2-dimensional images, requiring spatial-temporal feature learning before prototype construction.
  • Most FSAR methods use metric-based meta-learning to embed support and query videos into a shared space and classify them through feature similarity.
  • OTAM addresses temporal misalignment with a temporal alignment loss, while MML-FSAR uses stage-specific similarity metrics to improve generalization of high-level features.
  • HML-FSAR is motivated by remaining limitations in spatial-temporal feature modeling and feature metric learning.
  • Standard episodes divide action classes into disjoint meta-training and meta-testing sets and sample N-way K-shot support sets plus query sets.
  • The learning objective is a generalizable feature space where intra-class samples cluster tightly while inter-class boundaries remain clear and separable.

A. Overall Framework of HML-FSAR

HML-FSAR processes support and query videos through frame extraction, parallel spatial and temporal modeling, heterogeneous alignment, feature fusion and dictionary refinement before prototype generation. Hierarchical metrics supervise this pipeline progressively, beginning with frame-level compactness and extending to feature alignment and prototype learning.

  • Overall Framework of HML-FSAR: The framework contains FE, SE, TMHA, HA, STFF and DL modules for extracting, aligning, fusing and refining video representations.
  • Overall Framework of HML-FSAR: SE and TMHA operate in parallel to learn video-level global spatial features and preliminary spatial-temporal features, respectively.
  • Overall Framework of HML-FSAR: After alignment and self-attention-based fusion, dictionary learning refines fused features before subsequent prototype generation.
  • 1) Center Metric: Frame-level features are extracted by a pre-trained MAE feature extractor, with the [cls] token representing each frame.
  • Overall Framework of HML-FSAR: Support and query samples use identical parameters and computation pipelines throughout the subsequent modules and metric constraints.
  • 1) Center Metric: The center metric encourages frame-level features within each video to cluster closely, indirectly enlarging distances between different videos.

2) Alignment Metric:

The alignment metric addresses heterogeneous spatial and temporal representations by projecting them into a shared Euclidean space and aligning corresponding instances and batch-level distributions.

  • Alignment Metric:: The SE module mines cross-frame video-level spatial information to complement temporal features limited to frame-local spatial representations.It constructs covariance-based spatial representations and projects them through a two-layer SPDNet.
  • Alignment Metric:: TMHA processes frame-level features with a learnable sequence token to produce preliminary spatial-temporal features.The resulting PST feature is obtained from the output corresponding to the sequence token.
  • Alignment Metric:: HA projects enhanced-spatial and PST features from heterogeneous spaces into a shared Euclidean space Rd4 for fusion.The two inputs originate from a Riemannian-manifold representation and a Euclidean representation, respectively.
  • Alignment Metric:: The alignment metric combines instance-level and distribution-level terms to narrow spatial-temporal representation gaps.The instance term brings corresponding features closer, while the distribution term aligns global mini-batch distributions.

3) Contrastive Metric:

The contrastive metric makes fused spatial-temporal representations more discriminative by reducing distances within action classes and enlarging distances across classes.

  • Contrastive Metric:: Aligned enhanced-spatial and PST features are concatenated with a learnable fusion token and processed by STFF to produce fused features.The sequence is [rfusion, ¯ci, ¯vi], where all elements lie in the shared latent space Rd4.
  • Contrastive Metric:: The contrastive metric reduces distances between samples of the same action and enlarges distances between samples from different actions.This is intended to improve the discriminative performance of fused representations under scarce training samples.
  • Contrastive Metric:: The metric uses yij as a binary same-category indicator and h as a predefined distance threshold.yij = 1 when videos i and j share a category, and yij = 0 otherwise.

4) Dictionary Metric:

The dictionary metric purifies fused video features before prototype learning, combining reconstruction and discriminative constraints to reduce noise and prototype bias.

  • Dictionary Metric:: Few-shot samples, background interference and viewpoint variation can introduce redundant noise that biases averaged class prototypes.Such bias may cause prototype drift and degrade metric matching accuracy.
  • Dictionary Metric:: The DL module uses analysis and synthesis dictionaries to compute dictionary codes and purified reconstructed features from fused video-level features.The dictionaries are parameterized as bias-free linear layers, and ReLU is used in the reconstruction process.
  • Dictionary Metric:: The dictionary metric balances reconstruction quality with discriminative dictionary-code constraints.Its loss contains three terms weighted by hyperparameters λ1 and λ2.
  • Dictionary Metric:: Sample reconstruction filters noisy feature components to provide more robust features for subsequent prototype learning.Stop-gradient stabilizes dictionary parameter learning by routing gradients only through the reconstructed feature.
  • Dictionary Metric:: The dictionary formulation defines class-wise mean dictionary codes using the category set and sample indices.The notation includes N total categories and μc as the mean dictionary code for class c.
  • Dictionary Metric:: A dictionary orthogonality constraint uses the identity matrix and Frobenius norm to regularize the learned dictionaries.The cited formulation denotes the identity matrix as ID and the matrix norm as ∥·∥F.

5) Prototype Metric:

The prototype metric averages purified support features to form class prototypes and compares purified query features against them for few-shot classification.

  • Prototype Metric:: Class prototypes gc are generated by averaging purified support embeddings for each action class.The support subset Sc contains the support samples belonging to class c, with K support videos per class.
  • Prototype Metric:: Each query video passes through the feature extraction and dictionary-learning pipeline to obtain a purified reconstructed feature.The resulting query representation is used in the prototype metric.
  • Prototype Metric:: The prototype metric performs episodic query classification using a learnable temperature scaling parameter τ.M denotes the total number of query samples in the episode.

C. Network Optimization

The model is trained end-to-end with a multi-objective loss and evaluated on five few-shot action-recognition benchmarks spanning scene-level and temporally demanding settings.

  • Network optimization: The total loss combines prototype, center, alignment, contrastive, and dictionary losses, with β1–β4 balancing auxiliary constraints.These hyperparameters control trade-offs among the auxiliary loss terms.
  • Datasets: Evaluation covers HMDB51, UCF101, Kinetics, SSv2-Full, and SSv2-Small, with SSv2 emphasizing temporal modeling.HMDB51, UCF101, and Kinetics primarily emphasize scene-level semantics, whereas SSv2 relies more on motion cues.
  • Dataset protocols: The benchmark protocols use class splits of 31/10/10 for HMDB51, 70/10/21 for UCF101, and 64/12/24 for Kinetics and both SSv2 datasets.The ratios denote training, validation, and testing classes.
  • Implementation details: Training uses 10,000 episodes for HMDB51, UCF101, and Kinetics, and 60,000 episodes for both SSv2 benchmarks.All experiments use a learning rate of 1×10^-5 and 10,000 testing episodes per dataset.

B. Comparison with Various Methods

HML-FSAR outperforms prior methods across five benchmarks, including temporally challenging SSv2 datasets, while ablations show complementary gains from hierarchical metrics and network modules.

  • Benchmark comparison: 71.2%/86.1% on HMDB51, 93.5%/99.2% on UCF101, and 86.0%/96.5% on Kinetics are achieved under 1-shot/5-shot settings.The method improves over previous state-of-the-art approaches in most evaluated cases.
  • Benchmark comparison: 52.0%/66.7% on SSv2-Small and 58.3%/73.2% on SSv2-Full are obtained under 1-shot/5-shot settings.The reported gains include 4.8% and 3.6% over TEAM on SSv2-Small and 9.1% over MVP-Shot at 5-shot on SSv2-Full.
  • Comparison with vision-language methods: The method retains advantages over CLIP-FSAR and MVP-Shot at 5-shot across all five datasets and remains competitive at 1-shot.The paper attributes this result to more discriminative spatial-temporal features from hierarchical metric learning.
  • Ablation design: Ablation experiments examine metric schemes, network modules, sampled-frame counts, few-shot configurations, and hyperparameter sensitivity.These experiments are designed to dissect component contributions and robustness.
  • Metric ablation: The hierarchical metrics progressively improve performance by tightening frame compactness, heterogeneous alignment, inter-class margins, and few-shot feature robustness.The full multi-metric scheme performs best after stacking complementary constraints layer by layer.

2) Ablation on Key Components:

Component and setting ablations show that spatial-temporal modules, dictionary learning, richer frame sampling, and additional support data improve recognition, while harder tasks reduce accuracy.

  • Component ablation: Adding SE raises accuracy from 80.0% to 81.8% on Kinetics and from 43.1% to 45.3% on SSv2-Small.The result supports the contribution of cross-frame global spatial features beyond local frame-level representations.
  • Component ablation: TMHA alone reaches 83.6% on Kinetics and 48.9% on SSv2-Small, while SE, TMHA, HA, and STFF together reach 83.9% and 50.3%.These results indicate complementary contributions from temporal, spatial, and fusion modules.
  • Component ablation: Adding DL to the complete architecture increases accuracy to 86.0% on Kinetics and 52.0% on SSv2-Small.DL alone provides limited improvement, but contributes substantially when integrated with the other modules.
  • Frame sampling: Increasing sampled frames from 4 to 16 improves Kinetics accuracy from 82.8%/91.3% to 86.0%/96.5% and SSv2-Small accuracy from 47.3%/62.7% to 52.0%/66.7%.The paper adopts 16 sampled frames in subsequent experiments.
  • Few-shot settings: On Kinetics, accuracy rises from 86.0% to 96.5% from 1-shot to 5-shot but falls from 86.0% to 77.2% as task size grows from 5-way to 10-way.The method maintains relatively stable performance across the tested few-shot configurations.

5) Hyperparameter Sensitivity Analysis:

The analysis finds stable performance across broad hyperparameter ranges, with balanced loss weighting performing best. Feature visualizations show progressively tighter intra-class clusters and clearer inter-class separation through the proposed modules, while the full method achieves competitive or state-of-the-art benchmark performance.

  • Hyperparameter Sensitivity Analysis: Recognition accuracy remains stable as λ increases, indicating insensitivity to this hyper-parameter under both 1-shot and 5-shot settings.Accuracy fluctuates only slightly across the tested values.
  • Hyperparameter Sensitivity Analysis: The highest recognition accuracy is consistently achieved when all loss weights are set to 0.1.This setting balances center, alignment, contrastive, and dictionary losses during optimization.
  • Feature Visualization: After the SE module, same-class samples become more compact and inter-class boundaries clearer by capturing cross-frame global spatial information.The initial MAE backbone features show substantial overlap and dispersion across categories.
  • Feature Visualization: TMHA further separates actions with similar appearance but different motion patterns, although slight overlap remains between neighboring classes.Temporal features improve category separation while not eliminating all overlap.
  • Feature Visualization: After alignment and fusion, representations show improved intra-class compactness and inter-class separability, while the complete model forms compact clusters with larger margins and minimal overlap.The complete model’s visualization indicates complementary module effects and more discriminative representations.
  • Conclusion: Experimental results demonstrate competitive or state-of-the-art performance on five standard benchmarks.The conclusion attributes the framework’s feature optimization to complementary multi-stage metric constraints.
Loading 2609.05242v1…