PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatio-temporal Decoupled Knowledge Compensator for Few-Shot Action Recognition

Hongyu Qu, Xiangbo Shu, Rui Yan, Hailiang Gao, Wenguan Wang, Jinhui Tang

arXiv:2602.18043v1cs.CV

TL;DR

FSAR needs to recognize novel actions from few labeled videos, but coarse category names provide limited background knowledge for spatial and temporal concepts. DiST uses LLM-generated decoupled spatio-temporal descriptions with Spatial/Temporal Knowledge Compensators to learn object- and frame-level prototypes. It reports state-of-the-art performance, including 1.7%-6.8% accuracy gains under 5-way 1-shot evaluation, while its universal class-level attributes may include irrelevant objects.

  • Problem

    FSAR must recognize novel action categories from a few labeled videos, while coarse category names provide insufficient background knowledge for video understanding.

  • Method

    DiST decomposes action names into LLM-generated spatial and temporal descriptions and incorporates them through SKC/TKC to learn object- and frame-level prototypes.

  • Results

    1.7%-6.8% accuracy gains under the 5-way 1-shot setting were reported over existing state-of-the-art methods across five standard datasets.

  • Takeaways & Limitations

    Decoupled prior knowledge supports prototype learning at different levels for capturing fine-grained spatial details and dynamic temporal information.

  • Takeaways & Limitations

    The generated attributes are universal and class-level rather than instance-specific, so irrelevant attributes may affect model performance.

Abstract

from arXiv · show

Few-Shot Action Recognition (FSAR) is a challenging task that requires recognizing novel action categories with a few labeled videos. Recent works typically apply semantically coarse category names as auxiliary contexts to guide the learning of discriminative visual features. However, such context provided by the action names is too limited to provide sufficient background knowledge for capturing novel spatial and temporal concepts in actions. In this paper, we propose DiST, an innovative Decomposition-incorporation framework for FSAR that makes use of decoupled Spatial and Temporal knowledge provided by large language models to learn expressive multi-granularity prototypes. In the decomposition stage, we decouple vanilla action names into diverse spatio-temporal attribute descriptions (action-related knowledge). Such commonsense knowledge complements semantic contexts from spatial and temporal perspectives. In the incorporation stage, we propose Spatial/Temporal Knowledge Compensators (SKC/TKC) to discover discriminative object-level and frame-level prototypes, respectively. In SKC, object-level prototypes adaptively aggregate important patch tokens under the guidance of spatial knowledge. Moreover, in TKC, frame-level prototypes utilize temporal attributes to assist in inter-frame temporal relation modeling. These learned prototypes thus provide transparency in capturing fine-grained spatial details and diverse temporal patterns. Experimental results show DiST achieves state-of-the-art results on five standard FSAR datasets.

1 INTRODUCTION

FSAR addresses recognition of unseen action classes from few labeled videos, but coarse category names provide insufficient background knowledge. DiST decomposes names into spatio-temporal knowledge and uses it to learn object- and frame-level prototypes, achieving reported gains across five datasets.

  • Motivation: FSAR learns from base classes and evaluates generalization to novel action classes with unlabeled samples.
  • Motivation: Existing methods often use coarse category names as context, which is insufficient for comprehensive video action understanding.
  • DiST framework: DiST uses LLMs to decompose category names into spatial and temporal commonsense descriptions that complement semantic context.
  • DiST framework: Spatial and Temporal Knowledge Compensators incorporate decoupled knowledge into object-level and frame-level prototypes, respectively.SKC uses patch-level interaction, while TKC uses frame-level features and temporal knowledge.
  • Results: 1.7%-6.8% accuracy gains under the 5-way 1-shot setting were reported over existing state-of-the-art methods across five standard datasets.

2 RELATED WORK

Related FSAR methods mainly learn representations and matching strategies within metric-based meta-learning, while recent vision-language approaches rely on coarse category names. DiST instead uses decoupled spatio-temporal prior knowledge to support complementary prototype learning.

  • DiST Positioning: The framework decomposes matching into complementary object-level and frame-level prototypes to address local spatial details and dynamic temporal relations.
  • Few-Shot Action Recognition: Few-shot action recognition targets previously unseen action classes using a few labeled videos and commonly follows metric-based meta-learning.
  • Few-Shot Action Recognition: Existing FSAR work explores feature representation learning, matching strategies, temporal modeling, and patch-level spatial interactions.
  • Vision-Language Knowledge: Vision-language approaches transfer pretrained knowledge but rely heavily on semantically coarse or ambiguous category names.
  • DiST Positioning: DiST explicitly decouples class names into spatio-temporal attribute descriptions to provide action-relevant knowledge.

3 METHOD

DiST processes video and action names through CLIP and LLM-based spatio-temporal knowledge decomposition, then learns object- and frame-level prototypes for separate spatial and temporal matching. Its compensators aggregate visual features with decoupled attributes before combining the matching results for prediction, while using universal class-level attributes that may include irrelevant instance-specific concepts.

  • Overall Framework: DiST encodes RGB frame sequences with CLIP to obtain frame-level and patch-level visual features, while an LLM generates decoupled spatial and temporal action descriptions.The descriptions are encoded by CLIP's frozen text encoder as spatial and temporal attribute features.
  • Spatial Knowledge Compensator: SKC aggregates patch-level features into compact object-level prototypes and injects spatial attributes to focus on semantically relevant object regions.This sparse aggregation and attribute-injection design is intended to capture discriminative spatial patterns while reducing redundant patch information.
  • Temporal Knowledge Compensator: TKC forms frame-level prototypes by adding temporal knowledge, applying vision-attribute cross-attention, and using a temporal transformer to model inter-frame relationships.The resulting prototypes encode action dynamic information and adaptively perceive temporal changes under temporal-knowledge guidance.
  • Few-shot Metric: DiST performs dual-level matching by comparing object-level prototypes spatially and frame-level prototypes temporally before combining both matching scores into the merged query prediction.The spatial metric uses frame-to-frame distances, while the temporal branch applies a temporal alignment metric to support-query frame prototypes.
  • Discussion on LLM-generated universal attributes: DiST's LLM-generated attributes are universal and class-level rather than instance-specific, so predicted concepts such as a stool may be absent from individual videos.The authors note that irrelevant attributes may affect performance and propose future object-relevance estimation to filter noisy concepts.

4.1 Experimental Setup

DiST is evaluated on five standard FSAR datasets using established train/validation/test splits and 5-way few-shot protocols. Results average accuracy over 10,000 randomly selected test tasks.

  • Experiments cover Kinetics, SSv2-full, SSv2-small, HMDB51, and UCF101.
  • HMDB51 uses a 31/10/10 train/validation/test class split, while UCF101 uses 70/10/21 classes.
  • Evaluation uses 5-way 1-shot and 5-way 5-shot accuracy, averaged over 10,000 randomly selected test tasks.

4.2 Implementation Details

Implementation uses CLIP ViT-B/16 as the pretrained backbone, with fixed default attribute and prototype configurations. The paper reports ablation-study settings and a PyTorch training implementation.

  • DIST uses CLIP ViT-B initialized with pretrained CLIP ViT-B/16 parameters for comparison with prior methods.
  • The default configuration uses 6 spatial attributes, 3 temporal attributes, 9 object-level prototypes, and α = 0.5.
  • DIST is implemented in PyTorch and trained with Adam and a multi-step scheduler on two NVIDIA Tesla V100 GPUs.

4.3 Comparison with State-of-the-Arts

Across five standard datasets, DIST outperforms existing FSAR methods, including CLIP-FSAR, with larger margins in lower-shot settings. On HMDB51 and UCF101, its 5-way 3-shot performance is comparable to CLIP-FSAR’s 5-way 5-shot performance.

  • DIST outperforms all compared FSAR methods across multiple datasets and task settings.
  • DIST achieves larger performance margins over CLIP-FSAR in lower-shot settings.
  • On HMDB51 and UCF101, DIST’s 5-way 3-shot performance is comparable to CLIP-FSAR’s 5-way 5-shot performance.
  • The reported gains are attributed to decoupled LLM-derived spatiotemporal prior knowledge compensating for scarce visual information.

4.4 Ablation Study

Ablations show that the proposed SKC and TKC designs, decoupled attribute content, and their combination improve few-shot performance. The experiments also identify attribute-count and fusion configurations used by DIST.

  • Key Component Analysis: TKC improves performance over the baseline by 5.2% on HMDB51, 2.3% on SSv2-small, and 1.9% on UCF101.
  • Key Component Analysis: SKC improves performance over the baseline by 1.8% on HMDB51, 1.9% on SSv2-small, and 0.6% on UCF101.
  • Key Component Analysis: Combining SKC and TKC further improves performance, indicating complementarity between the two modules.
  • Attribute Configuration: Performance peaks at G = 6 spatial attributes, while excessive attributes may introduce redundant or noisy local cues.
  • Fusion and Class-Wise Analysis: Figure 4 examines fusion parameter α and reports improvement over CLIP-FSAR on all HMDB51 action classes.
  • Attribute Injection: Using SKC and TKC yields better results than directly concatenating visual and attribute features for multimodal fusion.
  • Attribute Content: LLM-generated spatial and temporal attributes outperform class names in the 1-shot HMDB51 setting by 1.2% and 1.6%, respectively.
  • Attribute Injection: DIST improves over CLIP-FSAR†, showing that gains are not solely due to replacing class labels with LLM-generated prompts.

4.4.6 Impact of the Number of Object-level Prototypes

The study examines prototype count, efficiency, matching, language-model choice, and visual-encoder fine-tuning in DiST. Results identify N = 9 as the default prototype count and report accuracy gains with limited computational overhead.

  • Impact of the Number of Object-level Prototypes: 80.8→82.6 on HMDB51 as object-level prototypes increase, with N = 9 selected for stable gains.The authors attribute excessive prototype counts to noisy background information and balance accuracy with computational efficiency.
  • Model Efficiency Analysis: 6.8% and 3.0% accuracy improvements over CLIP-FSAR on HMDB51 and Kinetics, respectively, with negligible additional FLOPs and memory.The comparison uses the same ViT-B visual encoder for both methods.
  • Fusion Parameter Sensitivity: α = 0.5 gives the best HMDB51 5-way 1-shot performance, indicating equal weighting of spatial and temporal matching scores.The authors describe spatial and temporal knowledge as complementary cues.
  • Language-Model Knowledge Base: Large language model variants consistently outperform the baseline using raw class names for spatial-temporal feature modeling.The comparison includes ChatGPT, LLaMA-2, and Vicuna under identical prompt templates and experimental settings.
  • Impact of Fine-tuning Visual Encoder: Fine-tuned DIST reaches 84.7% on HMDB51, 95.9% on Kinetics, and 60.7% on SSv2-small in 5-way 1-shot evaluation.These results substantially exceed the frozen-visual-encoder configuration.
  • Spatial Interaction Ablation: The Spatial Knowledge Compensator is evaluated against representative spatial interaction methods across multiple datasets under 1-shot settings.The supplied passage identifies patch-level enrichment and Short-Term Temporal Modeling as comparison methods but does not provide all numerical results.

1. Mouth 3. Container 2. Hand Spatial prompt

The supplied material labels a spatial-prompt visualization around mouth, container, and hand concepts, alongside a temporal-prompt panel and temporal scale values.

  • Mouth: The visualization includes a spatial prompt associated with the mouth.
  • Temporal prompt: A temporal prompt is identified in the visualization.
  • Container: The visualization includes a spatial prompt associated with the container.
  • Container: The supplied labels include the action of putting a container near the mouth.
  • Hand: The visualization includes a spatial prompt associated with the hand.
  • Temporal prompt: The temporal scale shown includes values from 0.1 through 0.4.

1. Bicycle 3. Feets 2. legs Spatial prompt

The experiments evaluate spatial interaction, backbone, parameter-efficient fine-tuning, class-wise performance, feature distribution, and prompt visualizations for DiST.

  • Spatial Interaction Comparison: DiST’s Spatial Knowledge Compensator consistently surpasses representative spatial interaction methods across HMDB51, SSv2-small, and UCF101 under 1-shot evaluation.The comparison includes patch-level enrichment and Short-Term Temporal Modeling.
  • Matching Metrics: The proposed spatial matching metric achieves the best results compared with one-to-one matching and Bi-MHM.
  • Generalization with ImageNet-pretrained Backbones: With ImageNet-pretrained backbones, DIST consistently outperforms CLIP-FSAR; using ResNet-50 yields a 2.4% gain on SSv2-small under 1-shot evaluation.
  • Generalization Under Parameter-efficient Fine-tuning: DIST consistently surpasses existing parameter-efficient fine-tuning approaches under 1-shot evaluation, even with similar backbones and fine-tuning protocols.The results support transfer from base to novel classes with limited trainable parameters.
  • Class-wise Performance Gains: DIST outperforms CLIP-FSAR across all HMDB51 action classes, with gains exceeding 10% for run, pour, and kick ball.The authors associate these categories with motion patterns and distinctive object interactions.
  • Visualization of Feature Distribution: After incorporating spatiotemporal-decoupled prior knowledge, DIST produces more compact intra-class distributions and more discriminative inter-class distributions in t-SNE visualizations.
  • Visualization of Spatial and Temporal Attributes: DIST’s attention focuses more on action-related objects and reduces attention to background and unrelated objects.
  • Visualization of Spatial and Temporal Attributes: The temporal attribute “Hold container” receives larger weights on the first three frames, indicating frame-specific dynamic semantics.

5 CONCLUSION

DIST uses LLM-derived spatiotemporal-decoupled prior knowledge with Spatial and Temporal Knowledge Compensators to learn object- and frame-level prototypes for FSAR. The paper reports state-of-the-art performance on four standard benchmarks and identifies richer structured LLM knowledge as a future direction.

  • Conclusion: DIST uses spatiotemporal-decoupled prior knowledge from LLMs to compensate for visual features in few-shot action recognition.
  • Conclusion: Spatial and Temporal Knowledge Compensators learn object-level and frame-level prototypes to capture fine-grained spatial details and dynamic semantics.
  • Conclusion: DIST achieves state-of-the-art performance on four standard benchmarks.
  • Conclusion: The paper points to integrating richer, more structured knowledge from LLMs as a future direction for low-shot video understanding.
Loading 2602.18043v1…