Source-linked AI summary
Few-Shot Video Classification via Temporal Alignment
Kaidi Cao, Jingwei Ji, Zhangjie Cao, Chien-Yi Chang, Juan Carlos Niebles
TL;DR
Few-shot video classification aims to recognize unseen classes from very few labeled examples, despite temporal ordering and duration variations that challenge video understanding. The paper proposes TAM, which aligns query and support videos while preserving temporal order and uses continuous relaxation for end-to-end optimization. TAM improves few-shot performance on Kinetics and Something-Something V2, including an approximately 8% top-1 accuracy gain over mean pooling in the single-example setting.
Problem
Few-shot video classification must recognize novel classes from scarce labels while handling temporal ordering and non-linear temporal variations that previous approaches often neglect.
Method
TAM dynamically aligns query videos with support-set proxies, averages per-frame distances along alignment paths, and uses continuous relaxation for differentiable end-to-end training.
Results
TAM significantly outperforms competitive baselines on Kinetics and Something-Something V2; with one example, it improves approximately 8% in top-1 accuracy over mean pooling.
Takeaways & Limitations
Explicitly preserving temporal ordering through adaptive alignment provides a data-efficient approach to few-shot video classification.
Takeaways & Limitations
ImageNet-pretrained-backbone experiment results are taken directly from CMN to ensure fair comparison.
Abstract
from arXiv · showhide
There is a growing interest in learning a model which could recognize novel classes with only a few labeled examples. In this paper, we propose Temporal Alignment Module (TAM), a novel few-shot learning framework that can learn to classify a previous unseen video. While most previous works neglect long-term temporal ordering information, our proposed model explicitly leverages the temporal ordering information in video data through temporal alignment. This leads to strong data-efficiency for few-shot learning. In concrete, TAM calculates the distance value of query video with respect to novel class proxies by averaging the per frame distances along its alignment path. We introduce continuous relaxation to TAM so the model can be learned in an end-to-end fashion to directly optimize the few-shot learning objective. We evaluate TAM on two challenging real-world datasets, Kinetics and Something-Something-V2, and show that our model leads to significant improvement of few-shot video classification over a wide range of competitive baselines.
1. Introduction
Few-shot video classification seeks to recognize unseen classes from very few labels, but video-specific temporal ordering and variable action durations make this difficult. TAM addresses these challenges by aligning video sequences while preserving temporal order, and improves performance on Kinetics and Something-Something V2.
- Few-shot learning targets recognition of novel classes using only a few labeled examples, reducing reliance on extensive manual video labeling.
- Temporal pooling can discard ordering information needed to distinguish actions and is especially problematic when few examples limit learning of local temporal patterns.
- Same-class videos can contain the same action steps with dramatically different durations, creating non-linear temporal variations that challenge few-shot classification.
- TAM computes alignment scores by averaging per-frame distances along a temporal alignment path, preserving temporal ordering between query videos and support-set proxies.
- TAM uses continuous relaxation for end-to-end differentiable training and directly optimizes the few-shot objective.
- Approximately 8% top-1 accuracy improvement over mean pooling is reported in the single-example setting across Kinetics and Something-Something V2.
2. Related Work
Prior few-shot video methods often collapse frame order or rely on representations and models requiring substantial data. TAM instead preserves temporal order and uses differentiable sequence alignment to support end-to-end learning.
- Few-shot learning methods commonly address scarce novel-class data through learned initializations, novel optimizers, or similarity-based recognition.
- CMN encodes videos into fixed-size matrix representations and classifies them by matching and ranking stored representations.
- Previous few-shot video methods collapse frame order, making them sub-optimal for datasets where sequence order is important.
- Many video representation methods depend on large-scale datasets and can fail when only a small number of training samples are available.
- Temporal relational models capture longer-term information but still pool or fuse frames into a single video feature vector.
- Discrete dynamic-programming alignment finds optimal paths but is non-differentiable, motivating continuous relaxation for end-to-end learning.
3. Methods
TAM formulates few-shot video classification as learning a distance function over temporally ordered video representations, aligning query and support sequences despite non-linear temporal variation. It uses differentiable dynamic-time-warping-style alignment and selects the support video with minimum distance.
- 3.1. Problem Formulation: Few-shot video classification aims to recognize novel classes from small support sets and generalize across new episodes.Episodes contain n novel classes, k labeled examples per class, and query videos to classify against the support set.
- 3.2. Model: TAM represents each video as a sequence of T snippet features, preserving temporal information rather than reducing the video to one image-like embedding.A CNN processes snippets individually, producing an embedding with dimension T × D_f; sparse snippets span the whole video for long-term temporal modeling.
- 3.2. Model: For two videos, TAM constructs a T × T frame-level distance matrix from cosine distances between their embedded snippets.Each matrix entry measures the distance between one frame of the first video and one frame of the second.
- 3.2. Model: TAM finds an alignment path by minimizing the inner product between a binary alignment matrix and the frame-level distance matrix, using a dynamic-time-warping variant.The alignment path preserves continuity and monotonicity, while relaxed boundary conditions permit flexible starting and ending points through matrix padding.
- 3.2. Model: Figure 3 visualizes alignment scores as paths through distance matrices, where darker entries indicate smaller frame-pair distances and green-bordered entries contribute to the score.The figure compares alternative ways of extracting an alignment score from the matrix.
- 3.2. Model: Continuous relaxation makes the alignment computation differentiable, allowing the full model to be trained end-to-end with the few-shot objective.The resulting alignment score is naturally normalized because each path sums exactly T scores, and inference selects the support video with minimum distance.
4. Experiments
The experiments evaluate TAM for few-shot video classification on Kinetics and Something-Something V2 using episodic testing and comparisons with temporal-agnostic and temporal-aware baselines. TAM significantly outperforms the baselines, while qualitative and ablation studies examine temporal alignment and smoothing choices.
- Datasets and evaluation: Experiments use Kinetics and Something-Something V2 to evaluate few-shot video classification across two groups of action-recognition data.Kinetics follows the CMN split, while a few-shot split is constructed for Something-Something V2.
- Datasets and evaluation: Each n-way, k-shot episode samples n support classes with k examples per class and reports mean accuracy over 10,000 randomly sampled episodes.The query set contains n unlabeled examples, yielding n(k + 1) examples per episode.
- Few-shot classification results: TAM significantly outperforms all baselines on both datasets in 1-shot and 5-shot video classification.Table 1 reports 5-way video classification accuracy on the meta-testing set.
- Qualitative results: Qualitative results show TAM can find a lower alignment score for the true support class when visual frame-level clues are insufficient.Compared with CMN, TAM uses temporal alignment to distinguish visually similar action clips.
- Ablation study: Adaptive temporal alignment improves baselines by more than 3% across 1-shot and 5-shot settings on both datasets.Diagonal is especially helpful on Something-Something V2, where individual frames contain fewer visual clues; the difference is smaller on Kinetics.
- Ablation study: Performance is generally stable across smoothing factors, with λ values from 0.05 to 0.1 working relatively well on both datasets.Very small λ values produce imbalanced gradients, whereas large values can oversmooth alignment differences.
5. Conclusion
TAM is a few-shot video framework designed to handle non-linear temporal variations with very little data. It dynamically aligns video sequences while preserving temporal ordering and achieves state-of-the-art results on two challenging real-world datasets.
- Conclusion: TAM learns distance measures and representations that are independent of non-linear temporal variations using very few data.The framework dynamically aligns two video sequences while preserving temporal ordering.
- Conclusion: Continuous relaxation makes TAM differentiable and allows end-to-end optimization for the few-shot learning objective.This design distinguishes TAM from approaches that do not directly optimize the few-shot objective end to end.
- Conclusion: TAM significantly outperforms a wide range of competitive baselines and achieves state-of-the-art results on Kinetics and Something-Something V2.The conclusion is supported by the paper’s reported results and ablations.