Source-linked AI summary
TARN: Temporal Attentive Relation Network for Few-Shot and Zero-Shot Action Recognition
Mina Bishay, Georgios Zoumpourlis, Ioannis Patras
TL;DR
Few-shot and zero-shot action recognition must handle scarce labels and comparisons between videos or videos and semantic representations. TARN uses attention-based temporal alignment and segment-level deep metric learning, achieving state-of-the-art few-shot results and competitive zero-shot performance without target-task fine-tuning or additional memory resources.
Problem
Action recognition for new classes is difficult because annotated examples are scarce, while video methods must also handle temporal structure and variable-length representations.
Method
TARN embeds videos or semantic representations, aligns segment-level representations with attention, and learns deep relation scores for few-shot and zero-shot matching.
Results
TARN achieves state-of-the-art results in few-shot action recognition and very competitive zero-shot performance across benchmark evaluations.
Takeaways & Limitations
Attention and segment-wise comparison benefit video-to-video and video-to-vector matching compared with video-wise representations, without additional resources or target-problem fine-tuning.
Takeaways & Limitations
The zero-shot comparison excludes methods using auxiliary data, fused features, or transductive access to unseen test classes, and uses attributes and/or word vectors as semantic representations.
Abstract
from arXiv · showhide
In this paper we propose a novel Temporal Attentive Relation Network (TARN) for the problems of few-shot and zero-shot action recognition. At the heart of our network is a meta-learning approach that learns to compare representations of variable temporal length, that is, either two videos of different length (in the case of few-shot action recognition) or a video and a semantic representation such as word vector (in the case of zero-shot action recognition). By contrast to other works in few-shot and zero-shot action recognition, we a) utilise attention mechanisms so as to perform temporal alignment, and b) learn a deep-distance measure on the aligned representations at video segment level. We adopt an episode-based training scheme and train our network in an end-to-end manner. The proposed method does not require any fine-tuning in the target domain or maintaining additional representations as is the case of memory networks. Experimental results show that the proposed architecture outperforms the state of the art in few-shot action recognition, and achieves competitive results in zero-shot action recognition.
1 Introduction
The paper targets few-shot and zero-shot video action recognition, where annotated examples for new action classes are scarce. TARN addresses this by comparing segment-level representations and reports strong performance without additional memory resources or target-task fine-tuning.
- Motivation: Few-shot and zero-shot learning address action recognition when new classes have only a few labelled examples or an action-related description.Deep architectures typically require large annotated datasets, while humans can recognize new actions from sparse supervision.
- Research gap: Video-based few-shot and zero-shot recognition is harder than image-based recognition because temporal variation adds an additional dimension.Prior video methods also used memory networks or a single embedding vector, which may inadequately capture temporal structure.
- Approach: TARN compares segment-wise visual features between videos for few-shot recognition and between video segments and class-wise semantic representations for zero-shot recognition.The architecture retains temporal information to produce video-wise classification.
- Contribution: TARN requires no additional resources such as memory networks and no training or fine-tuning on the target problem.The authors evaluate it on benchmark datasets for both few-shot and zero-shot action recognition.
- Results: The method achieves state-of-the-art results in few-shot action recognition and very competitive performance in zero-shot action recognition.This is the paper’s reported benchmark-level outcome across the two tasks.
2 Related work
Related work spans few-shot and zero-shot learning, sequence matching, and video-action recognition. The paper positions temporal segment matching as a way to address limitations of single-vector video representations and fixed similarity measures.
- Few-shot learning: Few-shot learning commonly uses meta-learning to acquire transferable knowledge from collections of tasks, including initial weights, embedding functions, or external memory.Similarity-based methods classify queries by comparing them with labelled examples, but many use fixed distance metrics.
- Zero-shot learning: Zero-shot action recognition bridges semantic class representations, such as attributes or word vectors, with visual representations of unseen classes.Existing approaches often average segment features into a single video vector before matching them to semantic embeddings.
- TARN architecture: In C-way K-shot recognition with K > 1, TARN averages the sample relation scores for each class to obtain the query-to-class relation score.The architecture comprises embedding and relation modules.
- Sequence matching: Sequence-matching research motivates comparing and aggregating parts of sequences rather than matching only whole-sequence representations.The paper draws on segment matching and attention-based alignment methods from video and text matching.
3 Proposed Architecture
TARN uses embedding and relation modules to compare variable-length video or semantic sequences at segment level. Attention aligns representations before a learned comparison and aggregation process produces relation scores.
- Architecture overview: TARN processes query videos against sample videos in few-shot learning or semantic attributes in zero-shot learning.Its output is a relation score for each video pair or video–semantic-attribute pair.
- Embedding module: The embedding module preserves temporal structure while encoding visual or semantic representations for later comparison.Video embeddings use C3D features from short segments followed by a bidirectional GRU; zero-shot learning uses separate visual and semantic embedding modules.
- Relation module: In few-shot learning, segment-by-segment attention aligns each sample video with the query so their segment embeddings have matching temporal length.The relation module pairs the query with each sample video before alignment.
- Temporal alignment: The attention matrix produces aligned sample representations whose rows are weighted sums of sample segments most similar to corresponding query segments.The aligned representations and query representations are then passed to a comparison layer.
- Deep metric learning: A comparison layer and nonlinear network learn a deep similarity or distance measure across segment comparisons.For few-shot learning, a unidirectional GRU captures temporal information across comparisons before an FC layer produces the relation score.
- Training and zero-shot extension: The architecture is trained end-to-end with episode-based training and binary cross-entropy, while zero-shot learning compares semantic vectors with query-video segments.The zero-shot relation network replaces the few-shot temporal GRU because there is no temporal alignment information to learn in that network.
4 Experimental results
Experiments evaluate TARN’s distance measures, temporal attention, segment-wise comparisons, and zero-shot settings across action-recognition benchmarks. TARN benefits from segment alignment and achieves strong few-shot and zero-shot results under the reported protocols.
- Few-shot experimental setting: The few-shot evaluation uses fixed 16-frame video segments, C3D FC7 features, bidirectional GRU embeddings, and episodic training with 20,000 training episodes.The sample set contains five classes per episode; validation and testing use 500 and 1,000 episodes, respectively.
- Distance-measure analysis: EucCos produces the highest TARN accuracy across all shot settings among the five tested comparison-layer measures.The other tested measures are Mult, Subt, NN, and SubMultNN; subsequent relation layers remain trainable and nonlinear.
- Few-shot ablations: Attention-based segment alignment and segment-wise embedding comparisons consistently outperform the TARN-single video-wise comparison across shots.The gains are especially substantial in the one-shot case, and TARN also exceeds the state-of-the-art method.
- Zero-shot experimental setting: The zero-shot experiments use UCF-101 and HMDB51 with attribute or word-vector semantics, while excluding auxiliary-data, feature-fusion, and transductive comparisons.These restrictions establish a common model-based comparison setting across the reported methods.
- Zero-shot ablations: Zero-shot ablations compare whole-video, aligned single-representation, and per-segment visual-to-semantic matching configurations on UCF-101 and HMDB51.The ablation study identifies multiple segment-to-attribute comparisons as the best-performing configuration.
- Zero-shot state-of-the-art comparison: Under the selected comparison setting, TARN leads UCF-101 51/50 and 81/20 splits by almost 0.5% and 3% over the second-best methods.On HMDB51, lower results occur only relative to methods using Improved Dense Trajectories or ECOC representations.
5 Conclusion
The conclusion presents TARN as a relation network for few-shot and zero-shot action recognition that aligns temporal representations and learns deep distances at segment level. The reported experiments associate these design choices with state-of-the-art few-shot and competitive zero-shot performance without target-task fine-tuning or additional resources.
- Architecture: TARN combines an embedding module with an attention-based relation module for temporal alignment and segment-level deep-distance learning.It matches video pairs for few-shot recognition and video-semantic representations for zero-shot recognition.
- Design findings: Using attention and segment-wise comparisons benefits video-to-video and video-to-vector matching relative to video-wise representations.This conclusion covers both few-shot and zero-shot matching scenarios.
- Overall results: TARN requires neither additional resources nor fine-tuning on the target problem, while achieving state-of-the-art few-shot and very competitive zero-shot results.The conclusion states these outcomes at the paper level without reporting specific benchmark values.