Source-linked AI summary
Temporal-Relational CrossTransformers for Few-Shot Action Recognition
Toby Perrett, Alessandro Masullo, Tilo Burghardt, Majid Mirmehdi, Dima Damen
TL;DR
Few-shot action recognition must learn new classes from only a handful of labeled examples, while video actions vary in temporal structure. TRX constructs query-specific prototypes by attending to ordered sub-sequences across all support videos with multiple tuple cardinalities. It achieves state-of-the-art results on four few-shot benchmarks, with tuple representations improving over single-frame comparisons on SSv2.
Problem
Few-shot action recognition seeks to learn new classes from only a handful of labeled examples, a challenge for fine-grained actions and temporally variable videos.
Method
TRX uses CrossTransformer attention to match ordered query frame tuples against sub-sequences from all support videos and combines models for different tuple cardinalities.
Results
TRX achieves state-of-the-art results on few-shot Kinetics, SSv2, HMDB51, and UCF101; tuple representations improve over single-frame comparisons by 5.8% on SSv2.
Takeaways & Limitations
Matching multiple support videos with ordered tuples captures temporal relations across different action speeds and temporal offsets.
Takeaways & Limitations
The method assumes tuples should be temporally ordered and, for simplicity, uniformly samples videos to the same length.
Abstract
from arXiv · showhide
We propose a novel approach to few-shot action recognition, finding temporally-corresponding frame tuples between the query and videos in the support set. Distinct from previous few-shot works, we construct class prototypes using the CrossTransformer attention mechanism to observe relevant sub-sequences of all support videos, rather than using class averages or single best matches. Video representations are formed from ordered tuples of varying numbers of frames, which allows sub-sequences of actions at different speeds and temporal offsets to be compared. Our proposed Temporal-Relational CrossTransformers (TRX) achieve state-of-the-art results on few-shot splits of Kinetics, Something-Something V2 (SSv2), HMDB51 and UCF101. Importantly, our method outperforms prior work on SSv2 by a wide margin (12%) due to the its ability to model temporal relations. A detailed ablation showcases the importance of matching to multiple support set videos and learning higher-order relational CrossTransformers.
1. Introduction
TRX addresses few-shot video recognition by matching temporally ordered sub-sequences across all support videos to build query-specific class prototypes. It models temporal relations at multiple tuple cardinalities and achieves state-of-the-art results across four few-shot benchmarks.
- Few-shot video recognition is challenging because fine-grained actions are difficult to label extensively, while actions vary in length and contain temporal dependencies.
- TRX compares each query sub-sequence with all support-set sub-sequences and aggregates the evidence into a query-specific class prototype.This contrasts with matching a single best support video or averaging all support videos within a class.
- Ordered frame tuples help match actions performed at different speeds and temporal offsets, supporting distinctions between fine-grained classes.
- TRX combines multiple CrossTransformers operating on different tuple cardinalities to exploit higher-order temporal relations, including pairs, triples, and quadruples.
- TRX achieves state-of-the-art results on few-shot Kinetics, Something-Something V2, HMDB51, and UCF101.
- 5.8% improvement on SSv2 is obtained by using tuple representations instead of single-frame comparisons, where temporal ordering is critical.
2. Related Work
Few-shot video recognition builds on metric, adaptation, generative, and temporal modeling approaches. TRX extends attention-based prototype construction to all support videos and ordered temporal tuples for higher-order relational matching.
- Few-shot classification methods include generative, adaptation-based, and metric-based approaches, with recent image methods combining metric learning and task-specific adaptation.
- CrossTransformer aligns query and support images through patch co-occurrences to create query-specific class prototypes before prototypical-network classification.
- Few-shot video methods use key-frame memory, feature generation, temporal alignment, spatial and temporal attention, and self-supervision.
- TRX differs from prior methods by attending to all support videos with ordered-tuple temporal-relational representations, matching actions across speeds and temporal shifts.
- Multiple CrossTransformers match tuples of different cardinalities, allowing TRX to represent higher-order temporal relations.
3. Method
TRX compares temporally ordered frame tuples between a query and all support videos, using separate CrossTransformers for different tuple cardinalities to build query-specific class prototypes and classify by accumulated distances.
- Temporal CrossTransformer: TRX constructs query-specific class prototypes by matching each query sub-sequence against all support-set sub-sequences and aggregating the resulting evidence.The method compares tuples rather than isolated frames, allowing multiple support videos to contribute to each prototype.
- Temporal CrossTransformer: Ordered frame pairs represent temporal relations, with query pairs compared against all possible support-video pairs to match actions at different speeds and locations.The pair representation uses frame embeddings plus positional encoding, and support pairs are formed from ordered frame indices.
- Temporal CrossTransformer: Multiple query pairs are evaluated because no single frame pair necessarily best represents an action; their distances are accumulated for classification.During inference, the class with the closest query-specific prototype is selected after matching is performed separately for each query pair.
- Temporal-Relational CrossTransformers: Higher-order tuples address cases where pairs are insufficient for fine-grained distinctions, such as separating picking an object up from moving an object.The paper motivates longer tuples because additional frames can capture whether an object is eventually put down or remains in hand.
- Temporal-Relational CrossTransformers: TRX generalizes pairs to ordered tuples of cardinality ω and uses one CrossTransformer per cardinality, such as pairs, triples, and quadruples.Each transformer has dimensionality-specific query, key, and value maps and outputs a distance for its tuple cardinality.
- Temporal-Relational CrossTransformers: Distances from the different TRX modules are accumulated, with averaging used within each module to balance the varying number of tuples.The model is trained jointly with a single cross-entropy loss, shared backbone parameters, and gradients from all tuple cardinalities.
4.1. Setup
The evaluation uses few-shot splits of four action-recognition datasets and focuses on standard 5-way 5-shot testing across randomly sampled tasks, with comparisons to prior methods.
- Datasets: The study evaluates Kinetics, Something-Something V2, HMDB51, and UCF101 using established few-shot splits.For the first two datasets, the setup selects 100 videos from 100 classes, divided into 64 training, 12 validation, and 24 test classes.
- Evaluation: The primary benchmark is 5-way 5-shot evaluation averaged over 10,000 randomly selected test tasks.The paper also includes an X-shot ablation with one-shot results for completeness.
- Baselines: Results are compared against four seminal and recent few-shot action-recognition methods that reported state-of-the-art performance.The comparisons are presented for the benchmark settings described in the evaluation setup.
- Implementation: Training uses SGD with learning rate 0.001 for 10,000 tasks, except for the larger SSv2* split, which uses 75,000 tasks.The model is trained on four Nvidia 2080Ti GPUs, with gradients averaged before backpropagation because a single task fits in memory.
4.2. Results
TRX outperforms prior work across four few-shot action-recognition datasets, with especially large gains on SSv2. Qualitative matches show that tuples can align actions across support videos, positions, and frame differences, though a failure case selects the wrong prototype.
- 12% and 10%: TRX outperforms prior work on SSv2 across different splits.SSv2 is identified as the most challenging dataset.
- 15% and 13%: TRX outperforms prior work on HMDB51 and UCF101, respectively.On Kinetics, TRX exceeds the state-of-the-art by 0.1%.
- TRX matches temporally ordered tuples across different support videos, positions, and frame differences.The qualitative examples use query pairs and triplets with Ω={2, 3}.
- In one failure case, the putting action matches correctly, but the query is assigned to the wrong prototype.The query about failing to put something into something because it does not fit is closest to the class “put something upright on the table.”
4.3. Ablations
The ablations show that ordered tuples, multiple tuple cardinalities, and multiple support videos are central to TRX, especially on temporally demanding SSv2. TRX also supports efficient variants through fewer sampled frames or randomly retained tuples, with dataset- and cardinality-dependent trade-offs.
- TRX with different Ω values: +4.5% improvement results from moving from single-frame to pair comparisons on SSv2.Triplets add +1.0% and quadruples add only +0.1%; combining Ω={2, 3} performs best.
- The impact of ordered tuples: Reversing query tuple order causes a -7.8% drop on SSv2, while Kinetics shows no drop.This supports the greater importance of temporal ordering for SSv2 than for Kinetics.
- Matching to multiple support set videos: TRX’s improvement over CMN grows with support-set size on SSv2, reaching +10.3% at 5-shot for Ω={2, 3}.The corresponding improvements are +3.9%, +7.3%, and +7.9% for 2-, 3-, and 4-shot settings.
- Matching to multiple support set videos: More than 50% of queries match tuples from 2–3 support videos, demonstrating that TRX uses multiple videos per class.The analysis considers maximum-attention tuple matches for pairs and quadruples on SSv2 and Kinetics.
- Runtime and tuple sampling: Accuracy remains comparable with at least 6 sampled frames, while runtime scales linearly with the number of sampled frames.The analysis varies sampled frames from 4 to 12 for Ω={2, 3} on the SSv2 5-way 5-shot task.
- Runtime and tuple sampling: Randomly retaining fewer tuples can achieve performance comparable to exhaustive selection while reducing GPU usage, but performance depends on cardinality.Performance degrades more quickly for Ω={2}, and selecting tuples that preserve performance remains future work.
5. Conclusion
The paper concludes that TRX constructs query-specific prototypes from ordered sub-sequences across all support videos, and that this design supports temporal matching across speeds and offsets. It reports state-of-the-art results across four few-shot action-recognition datasets and ablations supporting multiple videos, tuple representations, and varied cardinalities.
- Conclusion: TRX constructs query-specific class prototypes by comparing queries with sub-sequences from all support-set videos.Ordered frame tuples allow comparisons across different action speeds and temporal offsets.
- Conclusion: TRX achieves state-of-the-art results on Kinetics, SSv2, HMDB51, and UCF101 few-shot benchmarks.
- Conclusion: Ablations support matching multiple support videos, using tuples instead of single frames, and exploiting tuples with different cardinalities.
A. X-Shot results
The appendix examines TRX across shot counts and datasets, showing that its advantage is most evident when multiple support videos are available. Tuple-based TRX consistently outperforms frame-based TRX, while Kinetics benefits less from temporal modeling than SSv2.
- X-Shot results: TRX with Ω={2, 3} consistently outperforms Ω={1}, while TRX is designed specifically for K-shot problems where K > 1.
- X-Shot results: TRX scales better as the number of shots increases, with its margin over CMN widening on SSv2 and Kinetics.
B. The impact of positional encoding
Positional encoding provides small improvements for TRX, especially on SSv2, while its effect on Kinetics is limited. The appendix reports these comparisons for single-frame and higher-order tuple representations.
- The impact of positional encoding: +0.3% and +0.6% improvements on SSv2 result from positional encoding for single frames and higher-order tuples, respectively.
- The impact of positional encoding: On Kinetics, positional encoding leaves single-frame performance unchanged and improves tuple performance by +0.4%.Overall, the effect of positional encoding is marginal for TRX.