Source-linked AI summary
Temporal Relational Reasoning in Videos
Bolei Zhou, Alex Andonian, Aude Oliva, Antonio Torralba
TL;DR
Video activity recognition requires reasoning about transformations and temporal relations that CNNs often miss. The paper introduces TRN, which combines ordered frame features across multiple timescales, and reports competitive results across three temporal-reasoning datasets, including nearly 95% Jester Top1 accuracy.
Problem
CNNs struggle to reason about temporal transformations beyond object appearance, especially when observations are limited.
Method
TRN is a simple, interpretable, plug-and-play module that learns relations among ordered video-frame representations at multiple timescales.
Results
TRN-equipped networks achieve competitive results on Something-Something, Jester, and Charades using discrete frames; MultiScale TRN reaches 95.31% Jester Top1 accuracy.
Takeaways & Limitations
TRN-equipped networks discover interpretable visual common-sense knowledge in videos while enabling temporal relational reasoning for activity recognition.
Takeaways & Limitations
The experiments fix the base network architecture throughout to evaluate the TRN module’s effectiveness.
Abstract
from arXiv · showhide
Temporal relational reasoning, the ability to link meaningful transformations of objects or entities over time, is a fundamental property of intelligent species. In this paper, we introduce an effective and interpretable network module, the Temporal Relation Network (TRN), designed to learn and reason about temporal dependencies between video frames at multiple time scales. We evaluate TRN-equipped networks on activity recognition tasks using three recent video datasets - Something-Something, Jester, and Charades - which fundamentally depend on temporal relational reasoning. Our results demonstrate that the proposed TRN gives convolutional neural networks a remarkable capacity to discover temporal relations in videos. Through only sparsely sampled video frames, TRN-equipped networks can accurately predict human-object interactions in the Something-Something dataset and identify various human gestures on the Jester dataset with very competitive performance. TRN-equipped networks also outperform two-stream networks and 3D convolution networks in recognizing daily activities in the Charades dataset. Further analyses show that the models learn intuitive and interpretable visual common sense knowledge in videos.
1 Introduction
Temporal relational reasoning links entities and transformations across time, but remains difficult for neural networks. The paper addresses this gap with TRN, evaluated on activities that require temporal relations across multiple timescales.
- Temporal relational reasoning links observations across time to infer transformations and support intelligent decision-making.
- Activity recognition requires both short-term and long-term temporal relations, as illustrated by sprinting’s movements and event stages.
- Many existing video benchmarks can be recognized from still-frame appearance or optical flow without long-term temporal reasoning.
- CNNs struggle when observations are limited and activities depend on transformations rather than entity appearance.
- TRN is a simple, interpretable, plug-and-play module that learns temporal relations at multiple timescales from video observations.
- The paper evaluates TRN-equipped networks on Something-Something, Jester, and Charades, which require temporal relational reasoning.
2 Temporal Relation Networks
Temporal Relation Networks combine ordered frame features into learned temporal relations at multiple scales. Efficient sampling and shared end-to-end training make the module practical for CNN-based video recognition.
- TRN is designed as a simple, interpretable module that plugs into existing CNN architectures to enable temporal relational reasoning.
- Defining Temporal Relations: Pairwise temporal relations apply learned functions to representations of selected, ordered video frames.
- Defining Temporal Relations: For efficient computation, TRN uniformly samples and sorts frame pairs instead of summing over every possible pair.
- Defining Temporal Relations: Higher-order relations extend the same composite formulation from two frames to three or more ordered frames.
- Defining Temporal Relations: Multi-scale TRN accumulates relation terms across frame counts, with each term modeling temporal relationships at a distinct scale.
- The relation functions are end-to-end differentiable and train jointly with the CNN, while subsampling reduces computation and cached features support streaming inference.
3 Experiments
Experiments evaluate TRN-equipped networks across three datasets designed to require temporal relational reasoning, using a controlled CNN backbone. Results show strong gains over single-frame and temporal-order-agnostic baselines, competitive cross-dataset performance, and interpretable temporal evidence.
- 3 Experiments: TRN-equipped networks outperform a baseline without TRN by a large margin on temporal-relational activity-recognition tasks across Something-Something, Jester, and Charades.The evaluation covers human-object interaction, hand-gesture, and daily-activity recognition.
- 3.1 Datasets: The Something-Something dataset contains 174 human-object interaction classes whose distinctions depend on object transformations and temporal relations rather than appearance alone.Examples include tearing something into two pieces versus tearing it just a little, and turning something upside down versus pretending to do so.
- 3.2 Results on Something-Something Dataset: 42.01% and 55.52% validation accuracy are achieved by the 2-stream TRN on Something-V1 and Something-V2, respectively.The optical stream uses 8-frame TSN average pooling because it performed better than the proposed temporal relational pooling in that stream.
- 3.2 Results on Something-Something Dataset: TRN models outperform TSNs by a large margin, while adding more frames to the relation produces further significant improvements.TSN averages deep features and captures co-occurrence rather than temporal ordering, whereas TRN performance benefits from additional relation frames.
- 3.3 Results on Jester and Charades: 95.31% validation Top1 accuracy is achieved by MultiScale TRN on Jester, and MultiScale TRN outperforms methods including 2-stream networks and C3D on Charades.Jester evaluates 27 hand-gesture classes; Charades evaluates daily activities.
- 3.4 Interpreting Visual Common Sense Knowledge inside the TRN: Qualitative and feature analyses indicate that TRNs use multi-scale temporal ordering, distinguish activity categories, and select representative frames that capture an activity’s essence.The analyses include reversed-order predictions, shuffled-frame accuracy differences, t-SNE activity embeddings, and representative-frame rankings.
4 Conclusion
The paper proposes TRN as an interpretable module for temporal relational reasoning in videos and evaluates it across several recent datasets. The authors report competitive results using discrete frames and show that TRN modules discover visual common sense knowledge.
- TRN is a simple, interpretable network module designed to enable temporal relational reasoning in neural networks for videos.
- The authors evaluated TRN on several recent video datasets using only discrete frames and established competitive results.
- The analyses showed that TRN modules discover visual common sense knowledge in videos.