Source-linked AI summary
Timeception for Complex Action Recognition
Noureldien Hussein, Efstratios Gavves, Arnold W. M. Smeulders
TL;DR
The paper addresses the limited modeling of long-range, complex actions whose components have variable durations and weak temporal order. It introduces Timeception, combining efficient temporal-only convolutions with multi-scale kernels. Timeception models 1024 timesteps—eight times longer than the best related work—and performs effectively across Charades, Breakfast Actions, and MultiTHUMOS while tolerating temporal-extent variation.
Problem
Short-range action recognition dominates prior work, while complex actions require modeling minute-long temporal order and variable component durations.
Method
Timeception uses efficient temporal-only, depthwise-separable convolutions and multi-scale temporal kernels for complex-action recognition.
Results
Timeception models 1024 timesteps, eight times longer than the best related work, and achieves effective recognition on Charades, Breakfast Actions, and MultiTHUMOS.
Takeaways & Limitations
Timeception learns long-range temporal dependencies and tolerates variations in the temporal extents of complex actions.
Abstract
from arXiv · showhide
This paper focuses on the temporal aspect for recognizing human activities in videos; an important visual cue that has long been undervalued. We revisit the conventional definition of activity and restrict it to Complex Action: a set of one-actions with a weak temporal pattern that serves a specific purpose. Related works use spatiotemporal 3D convolutions with fixed kernel size, too rigid to capture the varieties in temporal extents of complex actions, and too short for long-range temporal modeling. In contrast, we use multi-scale temporal convolutions, and we reduce the complexity of 3D convolutions. The outcome is Timeception convolution layers, which reasons about minute-long temporal patterns, a factor of 8 longer than best related works. As a result, Timeception achieves impressive accuracy in recognizing the human activities of Charades, Breakfast Actions, and MultiTHUMOS. Further, we demonstrate that Timeception learns long-range temporal dependencies and tolerate temporal extents of complex actions.
1. Introduction
The paper distinguishes long, heterogeneous complex actions from short one-actions and targets their underexplored recognition through minute-long temporal modeling. Timeception combines multi-scale temporal kernels with temporal-only convolutions to handle varying component durations and long temporal ranges.
- Motivation: Complex actions combine several one-actions with weak temporal order, varied durations, and longer, heterogeneous unfolding.Examples include cooking a meal and cleaning the house, which must be sampled fully to avoid missing crucial parts.
- Research gap: Recent action-recognition research has focused mainly on short-range actions, leaving long-range complex actions comparatively understudied.The paper studies datasets such as Charades and EventNet for this setting.
- Challenges: Minute-long modeling requires attention to seconds-long details, but existing methods reach only 8 or 128 timesteps and scaling to 1000 timesteps is computationally costly.Statistical pooling also falls short because it does not learn temporal order.
- Challenges: Fixed-size spatiotemporal kernels are too rigid for complex actions whose component durations and temporal orders vary.The paper identifies tolerance to temporal extent and order as a second central challenge.
- Approach: Timeception introduces multi-scale temporal kernels to model component-duration variation and temporal-only convolutions better suited to complex actions.The design reduces reliance on full spatiotemporal convolution while targeting long-range temporal structure.
- Approach: Timeception layers learn minute-long action ranges of 1024 timesteps, eight times longer than the best related work.The paper presents this range as an efficient way to model complex actions.
2. Related Work
Prior temporal-modeling methods either pool or aggregate features without temporal patterns or model only short temporal footprints. The paper decomposes spatiotemporal convolutions into depthwise-separable temporal convolutions and adds multi-scale kernels for long-range, variable-duration actions.
- Temporal Modeling: Statistical pooling and vector aggregation summarize video features but completely neglect temporal patterns.Examples include max and average pooling, attention pooling, rank pooling, dynamic images, context gating, Fisher Vector, and VLAD extensions.
- Temporal Modeling: Neural temporal methods such as LSTMs and TA-DenseNet model temporal information, but recent substantial improvements have not been reported.The paper presents these methods as part of the broader temporal-modeling literature.
- Long-range modeling: Existing methods model at most 128 timesteps, whereas this work addresses complex actions with long-range dependencies of up to 1024 timesteps.The target temporal range is motivated by complex actions rather than short-range action benchmarks.
- Convolution Decomposition: Spatiotemporal convolution cost grows substantially at higher layers as channel counts increase, motivating convolution decomposition.The related work reviews separable, grouped, channel-reduced, and channel-shuffled convolution designs.
- Contribution: The paper decomposes spatiotemporal convolutions into depthwise-separable temporal convolutions for long-range modeling and uses multi-scale kernels for differing temporal extents.It reports these temporal convolutions as better suited than 2+1D convolutions for long-range temporal modeling.
3. Method
Timeception decomposes video processing into efficient temporal-only convolutions that model long-range dependencies and varying temporal extents in complex actions. It combines multi-scale kernels, channel grouping and shuffling, and stacked layers on CNN backbones to encode patterns across up to 1024 timesteps.
- Timeception layer: Timeception splits features into channel groups, applies temporal convolution modules, concatenates their outputs, and shuffles channels to recover cross-channel correlations efficiently.Each module combines temporal convolutions, max-pooling, channel reduction, and concatenation across the channel dimension.
- Long-range dependencies: Stacking temporal layers enables complex and abstract patterns to emerge across temporally distant one-actions throughout long videos.The layer is designed to learn dependencies across sequences of up to 1000 timesteps while tolerating variation in component durations.
- Temporal decomposition: Timeception separates temporal modeling from spatial processing, using temporal-only kernels after a CNN backbone to reduce convolutional complexity.The preceding CNN remains responsible for detecting spatial patterns, while Timeception focuses on temporal combinations.
- Computational efficiency: 1024 timesteps, approximately 40 seconds of video, are computationally feasible because a Timeception layer has parameter complexity approximately O(T + logL).This supports deeper temporal stacks for minute-long temporal modeling.
- Multi-scale temporal kernels: Multi-scale temporal kernels use either kernel sizes k ∈{1, 3, 5, 7} with d = 1 or dilation rates d ∈{1, 2, 3} with k = 3.These alternatives are designed to tolerate variation in the temporal extents of one-actions.
- The Final Model: The final model stacks four Timeception layers on a spatial 2D or short-range spatiotemporal 3D CNN backbone before classification.The implementation supports ResNet-152 and I3D backbone choices, with Timeception and MLP layers fine-tuned after backbone pretraining.
4. Experiments
Experiments on complex-action datasets test temporal-extent robustness, multi-scale kernels, long-range modeling, and efficiency. Timeception improves backbone performance while supporting temporal reasoning over up to 1024 timesteps and adapting to dynamic action durations.
- Datasets: Experiments use Charades, Breakfast Actions, and MultiTHUMOS to evaluate complex actions involving composition, temporal extent, and temporal order.Charades videos average 30 seconds and contain six one-actions; Breakfast Actions videos average 2.3 minutes.
- Tolerating Temporal Extents: Timeception tolerates unexpected temporal-extent variations better than fixed-size kernels on Charades with both I3D and ResNet backbones.Test videos are altered by repeating or dropping frames within segments at four granularities.
- Temporal Kernels: Multi-scale kernels outperform fixed-size kernels for complex-action modeling with both I3D and ResNet, while varying dilation rates produces little to no performance change.The comparison includes different kernel sizes and different dilation rates.
- Long-range Temporal Dependencies: 1024 timesteps are processed using Timeception on I3D, and stacking Timeception layers improves accuracy while controlling parameter growth.Four Timeception layers on I3D processing 1024 timesteps require half the parameters of a ResNet processing 128 timesteps.
- Learned Temporal Representations: At lower layers, long kernels capture fine-grained dependencies, whereas at higher layers they focus on coarse-grained correlations; higher layers also combine short- and long-range patterns.The learned weights indicate that both video-wide and local temporal reasoning remain relevant at the top layer.
- Effectiveness: Timeception maintains a reasonable parameter increase with depth, improves Charades mAP, and yields absolute gains of 8.8% on ResNet and 4.3% on I3D.It scales temporal capacity while maintaining the overall model size.
5. Conclusion
Timeception is a temporal convolution layer for complex action recognition that scales to minute-long modeling and accommodates varying temporal extents. It improves state-of-the-art performance on popular benchmarks.
- Timeception uses efficient temporal-only convolutions to scale complex action recognition to minute-long temporal modeling.The approach processes temporal patterns while reducing the complexity associated with spatiotemporal convolutions.
- Multi-scale temporal convolutions allow Timeception to tolerate changes in the temporal extents of complex actions.Complex actions may contain component actions with differing durations.
- Earlier Timeception layers learn fast temporal changes, whereas later layers focus on more global temporal transitions.This pattern emerges from visualizing the learned temporal weights.
- Timeception notably improves state-of-the-art accuracy on popular action-recognition benchmarks.