Source-linked AI summary
Temporal Pyramid Network for Action Recognition
Ceyuan Yang, Yinghao Xu, Jianping Shi, Bo Dai, Bolei Zhou
TL;DR
Action recognition often misses variation in visual tempo, while input-level frame pyramids require costly multi-branch processing. The paper proposes a feature-level Temporal Pyramid Network that integrates with 2D or 3D backbones, yielding consistent improvements across benchmarks, including a 2% gain for I3D-R50 on Kinetics-400 and larger gains for classes with greater tempo variance.
Problem
Existing action-recognition designs often miss visual-tempo variation, while input-level frame pyramids require computationally expensive multiple network branches.
Method
TPN forms a feature-level temporal pyramid by modulating and aggregating hierarchical features, and integrates with existing 2D or 3D backbones in a plug-and-play manner.
Results
TPN brings consistent improvements across three action-recognition benchmarks; I3D-R50 + TPN reaches 77.7% top-1 accuracy on Kinetics-400, and I3D-R50 gains 2%.
Takeaways & Limitations
TPN's gains are concentrated on action classes with larger visual-tempo variance, supporting feature-level aggregation as a way to capture varied tempos within one model.
Abstract
from arXiv · showhide
Visual tempo characterizes the dynamics and the temporal scale of an action. Modeling such visual tempos of different actions facilitates their recognition. Previous works often capture the visual tempo through sampling raw videos at multiple rates and constructing an input-level frame pyramid, which usually requires a costly multi-branch network to handle. In this work we propose a generic Temporal Pyramid Network (TPN) at the feature-level, which can be flexibly integrated into 2D or 3D backbone networks in a plug-and-play manner. Two essential components of TPN, the source of features and the fusion of features, form a feature hierarchy for the backbone so that it can capture action instances at various tempos. TPN also shows consistent improvements over other challenging baselines on several action recognition datasets. Specifically, when equipped with TPN, the 3D ResNet-50 with dense sampling obtains a 2% gain on the validation set of Kinetics-400. A further analysis also reveals that TPN gains most of its improvements on action classes that have large variances in their visual tempos, validating the effectiveness of TPN.
1. Introduction
Visual tempo varies across and within action classes and can be important for distinguishing visually similar actions. TPN addresses this challenge by aggregating multi-tempo information at the feature level instead of using costly input-level frame pyramids.
- Visual tempo describes how fast an action occurs and determines its effective duration at the temporal scale for recognition.
- Differences in visual tempo can distinguish visually similar actions such as walking, jogging, and running.
- Age, mood, energy level, and body weight contribute to intra-class variation in how quickly people perform the same action.
- Input-level frame pyramids sample videos at multiple rates and process each level with a separate backbone subnetwork to combine fast- and slow-tempo information.For example, sampling 64 frames at intervals 16 and 2 produces pyramid levels with 4 and 32 frames.
- TPN aggregates varied-tempo information at feature level, uses a single-rate input, and can be added plug-and-play to existing action-recognition models.The approach leverages feature hierarchies formed at different network depths and is intended for both 2D and 3D architectures.
- TPN improves both 2D and 3D networks consistently across Kinetics-400, Something-Something V1 and V2, and Epic-Kitchen.Its ablation analysis attributes most gains to action classes with significant visual-tempo variance.
2. Related Work
Video action-recognition methods commonly use either 2D per-frame processing with temporal aggregation or 3D convolutions for joint spatial-temporal modeling. Recent visual-tempo methods use input-level frame pyramids, but their separate backbone branches can be computationally expensive.
- Two-stream networks apply separate 2D CNNs to visual appearances and optical flows, then aggregate per-frame features temporally.
- TSN represents video clips by sampling frames from evenly divided temporal segments.
- 3D CNNs stack three-dimensional convolutions to jointly model temporal and spatial semantics.
- Input-level visual-tempo methods predefine sampling rates and process each frame-pyramid level with a separate backbone network.This design becomes computationally expensive as the number of pyramid levels increases.
3. Temporal Pyramid Network
TPN models visual tempos at the feature level by building and fusing a hierarchy of backbone features with aligned spatial semantics and calibrated temporal rates. It supports multiple feature sources and aggregation flows within a single, plug-and-play network.
- Motivation: Visual tempo varies across action instances and affects the temporal scale needed for recognition, including distinctions such as walking, jogging, and running.
- Motivation: Input-level frame pyramids require separate backbone branches for different sampling rates, making them computationally expensive as pyramid levels increase.
- Feature Source: TPN collects hierarchical features from a backbone, either by sampling one depth at multiple temporal rates or by selecting features from multiple depths.
- Feature Source: Spatial semantic modulation aligns multi-depth feature shapes and semantics using level-specific convolutions and auxiliary classification heads.
- Feature Source: Temporal rate modulation downsamples each updated feature by a controllable factor α_i, calibrating relative temporal scales for more effective aggregation.
- Information Flow: Information flow aggregates adjacent features through bottom-up, top-down, cascade, or parallel flows with temporal resampling, after which aggregated levels are rescaled and concatenated for prediction.
4. Experiments
TPN is evaluated across multiple action-recognition datasets, backbones, sampling schemes, and ablations. Results show consistent gains, with performance depending on feature sources, information flow, modulation components, and visual-tempo variation.
- Experimental Setup: TPN is evaluated on Kinetics-400, Something-Something V1 and V2, and Epic-Kitchen using both 2D and 3D backbones.Experiments use RGB-only inputs and include ablation studies and empirical analyses.
- Kinetics-400: 77.7% top-1 accuracy is achieved by TPN-R50 on Kinetics-400, exceeding methods with the same depth.TPN-R101 reaches 78.9% with 32×2 input, surpassing methods using the same number of input frames.
- Something-Something: TPN produces consistent improvements on TSN and TSM for Something-Something, while TSM-10116f with TPN reaches 67.7% Top-1 accuracy.The leaderboard result is dated 04/10/2020 and uses full resolution with two clips.
- Epic-Kitchen: 12.9% improvement is observed for verb classification on Epic-Kitchen when TSN is equipped with TPN.The comparison follows the Epic-Kitchen settings used in the cited baseline.
- Ablation Study: The parallel flow obtains 76.1%, while the isolation flow still improves performance by 0.58% under suitable feature modulations.The ablation indicates that features with different temporal receptive fields can help even when drawn from one backbone.
5. Conclusion
The paper proposes Temporal Pyramid Network as a generic module for capturing visual tempos in action instances, with plug-and-play use in 2D and 3D architectures.
- Temporal Pyramid Network captures the visual tempos of action instances.
- TPN is a feature-level module that can be applied to existing 2D and 3D architectures in a plug-and-play manner.
- TPN brings consistent improvements when combined with both 2D and 3D networks.
- Ablation analyses find that TPN obtains most improvements from action classes with significant variances in visual tempos.