Source-linked AI summary
Spatiotemporal Residual Networks for Video Action Recognition
Christoph Feichtenhofer, Axel Pinz, Richard P. Wildes
TL;DR
Video action recognition needs models that capture both appearance and motion while extending powerful image-based architectures into time. The paper introduces spatiotemporal ResNets with cross-stream and temporal residual connections, and reports state-of-the-art performance on two action-recognition datasets.
Problem
Video actions are spatiotemporal, but spatial appearance and temporal motion must be modeled and registered across time.
Method
The paper combines two-stream ConvNets with ResNets by adding cross-stream residual connections and temporal convolutions initialized as residual filters.
Results
The model achieves state-of-the-art performance on two popular action-recognition datasets and improves over the original two-stream method by 5.4% on UCF101 and 7% on HMDB51.
Takeaways & Limitations
End-to-end spatiotemporal ResNets support hierarchical learning of spacetime features while transferring pretrained image ConvNets into video models.
Abstract
from arXiv · showhide
Two-stream Convolutional Networks (ConvNets) have shown strong performance for human action recognition in videos. Recently, Residual Networks (ResNets) have arisen as a new technique to train extremely deep architectures. In this paper, we introduce spatiotemporal ResNets as a combination of these two approaches. Our novel architecture generalizes ResNets for the spatiotemporal domain by introducing residual connections in two ways. First, we inject residual connections between the appearance and motion pathways of a two-stream architecture to allow spatiotemporal interaction between the two streams. Second, we transform pretrained image ConvNets into spatiotemporal networks by equipping these with learnable convolutional filters that are initialized as temporal residual connections and operate on adjacent feature maps in time. This approach slowly increases the spatiotemporal receptive field as the depth of the model increases and naturally integrates image ConvNet design principles. The whole model is trained end-to-end to allow hierarchical learning of complex spatiotemporal features. We evaluate our novel spatiotemporal ResNet using two widely used action recognition benchmarks where it exceeds the previous state-of-the-art.
1 Introduction
Video action recognition treats actions as spatiotemporal objects, motivating methods that combine spatial and temporal processing. This work brings residual-network design principles into two-stream video ConvNets through temporal transformations and cross-stream connections.
- Video action recognition applies ConvNets to model spatial and temporal information in human actions.
- The method carries image-domain ConvNet design practices into video by transforming ResNet dimensionality-mapping filters into temporal filters.Stacking these filters provides a larger receptive field for later discriminative units.
- The approach converts spatial ConvNets into spatiotemporal models and leverages training data from image datasets such as ImageNet.
- The architecture builds on separate spatial appearance and temporal motion streams for recognizing input sequences.The streams process RGB appearance and motion-related information separately before interaction.
- Residual connections are introduced between the two streams to enable their spatiotemporal interaction.The figure indicates that residuals are not directed from the spatial stream into the temporal stream because this would bias both losses toward appearance information.
2 Related work
Related work spans hand-crafted features, jointly learned ConvNet representations, spatiotemporal extensions, and recurrent sequence models. The closest foundation is the two-stream ConvNet, which separates RGB and optical-flow processing before late fusion.
- Video action-recognition methods include hand-crafted features with decoupled classifiers and models that jointly learn features and classifiers.
- Prior spatiotemporal feature-learning methods used unsupervised models, stacked video frames, and 3D ConvNets with limited temporal support.Some studies found temporal modeling relatively insensitive, with purely spatial networks reaching similar performance in one comparison.
- Recurrent approaches combined ConvNet features with LSTM or related recurrent architectures for ordered sequence modeling and spatial attention.The cited work reports that LSTMs did not improve over temporal pooling in one comparison, while another recurrent approach yielded only a minor baseline increase.
- The two-stream ConvNet decomposes video into RGB and optical-flow components processed by separate deep networks, then combines their softmax scores by late fusion.
- The proposed work directly converts image ConvNets into 3D architectures and reports improved performance over the two-stream baseline.
3 Technical approach
The architecture combines two-stream ResNets with cross-stream residual connections and learnable temporal convolutions. These additions enable spatiotemporal interaction while progressively expanding temporal receptive fields and preserving end-to-end training.
- 3.1 Two-Stream residual networks: The model uses deep ResNet-50 streams with residual units, batch normalization, ReLU activations, and global average pooling for video classification.The network receives 224×224 inputs and reduces spatial resolution through stride-2 convolutions before classification.
- 3.1 Two-Stream residual networks: Cross-stream residual connections inject motion features into the appearance stream at multiple spatial resolutions, enabling learned spatiotemporal interaction.The connections are inserted before the second residual unit at each spatial resolution, including the conv5_x layers.
- 3.1 Two-Stream residual networks: The cross-stream skip connection also propagates gradients from the appearance stream into the motion stream during backpropagation.This adds an appearance-stream contribution to the motion-stream gradient.
- 3.2 Convolutional residual connections across time: Temporal residual connections transform spatial dimensionality-mapping filters into learnable temporal filters initialized to average feature responses across time.The transformed filters operate in both appearance and motion ResNets and are refined by backpropagation.
4 Evaluation
The ST-ResNet is evaluated on UCF101 and HMDB51 through staged training and comparisons with prior action-recognition methods. It improves over two-stream baselines and achieves best reported performance on both datasets.
- Evaluation datasets: UCF101 contains 13,320 videos across 101 action classes, while HMDB51 contains 6,766 videos across 51 actions and is generally more challenging.Both datasets are evaluated using their provided protocols.
- Two-Stream ResNet with additive interactions: 89.47% on UCF101 and 60.59% on HMDB51 are achieved by the initial two-stream ResNet50 baseline without cross-stream residual or temporal convolutional connections.This baseline uses ImageNet-initialized ResNet50 models for both streams.
- Two-Stream ResNet with additive interactions: Around 1% accuracy improvement on both datasets results from adding temporal max-pooling to ST-ResNet.The model uses 25 temporal chunks during testing, enabling larger temporal windows.
- Comparison with the state-of-the-art: 5.4% on UCF101 and 7% on HMDB51 are the improvements over the original two-stream method.The comparison uses all three splits and fully convolutional testing in a single forward pass.
- Comparison with the state-of-the-art: Adding trajectory-based IDT features further boosts ST-ResNet accuracy on HMDB51, but produces a smaller improvement on UCF101.The authors attribute some benefit to explicit compensation for camera motion.
5 Conclusion
The paper presents spatiotemporal ResNets for video action recognition by combining two-stream pathways with residual architectures. The resulting end-to-end model learns hierarchical spacetime features and achieves state-of-the-art performance on two popular datasets.
- Conclusion: ST-ResNet connects appearance and motion channels in a two-stream architecture to support hierarchical learning of spacetime features.Residual connections enable interaction between the two streams.
- Conclusion: The model transforms dimensionality-mapping filters in pretrained spatial networks into temporal convolutions initialized as residual filters over time.This transfers both streams from the spatial to the spatiotemporal domain.
- Conclusion: The whole system is trained end-to-end and achieves state-of-the-art performance on two popular action-recognition datasets.The conclusion presents this as the central empirical outcome.