Source-linked AI summary
Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, Luc Van Gool
TL;DR
Video action recognition has not consistently surpassed traditional handcrafted methods, partly because common ConvNet approaches miss long-range temporal structure and rely on costly dense sampling. This paper introduces Temporal Segment Networks, which use sparse temporal sampling and video-level supervision to model entire videos efficiently, significantly outperforming the state of the art on UCF101 and HMDB51.
Problem
Video ConvNets have not achieved a significant advantage over handcrafted features and often lack efficient modeling of long-range temporal structure.
Method
Temporal Segment Networks model entire videos through sparse segment-based snippet sampling, segmental consensus, and video-level supervision.
Results
The method significantly outperformed the state of the art on the UCF101 and HMDB51 action-recognition datasets.
Takeaways & Limitations
TSN provides an effective and computationally reasonable framework for capturing long-term temporal structure while training deep networks on limited data.
Abstract
from arXiv · showhide
Deep convolutional networks have achieved great success for visual recognition in still images. However, for action recognition in videos, the advantage over traditional methods is not so evident. This paper aims to discover the principles to design effective ConvNet architectures for action recognition in videos and learn these models given limited training samples. Our first contribution is temporal segment network (TSN), a novel framework for video-based action recognition. which is based on the idea of long-range temporal structure modeling. It combines a sparse temporal sampling strategy and video-level supervision to enable efficient and effective learning using the whole action video. The other contribution is our study on a series of good practices in learning ConvNets on video data with the help of temporal segment network. Our approach obtains the state-the-of-art performance on the datasets of HMDB51 ( $ 69.4\% $) and UCF101 ($ 94.2\% $). We also visualize the learned ConvNet models, which qualitatively demonstrates the effectiveness of temporal segment network and the proposed good practices.
1 Introduction
The introduction identifies limited advantages of end-to-end ConvNets for video action recognition, chiefly because they emphasize appearances and short-term motion rather than long-range temporal structure. It motivates a video-level framework and training practices for learning effective ConvNet models from limited samples.
- Motivation: End-to-end deep ConvNets have not achieved a significant advantage over traditional methods for video-based action recognition.Although ConvNets learn discriminative representations from raw visual data, their success in image classification has not transferred comparably to video action recognition.
- Challenges: Long-range temporal structure is crucial for understanding action dynamics, but mainstream ConvNet frameworks mainly capture appearances and short-term motions.This limitation leaves them unable to incorporate long-range temporal structure effectively.
- Objectives: The study therefore targets an effective, efficient video-level representation-learning framework that captures long-range temporal structure and learns ConvNet models from limited training samples.The method is built on the two-stream architecture while addressing these two challenges.
- Training practices: The proposed training practices include cross-modality pre-training, regularization, and enhanced data augmentation.The paper adopts very deep ConvNet architectures and studies these practices to overcome difficulties caused by limited training samples.
- Experimental scope: The study empirically compares four input modalities for two-stream ConvNets: a single RGB image, stacked RGB difference, stacked optical flow field, and stacked warped optical flow field.Experiments are conducted on the UCF101 and HMDB51 action-recognition datasets.
- Results: Experiments on UCF101 and HMDB51 report that models learned using the temporal segment network significantly outperform the state of the art on both datasets.The authors also visualize the learned two-stream models to provide insights for future action-recognition research.
2 Related Works
Prior work on action recognition is organized around convolutional-network architectures and temporal-structure modeling. Existing temporal-structure methods could not assemble an end-to-end learning scheme, whereas TSN emphasizes this principle as the first such framework.
- Previous action-recognition work falls into two categories: convolutional networks and temporal structure modeling.
- Convolutional Networks for Action Recognition: ConvNet research explored deep architectures, two-stream networks using spatial and temporal nets, and 3D ConvNets for video action recognition.
- Temporal Structure Modeling: Temporal-structure research modeled atomic actions, latent temporal decompositions, dynamic-poselet relations, and temporal evolution of BoVW representations.
- Temporal Structure Modeling: Existing temporal-structure methods remained unable to assemble an end-to-end learning scheme for modeling temporal structure.
- Temporal Structure Modeling: The proposed temporal segment network emphasizes temporal-structure modeling and is presented as the first framework for end-to-end temporal-structure modeling.
3 Action Recognition with Temporal Segment Networks
This section presents temporal segment networks for video-level action recognition through sparse snippet sampling across equal-duration segments and segmental consensus. It also examines ConvNet architectures, input modalities, and training practices for effective learning with limited action-recognition data.
- Temporal segment network framework: TSN divides each video into K equal-duration segments, randomly samples one short snippet from each, and fuses snippet predictions into a video-level prediction.The framework combines spatial and temporal stream ConvNets, with shared parameters across snippets and modality-level fusion for the final prediction.
- Temporal segment network framework: K = 3 snippets are used in experiments, while the consensus function aggregates scores for each action class across snippets.The aggregation function remains an open question; the experiments use the simplest class-wise form of consensus.
- Temporal segment network framework: Differentiable temporal segment networks jointly optimize model parameters with back-propagation, allowing learning from entire videos rather than short snippets.With fixed K across videos, the method also provides a sparse temporal sampling strategy.
- Good practices: BN-Inception is selected as the two-stream ConvNet building block because it balances accuracy and efficiency.The architecture is adapted for temporal segment networks to address the limitations of the relatively shallow original two-stream ConvNets.
- Good practices: Additional input modalities studied include RGB difference and warped optical flow, with warping compensating camera motion to suppress background movement.RGB difference describes appearance changes, while warped optical flow is obtained by estimating a homography matrix and compensating for camera motion.
- Good practices: Partial BN freezes Batch Normalization mean and variance parameters except in the first layer after pre-trained initialization to reduce over-fitting during transfer.The first layer is re-estimated because optical-flow inputs have a different distribution from RGB images.
4 Experiments
Experiments evaluate TSN on HMDB51 and UCF101, investigate training strategies, input modalities, consensus functions, and architectures, and then assess the final method and learned representations. The results support cross-modality training, complementary modalities, average pooling, long-term temporal modeling, and pre-training as effective practices.
- Evaluation setup: Experiments use HMDB51 and UCF101, with UCF101 containing 101 classes and 13,320 clips and HMDB51 containing 51 categories and 6,766 clips.UCF101 evaluation follows the THUMOS13 scheme with three training/testing splits.
- Good practices: The study compares training from scratch, spatial pre-training, cross-modality pre-training, and their combination with partial BN and dropout to reduce over-fitting.Training from scratch performs much worse than the original two-stream ConvNets baseline, while the combined strategies are evaluated as progressively stronger settings.
- Good practices: 87.3% is achieved by combining RGB images and RGB differences, indicating that these modalities encode complementary information.The modality experiments use the good practices verified in the training-strategy study.
- Temporal segment network: Average pooling performs best among max pooling, average pooling, and weighted average, so it is selected as TSN’s default segmental consensus function.The consensus-function study uses three temporal segments and compares the candidates on UCF101 split 1.
- Temporal segment network: TSN improves recognition even after the proposed good practices are applied, supporting the importance of modeling long-term temporal structures.The analysis evaluates TSN with RGB images and optical flow fields and includes component-wise recognition accuracies.
- Model visualization: Pre-training enables spatial and temporal ConvNets to capture structured visual patterns, whereas models trained without pre-training barely generate meaningful visual structure and short-term models focus on scenery or objects.Single-frame models can mistake water, diving platforms, or wave motion for evidence of the action in the Diving class.
5 Conclusions
The paper presents Temporal Segment Network (TSN), a video-level framework for modeling long-term temporal structure. Its state-of-the-art performance with reasonable computational cost is attributed to sparse segmental sampling and explored good practices.
- 5 Conclusions: TSN is a video-level framework designed to model long-term temporal structure.The framework uses a segmental architecture with sparse sampling.
- 5 Conclusions: TSN brought state-of-the-art performance to a new level on two challenging datasets while maintaining reasonable computational cost.The conclusion explicitly reports this outcome across two datasets.
- 5 Conclusions: The reported effectiveness and efficiency are largely attributed to the segmental architecture with sparse sampling and a series of explored good practices.Sparse sampling provides an effective and efficient way to capture long-term temporal structure.