Source-linked AI summary
Two-Stream Convolutional Networks for Action Recognition in Videos
Karen Simonyan, Andrew Zisserman
TL;DR
Video action recognition must combine appearance in still frames with motion across frames, while prior deep ConvNet approaches underperformed strong hand-crafted representations. This paper introduces separate spatial and optical-flow temporal ConvNets with late fusion, achieving competitive state-of-the-art performance on UCF-101 and HMDB-51.
Problem
Video action recognition requires capturing both appearance in still frames and motion between frames, which prior stacked-frame ConvNets captured poorly relative to strong hand-crafted representations.
Method
The paper combines separate spatial and temporal ConvNets through late fusion, using still frames for appearance and stacked dense optical flow for motion.
Results
On UCF-101 and HMDB-51, the two-stream architecture significantly outperforms earlier deep video models and is competitive with state-of-the-art shallow representations.
Takeaways & Limitations
The complementary spatial and temporal streams provide a competitive deep-learning approach for video action recognition despite relatively small training datasets.
Takeaways & Limitations
The architecture misses trajectory-centered spatio-temporal pooling and does not explicitly handle camera motion beyond mean-displacement compensation.
Abstract
from arXiv · showhide
We investigate architectures of discriminatively trained deep Convolutional Networks (ConvNets) for action recognition in video. The challenge is to capture the complementary information on appearance from still frames and motion between frames. We also aim to generalise the best performing hand-crafted features within a data-driven learning framework. Our contribution is three-fold. First, we propose a two-stream ConvNet architecture which incorporates spatial and temporal networks. Second, we demonstrate that a ConvNet trained on multi-frame dense optical flow is able to achieve very good performance in spite of limited training data. Finally, we show that multi-task learning, applied to two different action classification datasets, can be used to increase the amount of training data and improve the performance on both. Our architecture is trained and evaluated on the standard video actions benchmarks of UCF-101 and HMDB-51, where it is competitive with the state of the art. It also exceeds by a large margin previous attempts to use deep nets for video classification.
1 Introduction
The paper extends ConvNets for video action recognition by combining separate spatial and temporal streams to capture complementary appearance and motion information. On UCF101 and HMDB-51, the architecture substantially improves over earlier deep approaches and competes with state-of-the-art shallow representations despite limited training data.
- Motivation: Video action recognition is challenging because temporal motion provides an important clue beyond still-image appearance, while video frames also provide natural jittering-based data augmentation.The motivation is to exploit both spatial content and temporal information in video.
- Contribution: The paper proposes separate spatial and temporal recognition streams to extend ConvNets from still-image representation to video action recognition.This architecture addresses the limitations of earlier models that used stacked video frames as input.
- Results: Experiments on UCF101 and HMDB-51 show that the two streams are complementary, significantly outperform earlier deep architectures, and remain competitive with state-of-the-art shallow representations despite relatively small datasets.The paper also develops multi-task learning to combine training data across datasets.
- Prior work: State-of-the-art shallow representations use dense trajectories computed with optical flow, with Motion Boundary Histogram achieving the best performance in that trajectory-based pipeline.These methods represent a major hand-crafted baseline for comparison.
- Prior work: Earlier deep video-recognition methods commonly stacked consecutive frames and expected the first layers to learn motion-dependent features implicitly, which can be difficult.The paper instead develops a temporal stream operating on multiple-frame dense optical flow.
2 Two-stream architecture for video recognition
The architecture decomposes video recognition into spatial and temporal streams, capturing frame appearance and motion across frames with separate deep ConvNets. The spatial stream recognizes actions from individual frames and remains useful on its own because static objects provide action-related clues.
- Two-stream architecture: The two-stream architecture separates video into spatial appearance and temporal motion components, each implemented with a deep ConvNet.Spatial information comes from individual frame appearance, while temporal information captures motion across frames, including camera and object movement.
- Spatial stream: The spatial stream ConvNet operates on individual video frames, effectively performing action recognition from still images.This stream builds on image-classification ConvNet advances.
- Spatial stream: Static appearance is a useful action-recognition clue because some actions are strongly associated with particular objects.Action classification from still frames is described as fairly competitive on its own.
3 Optical flow ConvNets
The temporal recognition stream uses ConvNets over stacked optical-flow fields, explicitly representing motion across multiple frames. Its learned filters capture spatial and temporal motion structure while generalising several hand-crafted flow-based descriptors.
- Temporal ConvNet: The temporal stream feeds a ConvNet stacked optical-flow displacement fields, explicitly representing inter-frame motion rather than requiring the network to estimate it implicitly.The input consists of optical-flow fields between several consecutive frames.
- Optical flow stacking: Stacking flow components from L consecutive frames produces a ConvNet input with 2L channels encoding motion at each point over the sequence.The input volume has dimensions Iτ ∈ R^w×h×2L.
- Trajectory stacking: The section considers trajectory stacking, which samples flow vectors along motion trajectories instead of at fixed image locations across frames.The trajectory representation follows points pk along trajectories beginning at a location in frame τ.
- Relation to prior representations: The temporal ConvNet generalises flow-based descriptors such as HOF, MBH, kinematic features, and trajectory features within a discriminatively trained convolutional model.These descriptors can be derived from optical-flow displacement fields or their gradients, while trajectory features are represented through trajectory-based sampling.
- Learned motion features: Some learned filters compute spatial derivatives of optical flow, while others compute temporal derivatives, capturing motion changes across image location and time.The visualised temporal ConvNet uses 96 filters over 20 channels from 10 stacked optical-flow fields.
4 Multi-task learning
Multi-task learning addresses limited video training data by jointly training a shared ConvNet representation for HMDB-51 and UCF-101, using each dataset as an additional regularising task.
- Motivation: UCF-101 and HMDB-51 provide only 9.5K and 3.7K videos, respectively, making temporal ConvNet training prone to over-fitting.Unlike the spatial stream, the temporal stream cannot be pre-trained on a large still-image dataset.
- Method: Multi-task learning jointly learns a video representation applicable to HMDB-51 and UCF-101 classification, with additional tasks acting as regularisers and providing more training data.The method is presented as a principled alternative to simply combining datasets.
- Method: The modified ConvNet has two softmax layers above the last fully-connected layer, producing separate HMDB-51 and UCF-101 classification scores.Each softmax layer is associated with its respective dataset.
- Method: The overall training loss is the sum of the two task-specific losses, each computed only from videos in its corresponding dataset, and gradients are obtained by back-propagation.The shared network weights are updated using derivatives of the combined loss.
5 Implementation details
The implementation uses CNN-M-2048-style spatial and temporal ConvNets, trained with momentum SGD on randomly sampled video frames and evaluated by aggregating predictions across temporally spaced frames and crops. ImageNet pre-training, multi-GPU parallelism, and pre-computed compressed optical flow support the training pipeline.
- Training: Training uses mini-batch stochastic gradient descent with momentum 0.9, sampling 256 videos uniformly across classes and one random frame from each video per iteration.The same general training procedure is used for the spatial and temporal networks.
- Testing: At test time, 25 equally spaced frames each produce 10 cropped and flipped ConvNet inputs, whose class scores are averaged across frames and crops.The 10 inputs use four corners and the center of each frame, with cropping and flipping.
- Pre-training on ImageNet ILSVRC-2012: 13.5% top-5 error on the ILSVRC-2012 validation set compares favorably with the 16.0% reported for a similar network.The spatial ConvNet uses cropping, flipping, and RGB jittering; the authors attribute the improvement mainly to sampling from the whole image rather than only its center.
- Multi-GPU training: Multi-GPU data parallelism splits each SGD batch across GPUs, reducing temporal ConvNet training to 1 day on four NVIDIA Titan cards, a 3.2 times speed-up over one GPU.The implementation is derived from Caffe and includes significant modifications for parallel training.
- Optical flow preprocessing: Optical flow is computed off-the-shelf in 0.06s per frame pair, pre-computed before training, and JPEG-compressed after rescaling horizontal and vertical components to [0, 255].The flow is rescaled back after decompression, avoiding storage of displacement fields as floats.
6 Evaluation
The evaluation uses standard three-split mean-accuracy protocols on UCF-101 and HMDB-51 to compare spatial, temporal, multi-task, and two-stream ConvNets. Results show benefits from appropriate optical-flow inputs, multi-task training, and combining complementary streams, while revealing class-specific confusion patterns.
- Datasets and evaluation protocol: UCF-101 contains 13K videos across 101 actions, while HMDB-51 contains 6.8K videos across 51 actions; performance is mean classification accuracy over three splits.Each split provides 9.5K UCF-101 or 3.7K HMDB-51 training videos.
- Temporal ConvNets: 56.4% accuracy from slow fusion of 11 RGB frames exceeded 52.3% from a single-frame ConvNet, but remained well below the optical-flow network.The comparison used networks trained from scratch on UCF-101.
- Multi-task learning of temporal ConvNets: Multi-task learning performed best for HMDB-51 temporal ConvNets by exploiting all available training data from UCF-101 and HMDB-51.The HMDB-51 training split is 2.6 times smaller than UCF-101’s, motivating additional training data.
- Multi-task learning of temporal ConvNets: 81.5% accuracy on the first UCF-101 split improved on 81.0% when the same setup omitted the additional HMDB-51 classification task.The multi-task network classified full HMDB-51 data and a single UCF-101 split.
- Two-stream ConvNets: The two-stream model reached 87.0% accuracy on the first UCF-101 split, with spatial and temporal streams providing complementary recognition information.The streams were fused through softmax-score averaging or a linear SVM because jointly training fully connected layers caused over-fitting.
- Confusion matrix and per-class recall for UCF-101 classification: Hammering had the worst class recall, being confused with HeadMassage by the spatial stream and BrushingTeeth by the temporal stream.The reported causes were shared human faces and recurring hand-up-and-down motion patterns, respectively.
7 Conclusions and directions for improvement
The paper concludes that its two-stream ConvNet model achieves competitive video-classification performance, with optical-flow temporal training outperforming raw stacked-frame training. Future improvements include scaling temporal training data and incorporating trajectory-based pooling and explicit camera-motion handling.
- Conclusions: The proposed model combines separate spatial and temporal ConvNet streams and achieves competitive performance for deep video classification.The architecture is presented as a deep video classification model with separate recognition streams.
- Conclusions: Training a temporal ConvNet on optical flow is significantly better than training it on raw stacked frames.The raw-frame approach is described as probably too challenging and potentially requiring architectural changes.
- Directions for improvement: Extra training data benefits the temporal ConvNet, motivating training on large video datasets despite the challenge of handling multiple TBs of data.The authors specifically identify the gigantic volume of training data as a significant challenge.
- Directions for improvement: The current architecture misses local feature pooling over spatio-temporal tubes centered at trajectories.Although the input captures optical flow along trajectories, the network’s spatial pooling does not account for them.
- Directions for improvement: Explicit handling of camera motion is identified as another potential area for improvement.The passage presents camera-motion handling as a missing ingredient of the current architecture.