Source-linked AI summary
Beyond Short Snippets: Deep Networks for Video Classification
Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, George Toderici
TL;DR
The paper asks how CNN image information can be combined across full videos rather than short snippets, while retaining useful temporal and motion cues. It evaluates temporal feature pooling and CNN-connected LSTM sequence models, achieving state-of-the-art results on Sports-1M and UCF-101, with performance depending on optical-flow use and sequence architecture.
Problem
Short or independently processed frame inputs provide incomplete video information, motivating models that capture global temporal evolution across variable-length videos.
Method
The paper combines frame-level CNN activations over long videos using temporal feature-pooling architectures and LSTM recurrent networks, with optical-flow images incorporated through late fusion.
Results
The resulting networks achieve state-of-the-art performance on the Sports-1M and UCF-101 benchmarks.
Takeaways & Limitations
Learning over entire videos rather than short clips supports stronger video classification, while optical flow benefits depend on the benchmark and sequence-processing architecture.
Takeaways & Limitations
Current models do not backpropagate temporal information through time in the lower CNN layers, leaving deeper temporal integration for future work.
Abstract
from arXiv · showhide
Convolutional neural networks (CNNs) have been extensively applied for image recognition problems giving state-of-the-art results on recognition, detection, segmentation and retrieval. In this work we propose and evaluate several deep neural network architectures to combine image information across a video over longer time periods than previously attempted. We propose two methods capable of handling full length videos. The first method explores various convolutional temporal feature pooling architectures, examining the various design choices which need to be made when adapting a CNN for this task. The second proposed method explicitly models the video as an ordered sequence of frames. For this purpose we employ a recurrent neural network that uses Long Short-Term Memory (LSTM) cells which are connected to the output of the underlying CNN. Our best networks exhibit significant performance improvements over previously published results on the Sports 1 million dataset (73.1% vs. 60.9%) and the UCF-101 datasets with (88.6% vs. 88.0%) and without additional optical flow information (82.6% vs. 72.8%).
1. Introduction
The paper addresses the difficulty of classifying full videos from incomplete frame-level evidence by learning global temporal descriptions with CNN-based pooling and LSTM architectures. It also uses sparse frame sampling and optical flow to balance long-range information with computational cost.
- Video classification is computationally demanding because videos contain hundreds to thousands of frames, many irrelevant to the action.
- Averaging predictions from independently recognized frames can confuse classes because each frame captures only a small, potentially irrelevant part of the video.
- The proposed feature-pooling and LSTM architectures learn global temporal descriptions while keeping the number of parameters constant as video length varies.
- Sampling one frame per second reduces computation but loses implicit motion information, which the method compensates for using optical flow images from adjacent frames.
- The architectures improve classification by using more frames, sharing parameters through time, and combining optical flow with LSTMs even when flow is noisy.
2. Related Work
Earlier CNN-based video methods generally learned motion from short clips or aggregated frame predictions, while traditional approaches built global descriptors from handcrafted appearance and motion features. This paper instead aggregates CNN image features over much longer video periods using pooling and recurrent models.
- Traditional video recognition methods combine handcrafted appearance and motion features around interest points, dense grids, or tracked trajectories into global descriptors.
- Prior CNN approaches commonly use 3D convolutions on clips lasting only a few seconds and then aggregate predictions at the video level.
- A prior network was only marginally better than a single-frame baseline, indicating that learning motion features from raw frames is difficult.
- The paper aggregates strong CNN image features over tens of seconds with feature pooling and recurrent neural networks instead of learning spatio-temporal features only over short periods.
- LSTM memory cells are used because standard recurrent networks have difficulty learning over long sequences due to vanishing and exploding gradients.
3. Approach
The approach combines frame-level CNN features across long videos using temporal feature pooling or recurrent LSTMs, with CNN parameters shared across frames. Feature pooling is order-agnostic, whereas LSTMs explicitly model ordered CNN activation sequences and long-range temporal relationships.
- Temporal modeling: Both temporal architectures share CNN parameters across frames, while optical-flow models add explicit motion information to compensate for one-frame-per-second sampling.The LSTM explicitly considers sequence order, unlike order-invariant max-pooling.
- Feature pooling: The pooling architectures aggregate frame-level CNN features using max-pooling, average pooling, or fully connected layers at different network depths.The study varies pooling location and method, while gradients from upper layers can help learn useful image features.
- Feature pooling: Conv Pooling preserves convolutional spatial information by max-pooling the final convolutional layer across video frames.Late, Slow, Local, and Time-Domain Convolution variants instead combine features at later layers, hierarchically, locally, or within short temporal windows.
- LSTM architecture: The recurrent approach uses LSTMs to process ordered CNN activations, retaining, modifying, and emitting memory through gated cells.The deep architecture stacks LSTM layers, and weighted predictions usually perform best as the hidden state incorporates more frames.
4. Results
Experiments on Sports-1M and UCF-101 evaluate architecture choices, frame counts, optical flow, and video-level prediction. Performance generally improves when models aggregate more temporal context, while optical flow helps LSTMs more than feature-pooling models on Sports-1M.
- Evaluation setup: The experiments evaluate proposed architectures on Sports-1M and UCF-101, including frame counts, frame rates, and optical-flow effects.Sports-1M contains roughly 1.2 million videos across 487 classes, with 1.1 million remaining after removals.
- Sports-1M results: Conv-Pooling achieved the strongest Sports-1M feature-pooling result with 68.7 clip Hit@1 and 71.1 video Hit@1 using 120-frame AlexNet models.It outperformed Late, Slow, Local, and Time-Domain Convolution pooling variants.
- Sports-1M results: Fine-tuning increased a 30-frame GoogLeNet-LSTM model from 67.5 to 69.5 Hit@1 on Sports-1M.The comparison used no data augmentation and classified the entire 300-second video.
- Temporal context: 70.8 clip Hit@1 from the best model represented a 70% improvement over Slow Fusion, while 120-frame models significantly outperformed 30-frame models.These results support aggregating more frames to capture more of a video's content.
- Motion information: Optical flow raised LSTM accuracy to 73.1% on Sports-1M, although the optical-flow model alone reached only 59.7% versus 72.1% for the image-based model.Fusion produced no significant improvement for Conv-Pooling networks.
- UCF-101 results: On UCF-101, lowering the frame rate from 30fps to 6fps slightly improved 30-frame Conv-Pooling performance by providing five seconds of context.The models used 30 frames, corresponding to one second at 30fps or five seconds at 6fps.
5. Conclusion
The paper combines frame-level CNN outputs over long video sequences using feature pooling and LSTM architectures, achieving strong benchmark performance while supporting whole-video processing. It also identifies motion-information trade-offs and a limitation in how temporal gradients reach the CNN layers.
- Methods: Feature Pooling max-pools local information through time, while LSTM updates a hidden state for each subsequent frame.Both methods aggregate frame-level CNN outputs into video-level predictions.
- Long-range processing: 120 frames over up to two minutes produced optimal classification performance, extending training beyond the short clips used in previous work.The methods can also process an entire video in one shot when speed is important.
- Results: State-of-the-art performance was achieved on both the Sports-1M and UCF-101 benchmarks.This supports learning over entire videos rather than only short clips.
- Motion information: Optical flow is necessary for state-of-the-art UCF-101 results but is not always helpful for unconstrained Sports-1M videos.For Sports-1M, LSTMs using both image frames and optical flow achieve the highest published performance measure.
- Limitation: Temporal backpropagation currently reaches the top layers but not the lower CNN layers.The authors identify deeper integration of temporal sequence information into the CNNs as a future direction.