Source-linked AI summary
Long-term Temporal Convolutions for Action Recognition
Gül Varol, Ivan Laptev, Cordelia Schmid
TL;DR
Existing action-recognition methods often model only short clips, despite actions spanning much longer temporal extents. This paper uses long-term temporal convolutions and compares input representations, achieving state-of-the-art performance while showing that optical-flow quality strongly affects results.
Problem
Many CNN-based methods learn action representations from 1–16 frames, although human actions often span tens or hundreds of frames with characteristic long-term structure.
Method
The paper evaluates space-time CNNs with Long-term Temporal Convolutions, increasing temporal extent while reducing spatial resolution, and compares RGB pixels with optical-flow inputs.
Results
Long-term temporal convolutions achieve state-of-the-art performance on UCF101 and HMDB51, while higher-quality optical flow further improves action-recognition results.
Takeaways & Limitations
With limited training data, motion-based representations outperform RGB, and optical-flow quality significantly impacts recognition accuracy.
Takeaways & Limitations
RGB networks remain difficult to train from scratch; even extending inputs to 100 frames stays below ImageNet-pretrained frame-based methods on UCF101 split 1.
Abstract
from arXiv · showhide
Typical human actions last several seconds and exhibit characteristic spatio-temporal structure. Recent methods attempt to capture this structure and learn action representations with convolutional neural networks. Such representations, however, are typically learned at the level of a few video frames failing to model actions at their full temporal extent. In this work we learn video representations using neural networks with long-term temporal convolutions (LTC). We demonstrate that LTC-CNN models with increased temporal extents improve the accuracy of action recognition. We also study the impact of different low-level representations, such as raw values of video pixels and optical flow vector fields and demonstrate the importance of high-quality optical flow estimation for learning accurate action models. We report state-of-the-art results on two challenging benchmarks for human action recognition UCF101 (92.7%) and HMDB51 (67.2%).
1 INTRODUCTION
Human actions contain class-specific spatial and long-term temporal patterns, but short-clip CNNs often fail to represent their full duration. The paper addresses this with long-term temporal convolutions and examines motion representations.
- 1 INTRODUCTION: Typical actions span several seconds and tens or hundreds of frames, exceeding the short intervals used by many CNN methods.Short clips can break characteristic spatial and long-term temporal structure, making recognition more difficult.
- 1 INTRODUCTION: Long-term temporal convolutions learn video representations over extended periods while trading temporal extent against spatial resolution to keep networks tractable.The architecture uses spatio-temporal convolutions and increases representation duration at reduced spatial resolution.
- 1 INTRODUCTION: The study compares raw video pixels with optical-flow vector fields as low-level representations for action recognition.This comparison tests how appearance and motion inputs affect learned video representations.
- 1 INTRODUCTION: The experiments highlight the importance of high-quality motion estimation and report state-of-the-art performance on UCF101 and HMDB51.The paper identifies long-term temporal convolutions and high-quality optical flow as its two principal contributions.
2 RELATED WORK
Earlier action-recognition methods relied heavily on local motion features, while CNN extensions often modeled only frames or short clips. This work extends 3D CNNs to longer temporal convolutions and evaluates optical-flow input.
- 2 RELATED WORK: Traditional action-recognition pipelines aggregate local video descriptors with Bag-of-Features or Fisher Vector representations.Motion features, particularly Motion Boundary Histograms, have been important in practice, while explicit temporal structure has been less common.
- 2 RELATED WORK: CNN-based video methods include static frame encodings, short clips, and raw-pixel or precomputed optical-flow inputs.Motion-based CNN representations typically outperform RGB representations, consistent with earlier hand-crafted features.
- 2 RELATED WORK: Existing 3D CNNs generally use RGB and short intervals, such as 16-frame clips or shorter frame sequences.These approaches provide spatio-temporal modeling but do not cover actions at their full temporal scale.
- 2 RELATED WORK: The paper extends 3D CNNs with significantly longer temporal convolutions and explores optical-flow input as two extensions evaluated against previous methods.The related-work comparison frames both longer temporal context and motion input as experimentally advantageous directions.
3 LONG-TERM TEMPORAL CONVOLUTIONS
The method uses a space-time CNN with adjustable temporal input extents and compares RGB with optical-flow representations. Training and testing use sampled clips, augmentation, and multi-crop aggregation.
- 3 LONG-TERM TEMPORAL CONVOLUTIONS: The network contains five space-time convolutional layers followed by three fully connected layers, with ReLU and space-time max pooling after each convolution.Convolutional layers use 3 × 3 × 3 filters, while pooling is 2 × 2 × 2 except in the first layer.
- 3 LONG-TERM TEMPORAL CONVOLUTIONS: The study compares 16-frame and 60-frame inputs, then varies temporal and spatial resolutions to analyze longer temporal extents.The 60-frame networks reduce spatial resolution to preserve network complexity.
- 3 LONG-TERM TEMPORAL CONVOLUTIONS: Inputs use either raw RGB values or two-channel flow fields representing motion in the x and y directions.Flow magnitudes are scaled to remain correct after spatial subsampling.
- 3 LONG-TERM TEMPORAL CONVOLUTIONS: MPEG, Farneback, and Brox provide three motion inputs for testing how optical-flow quality affects recognition.MPEG flow is fast but low-resolution and incomplete at some frames, Farneback is relatively fast, and Brox is the most sophisticated.
- 3 LONG-TERM TEMPORAL CONVOLUTIONS: Training samples random fixed-size video clips, while testing averages scores across temporally strided clips and ten spatial crops with horizontal flips.The augmentation includes random clipping, multiscale spatial cropping, and 50% horizontal flipping.
4 EXPERIMENTS
Experiments on UCF101 and HMDB51 evaluate optical flow quality, data augmentation, temporal extent, and network combinations. Long-term temporal convolutions consistently improve action recognition, while optical flow and complementary multiscale networks strengthen performance.
- Evaluation setup: UCF101 and HMDB51 experiments evaluate clip and video accuracy across temporal resolutions, modalities, augmentation settings, and network combinations.Video accuracy averages per-clip softmax scores across each video and then across the three standard splits.
- Optical flow: Nearly 20% improvement follows from using Brox optical flow instead of lower-quality flow, with flow outperforming RGB even at lower quality.The improvement is consistent for both individual clips and full videos, motivating Brox flow in subsequent experiments.
- Data augmentation: 4% gain results from combining random clipping, multiscale cropping, and higher dropout, while each modification independently improves the 60-frame UCF101 model.The reported individual gains are 3.1% for random clipping, 1.6% for multiscale cropping, and 2% for higher dropout.
- Temporal extent: 60-frame LTC networks significantly improve over 16-frame networks for both RGB and flow across tested setups, with larger gains for clip than video accuracy.Video evaluation aggregates information over the whole video, reducing the relative impact of longer individual inputs.
- Class-level effects: 25 of 101 UCF101 classes improve monotonically with longer temporal extents, while no class shows monotonic decline across increasing extents.FloorGymnastics and JavelinThrow are highlighted as classes with significant gains, whereas Shotput performs worse with longer inputs.
- Comparison with prior methods: LTC improves on both datasets, and combined LTCF low+RGB scores outperform the Two-stream average-fusion baseline by 4.8% and 6.8%.Extending a pre-trained RGB network to 100 frames improves UCF101 from 80.2% to 82.4%.
5 CONCLUSIONS
Long-term temporal convolutions improve action-recognition performance by modeling video over extended temporal spans. The approach also highlights the importance of optical-flow quality, achieving state-of-the-art results on UCF101 and HMDB51.
- LTC significantly improves action-recognition performance by using space-time convolutions over a large number of video frames.