Source-linked AI summary
Convolutional Two-Stream Network Fusion for Video Action Recognition
Christoph Feichtenhofer, Axel Pinz, Andrew Zisserman
TL;DR
Video action recognition had lagged behind other ConvNet applications because datasets are limited and existing architectures underuse temporal information. This paper develops a spatiotemporal fusion architecture, finding that convolutional and temporal fusion can improve benchmark performance while avoiding a substantial parameter increase.
Problem
Action-recognition ConvNets had not achieved comparable gains to other applications, partly because datasets are small or noisy and architectures often fail to exploit temporal information.
Method
The paper studies spatial registration, fusion location, and temporal fusion, then combines the selected strategies in a two-stream architecture using convolutional fusion, 3D convolutions, and 3D pooling.
Results
The proposed architecture achieves state-of-the-art performance on UCF101 and HMDB51 without significantly increasing parameters over previous methods.
Takeaways & Limitations
Learning correspondences between highly abstract spatial and temporal ConvNet features is supported as important for video action recognition.
Takeaways & Limitations
Because current datasets are too small or noisy, the paper cautions that some conclusions should be treated with caution.
Abstract
from arXiv · showhide
Recent applications of Convolutional Neural Networks (ConvNets) for human action recognition in videos have proposed different solutions for incorporating the appearance and motion information. We study a number of ways of fusing ConvNet towers both spatially and temporally in order to best take advantage of this spatio-temporal information. We make the following findings: (i) that rather than fusing at the softmax layer, a spatial and temporal network can be fused at a convolution layer without loss of performance, but with a substantial saving in parameters; (ii) that it is better to fuse such networks spatially at the last convolutional layer than earlier, and that additionally fusing at the class prediction layer can boost accuracy; finally (iii) that pooling of abstract convolutional features over spatiotemporal neighbourhoods further boosts performance. Based on these studies we propose a new ConvNet architecture for spatiotemporal fusion of video snippets, and evaluate its performance on standard benchmarks where this architecture achieves state-of-the-art results.
1. Introduction
Video action recognition remains behind human performance, partly because existing ConvNet approaches underuse temporal information and rely on limited or noisy datasets. This paper develops multilevel spatial and temporal fusion of appearance and motion streams to address those gaps.
- Video action recognition remains far from human performance despite substantial ConvNet progress in related vision tasks.
- Small or noisy datasets and viewpoint and motion variation make video action recognition harder than image classification.UCF-101 has only 100 examples per class, compared with ImageNet’s 1000.
- Existing ConvNet architectures often fail to fully exploit temporal information, leaving performance dominated by spatial appearance recognition.
- Two-stream ConvNets separately model appearance and optical-flow motion, but prior methods do not register these cues spatially or model their evolution over time.
- The paper investigates spatial registration, fusion location, and temporal fusion, then combines the selected designs into a spatiotemporal architecture.
2. Related work
Prior work extends ConvNets across time, pools temporal features, and separates appearance from optical-flow motion. The paper builds most directly on two-stream networks while addressing their limited spatial registration and temporal-evolution modeling.
- Video ConvNet research extends 2D filters across time through early fusion, slow fusion, and 3D convolution over short frame sequences.
- C3D learns 3D ConvNets over a limited temporal support of 16 consecutive frames using 3×3×3 filter kernels.
- Temporal pooling of convolutional features outperformed slow, local, late pooling, and temporal convolution, while LSTMs did not improve over temporal pooling.
- The two-stream architecture feeds RGB and stacked optical-flow inputs into separate ConvNets and combines their softmax scores by late fusion.
- Two-stream methods remained effective with limited training data, but smaller temporally consistent datasets risk severe overfitting and larger datasets can contain label noise.
- Bilinear fusion correlates two ConvNet layers through an outer product at each image location before pooling the resulting feature.
3. Approach
The approach fuses spatial and temporal ConvNet streams at corresponding feature locations and investigates fusion functions, fusion depth, and temporal integration. These choices motivate a spatiotemporal architecture using 3D convolution and pooling while retaining both streams.
- Motivation: The two-stream architecture is limited by classification-score fusion and restricted temporal scale, preventing pixel-wise feature correspondence and modelling of temporal evolution.The spatial stream processes single frames, while the temporal stream processes a short stack of adjacent optical-flow frames.
- Spatial fusion: Spatial fusion aligns feature maps at the same pixel locations, after which subsequent filters can learn correspondences between channels.Matching spatial dimensions are required; upconvolution or zero-padding can align maps when necessary.
- Fusion functions: Sum and max fusion combine corresponding feature-map values but impose an arbitrary channel correspondence between the two networks.Subsequent learning can optimize filters around that correspondence.
- Fusion functions: Concatenation preserves both streams’ channels for subsequent filters, whereas convolution fusion learns weighted same-location combinations and reduces 2D channels to D.The convolution fusion kernel has dimensions 1 × 1 × 2D and can learn channel correspondences under a joint loss.
- Fusion functions: Bilinear fusion forms an outer product at each pixel and sums over locations, capturing multiplicative channel interactions but producing D^2 features and discarding spatial information.Its high dimensionality commonly motivates applying it at ReLU5 with fully connected layers removed and normalization applied.
- Fusion depth: Fusion after ReLU5 removes nearly half the parameters in the two-stream VGG-M architecture, while fusing at multiple layers preserves both towers without the same parameter reduction.Fusing at different convolutional layers has roughly similar parameter impact because most parameters reside in fully connected layers.
- Proposed architecture: The proposed architecture fuses streams after the last convolutional layer using 3D convolution followed by 3D pooling, retains the temporal tower, and averages both streams’ predictions.Its spatial-fusion method, layer, and temporal-fusion design are selected from the corresponding empirical studies.
4. Evaluation
The evaluation compares spatial and temporal fusion choices, network depth, and spatiotemporal pooling on UCF101 and HMDB51. The proposed fusion architecture reaches strong benchmark performance while trading parameters and computation against accuracy.
- Spatial fusion: 85.96% accuracy with Conv fusion at ReLU5 matched softmax fusion while using 97.58M rather than 181.42M parameters on UCF101 split 1.ReLU5 fusion therefore retained classification accuracy with substantially fewer parameters than softmax fusion.
- Fusion location: ReLU5 fusion outperformed fusion at fully connected layers, while adding FC8 fusion achieved similar accuracy with nearly twice the parameters.The authors attribute the ReLU5 advantage to preserving spatial correspondences between appearance and motion features.
- Fusion location: 85.96% versus 86.04% showed that ReLU5 fusion alone nearly matched additional FC8 fusion at much lower parameter cost.The comparison used Conv fusion initialized to sum activations from preceding layers.
- Network depth: Deeper spatial models improved performance by 8.11% on UCF101 and 10.29% on HMDB51, while deeper temporal models gained 3.91% and 3.73%.The evaluation compared VGG-M-2048 with VGG-16 two-stream networks.
- Temporal fusion: 3D pooling after fusion improved performance on both datasets, and a 3D fusion filter provided a further recognition boost.The temporal-fusion experiments used VGG-16 for the spatial stream and VGG-M for the temporal stream.
- State-of-the-art comparison: 92.5% accuracy on UCF101 exceeded the reported 88.2% and 88.6% results of another two-stream approach, while retaining two streams after temporal fusion.A single stream after temporal fusion reached 91.8% with fewer parameters and a simpler architecture.
5. Conclusion
The paper proposes a spatiotemporal two-stream architecture that fuses spatial and temporal ConvNet features through convolutional and temporal fusion layers, achieving strong benchmark performance with little parameter growth. However, conclusions should be treated cautiously because the evaluated datasets may be too small or noisy.
- The proposed architecture combines a convolutional fusion layer between two streams with a temporal fusion layer using 3D convolutions and pooling.
- The architecture does not significantly increase parameter count over previous methods while exceeding the state of the art on two standard benchmark datasets.
- The results suggest that learning correspondences between highly abstract ConvNet features in space and time is important for video action recognition.
- Combining ConvNet predictions with FV-encoded IDT features still improves performance, although the authors suspect this difference may disappear with substantially more training data.
- Some conclusions should be treated with caution because current datasets are either too small or too noisy.