Source-linked AI summary
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, Kevin Murphy
TL;DR
Video classification has lagged behind image classification while 3D CNNs impose high computation and overfitting costs. The paper explores network design choices and combines low-cost 2D convolutions, separable convolution, and feature gating, yielding competitive results across multiple recognition and localization benchmarks.
Problem
Video classification has improved less than static image classification, while 3D CNNs are more expensive than 2D CNNs and prone to overfit.
Method
The paper systematically explores replacing lower-level 3D convolutions with 2D convolutions, separating spatial and temporal convolution, and adding spatio-temporal feature gating.
Results
The resulting system achieves significant accuracy gains over baseline methods across Kinetics, Something-something, UCF-101, HMDB, and action-localization benchmarks.
Takeaways & Limitations
Temporal representation learning on high-level semantic features can support a faster and accurate video classification system across datasets with different properties.
Takeaways & Limitations
The I2D comparison model is not a pure 2D model because it retains two temporally strided max-pooling layers.
Abstract
from arXiv · showhide
Despite the steady progress in video analysis led by the adoption of convolutional neural networks (CNNs), the relative improvement has been less drastic as that in 2D static image classification. Three main challenges exist including spatial (image) feature representation, temporal information representation, and model/computation complexity. It was recently shown by Carreira and Zisserman that 3D CNNs, inflated from 2D networks and pretrained on ImageNet, could be a promising way for spatial and temporal representation learning. However, as for model/computation complexity, 3D CNNs are much more expensive than 2D CNNs and prone to overfit. We seek a balance between speed and accuracy by building an effective and efficient video classification system through systematic exploration of critical network design choices. In particular, we show that it is possible to replace many of the 3D convolutions by low-cost 2D convolutions. Rather surprisingly, best result (in both speed and accuracy) is achieved when replacing the 3D convolutions at the bottom of the network, suggesting that temporal representation learning on high-level semantic features is more useful. Our conclusion generalizes to datasets with very different properties. When combined with several other cost-effective designs including separable spatial/temporal convolution and feature gating, our system results in an effective video classification system that that produces very competitive results on several action classification benchmarks (Kinetics, Something-something, UCF101 and HMDB), as well as two action detection (localization) benchmarks (JHMDB and UCF101-24).
1 Introduction
The paper studies how video networks should represent space and time while balancing accuracy against computational cost. It compares 2D/3D layer arrangements and introduces separable convolution and feature gating to build efficient, accurate systems.
- Video classification faces challenges in spatial representation, temporal representation, and trading model complexity against training and testing speed.
- I3D inflates 2D Inception filters into 3D convolutions but is computationally expensive.
- Top-Heavy-I3D retains 3D temporal convolutions in upper layers while using 2D convolutions in lower layers, making it faster than Bottom-Heavy-I3D.
- Top-Heavy-I3D models are often more accurate despite ignoring low-level motion cues.
- S3D replaces kt × k × k filters with sequential 1 × k × k and kt × 1 × 1 convolutions, reducing parameters and computation while improving accuracy over I3D.
- S3D-G combines the explored designs with spatio-temporal feature gating and improves accuracy over baseline methods across multiple classification and localization datasets.
2 Related work
Related work extends 2D image-classification architectures to video, while addressing the high computational cost of 3D convolution through mixed designs, separable convolution, feature gating, and optical-flow streams.
- I3D inflates Inception V1 filters into 3D convolutions and initializes the resulting model using pretrained 2D network weights.
- Concurrent mixed-convolution models use 3D convolutions in either lower or upper layers and 2D convolutions elsewhere, with conclusions differing from this paper’s findings.
- S3D factorizes 3D convolution into spatial 2D convolution followed by temporal 1D convolution to save computation.
- The paper combines separable convolution with top-heavy and bottom-heavy designs, reporting complementary gains from separability and top-heavy architecture.
- Feature gating models dependencies between feature channels through a multiplicative transformation and is placed after temporal convolutions in S3D.
- Two-stream approaches combine raw RGB and precomputed optical-flow inputs, typically improving video classification at higher cost.
3 Experiment Setup
The experiments evaluate the proposed designs on large action-classification datasets with different recognition demands, then test transferability and localization performance under a common video-training protocol.
- Kinetics contains 400 action classes and 240K training examples, with videos temporally trimmed to around 10 seconds.
- Mini-Kinetics-200 selects the 200 categories with the most training examples from Kinetics.
- Something-something contains 110k videos across 174 low-level actions lasting 2 to 6 seconds and requires fine-grained distinctions between physical interactions.
- The contrasting Kinetics and Something-something properties test whether the same architectural principles hold across datasets.
- The study also evaluates transferability on two smaller classification datasets and performance on two action-detection datasets.
- Training densely samples 64 frames, uses 224×224 random crops, and optimizes models with synchronous SGD on 56 GPUs.
- Evaluation reports top-1 and top-5 accuracy alongside theoretical FLOPS measured on 64-frame, 224 × 224 video inputs.
4 Network surgery
The network-surgery experiments replace selected 3D convolutions with 2D or separable alternatives to study speed–accuracy trade-offs. Top-heavy designs, which retain temporal processing in higher layers, generally provide the strongest efficiency and accuracy combination.
- Replacing all 3D convolutions with 2D: Replacing every 3D filter in I3D with a 2D filter produces the I2D model, which is nearly a single-frame 2D classifier.Two temporally strided max-pooling layers are retained for computational and training-protocol reasons.
- Replacing all 3D convolutions with 2D: I2D performance is unchanged when validation frames are temporally reversed, whereas reversal hurts I3D on Something-something but not Kinetics-Full.This indicates that temporal order matters more for Something-something’s fine-grained action distinctions.
- Partial 3D convolution: Top-heavy I3D models are faster and more accurate than bottom-heavy models at the same FLOPS.Top-heavy models apply 3D convolutions after spatial pooling, reducing the size of feature maps before expensive temporal processing.
- Analysis of learned filters: Learned low-level I3D filters largely ignore temporal offsets, while higher-level filters distribute weights across temporal positions.The weight analysis supports retaining temporal convolutions mainly in higher network layers.
- Separating temporal and spatial convolution: S3D factorizes each kt × k × k convolution into 1 × k × k followed by kt × 1 × 1, reducing parameters and computation while improving accuracy.On Kinetics-Full, parameters fall from 12.06M to 8.77M and top-1 accuracy rises from 71.1% to 72.2%; on Something-something, accuracy rises from 45.8% to 47.3%.
- Speed–accuracy trade-off: Separable top-heavy models offer the best speed–accuracy trade-off, with Fast-S3D reaching 2.5 times I3D’s efficiency while maintaining comparable Mini-Kinetics-200 accuracy.Fast-S3D uses separable 3D convolutions in the top two layers and 2D convolutions elsewhere, achieving 43.47 versus 107.9 GFLOPS and 78.0% versus 78.4% accuracy.
- Separating temporal and spatial convolution: Adding temporal separable convolution to all four Inception branches improves Mini-Kinetics-200 accuracy from 78.4% to 78.9%.Standard I3D supplies temporal information only through the two branches containing 3×3 convolutions.
- Representation analysis: Top-heavy S3D produces more discriminative representations than bottom-heavy S3D for visually similar Something-something actions.The comparison is made at Max5a, particularly for subtle actions such as picking, putting, and poking.
5 Generalization to other modalities, data and tasks
S3D-G generalizes across input modalities, video datasets, and recognition tasks, improving over I3D and prior methods in the reported experiments. Its gains extend from optical-flow classification to transfer learning and spatiotemporal action detection.
- Using optical flow features: 77.22% accuracy from RGB-flow ensembling gives a 3% boost over I3D trained on the same data.With optical flow alone, S3D-G improves performance from 63.91% to 68.00%.
- Using optical flow features: S3D-G improves over I3D on optical-flow inputs, increasing performance from 63.91% to 68.00%.
- Fine-tuning on other video classification datasets: On UCF-101, S3D-G pretrained only on Kinetics outperforms I3D and matches R(2+1)D, while on HMDB-51 it outperforms prior methods.I3D and R(2+1)D use large-scale pretraining datasets, whereas S3D-G uses Kinetics only.
- Spatiotemporal action detection in video: S3D-G is consistently better than I3D on JHMDB and UCF-101-24 action detection, while both 3D networks outperform earlier architectures.The evaluation reports frame-mAP at IoU 0.5 on the two detection datasets.
6 Conclusion
The paper improves the efficiency of 3D CNN video classification by combining top-heavy design, temporally separable convolution, and spatio-temporal feature gating.
- The system combines three key ideas: top-heavy model design, temporally separable convolution, and spatio-temporal feature gating.These modifications significantly improve the efficiency of I3D and can be applied to other architectures.