PaperlayerView exact revision on arXiv

Source-linked AI summary

A Closer Look at Spatiotemporal Convolutions for Action Recognition

Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, Manohar Paluri

arXiv:1711.11248v3cs.CV

TL;DR

Frame-based 2D CNNs remain competitive in action recognition, leaving the role of temporal reasoning an open design question. This paper compares spatiotemporal convolutions in residual networks and finds that R(2+1)D achieves results comparable or superior to the state of the art across four action-recognition benchmarks.

  • Problem

    The role of temporal reasoning in action recognition remains uncertain because frame-based 2D CNNs achieve performance close to the state of the art.

    Source: §1 · p. 1

  • Method

    The paper empirically compares 2D, 3D, mixed, and spatial-then-temporal factorized convolutions in residual networks, introducing the R(2+1)D block.

    Source: §2 · p. 2

  • Results

    R(2+1)D achieves results comparable or superior to the state of the art on Sports1M, Kinetics, UCF101, and HMDB51.

    Source: §5 · p. 8

  • Takeaways & Limitations

    On Kinetics, the reported gap between 2D and 3D or mixed models supports the importance of motion modeling for action recognition.

    Source: §4.2 · p. 6

    The mixed-convolution variants rely on the hypothesis that motion modeling may be useful early but unnecessary at higher semantic levels.

    Source: §3.4 · p. 3

Abstract

from arXiv · show

In this paper we discuss several forms of spatiotemporal convolutions for video analysis and study their effects on action recognition. Our motivation stems from the observation that 2D CNNs applied to individual frames of the video have remained solid performers in action recognition. In this work we empirically demonstrate the accuracy advantages of 3D CNNs over 2D CNNs within the framework of residual learning. Furthermore, we show that factorizing the 3D convolutional filters into separate spatial and temporal components yields significantly advantages in accuracy. Our empirical study leads to the design of a new spatiotemporal convolutional block "R(2+1)D" which gives rise to CNNs that achieve results comparable or superior to the state-of-the-art on Sports-1M, Kinetics, UCF101 and HMDB51.

1. Introduction

The introduction questions whether temporal reasoning is essential given the strength of frame-based CNNs, then presents residual 3D and factorized spatiotemporal convolutions as effective approaches to action recognition.

  • Despite being unable to model temporal information, frame-based ResNet-152 [25] performs remarkably close to state of the art on Sports-1M, raising questions about temporal reasoning’s value.The introduction also notes that I3D [4] leads action recognition, but its margin over hand-crafted iDT is less striking than image-recognition gains.Source: §1 · p. 1
  • 3D ResNets significantly outperform same-depth 2D ResNets on large-scale Sports1M and Kinetics [17] benchmarks.The comparison is within residual learning, with networks trained and evaluated on the benchmarks.Source: §1 · p. 1
  • Mixed-convolution ResNets gain roughly 3-4% in clip-level accuracy over comparable-capacity 2D ResNets and match 3D ResNets with 3 times as many parameters.Mixed convolution uses 3D convolutions in early layers and 2D convolutions in top layers.Source: §1 · p. 1
  • A (2+1)D block applies 2D spatial and 1D temporal convolutions successively, adding a nonlinearity that doubles nonlinearities at equal parameter count and facilitates optimization.The factorized blocks achieve lower training and testing losses than full 3D filters.Source: §1 · pp. 1–2
  • ResNets using (2+1)D blocks throughout achieve state-of-the-art performance on Kinetics and Sports-1M.The blocks factorize spatial and temporal components.Source: §1 · p. 2

2. Related Work

Video action recognition has progressed from hand-crafted features to frame-based, two-stream, and 3D CNN approaches. This work compares spatiotemporal convolution designs and presents R(2+1)D, which factorizes each spatiotemporal layer into spatial and temporal convolutions.

2. Related Work

  • Earlier video representations included hand-crafted STIPs, SIFT-3D, HOG3D, Motion Boundary Histogram, Cuboids, and Action-Bank; iDT was widely considered state-of-the-art.These representations use different encodings, including histograms and pyramids.Source: §2 · p. 2
  • Deep video methods integrated frame-based CNN features through pooling, feature encoding [11], or recurrent networks [7] [32] [2], while 3D CNNs later showed strong action-recognition results.Karpathy et al.’s slow-fusion model added temporal connectivity, but frame-based networks performed on par with networks processing full spatiotemporal video volumes; 3D CNN features also generalized to detection [28], captioning [24], and gesture detection.Source: §2 · p. 2
  • Two-stream methods fuse optical-flow and RGB CNN features [29], with later variants adding ResNet architectures [13] and stream connections [9]; I3D combines two-stream processing with 3D convolutions and held the top Kinetics results.Other approaches include Temporal Segment Networks [39], Action Transformations, and Convolutional Fusion [10].Source: §2 · p. 2
  • On Sports-1M RGB, R(2+1)D reaches 57.0% Clip@1 versus P3D’s 47.9%, a 9.1% margin despite using 34 rather than 152 layers.The paper reports state-of-the-art results for R(2+1)D on four action-recognition benchmarks.Source: §2 · p. 2
  • R(2+1)D factorizes each spatiotemporal convolution into spatial then temporal convolutions, unlike FST CN’s network factorization [33] and P3D’s interleaved block types [25].R(2+1)D uses one homogeneous residual-block type without bottlenecks; careful spatial-temporal dimensionality choices yield a compact model.Source: §2 · p. 2

3. Convolutional residual blocks for video

The section compares spatiotemporal convolution variants in vanilla residual networks, whose blocks use two convolutions with ReLU activations and whose top layer uses global average pooling and a fully connected classifier [13].

3.1. R2D: 2D convolutions over the entire clip

  • R2D treats the L frames as channels and applies filters only spatially, so its first convolution collapses temporal information and later layers cannot reason over time.Source: §3.1 · p. 3

3.2. f-R2D: 2D convolutions over frames

  • f-R2D applies shared 2D convolutional blocks independently to each frame, leaving temporal fusion to global spatiotemporal pooling.Source: §3.2 · p. 3

3.3. R3D: 3D convolutions

  • R3D preserves and propagates temporal information by convolving filters across both time and space, using temporal extent t = 3.Source: §3.3 · p. 3

3.4. MCx and rMCx: mixed 3D-2D convolutions

  • MCx switches from 3D to 2D convolutions in progressively earlier groups, while rMCx reverses this pattern, using 2D early and 3D in deeper groups.The variants test whether temporal modeling is more useful in early or late layers; MC1 is omitted because it is equivalent to f-R2D on clip inputs.Source: §3.4 · pp. 3–4

3.5. R(2+1)D: (2+1)D convolutions

  • R(2+1)D replaces each 3D convolution with a spatial 2D convolution followed by a temporal 1D convolution, using an intermediate subspace sized for approximately equal parameter count.The decomposition can be applied to any 3D convolutional layer, with spatial and temporal striding decomposed correspondingly.Source: §3.5 · p. 4
  • R(2+1)D has lower training and testing error than same-depth, same-capacity R3D, with the training-error gap especially large at 34 layers.The spatial-then-temporal decomposition adds a ReLU, doubling nonlinearities without changing parameter count, and makes optimization easier; the reported optimization advantage increases with depth.Source: Figure 3 · p. 4, §3.5 · p. 4
  • R(2+1)D significantly outperforms R3D, R2D, and P3D on Sports-1M.Unlike P3D-A’s bottleneck and the P3D architecture’s interleaved block types, R(2+1)D uses the same decomposition throughout the network and is designed to match 3D-convolution parameter counts.Source: §3.5 · p. 5

4. Experiments

The experiments compare spatiotemporal convolution variants and evaluate R(2+1)D on action-recognition benchmarks and transfer tasks.

4.1. Experimental setup

  • Experiments compared residual 2D, 3D, mixed, reversed-mixed, and (2+1)D convolutions, using Kinetics and Sports-1M for primary evaluation and UCF101 and HMDB51 for transfer.The paper describes these datasets as benchmarks for training deep models from scratch and evaluating transfer learning.Source: §4 · p. 5, §4.1 · p. 5

4.2. Comparison of spatiotemporal convolutions

  • At similar computational cost, R(2+1)D achieved the highest Kinetics accuracy, gaining about 3−3.8% over R3D across 8- and 16-frame inputs.Its training and testing errors were also lower than R3D’s, suggesting that spatiotemporal decomposition eases optimization, especially at 34 layers; the gap between 2D and motion-modeling networks widened with longer clips.Source: Figure 4 · p. 6, §4.2 · p. 6

4.3. Revisiting practices for video-level prediction

  • With R(2+1)D, clip accuracy kept rising as input length increased, but video-level accuracy peaked at 32 frames.Testing an 8-frame model on 32-frame clips reduced clip accuracy by 1.2% and video accuracy by 5.8%; finetuning that model on 32 frames yielded 59.8% versus 60.1% from scratch and a 7% gain over the 8-frame model. Using 20 crops was about 0.5% worse than 100 crops but 5x faster; training on shorter clips before finetuning longer clips gave the best training-time/accuracy trade-off.Source: §4.3 · p. 7, Figure 5 · p. 7, Table 3 · p. 7

4.4. Action recognition with a 34-layer R(2+1)D net

  • R(2+1)D-34 reached 73.3% video-level accuracy on Sports-1M, with clip-level accuracy exceeding C3D by 10.9% and P3D by 9.1%.The paper reports this as the best published video-level result on Sports-1M.Source: Table 4 · p. 7, §4.4 · p. 8
  • On Kinetics, R(2+1)D-34 beat I3D by 4.5% from-scratch on RGB, and Sports-1M-pretrained R(2+1)D beat ImageNet-pretrained I3D on individual streams.For fused streams, Sports-1M-pretrained R(2+1)D was 0.3% worse than I3D.Source: Table 5 · p. 8, §4.4 · p. 8
  • On UCF101 and HMDB51, Kinetics-pretrained R(2+1)D was comparable to I3D on RGB and slightly worse on optical flow.I3D used ImageNet in addition to Kinetics for pretraining; the paper attributes its flow advantage to using TV-L1 rather than Farneback flow.Source: Table 6 · p. 8, §4.4 · p. 8

5. Conclusions

The empirical study finds that R(2+1)D achieves results comparable or superior to the state of the art on four action-recognition benchmarks, while identifying limits that motivate future architecture research.

5. Conclusions

  • R(2+1)D achieves results comparable or superior to the state of the art on Sports1M, Kinetics, UCF101, and HMDB51.The paper presents this as the outcome of its empirical study of spatiotemporal convolutions for video action recognition.Source: §5 · p. 8
  • The study examines the effects of different spatiotemporal convolutions for action recognition in video.Source: §5 · p. 8
  • The study focuses on ResNet and homogeneous use of the (2+1)D decomposition; future work will seek more suitable architectures for the approach.The authors hope the analysis will inspire network designs that harness the efficacy and modeling flexibility of spatiotemporal convolutions.Source: §5 · p. 8

Appendix

At conv1, R(2+1)D decomposes R3D’s 3D filters into spatial and temporal filters with intervening ReLUs while preserving the parameter count.

Appendix

  • At conv1, R(2+1)D uses 45 1×7×7 spatial filters and 64 3×1×1 temporal filters, with ReLUs between them, matching R3D’s parameter count.Figure 6 visualizes the spatial filters upscaled 4× and the temporal filters as 45 × 3 matrices showing how they combine spatial-filter channels across three frames.Source: Appendix · p. 9

Related papers

Loading 1711.11248v3…