PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks

Zhaofan Qiu, Ting Yao, Tao Mei

arXiv:1711.10305v1cs.CV

TL;DR

Video representation learning must capture temporal evolution without the computational and memory burden of very deep 3D CNNs. The paper introduces P3D ResNet, which combines spatial and temporal convolutions in diverse residual blocks, and evaluates its learned representations across multiple benchmarks and tasks. P3D ResNet outperforms several competing approaches, including 3D CNN and frame-based 2D CNN baselines on Sports-1M.

  • Problem

    Very deep 3D CNNs capture spatial and temporal video information but require expensive computation and memory, while frame-level aggregation leaves low-level temporal correlations underused.

  • Method

    P3D ResNet simulates 3 × 3 × 3 convolutions with spatial 1 × 3 × 3 and temporal 3 × 1 × 1 convolutions, using varied bottleneck blocks throughout a residual network.

  • Results

    P3D ResNet representations show superior performance across five benchmarks and three tasks, including 88.6% accuracy in one reported evaluation.

  • Takeaways & Limitations

    The learned P3D ResNet representations generalize across video action recognition, action similarity labeling, and scene recognition tasks.

  • Takeaways & Limitations

    Future work remains to study attention, the effect of increasing training-clip frames, and additional inputs such as optical flow or audio.

Abstract

from arXiv · show

Convolutional Neural Networks (CNN) have been regarded as a powerful class of models for image recognition problems. Nevertheless, it is not trivial when utilizing a CNN for learning spatio-temporal video representation. A few studies have shown that performing 3D convolutions is a rewarding approach to capture both spatial and temporal dimensions in videos. However, the development of a very deep 3D CNN from scratch results in expensive computational cost and memory demand. A valid question is why not recycle off-the-shelf 2D networks for a 3D CNN. In this paper, we devise multiple variants of bottleneck building blocks in a residual learning framework by simulating $3\times3\times3$ convolutions with $1\times3\times3$ convolutional filters on spatial domain (equivalent to 2D CNN) plus $3\times1\times1$ convolutions to construct temporal connections on adjacent feature maps in time. Furthermore, we propose a new architecture, named Pseudo-3D Residual Net (P3D ResNet), that exploits all the variants of blocks but composes each in different placement of ResNet, following the philosophy that enhancing structural diversity with going deep could improve the power of neural networks. Our P3D ResNet achieves clear improvements on Sports-1M video classification dataset against 3D CNN and frame-based 2D CNN by 5.3% and 1.8%, respectively. We further examine the generalization performance of video representation produced by our pre-trained P3D ResNet on five different benchmarks and three different tasks, demonstrating superior performances over several state-of-the-art techniques.

1. Introduction

The paper frames video representation learning as difficult because temporal variation is not handled as naturally as spatial image content. It proposes P3D ResNet blocks that combine spatial and temporal convolutions within a deep residual architecture.

  • Video is harder than image recognition because temporal sequences contain large variations and complexities that complicate generic spatio-temporal representation learning.
  • 3D CNNs capture spatial appearance and temporal evolution, but their computational cost and model size make very deep networks difficult to train.An 11-layer C3D model is 321MB, compared with 235MB for 152-layer 2D ResNet.
  • Frame pooling and RNN approaches connect temporal information mainly at high-level features, leaving low-level correlations such as edges underused.
  • P3D blocks replace standard 3 × 3 × 3 convolutions with spatial 1 × 3 × 3 and temporal 3 × 1 × 1 convolutions arranged in parallel or cascaded forms.
  • P3D ResNet places varied blocks throughout a ResNet-like architecture to increase structural diversity and construct temporal connections from bottom to top.
  • Experiments evaluate P3D ResNet on five benchmarks and three tasks, reporting performance above several state-of-the-art models.

2. Related Work

Prior video-representation methods include handcrafted local descriptors and deep architectures based on frame sequences, optical flow, or direct 3D convolutions. The paper identifies incomplete modeling of temporal evolution across consecutive frames as a limitation of many existing approaches.

  • Handcrafted methods detect spatio-temporal interest points and describe them using extensions of image representations for temporal 3D volumes.
  • Deep learning methods use frame-level CNN windows, two-stream RGB and optical-flow networks, and extensions such as multi-granular structure or temporal segment networks.
  • Many deep methods treat video as a sequence of frame or optical-flow images rather than fully exploiting temporal evolution across consecutive frames.
  • 3D CNNs directly learn spatio-temporal representations from short video clips by modeling spatial information and temporal connections jointly.

3. P3D Blocks and P3D ResNet

The paper develops Pseudo-3D residual blocks that decompose spatial and temporal processing, then combines three block designs in a ResNet-like video architecture. Experiments compare these variants and the mixed P3D ResNet in terms of accuracy, model size, and speed.

  • 3.1. 3D Convolutions: Pseudo-3D convolutions decouple a 3×3×3 operation into 1×3×3 spatial and 3×1×1 temporal filters, reducing model size while enabling 2D CNN pre-training.The spatial filters encode frame appearance, while the temporal filters connect adjacent feature maps across time.
  • 3.2. Pseudo-3D Blocks: P3D blocks replace 2D Residual Units with spatial and temporal filters whose direct connections are varied across pathways and outputs.The two design choices concern whether spatial filters feed temporal filters directly and whether both filter types directly influence the final output.
  • 3.2. Pseudo-3D Blocks: P3D-A cascades temporal filters after spatial filters, so the two filter types directly influence each other while only temporal filters connect directly to the output.Its residual form is represented as xt+1 = xt + T(S(xt)).
  • 3.2. Pseudo-3D Blocks: P3D-B places spatial and temporal filters on parallel pathways, while P3D-C adds a shortcut from spatial filters to the final output.P3D-B directly accumulates both filter outputs; P3D-C combines this spatial shortcut with the cascaded temporal path.
  • 3.3. Pseudo-3D ResNet: The authors construct single-design P3D ResNet variants by replacing every ResNet-50 Residual Unit, then propose a mixed architecture interleaving P3D-A, P3D-B, and P3D-C.The complete P3D ResNet follows the block order P3D-A→P3D-B→P3D-C to pursue structural diversity.
  • 3.3. Pseudo-3D ResNet: 8.6 ∼9.0 clips per second is reported for the P3D ResNet variants, while all three variants outperform ResNet-50 with only a small model-size increase on UCF101.The comparisons evaluate performance and time efficiency for the ResNet-50 baseline and Pseudo-3D variants.

4. Spatio-Temporal Representation Learning

The authors evaluate P3D ResNet as a Sports-1M-trained video representation and compare its classification performance with established architectures. They also examine structural diversity, visualization, and the construction of a generic 2,048-dimensional representation.

  • Training and evaluation: P3D ResNet is trained on Sports-1M, a large-scale benchmark containing about 1.13 million videos across 487 sports labels.The training uses the deeper 152-layer P3D ResNet architecture.
  • Training and evaluation: 20 clips of 16 frames are sampled per video, and each clip is evaluated with a single center crop before averaging clip-level scores into a video-level prediction.
  • Classification results: P3D ResNet improves top-1 video-level accuracy over ResNet-152 and C3D by 1.8% and 5.3%, respectively.The comparison is reported on the evaluation test set in Table 2.
  • Classification results: P3D ResNet outperforms Deep Video (Slow Fusion), while Convolutional Pooling remains stronger when pooling 120-frame clips sampled at 1 fps.P3D ResNet instead uses 16 consecutive frames covering less than 0.5 seconds.
  • Model analysis: DeepDraw visualizations generate input clips through iterative gradient ascent to expose class knowledge for tai chi, horizontal bar, motorcycle racing, and boxing.
  • Model analysis: The generic P3D ResNet representation averages 2,048-dimensional pool5 activations from 20 clips to produce one 2,048-dimensional vector per video.

5. Video Representation Evaluation

The learned P3D ResNet representation is evaluated across action recognition, action similarity labeling, and scene recognition on five benchmarks. It consistently improves over several baselines, while remaining robust to reduced representation dimensionality and producing more semantically separated embeddings than ResNet-152.

  • Action recognition: On ActivityNet validation, P3D ResNet improves Top-1 accuracy over IDT, C3D, VGG 19, and ResNet-152 by 10.4%, 9.3%, 8.5%, and 3.7%, respectively.The paper attributes this to combining image-pretrained spatial convolutions with video-learned temporal convolutions.
  • Action similarity labeling: P3D ResNet also improves action similarity labeling over C3D and ResNet-152, combining image-domain spatial pretraining with video-domain temporal learning.
  • Scene recognition: P3D ResNet outperforms handcrafted features by 16.9% on Dynamic Scene and 3.3% on YUPENN, with additional YUPENN gains of 1.4% over C3D and 0.3% over ResNet-152.
  • Representation analysis: On UCF101, P3D ResNet consistently outperforms other representations across dimensions from 500 to 10 after PCA reduction.ResNet-152 degrades more sharply as dimensionality decreases, whereas P3D ResNet is described as more robust to this change.
  • Representation analysis: t-SNE visualization of 10K UCF101 videos shows that P3D ResNet embeddings are more semantically separated than ResNet-152 embeddings.

6. Conclusion

P3D ResNet learns spatio-temporal video representations by combining 2D spatial and 1D temporal convolutions in diverse residual-block placements. Experiments show effectiveness across five datasets and three tasks, while future work targets attention, clip length, and additional input modalities.

  • 6. Conclusion: P3D ResNet simplifies 3D convolutions by combining 2D spatial filters with 1D temporal connections in a residual framework.Different block variants are integrated at different placements to promote structural diversity.
  • 6. Conclusion: Experiments on five datasets show effective spatio-temporal representations for action recognition, action similarity labeling, and scene recognition.Performance improvements are observed against other feature-learning techniques.
  • 6. Conclusion: Attention mechanisms, varying the number of training frames, and extending inputs to optical flow or audio remain future research directions.These directions are proposed to further enhance representation learning and broaden the input types handled by P3D ResNet.
Loading 1711.10305v1…