PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatiotemporal Pyramid Network for Video Action Recognition

Yunbo Wang, Mingsheng Long, Jianmin Wang, Philip S. Yu

arXiv:1903.01038v1cs.CV

TL;DR

Video recognition methods lack effective modeling of spatial-temporal correlations across abstraction levels, with spatial streams vulnerable to similar backgrounds and temporal streams to short-term action ambiguity. The paper proposes a hierarchical spatiotemporal pyramid with long-term temporal fusion, motion-guided attention, unified training, and compact bilinear interactions. The network achieves state-of-the-art performance on UCF101 and HMDB51, including average-accuracy gains of 0.6% and 0.4%, respectively.

  • Problem

    Existing video CNNs do not fully exploit spatiotemporal features, while two-stream models can fail on similar backgrounds, short temporal snippets, and subtle cross-stream relationships.

  • Method

    The paper builds a hierarchical spatiotemporal pyramid that combines longer-range temporal fusion, motion-guided spatial attention, unified optimization, and compact bilinear fusion.

  • Results

    The network achieves state-of-the-art performance on UCF101 and HMDB51; its best implementation improves average accuracy by 0.6% and 0.4%, respectively.

  • Takeaways & Limitations

    Hierarchical fusion lets spatial and temporal cues reinforce one another while compact bilinear fusion models their interactions efficiently.

Abstract

from arXiv · show

Two-stream convolutional networks have shown strong performance in video action recognition tasks. The key idea is to learn spatiotemporal features by fusing convolutional networks spatially and temporally. However, it remains unclear how to model the correlations between the spatial and temporal structures at multiple abstraction levels. First, the spatial stream tends to fail if two videos share similar backgrounds. Second, the temporal stream may be fooled if two actions resemble in short snippets, though appear to be distinct in the long term. We propose a novel spatiotemporal pyramid network to fuse the spatial and temporal features in a pyramid structure such that they can reinforce each other. From the architecture perspective, our network constitutes hierarchical fusion strategies which can be trained as a whole using a unified spatiotemporal loss. A series of ablation experiments support the importance of each fusion strategy. From the technical perspective, we introduce the spatiotemporal compact bilinear operator into video analysis tasks. This operator enables efficient training of bilinear fusion operations which can capture full interactions between the spatial and temporal features. Our final network achieves state-of-the-art results on standard video datasets.

1. Introduction

CNN-based video methods model spatial and temporal cues but are not specifically designed to fully exploit spatiotemporal features. The proposed architecture addresses stream-specific failures through longer temporal context, motion-guided spatial attention, and hierarchical fusion.

  • CNN-based video methods jointly model spatial and temporal cues but cannot fully exploit spatiotemporal features.
  • Averaging spatial and temporal classifier outputs is insufficient because one stream often fails while the other remains correct.
  • The proposed end-to-end architecture uses hierarchical fusion strategies and a unified spatiotemporal loss to jointly optimize the network.
  • Longer optical-flow sequences help distinguish actions that resemble one another in short snippets but differ over extended time.The paper illustrates this issue with Pull-ups and RopeClimbing in UCF101.
  • Motion information guides a spatiotemporal attention module toward significant locations when similar backgrounds make spatial classification ambiguous.On UCF101, the optical-flow network outperforms the spatial stream by at least 5% for the cited ambiguous classes.
  • Compact bilinear fusion captures interactions between spatial and temporal features while reducing traditional bilinear-fusion parameters from millions to several thousands.

2. Related Work

Video recognition research progressed from spatiotemporal convolutions and temporal connectivity schemes to two-stream CNNs using optical flow. Prior two-stream methods remain limited by short temporal windows, separate stream training, and weak spatiotemporal interaction modeling.

  • Early approaches extended convolution into the temporal domain or compared late, early, and slow fusion across time.These methods produced only modest improvements and did not fully utilize motion information.
  • Two-stream CNNs added an optical-flow stream to capture motion alongside RGB appearance features.
  • The original two-stream method uses only 10 consecutive optical-flow frames and therefore misses long-term temporal cues.
  • Its spatial and temporal streams are trained separately, with final predictions obtained by averaging two classifier outputs rather than learning subtle cross-stream relationships.

3. Spatiotemporal Pyramid Network

The spatiotemporal pyramid network hierarchically fuses long-range temporal, spatial, and attention features using compact bilinear operations. Its fusion layers support end-to-end training and produce a joint video representation.

  • Spatiotemporal Compact Bilinear Fusion: Bilinear fusion captures multiplicative interactions between spatial and temporal features, while STCB projects them into lower-dimensional compact representations.The projection avoids explicit outer-product computation and reduces parameters from millions to several thousands using Count Sketch and FFT-based convolution.
  • Temporal Fusion: The network broadens temporal context by combining optical-flow representations sampled across a longer video sequence.Multiple CNNs with shared parameters sample frames at interval τ, with each chunk covering previous and next L/2 frames.
  • Spatiotemporal Attention: A spatiotemporal attention module merges spatial feature maps with corresponding motion features to identify salient activity regions.Attention weights are produced over feature-map locations and normalized before weighted pooling replaces ordinary pooling.
  • Pyramid Architecture: The architecture integrates the fusion layers between convolutional and fully connected layers and can be initialized from pretrained CNN models across several backbones.The described backbones include BN-Inception, ResNets, and VGGnet.
  • Pyramid Architecture: The pyramid uses three STCB stages: temporal fusion, motion-guided spatial attention, and final fusion of spatial, temporal, and attended features.The top stage produces a 4096-dimensional vector from the three preceding outcomes.

4. Experiments

Experiments evaluate base architectures and progressively combine compact bilinear fusion, multi-path temporal fusion, and attention pooling. The resulting architecture improves over two-stream baselines and state-of-the-art systems on UCF101 and HMDB51.

  • 4.2. Base Architectures: BN-Inception is the top-performing base architecture, while deeper networks benefit RGB more than optical flow.Optical-flow performance decreases slightly with depth because of distribution differences and over-fitting on limited UCF101 data.
  • 4.3. Spatiotemporal Compact Bilinear Fusion: Late fusion between the last convolutional layer and classifier outperforms fusion moved to earlier layers.The experiments attribute this to the class-specific nature of the last convolutional features.
  • 4.3. Spatiotemporal Compact Bilinear Fusion: Spatiotemporal compact bilinear fusion achieves the highest fusion accuracy, improving performance by around 1.5 points.A 4096-dimensional output is appropriate for both video datasets, balancing compression and representation quality.
  • 4.4. Temporal Fusion: A 3-path temporal network with spatiotemporal compact bilinear fusion outperforms single-path and larger-path alternatives.The 3-path model is 2.3 points higher than the single-path model; more paths can weaken spatial-temporal correlation over very long sequences.
  • 4.5. Spatiotemporal Attention: Attention pooling boosts the spatial pathway by 2.1 points and helps address errors from similar or chaotic backgrounds.Using temporal representations only for attention weights performs below average pooling, whereas earlier compact bilinear fusion performs surprisingly well.
  • 4.7. Final Results: Ablation results show cumulative gains: spatiotemporal fusion adds 1.5 points, multi-path temporal fusion adds 0.4 points, and the final model reaches 94.2%.Compared with prior state of the art, the best BN-Inception implementation improves average accuracy by 0.6% on UCF101 and 0.4% on HMDB51.

5. Conclusions

The paper proposes a hierarchical spatiotemporal pyramid network that combines spatial and temporal features so they reinforce each other. It introduces spatiotemporal compact bilinear fusion and reports state-of-the-art performance on UCF101 and HMDB51.

  • The spatiotemporal pyramid network combines spatial and temporal features through hierarchical fusion strategies at different abstraction levels.The fusion modules are trained jointly to complement each other.
  • Spatiotemporal compact bilinear fusion learns element-wise interactions between spatial and temporal features for video analysis.The paper compares it with concatenation and element-wise sum.
  • The network achieves state-of-the-art performance on UCF101 and HMDB51.
Loading 1903.01038v1…