PaperlayerView exact revision on arXiv

Source-linked AI summary

TS-LSTM and Temporal-Inception: Exploiting Spatiotemporal Dynamics for Activity Recognition

Chih-Yao Ma, Min-Hung Chen, Zsolt Kira, Ghassan AlRegib

arXiv:1703.10667v1cs.CV

TL;DR

Prior work had not systematically clarified how architectures exploit spatiotemporal dynamics or how differing two-stream baselines affect comparisons. This paper establishes a strong ResNet-101 baseline, evaluates LSTM and Temporal-ConvNet designs, and proposes TS-LSTM and Temporal-Inception. Both approaches can achieve high or state-of-the-art accuracy, but their architectures require careful design, with LSTMs requiring pre-segmented inputs to fully exploit temporal information.

  • Problem

    Prior methods did not systematically analyze architectures for exploiting spatiotemporal dynamics, and differing two-stream baselines obscured method comparisons.

  • Method

    The paper builds a ResNet-101 two-stream baseline, evaluates RNNs and Temporal-ConvNets, and proposes Temporal Segment LSTM and Inception-style Temporal-ConvNet models.

  • Results

    Both LSTMs and Temporal-ConvNets can exploit spatiotemporal dynamics and improve performance; TS-LSTM surpasses existing methods, while Temporal-Inception achieves state-of-the-art performance.

  • Takeaways & Limitations

    Careful architectural design matters: integrating temporal segments with LSTMs and using multi-scale temporal convolutions are effective ways to model activity dynamics.

  • Takeaways & Limitations

    LSTMs require pre-segmented inputs to fully exploit temporal information, and deeper LSTM layers can overfit rather than improve recognition.

Abstract

from arXiv · show

Recent two-stream deep Convolutional Neural Networks (ConvNets) have made significant progress in recognizing human actions in videos. Despite their success, methods extending the basic two-stream ConvNet have not systematically explored possible network architectures to further exploit spatiotemporal dynamics within video sequences. Further, such networks often use different baseline two-stream networks. Therefore, the differences and the distinguishing factors between various methods using Recurrent Neural Networks (RNN) or convolutional networks on temporally-constructed feature vectors (Temporal-ConvNet) are unclear. In this work, we first demonstrate a strong baseline two-stream ConvNet using ResNet-101. We use this baseline to thoroughly examine the use of both RNNs and Temporal-ConvNets for extracting spatiotemporal information. Building upon our experimental results, we then propose and investigate two different networks to further integrate spatiotemporal information: 1) temporal segment RNN and 2) Inception-style Temporal-ConvNet. We demonstrate that using both RNNs (using LSTMs) and Temporal-ConvNets on spatiotemporal feature matrices are able to exploit spatiotemporal dynamics to improve the overall performance. However, each of these methods require proper care to achieve state-of-the-art performance; for example, LSTMs require pre-segmented data or else they cannot fully exploit temporal information. Our analysis identifies specific limitations for each method that could form the basis of future work. Our experimental results on UCF101 and HMDB51 datasets achieve state-of-the-art performances, 94.1% and 69.0%, respectively, without requiring extensive temporal augmentation.

1. Introduction

Human action recognition benefits from temporal information but remains computationally demanding, while traditional two-stream ConvNets do not model correlations between spatial and temporal streams. The paper evaluates recurrent and temporal-convolutional designs, proposing TS-LSTM and Temporal-Inception to improve spatiotemporal modeling.

  • Temporal correlations between video frames provide motion information, but processing hundreds of frames makes action recognition computationally demanding.
  • Traditional two-stream ConvNets separately process RGB and optical-flow images and fuse stream prediction scores without modeling cross-stream visual correlations.
  • The paper establishes a strong two-stream ConvNet baseline and systematically evaluates recurrent and Temporal-ConvNet approaches for exploiting temporal information.
  • TS-LSTM: TS-LSTM combines LSTM-based fusion of spatial and temporal features with temporal segments, while direct LSTM performs similarly to naive mean or max pooling.
  • Temporal-ConvNet: Temporal-Inception extends stacked temporal convolutions to multiple scales and achieves state-of-the-art performance using feature vectors rather than feature maps.
  • The analysis identifies limitations in LSTMs’ ability to exploit temporal information without manipulating or pre-segmenting their inputs.

2. Related work

Related work includes 3D ConvNets, ConvNet–RNN hybrids, and extensions of two-stream ConvNets, but comparisons are complicated by differing architectures and procedures. This paper provides a common baseline for analyzing Temporal Segment LSTM and Temporal-ConvNet designs.

  • Action-recognition systems commonly combine high-dimensional frame-level features, whether hand-crafted or learned, before classification.
  • 3D ConvNets: 3D ConvNets learn spatiotemporal features from stacked frames, but replacing 2D kernels with 3D kernels increases GPU-memory requirements.
  • ConvNets with RNNs: ConvNet–RNN methods feed frame-level spatial features to LSTMs, enabling variable-length inputs and long-term dependencies compared with fixed-input traditional models.
  • Two-stream ConvNets: Two-stream extensions combine RGB and optical-flow information through LSTMs or 3D operations, whereas Temporal-ConvNet uses feature vectors rather than feature maps.
  • The paper addresses prior inconsistency by creating a common strong two-stream baseline and analyzing design choices for RNN and Temporal-ConvNet approaches.

3. Approach

The approach builds on a two-stream ConvNet and compares recurrent and temporal-convolutional frameworks for integrating spatial and temporal features. TS-LSTM uses temporal segmentation and pooling before LSTM processing, while Temporal-Inception applies multi-scale temporal convolutions to feature matrices.

  • Two-stream ConvNets: The baseline uses separate spatial and temporal ConvNets, taking RGB images and stacked optical-flow images as inputs.The two streams are fused through their prediction scores in the traditional architecture.
  • Temporal Segment LSTM (TS-LSTM): TS-LSTM divides sampled video frames into segments, pools each segment, and sequentially feeds the pooled outputs to an LSTM.Mean and max pooling are considered; temporal mean pooling performs similarly to max pooling, while experiments use max pooling.
  • Temporal Segment LSTM (TS-LSTM): The TS-LSTM combines temporal pooling with recurrent modeling because directly using LSTMs performs similarly to naive mean or max pooling.The LSTM learns nonlinear feature combinations and segmental representations over time.
  • Temporal-ConvNet: Temporal-ConvNet applies convolutional layers to feature matrices whose columns represent high-level features across sampled frames.The architecture targets temporal correlations across feature vectors rather than spatial dependencies within each vector.
  • Temporal-Inception: Temporal-Inception uses multi-flow modules with different one-dimensional temporal kernels to encode multiple temporal scales and reduce temporal dimension.Each multi-flow module contains two temporal convolutional layers, and the architecture maps feature matrices to class predictions through fully connected layers.

4. Evaluation

Experiments on UCF101 and HMDB51 evaluate baseline two-stream ConvNets, TS-LSTM, and Temporal-ConvNet designs. Temporal segments, feature integration, and carefully designed temporal convolutions improve modeling, while vanilla or overly deep LSTMs have limitations.

  • Experiments evaluate spatial-stream, temporal-stream, and two-stream ConvNets across three splits of UCF101 and HMDB51.
  • TS-LSTM: Deeper LSTM layers often overfit, and the evaluated LSTM usage requires pre-segmented inputs despite LSTMs' theoretical ability to model varied temporal patterns.
  • TS-LSTM: Temporal segments improve classification with pre-saved feature vectors, while three and five segments perform similarly and may depend on action type.Temporal mean pooling performed similarly with the reported segment settings.
  • TS-LSTM: Adding an FC layer before temporal max pooling controls feature dimensionality, exploits spatial-temporal correlations, and helps prevent overfitting.
  • TS-LSTM: 0.2% improvement over the 92.6%-accuracy two-stream baseline was obtained with vanilla LSTM, whose performance was similar to naive temporal max pooling.The gain decreased when the baseline two-stream ConvNet was stronger.
  • Temporal-ConvNet: Multiple temporal convolution layers, concatenated multi-flow outputs, batch normalization, and dropout improve Temporal-ConvNet designs.The architecture must be chosen carefully because convolution across time can extract temporal patterns but design choices remain crucial.
  • Temporal-ConvNet: Temporal-Inception outperforms multi-flow Temporal-VGG by combining temporal information from varied receptive fields.Kernel sizes of 5 and 7 achieved the best prediction accuracy in the reported multi-flow experiments.
  • Final Performance: 93.7 to 94.1% TS-LSTM and 93.2 to 93.9% Temporal-Inception accuracy were achieved when more video frames were provided.The baseline using the first 10 seconds achieved 92.9% accuracy, slightly better than using full videos.

5. Conclusion & Discussion

The paper compares Temporal Segment LSTM and Temporal-ConvNet approaches for modeling dynamic temporal information. Both achieve state-of-the-art performance, while the authors identify LSTM modeling and regularization as areas for further work.

  • Temporal segments combined with LSTMs achieve state-of-the-art accuracy on UCF101 and HMDB51.
  • Temporal-ConvNet learns global video-level representations by convolving temporally constructed feature vectors.
  • Temporal-Inception achieves state-of-the-art performance using high-level feature vectors sampled equally from videos.
  • Future work includes larger datasets and better regularization of LSTMs.

A.1. Two-stream ConvNets Comparison on UCF101

The appendix compares two-stream ConvNet baselines trained with different network backbones and demonstrates a strong ResNet-101 baseline.

  • ResNet-101 provides a strong baseline for two-stream ConvNet action recognition.

A.2. Complete Experimental Results for TS-LSTM

The appendix reports TS-LSTM design findings, including the effects of pooling, temporal segmentation, feature dimension, and LSTM depth on UCF101 split 1.

  • Vanilla LSTM improves only modestly over the 92.6% baseline and can overfit training samples.
  • Naive temporal pooling performs similarly to vanilla LSTM, whereas integrating temporal segments with LSTMs increases accuracy.
  • Three and five temporal segments produce similar performance on UCF101 split 1, though the useful number may depend on the video classification problem.

A.3. Complete Experimental Results for Temporal-ConvNet

Temporal-ConvNet experiments show that architecture design, temporal dimension reduction, flow fusion, and kernel size critically affect performance and overfitting. Temporal-Inception benefits from multi-scale temporal processing and carefully chosen feature dimensions.

  • Architecture design: Temporal-Inception fuses multi-flow outputs at each layer and uses four TCLs to reduce temporal dimensionality, improving accuracy while avoiding overfitting.Temporal-VGG instead fuses flows at the end, while single- or double-TCL designs can retain excessively large dimensions.
  • Architecture design: 1024 is selected as the final fully connected feature dimension to balance discriminative capability against overfitting.
  • Dimension reduction: Average pooling, max pooling, and convolutional fusion reduce the increased filter dimension caused by concatenating multi-flow modules.The text contrasts reducing each module separately with reducing the dimension after concatenation.
  • Temporal kernels: Kernel sizes 5 and 7 combined with Temporal-Inception provide the best capability to represent actions with different temporal durations.Larger kernels are used for actions with longer temporal duration.
  • Dimension reduction: Two-stride convolution improves performance on split 1 but does not improve overall accuracy, despite potentially preserving more information than max pooling.

A.4. Statistics of the UCF101 dataset

The UCF101 video-length analysis tests how restricting temporal access affects recognition. Because many videos exceed the ten-second limit, the experiment assesses whether proposed temporal models continue benefiting from additional video information.

  • Video-length statistics: The experiment limits each method to the first 10 seconds of each video, corresponding to a maximum of 250 frames.
  • Video-length statistics: 21% of UCF101 videos are cut short by the 250-frame limit, representing 2805 of 13320 videos.
  • Experimental implication: Performance continues to improve as the proposed methods process additional temporal data, indicating effective use of temporal information.

A.5. Video Analysis of the UCF101 dataset

Video examples illustrate how TS-LSTM and Temporal-Inception use cross-frame information to resolve confusions that affect the frame-based baseline. They correctly classify challenging HighJump and PizzaTossing examples involving similar appearances or motions.

  • HighJump: The frame-based baseline misclassifies HighJump examples as JavelinThrow, LongJump, FloorGymnastics, or PoleVault because it lacks cross-frame information.
  • HighJump: TS-LSTM and Temporal-Inception correctly classify the HighJump examples, demonstrating their ability to extract temporal information.
  • PizzaTossing: PizzaTossing examples are confused with Punch, Nunchucks, and SalsaSpin when appearance or object motion is similar across categories.
  • PizzaTossing: Both proposed methods correctly classify the PizzaTossing examples by accounting for spatial and temporal correlations.

A.6. t-SNE Visualization

t-SNE visualizations show that TS-LSTM and Temporal-Inception produce more distinct feature clusters than the baseline. Class-specific examples also show substantial accuracy gains, with the strongest method varying by action category.

  • Feature-space structure: TS-LSTM and Temporal-Inception group test samples into more distinct clusters than the baseline two-stream ConvNet.
  • Class examples: 97.3% HighJump accuracy is achieved by TS-LSTM, compared with 62.2% for the baseline and 94.6% for Temporal-Inception.
  • Class examples: 97.0% PizzaTossing accuracy is achieved by Temporal-Inception, compared with 66.7% for the baseline and 90.9% for TS-LSTM.
Loading 1703.10667v1…