PaperlayerView exact revision on arXiv

Source-linked AI summary

Convolutional Tensor-Train LSTM for Spatio-temporal Learning

Jiahao Su, Wonmin Byeon, Jean Kossaifi, Furong Huang, Jan Kautz, Animashree Anandkumar

arXiv:2002.09131v5cs.LGcs.CVstat.ML

TL;DR

Long-term video prediction requires learning long-range spatio-temporal correlations, while higher-order recurrent transitions can be prohibitively complex. The paper introduces a higher-order convolutional LSTM using convolutional tensor-train decomposition and preprocessing to incorporate longer histories efficiently. It reports an 8% accuracy increase over the state-of-the-art on Something-Something v2 and state-of-the-art multi-step video prediction on Moving-MNIST-2 and KTH.

  • Problem

    First-order recurrent models use only the previous time step, while higher-order transitions for spatio-temporal learning face exponential complexity.

    Source: §1 · p. 2

  • Method

    The model combines longer histories through convolutional LSTM updates, using convolutional tensor-train factorization and preprocessing to control complexity.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    An 8% absolute accuracy increase over the state-of-the-art on Something-Something v2 accompanies new state-of-the-art multi-step prediction on Moving-MNIST-2 and KTH.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The model reports better results than previous works while using only a fraction of their parameters.

    Source: §1 · p. 2

Abstract

from arXiv · show

Learning from spatio-temporal data has numerous applications such as human-behavior analysis, object tracking, video compression, and physics simulation.However, existing methods still perform poorly on challenging video tasks such as long-term forecasting. This is because these kinds of challenging tasks require learning long-term spatio-temporal correlations in the video sequence. In this paper, we propose a higher-order convolutional LSTM model that can efficiently learn these correlations, along with a succinct representations of the history. This is accomplished through a novel tensor train module that performs prediction by combining convolutional features across time. To make this feasible in terms of computation and memory requirements, we propose a novel convolutional tensor-train decomposition of the higher-order model. This decomposition reduces the model complexity by jointly approximating a sequence of convolutional kernels asa low-rank tensor-train factorization. As a result, our model outperforms existing approaches, but uses only a fraction of parameters, including the baseline models.Our results achieve state-of-the-art performance in a wide range of applications and datasets, including the multi-steps video prediction on the Moving-MNIST-2and KTH action datasets as well as early activity recognition on the Something-Something V2 dataset.

1 Introduction

Learning spatio-temporal data is challenging because video understanding must capture content and dynamics together, while first-order recurrent models use only the previous step. The paper proposes a higher-order convolutional LSTM with convolutional tensor-train decomposition and reports improved results across activity recognition and video prediction.

1 Introduction

  • First-order recurrent models update from only the previous time step, making long-range temporal correlations intrinsically difficult to capture.Video understanding also requires spatial-temporal representations that capture content and dynamics simultaneously.Source: §1 · pp. 1–2
  • Higher-order transitions can capture longer histories, but their tensor representations typically cause exponential complexity, especially for spatio-temporal ConvLSTM.Prior tensorized higher-order LSTMs address sequence forecasting but do not directly extend to general spatio-temporal data.Source: §1 · p. 2
  • Higher-order ConvLSTM incorporates a long history while preserving spatial structure, and convolutional tensor-train decomposition makes it tractable with fewer parameters than previous works.The factorization recursively approximates convolutional kernels across time and introduces an information bottleneck.Source: §1 · p. 2
  • An 8% absolute accuracy increase over the state-of-the-art on Something-Something v2 accompanies new state-of-the-art multi-step prediction on Moving-MNIST-2 and KTH.The paper also reports outperforming 3D-CNN and ConvLSTM by a large margin.Source: §1 · p. 2
  • The paper reports consistent performance improvements when training higher-order models with preprocessing, gradient clipping, and learning scheduling.These are presented as a procedure for incorporating longer temporal context and improving training.Source: §1 · p. 2

2 Background: Convolutional LSTM and Higher-order LSTM

This section reviews recurrent models for sequence and spatio-temporal learning, then contrasts first-order LSTMs with higher-order updates. Higher-order updates incorporate more past states, but polynomial transitions can incur exponential complexity when implemented naively.

Background

  • An LSTM updates its hidden and cell states from the immediate previous states and current input, making it a first-order Markovian model.The paper notes difficulty capturing long-term dynamics as one limitation of LSTMs.Source: §2 · pp. 2–3
  • ConvLSTM extends LSTM to spatio-temporal data by representing states, cell memory, gates, and parameters as high-dimensional tensors and using convolution.The convolution operation is defined between states and parameters.Source: §2 · p. 3
  • Higher-order LSTM incorporates more previous states in each update, partially addressing the long-term-dynamics limitation of first-order LSTM.The update combines N prior hidden states through Φ.Source: §2 · p. 3
  • A polynomial higher-order transition has exponential space and computational complexity when implemented naively, unlike the linear function's linear growth in N.The polynomial transition contracts a tensor with the outer product of previous states.Source: §2 · p. 3

3 Methodology: Convolutional Tensor-Train LSTM

The methodology develops a higher-order ConvLSTM that combines previous hidden states while retaining spatial structure and controlling complexity. It uses convolutional tensor-train factorization and preprocessing to make longer-history updates efficient and trainable.

3.1 Extending ConvLSTM to Higher-orders

  • The higher-order transition must preserve spatial structure, expand receptive field with temporal distance, and keep space and time complexity O(N).Existing polynomial tensor-train transitions meet the complexity condition but do not preserve spatial structure; convolutional linear extensions with fixed filters fail the receptive-field condition, while growing filters incur O(N^3) complexity.Source: §3.1 · p. 3

3.2 Designing an Effective and Efficient Higher-order ConvLSTM

  • The convolutional tensor-train decomposition jointly represents convolutional kernels across time with smaller factors while maintaining spatial structure.Its computation is linear in the number of time steps, satisfying the stated complexity requirement.Source: §3.2 · p. 4
  • Preprocessing groups neighboring hidden states with a sliding window and convolution, reducing the steps and channels passed into the tensor-train module.This separates the history length from decomposition order and permits lower ranks, addressing long-order gradient problems and parameter limits.Source: §3.2 · pp. 4–5
  • Conv-TT-LSTM combines preprocessing with a tensor-train module that aggregates spatio-temporal information and computes the LSTM gates.The illustrated single-channel addition accumulates outputs from multiple channels when the model has multiple channels.Source: Figure 1 · p. 4, §3.2 · p. 5

W ∗X(t) + CTT

  • The final implementation reduces parameters, can be more compact than first-order ConvLSTM, and uses a low-rank constraint as an implicit regularizer.The tensor-train structure also encodes correlations associated with the natural flow of time.Source: W ∗X(t) + CTT · p. 5

4 Experiments

Experiments evaluate Conv-TT-LSTM on multi-step video prediction and early activity recognition, with the model outperforming existing approaches across the evaluated tasks and datasets.

4 Experiments

  • Across video prediction and early activity recognition datasets, Conv-TT-LSTM outperforms existing approaches.The experiments cover Moving-MNIST-2, KTH action, and Something-Something V2.Source: §4 · p. 5

4.1 Implementation Details

  • Training uses task-specific learning-rate scheduling and gradient clipping, which the authors report as important for stable, improved training of higher-order models.All models use ADAM; gradient clipping uses value 1 with a learning rate of 1e−3.Source: §4.1 · p. 5
  • Video prediction tests 10–40 recursively predicted future frames after 10 input frames; early recognition predicts the next 10 frames from 25%–50% of a video while classifying its activity.The video-prediction datasets are KTH at 128×128 and Moving-MNIST-2 at 64×64; recognition uses a 41-category subset.Source: §4.1 · p. 5
  • The recognition model combines prediction loss with classification cross-entropy, while its architecture places eight RNN layers between two-layer convolutional encoder and decoder modules.Video prediction optimizes an ℓ1+ℓ2 loss; the recognition loss weights prediction loss by λ.Source: §4.1 · p. 5

4.2 Experimental Results

  • On Something-Something V2, Conv-TT-LSTM consistently beats ConvLSTM, 3D-CNN, and E3D-LSTM across input-frame ratios; examples show 25% and 50% inputs with prediction confidence.Per-activity gains are substantial for Moving and Tearing, but marginal for static activities such as Holding and Pouring.Source: §4.2 · p. 6, Table 1 · p. 7, Figure 4 · p. 7
  • For long-term Moving-MNIST prediction, Conv-TT-LSTM produces sharp, realistic digits while reproduced PredRNN++ and E3D-LSTM outputs become vague and blurry, especially after 20 steps.Figure 3 shows 30-frame Moving-MNIST and 20-frame KTH predictions given 10 input frames.Source: §4.2 · p. 6, Figure 3 · p. 7
  • On Moving-MNIST-2, Conv-TT-LSTM beats ConvLSTM at 10- and 30-frame prediction and outperforms prior approaches on SSIM and LPIPS with less than one-fifth their parameters.Figure 2 compares SSIM and LPIPS, with bubble size representing model size.Source: Figure 2 · p. 6, §4.2 · p. 6
  • On KTH, Conv-TT-LSTM beats ConvLSTM at 20 and 40 frames and leads prior models on LPIPS at both horizons, though its 20-frame SSIM is lower than state-of-the-art models.Visual examples show sharper frames and better-preserved human silhouettes, with slight shifting artifacts over time.Source: §4.2 · p. 6, Table 4 · p. 8

5 Discussion

The discussion tests the proposed modules against ablations and a compressed ConvLSTM baseline, and compares the model’s efficiency and design with other approaches. It also notes that fewer FLOPs do not necessarily mean faster execution.

5 Discussion

  • Ablations that remove higher-order modeling or replace 5 × 5 filters with 1 × 1 filters decrease performance, at best matching ConvLSTM.The authors conclude that both higher-order modeling and convolutional operations are necessary.Source: §5 · p. 8
  • Reducing channels while retaining steps can cause gradient vanishing or exploding, while reusing all prior states yields a tube-shaped receptive field that does not distinguish recent from remote history.Source: §5 · p. 8
  • Conv-TT-LSTM has fewer FLOPs and parameters than the compared models, but its sequential module can limit latency gains.On KTH, Conv-TT-LSTM takes 27.3 minutes per epoch (37.83 GFLOPs), versus ConvLSTM’s 26.2 minutes (55.83 GFLOPs).Source: §5 · pp. 8–9
  • Unlike Yang et al. [40], CTTD compresses higher-order convolutional hidden-to-hidden kernels and uses convolutions to preserve spatial structure.Yang et al. [40] compresses the input-hidden transition of a first-order fully connected LSTM.Source: §5 · p. 9
  • With similar parameter counts, Conv-TT-LSTM outperforms compressed ConvLSTM on MNIST and KTH, except on LPIPS for KTH.The compressed baseline uses tensor-train storage for the input-hidden transition.Source: §5 · p. 9
  • The authors contrast Conv-TT-LSTM’s compact, structure-preserving operator with Transformers’ memory and computation demands and need for auxiliary components on spatial data.They note that attention does not preserve spatial structure.Source: §5 · p. 9

6 Related Work

Prior work applies tensor decompositions to neural networks and recurrent models, while this paper introduces convolutional tensor-train decomposition for higher-order convolutional recurrence. It situates that contribution among video prediction methods built around ConvLSTM.

6 Related Work

  • Tensor-train RNNs have compressed recurrent input-state and state-state matrices by reshaping them into tensors, including for video classification [40].Source: §6 · p. 9
  • CTTD targets efficient higher-order ConvLSTM and preserves spatial structure by factorizing convolutional kernels with convolutions, unlike Yang et al.’s first-order fully connected LSTM [40].Source: §6 · p. 10
  • Prior spatio-temporal prediction work includes short-term video prediction and motion-content decomposition, with many methods using ConvLSTM as a base module [42] [43] [44] [25] [45] [46].Byeon et al. [43] demonstrated a deep ConvLSTM baseline used as the base architecture in this paper.Source: §6 · p. 10

7 Conclusion

The conclusion presents a fully convolutional higher-order LSTM whose convolutional tensor-train decomposition jointly parameterizes convolutions and encodes temporal dependencies. The supplement reports a constructive linear-time algorithm and additional implementation and ablation material.

7 Conclusion

  • The proposed fully convolutional higher-order LSTM uses convolutional tensor-train decomposition to jointly parameterize convolutions and encode temporal dependencies compactly.The paper reports performance gains on video prediction and early activity recognition, including Something-Something V2, Moving-MNIST-2, and KTH.Source: §7 · p. 10

Appendix: Convolutional Tensor-Train LSTM for Spatio-temporal Learning

A An Efficient Algorithm for Convolutional Tensor-Train Module

B Experimental Details

The experimental-details section introduces the setups for video prediction and early activity recognition. It compares preprocessing windows and describes the architectures used for both tasks.

B Experimental Details

  • The appendix section outlines experiment setups covering datasets, model architectures, learning strategies, and evaluation metrics for video prediction and early activity recognition.Source: B Experimental Details · p. 16

B.1 Preprocessing Module

  • Sliding windows use fewer preprocessing parameters, preserve chronological order, and slightly outperform fixed windows on Moving-MNIST-2, motivating their selection.The fixed-window approach’s preprocessed states are equivalent and do not reflect their times in the sequence.Source: §B.1 · p. 17

B.2 Model Architectures

  • Video prediction uses a 12-layer recurrent stack; early recognition combines 2D-CNN encoding and decoding, video prediction, and a 3D-CNN classifier.The early-recognition experiments compare ConvLSTM, Conv-TT-LSTM, and causal 3D-convolutional realizations of each layer.Source: §B.2 · p. 17

B.3 Hyper-parameters selection.

The hyperparameter search selects settings for Conv-TT-LSTM and tensor-train-compressed ConvLSTM, balancing performance against training stability and diminishing returns.

B.3 Hyper-parameters selection

  • Conv-TT-LSTM uses order N = 3, ranks C(i) = 8, and M = 5 after performance plateaus at larger ranks and step counts and high-order training becomes unstable.Performance typically increases with order, but N = 5 causes gradient instability.Source: §B.3 · p. 18, Table 6 · p. 18
  • Conv-TT-LSTM uses learning rate 10^-3 with gradient clipping value 1 because higher rates are unstable and 10^-4 learns poorly.The authors use these settings for all experiments.Source: §B.3 · p. 18
  • Tensor-train-compressed ConvLSTM uses K = 5 and learning rate 10^-3; order N = 2 and rank R = 8 win by a small margin.The compressed models are reported as robust to different orders and ranks.Source: §B.3 · p. 18, Table 7 · p. 18

B.4 Datasets

The experiments use Moving-MNIST-2, KTH, and Something-Something V2 for video prediction and activity recognition, evaluated with pixel-based and perceptual similarity metrics.

B.4 Datasets

  • Moving-MNIST-2 videos contain two moving 28 × 28 MNIST digits on a 64 × 64 canvas, with 10,000 training, 3,000 validation, and 5,000 test videos.Digits start at random locations, move at constant velocity, and bounce at canvas boundaries.Source: §B.4 · p. 18
  • KTH contains 25 people performing six actions; the setup trains on people 1–16, tests on 17–25, and predicts 10 frames from 10 inputs.Frames are resized to 128 × 128 pixels.Source: §B.4 · p. 18
  • Something-Something V2 uses the official 41-category subset, with 55,111 training and 7,518 test videos and 10% of training reserved for validation.Videos are 2–6 seconds long at 24 fps.Source: §B.4 · p. 19

B.5 Evaluation Metrics

  • Evaluation uses MSE or PSNR, SSIM, and LPIPS; the paper notes pixel-wise metrics favor vague, blurry predictions while LPIPS aligns better with human perception.LPIPS measures similarity between deep features.Source: §B.5 · p. 19

B.6 Ablation Studies

Ablations show Conv-TT-LSTM outperforming ConvLSTM across tested architecture, loss, and learning-schedule settings, with sliding windows best under the selected setup.

B.6 Ablation Studies

  • Conv-TT-LSTM improves on ConvLSTM across all tested settings, varying layer count, loss function, and scheduled sampling versus teacher forcing.The table identifies these as ablations of the experimental setup.Source: §B.6 · p. 19, Table 8 · p. 19
  • The ablations support the authors’ claim that their selected setup is optimal within the tested Moving-MNIST-2 search space.The tested changes include reducing the model from 12 layers to 4, using L1 alone, and replacing scheduled sampling with teacher forcing.Source: §B.6 · p. 19
  • Under the optimal experimental setting, the sliding-window approach outperforms the fixed-window approach.FW denotes fixed window and SW denotes sliding window.Source: Table 8 · p. 19

C Additional Experimental Results

Per-frame comparisons show Conv-TT-LSTM’s advantage over baselines on Moving-MNIST-2 and KTH, while supplementary figures and clips provide additional prediction and recognition examples.

C Additional Experimental Results

Related papers

Loading 2002.09131v5…