PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learning

Cheng Tan, Siyuan Li, Zhangyang Gao, Wenfei Guan, Zedong Wang, Zicheng Liu, Lirong Wu, Stan Z. Li

arXiv:2306.11249v2cs.CVcs.AI

TL;DR

Spatio-temporal predictive learning lacks a comprehensive benchmark, limiting systematic comparisons across varied methods and tasks. OpenSTL implements and evaluates recurrent-based and recurrent-free approaches, then extends recurrent-free models with MetaFormers. Its findings indicate that recurrence can help capture temporal dependencies but is not always necessary, as recurrent-free models can balance performance and efficiency.

  • Problem

    The field lacks a comprehensive benchmark, and whether recurrent-free models can match recurrent-based models without explicit temporal modeling remains unresolved.

    Source: §1 · p. 2

  • Method

    OpenSTL benchmarks recurrent-based and recurrent-free methods across tasks and extends recurrent-free architectures with MetaFormers.

    Source: §1 · p. 2, §3.1.2 · p. 5

  • Results

    Recurrent-based models capture temporal dependencies, but recurrent-free models achieve comparable performance with significantly higher efficiency on synthetic datasets.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Recurrent-free models can be a viable alternative with a good balance of efficiency and performance, especially for computationally expensive tasks.

    Source: §5 · p. 10

    The paper leaves open how to effectively combine recurrent-based and recurrent-free strengths, including whether extending existing MetaFormers can improve recurrent-free models.

    Source: §5 · p. 10

Abstract

from arXiv · show

Spatio-temporal predictive learning is a learning paradigm that enables models to learn spatial and temporal patterns by predicting future frames from given past frames in an unsupervised manner. Despite remarkable progress in recent years, a lack of systematic understanding persists due to the diverse settings, complex implementation, and difficult reproducibility. Without standardization, comparisons can be unfair and insights inconclusive. To address this dilemma, we propose OpenSTL, a comprehensive benchmark for spatio-temporal predictive learning that categorizes prevalent approaches into recurrent-based and recurrent-free models. OpenSTL provides a modular and extensible framework implementing various state-of-the-art methods. We conduct standard evaluations on datasets across various domains, including synthetic moving object trajectory, human motion, driving scenes, traffic flow and weather forecasting. Based on our observations, we provide a detailed analysis of how model architecture and dataset properties affect spatio-temporal predictive learning performance. Surprisingly, we find that recurrent-free models achieve a good balance between efficiency and performance than recurrent models. Thus, we further extend the common MetaFormers to boost recurrent-free spatial-temporal predictive learning. We open-source the code and models at https://github.com/chengtan9907/OpenSTL.

1 Introduction

Spatio-temporal predictive learning forecasts future frames from past frames without supervision, but the field lacks a comprehensive benchmark and a settled answer about whether recurrence is necessary. OpenSTL compares recurrent-based and recurrent-free approaches across diverse tasks and extends recurrent-free models with MetaFormers.

1 Introduction

  • OpenSTL addresses the lack of a comprehensive benchmark and the unresolved question of whether recurrent architectures are necessary for spatio-temporal predictive learning.The question is whether recurrent-free models can match recurrent-based models without explicit temporal modeling.Source: §1 · p. 2
  • OpenSTL uses a modular, extensible framework to revisit leading approaches and support fair comparisons.Source: §1 · p. 2
  • The benchmark includes 14 representative algorithms and 24 models, classified as recurrent-based or recurrent-free.Source: §1 · p. 2
  • Experiments span synthetic moving-object trajectories, human motion, driving scenes, traffic flow, and weather forecasting.The datasets range from synthetic to real-world data and from micro to macro scales.Source: §1 · p. 2
  • The paper generalizes MetaFormer-like architectures to boost recurrent-free learning and frames it as vision-backbone design for general applications.Source: §1 · p. 2

2 Background and Related work

The section defines prediction as mapping observed frames to future frames, then reviews recurrent and recurrent-free approaches. It describes their temporal modeling strategies and positions OpenSTL as a unified framework for comparing them.

2.1 Problem definition

  • Given T observed frames, the task is to predict the subsequent T′ frames by learning a neural mapping that minimizes prediction discrepancy.Observed and predicted sequences are represented as tensors with time, channel, height, and width dimensions.Source: §2.1 · p. 3

2.2 Recurrent-based models

  • Recurrent-based models apply shared frame-to-frame mappings while carrying a memory state that contains historical information.Examples include PredRNN's ST-LSTM, PredRNN++'s gradient highway unit, and MIM's high-order non-stationarity learning.Source: §2.1 · p. 3, §2.2 · p. 3

2.3 Recurrent-free models

  • Recurrent-free models range from convolutional approaches to SimVP, PastNet, and IAM4VP; OpenSTL implements representative methods under one framework.The paper also explores extending MetaFormers to connect vision backbones with spatio-temporal learning.Source: §2.3 · p. 3

3 OpenSTL

OpenSTL organizes spatio-temporal predictive learning methods by how they model temporal dependencies and extends recurrent-free models with MetaFormer-based modules. It supports diverse tasks and metrics through a modular framework.

3.1.1 Overview

  • OpenSTL implements 14 methods—11 recurrent-based and 3 recurrent-free—and their main distinction is how they model temporal dependencies.Most use Conv2D for spatial modeling; E3D-LSTM and CrevNet use Conv3D instead.Source: §3.1.1 · p. 4, Table 1 · p. 4

3.1.2 Rethink the recurrent-free models

  • Recurrent-free models encode observed frames into low-dimensional spatial features, model time in an intermediate module, then decode future frames.The MetaFormer extension changes input channels from C to T × C and implements ten architectures as temporal modules.Source: §3.1.2 · pp. 4–5, Figure 2 · p. 5

3.2 Supported Tasks

  • The benchmark covers synthetic moving-object trajectories, human motion, driving scenes, traffic flow, and weather forecasting across synthetic and real-world scales.It includes Moving MNIST variants and robustness settings; KTH and Human3.6M; KITTI-to-Caltech evaluation; TaxiBJ inflow and outflow; and single- and multi-variable WeatherBench setups.Source: §3.2 · p. 5

3.3 Evaluation Metrics

  • Supported tasks use error, similarity, perceptual, and computational metrics suited to their characteristics.These include MSE, MAE, weather RMSE, SSIM, PSNR, LPIPS, parameter and FLOP counts, and FPS measured on a single NVIDIA V100 GPU.Source: §3.3 · p. 6

3.4 Codebase Structure

  • OpenSTL provides a modular, extensible framework designed around OpenMMLab principles and incorporating code elements from OpenMixup and USB.The paper describes it as more user-friendly, organized, and comprehensive than existing open-source spatio-temporal predictive learning codebases.Source: §3.4 · p. 6

4 Experiment and Analysis

OpenSTL evaluates recurrent-based and recurrent-free methods across synthetic, real-world video, traffic, and weather tasks, then tests robustness under altered inputs.

4.1 Synthetic Moving Object Trajectory Prediction

  • On synthetic Moving MNIST, recurrent models sometimes achieve lower MSE and MAE, but use nearly five times the FLOPs and run approximately seven times slower.Across the additional FashionMNIST and MNIST-CIFAR variants, recurrent-free models retain comparable performance with higher efficiency.Source: §4.1 · p. 7, Table 3 · p. 7

4.2 Real-world Video Prediction

  • In real-world video prediction, recurrent-free models balance efficiency and performance; some recurrent models achieve lower MSE and MAE but use nearly 20 times the FLOPs.The reported comparison is on Kitti&Caltech; experiments also cover KTH and Human3.6M.Source: §4.2 · p. 8, Table 4 · p. 8

4.3 Traffic and Weather Forecasting

  • At low resolution, recurrent-free models outperform recurrent-based models on all four single-variable WeatherBench factors; similar trends appear in high-resolution and multi-variable settings.Figure 3 compares MAE and RMSE for temperature, humidity, wind component, and cloud cover.Source: §4.3 · pp. 8–9, Figure 3 · p. 9

4.4 Robustness Analysis

  • Recurrent-free methods show remarkable robustness to missing and perceptual noise, with little degradation versus no noise; recurrent-based methods drop substantially, while irregular digit speeds significantly set back all methods.Source: §4.4 · p. 9

5 Conclusion and Discussion

OpenSTL benchmarks recurrent-based and recurrent-free approaches across diverse tasks and introduces MetaFormers for recurrent-free temporal modeling. Its results show that recurrence can help capture temporal dependencies but is not always necessary, particularly when computation is costly.

5 Conclusion and Discussion

  • OpenSTL benchmarks 14 representative methods and 24 models across varied tasks, categorizing them as recurrent-based or recurrent-free.The paper introduces a general recurrent-free architecture and MetaFormers for temporal modeling.Source: §5 · p. 10
  • Recurrent-free models balance efficiency and performance in synthetic and real-world video tasks, and outperform recurrent counterparts in weather forecasting.Recurrent models capture high-frequency dependencies through sequential frame tracking, while recurrent-free models combine frames with a global temporal inductive bias suited to low-frequency dependencies.Source: §5 · p. 10
  • An open problem is how to combine recurrent and recurrent-free strengths and whether extending existing MetaFormers can improve recurrent-free models.The paper notes a correspondence between MetaVP spatial encoding and temporal modeling and MetaFormer token mixing and channel mixing.Source: §5 · p. 10

A Installation and Data Preparation

The appendix describes installation and data preparation, then outlines OpenSTL’s layered codebase for model implementation and use.

A.1 Installation

  • OpenSTL can be installed from its GitHub repository in development mode using the provided Conda environment, or as a PyPI package.Development mode applies local code modifications; the package installation must be repeated to apply them.Source: §A.1 · p. 14

A.2 Data Preparation

  • Users are advised to symlink their dataset root to OpenSTL/data and adjust configuration paths when their folder structure differs.OpenSTL provides tools to download and preprocess datasets.Source: §A.2 · p. 14

B Codebase Overview

  • The codebase separates core components, algorithm implementations, and user-interface tools into three layers.The core layer unifies data loading, model modules, and metrics; the algorithm layer provides shared interfaces, while the user interface supports training, evaluation, testing, and visualizations.Source: B Codebase Overview · pp. 14–15, Figure 4 · p. 14

C Implementation Details

Implementation details summarize dataset-specific hyperparameters and explain the spatial and temporal settings used by recurrent-free models.

C Implementation Details

  • Table 8 lists supported-model hyperparameters across seven datasets, including input and prediction lengths, hidden sizes, block counts, epochs, optimizer, drop path, and learning rate.The learning rate and drop path are selected from candidate values, and the best result for each experiment is reported.Source: C Implementation Details · p. 15, Table 8 · p. 15
  • Input length T and prediction length T′ match for MMNIST, Human, TaxiBJ, and Weather-M, but differ for KITTI and KTH.Their specific values depend on the dataset.Source: C Implementation Details · p. 15
  • For recurrent-free models, hidS and hidT are spatial and temporal hidden-layer sizes, while NS and NT count blocks in those components.These four hyperparameters largely remain consistent across datasets; recurrent-model details are provided on GitHub.Source: C Implementation Details · p. 15

D.1 Synthetic Moving Object Trajectory Prediction

Across synthetic moving-object tasks, recurrent-based models fare well on simpler motion, while more complex backgrounds narrow the gap between recurrent-based and recurrent-free approaches.

Moving MNIST

  • On Moving MNIST, most recurrent-based models produce high-quality predictions, while recurrent-free models are comparable but blurrier in the final frames.PredNet and DMVFN are exceptions among recurrent-based models; the observed pattern suggests recurrent models excel at capturing temporal dependencies.Source: §D.1 · p. 16, Figure 5 · p. 16

Moving Fashion MNIST

  • Moving Fashion MNIST results are consistent with Moving MNIST, with recurrent-based models performing well in long-range temporal modeling.Quantitative results are in Table 9 and qualitative examples in Figure 6.Source: §D.1 · p. 17, Table 9 · p. 17, Figure 6 · p. 18

Moving MNIST-CIFAR

  • On Moving MNIST-CIFAR, complex backgrounds increase the need for spatial modeling, narrowing the performance gap between recurrent-based and recurrent-free models.The quantitative results are reported in Table 10, with qualitative visualizations in Figure 7.Source: §D.1 · p. 17, Table 10 · p. 17, Figure 7 · p. 19

D.2 Real-world Video Prediction

Real-world video results vary with the task: recurrent-free models are effective on low-frequency human motion and traffic flow, while recurrent-based models remain competitive on other video settings.

D.2 Real-world Video Prediction

  • On Kitti&Caltech, PredNet and DMVFN perform reasonably well at next-frame prediction despite limited temporal modeling capabilities.Figure 8 provides a qualitative visualization.Source: §D.2 · p. 20, Figure 8 · p. 20
  • On KTH, recurrent-free models achieve comparable performance with low computational cost, balancing performance and efficiency.Quantitative results are in Table 11, and Figure 9 shows qualitative predictions.Source: §D.2 · p. 20, Table 11 · p. 20, Figure 9 · p. 21
  • For Human3.6M, subtle adjacent-frame changes create low-frequency dynamics that recurrent-free models can predict efficiently and accurately.Quantitative results are in Table 12, and Figure 10 provides a qualitative visualization.Source: §D.2 · p. 22, Table 12 · p. 22, Figure 10 · p. 23

D.3.1 TaxiBJ

  • On TaxiBJ's low-frequency traffic-flow data, recurrent-free models show promising results compared with their counterparts.Quantitative results are in Table 13, with qualitative visualizations in Figure 11.Source: §D.3.1 · p. 22, Table 13 · p. 22, Figure 11 · p. 24

D.3.2 WeatherBench

WeatherBench results favor recurrent-free models across several single-variable tasks and at high resolution; in multi-variable forecasting, MogaNet leads across reported metrics.

Single-variable Temperature Forecasting

  • Recurrent-free models clearly outperform recurrent-based models in single-variable temperature forecasting on both performance and efficiency.Table 14 reports quantitative results, and Figure 12 provides a qualitative visualization.Source: §D.3.2 · p. 25, Table 14 · p. 25

Single-variable Humidity Forecasting

  • In single-variable humidity forecasting, results are almost consistent with the temperature task.Quantitative results are in Table 15, and Figure 13 provides a qualitative visualization.Source: §D.3.2 · p. 25, Table 15 · p. 25

Single-variable Wind Component Forecasting

  • Most recurrent-free models outperform recurrent-based models in single-variable wind-component forecasting.Table 16 contains quantitative results; Figures 14 and 15 visualize latitude and longitude wind predictions.Source: §D.3.2 · p. 28, Table 16 · p. 28, Figure 14 · p. 29

Single-variable Cloud Cover Forecasting

Single-variable Temperature Forecasting with High Resolution

  • At 128×256 resolution, SimVPv2 surpasses recurrent-based models by large margins in single-variable temperature forecasting.Quantitative results are reported in Table 18.Source: §D.3.2 · p. 32, Table 18 · p. 32

Multiple-variable Forecasting

  • MogaNet leads across various metrics in multi-variable forecasting of temperature, humidity, and wind factors.Results are in Tables 19–22; Figure 17 compares MAE and RMSE across the four WeatherBench tasks.Source: §D.3.2 · p. 32, Table 19 · p. 32, Table 20 · p. 33

Related papers

Loading 2306.11249v2…