PaperlayerView exact revision on arXiv

Source-linked AI summary

Long-Range Transformers for Dynamic Spatiotemporal Forecasting

Jake Grigsby, Zhe Wang, Nam Nguyen, Yanjun Qi

arXiv:2109.12218v3cs.LGstat.ML

TL;DR

Conventional temporal attention can miss distinct variable relationships, while graph-based methods may rely on predefined, static graphs and separate spatial and temporal updates. Spacetimeformer represents each variable at each timestep as a token, enabling joint, data-driven spatiotemporal attention. It achieves competitive results on long-sequence forecasting benchmarks, while scaling to real-world datasets remains an open concern.

  • Problem

    Temporal attention groups variables into timestep tokens, while graph-based approaches may rely on static predefined relationships and separate spatial and temporal updates.

    Source: §1 · pp. 1–2

  • Method

    Spacetimeformer flattens multivariate inputs into variable-time tokens so a Transformer can jointly learn spatiotemporal relationships from data.

    Source: §1 · p. 2

  • Results

    Spacetimeformer achieves competitive results on long-sequence forecasting and benchmarks spanning traffic, electricity production, temperature prediction, and metro ridership.

    Source: §1 · p. 2, §5 · p. 7

  • Takeaways & Limitations

    The approach learns spatiotemporal relationships without a predefined graph and can scale to high-dimensional spatial problems.

    Source: §5 · p. 7

    Because the representation multiplies sequence length by the number of variables, whether it scales to real-world datasets remains an open question.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Multivariate time series forecasting focuses on predicting future values based on historical context. State-of-the-art sequence-to-sequence models rely on neural attention between timesteps, which allows for temporal learning but fails to consider distinct spatial relationships between variables. In contrast, methods based on graph neural networks explicitly model variable relationships. However, these methods often rely on predefined graphs that cannot change over time and perform separate spatial and temporal updates without establishing direct connections between each variable at every timestep. Our work addresses these problems by translating multivariate forecasting into a "spatiotemporal sequence" formulation where each Transformer input token represents the value of a single variable at a given time. Long-Range Transformers can then learn interactions between space, time, and value information jointly along this extended sequence. Our method, which we call Spacetimeformer, achieves competitive results on benchmarks from traffic forecasting to electricity demand and weather prediction while learning spatiotemporal relationships purely from data.

1 INTRODUCTION

Multivariate forecasting models seek to predict future values from historical context, but conventional temporal attention can overlook distinct relationships between variables. Spacetimeformer instead uses separate variable-time tokens so Transformers can learn spatiotemporal relationships jointly from data.

1 INTRODUCTION

  • Temporal-attention models represent all variables at a timestep in one token, which can overlook distinct relationships between variables.Jointly modeling variables can reveal dependency relationships that provide early warning signs of future behavioral changes.Source: §1 · p. 1
  • Graph-based approaches commonly depend on predefined relationships, which can be difficult to define and cannot capture changing spatial structure when graphs remain static.The paper notes that variable relationships may change over time and may be better modeled by a context-dependent graph.Source: §1 · pp. 1–2
  • Spacetimeformer flattens inputs of shape (L, N) into (L×N, 1), enabling Transformer attention to learn dynamic variable relationships jointly across space and time.The paper reports competitive results against specialized time-series and spatial-temporal GNN baselines across forecasting benchmarks.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

The background reviews long-range attention, time-series Transformers, and graph-based spatial-temporal forecasting. It identifies limitations in attention length, variable grouping, and reliance on static predefined graphs that motivate joint spatiotemporal modeling.

2.1 Long-Range Transformers

  • Attention runtime and memory grow quadratically with input length, motivating sparse-attention heuristics and sub-quadratic approximations such as Performer.Performer approximates attention in linear time and space using random orthogonal features.Source: §2.1 · p. 2

2.2 Time Series Forecasting and Transformers

  • Time-series Transformers use one token per timestep, grouping all variables and forcing them to receive the same information from other timesteps despite potentially distinct patterns.The paper frames this as a limitation of temporal attention for modeling variable relationships.Source: §2.2 · pp. 2–3

2.3 Spatial-Temporal Forecasting and GNNs

  • GNN-based methods commonly require predefined spatial graphs, alternate spatial and temporal updates, and retain static graphs across timesteps.The paper's goal is a sequence-to-sequence model with dynamic spatiotemporal message passing that can compete without predefined variable relationships.Source: §2.3 · p. 3

3 SPATIOTEMPORAL TRANSFORMERS

Spacetimeformer recasts multivariate forecasting as a longer sequence of variable-time tokens, then embeds and processes that sequence with a Transformer. Fast attention and optional architectural strategies address the resulting sequence-length demands.

3.1 Spatiotemporal Sequences

  • Flattening each multivariate value vector into separate variable tokens creates a sequence with direct attention paths among variables at every timestep, allowing relationships to vary with time and input values.The resulting attention matrix represents a spatiotemporal graph without relying on predefined variable graphs.Source: §3.1 · p. 3

3.2 Spatiotemporal Embeddings

  • The embedding scheme combines variable values with learned Time2Vec features, position and variable embeddings, and a given embedding that marks originally missing values.Each variable receives a separate token with its timestamp information and original timestep position.Source: §3.2 · pp. 3–4

3.3 Scaling to Long Sequence Lengths

  • The longer input can exceed 26,000 tokens, so the paper primarily uses Performer FAVOR+ fast attention and also describes convolutional and shifted-window strategies.Convolutions can shorten representations, while shifted windows enable information to spread between distant chunks across layers.Source: §3.3 · p. 4
  • Most reported results use fast attention alone with less than 40GB of GPU memory; convolutions are reserved for the largest datasets, and windows mainly help with padded-sequence masking.The authors aim to use as few optimizations as possible to learn graphs across entire input sequences.Source: §3.3 · pp. 4–5

3.4 Spacetimeformer

  • Local attention lets each token attend to timesteps of its own variable before global attention lets it attend across the full spatiotemporal sequence.The local layer is an architectural bias, not a separation of spatial and temporal updates.Source: §3.4 · p. 5
  • The decoder folds predictions back into the original format and supports varied forecasting losses; optional time-series tricks are off by default and used for non-stationary domains.Those tricks include seasonal decomposition, input normalization, and predicting a residual from a simple linear model.Source: §3.4 · p. 5

4 EXPERIMENTS

Experiments span toy sequence tasks and weather, electricity, solar, traffic, and metro forecasting, testing both predictive performance and learned spatial-temporal relationships.

4 EXPERIMENTS

  • Experiments compare against TSF and GNN baselines, including a Temporal ablation, and average metrics across target timesteps and at least three training runs.The Temporal ablation uses the Spacetimeformer architecture with a standard temporal sequence embedding.Source: §4 · p. 5

4.1 Toy Examples

  • On the shifted multivariate copy task, spatiotemporal attention learns all variable relationships in one head and produces accurate outputs, unlike Temporal's limited per-head relationships.The sine-wave forecasting task also reports that the method learns an uncompromising spatiotemporal relationship and generates the most accurate predictions.Source: §4.1 · p. 5

4.2 Time Series Forecasting

  • On NY-TX Weather, Spacetimeformer's embedding scheme gives the most accurate forecasts, while MTGNN's convolution-only output struggles at 80- and 160-hour horizons.The experiment compares 40-, 80-, and 160-hour target sequences; the reported gap over the Temporal method appears to increase at longer sequence lengths.Source: §4.2 · pp. 5–6, Table 1 · p. 6
  • On ETTm1, Spacetimeformer improves on Informer and is competitive with later variants, but results also highlight robustness to distribution shift and gains from normalization.Reversible instance normalization more than halves the LSTM baseline's prediction error; LinearAR closely matches ETSFormer.Source: §4.2 · p. 6, Table 2 · p. 6

4.3 Spatial-Temporal Forecasting

  • Spacetimeformer is significantly more accurate than TSF baselines on AL Solar, while MTGNN's temporal predictions are less accurate.The dataset covers 137 locations with a 4-hour prediction horizon.Source: §4.3 · p. 6, Table 3 · p. 6
  • On Metr-LA and Pems-Bay, Spacetimeformer enters the performance band of dedicated GNN methods without predefined graphs; it is also competitive with graph-based methods on HZMetro.The traffic forecasts cover the next hour; HZMetro forecasts the next hour across 160 variables.Source: §4.3 · p. 6, Table 4 · p. 7, Table 5 · p. 7
  • Some attention heads recover NY-TX weather stations' spatial relationships, while learned patterns form variable blocks and include strided temporal relations.Figure 4 illustrates the station relationships and strided temporal attention.Source: §4.3 · p. 6, Figure 4 · p. 7

5 CONCLUSION AND FUTURE DIRECTIONS

The paper presents a unified forecaster that jointly learns temporal and spatial relationships through a long-range Transformer over elongated spatiotemporal sequences. It reports competitive long-sequence forecasting, identifies scaling and more dynamic applications as future directions, and outlines Transformer attention as token-to-token information passing.

5 CONCLUSION AND FUTURE DIRECTIONS

  • Spacetimeformer jointly learns temporal and spatial relationships and achieves competitive long-sequence forecasting results while scaling to high-dimensional spatial problems without a predefined graph.The authors identify larger domains and model sizes, more rapidly changing variable behavior, irregularly sampled series, exogenous variables, and multi-dataset generalization as future directions.Source: §5 · p. 7
  • The main experiments use established benchmarks with relatively static spatial relationships, while the authors expect rapidly changing domains could better use Spacetimeformer’s dynamic, learnable graph.They also note potential for larger domains and model sizes, flexible input formats, and multi-dataset generalization.Source: §5 · p. 7

A.1 Transformers and Self Attention

  • Attention represents token-to-token connections with weights computed from query–key dot-product similarities, then uses those weights to combine value vectors into updated representations.In encoder-decoder Transformers, encoders self-attend to context, while decoder layers alternate target self-attention and cross-attention to encoded context before producing predictions.Source: §A.1 · pp. 10–11

A.2 Spatial-Temporal Graph Neural Networks

This section distinguishes temporal, spatial-temporal, and jointly spatiotemporal approaches by how they represent variable relationships, use predefined graphs, and adapt connections over time. Spacetimeformer uses efficient attention to generate a dense, dynamic graph connecting nodes across all timesteps.

A.2 Spatial-Temporal Graph Neural Networks

  • Graph-based forecasting represents each timestep as a graph of variable nodes and predicts future node values from past values, commonly using a fixed adjacency matrix.In traffic forecasting, nodes can be road sensors, with weighted edges corresponding to road lengths.Source: §A.2 · p. 11
  • Traffic Transformer exemplifies spatial-then-temporal processing: predefined adjacency drives graph convolution before a Transformer handles temporal learning.DCRNN combines diffusion-based graph layers with recurrent temporal processing.Source: §A.2 · p. 11
  • Hardcoded adjacency cannot adapt spatial connections to specific times, motivating dynamic adjacency sequences for settings such as changing roadway conditions.The passage names closures, accidents, and high traffic volume as examples that can change roadway connections.Source: §A.2 · p. 11
  • Graph WaveNet [76], MTGNN [75], and AGCRN learn adjacency from node embeddings but produce static graphs after training, while STAWnet makes relationships depend on current time and values.ASTGCN can reweight provided spatial relationships based on time and node attributes, but cannot learn edges absent from the supplied graph; GMAN enables a fully dynamic spatial graph.Source: §A.2 · p. 11
  • Alternating spatial and temporal layers route information through another node’s representation, whereas joint spatiotemporal attention creates direct paths between relevant tokens.The indirect route can depend on attention involving unrelated tokens, as illustrated by Figure 5.Source: §A.2 · p. 11, Figure 5 · p. 12
  • Spacetimeformer processes joint spatiotemporal relationships across a learnable dynamic graph, using efficient attention to generate dense connections among nodes at all timesteps.Unlike alternating spatial and temporal updates, this permits direct information paths between variables at different times.Source: §A.2 · pp. 11–12, Figure 5 · p. 12

A.3 Connection to Vision Transformers

The section recasts multivariate forecasting as grayscale-image completion and relates its spatiotemporal modeling to Vision Transformer design choices. It highlights the distinct ordering of time and variables and the use of convolutional, local-attention, and global-attention mechanisms.

A.3 Connection to Vision Transformers

  • Spacetimeformer completes the rightmost columns of a grayscale image from the leftmost columns, and its spatiotemporal architecture reduces MNIST prediction error by 8% versus a temporal architecture.The authors caution that MNIST is simple and expect a larger performance gap on higher-resolution images.Source: §A.3 · p. 12, Figure 6 · p. 13
  • Unlike image rows and columns, whose order is meaningful, the time-series spatial axis is assumed arbitrary, making square patches liable to group variables according to dataset ordering.The authors describe rectangular 1 × k time slices as an option; initial convolutions on large datasets act like overlapping patches that reduce sequence length.Source: §A.3 · p. 12
  • Spacetimeformer combines intermediate convolutions and local attention with true global layers, relating its design to vision models that mix local structure and global connections.Its local attention uses time slices, and shifted windowed attention is implemented but not used in the primary experiments because fast attention provides sufficient memory savings.Source: §A.3 · pp. 12–13

B.1 Real-World Dataset Details

The evaluation uses real-world forecasting datasets and, where possible, follows prior train/validation/test splits. The datasets span traffic, weather, solar, metro ridership, and electricity, with sequence lengths and variable counts varying by benchmark.

B.1 Real-World Dataset Details

  • Where possible, the evaluation follows prior train/validation/test splits, often assigning earlier observations to training and the most recent observations to testing.This time-ordered evaluation is intended to measure future predictive power and is especially important for non-stationary datasets.Source: §B.1 · p. 14
  • The dataset summary reports sequence length as the largest combined context and target window used in results.The passage defines sequence length as L.Source: Table 7 · p. 14
  • The real-world benchmarks include weather, traffic, solar power, metro ridership, electricity-transformer, and German weather series with differing context and target lengths.Examples include 12-step context and target windows for Metr-LA and Pems-Bay, and ETTm1 target lengths of 24, 48, 96, 288, and 672.Source: §B.1 · p. 14

B.2 Code and Implementation Details

The implementation supports variable-time token embeddings, configurable Transformer components, and a range of forecasting losses. Data preparation and evaluation follow established normalization and split practices, with reported metrics averaged across forecast horizons.

Compute and code

  • The code is open-sourced, with baseline implementations drawn from public code or implemented from scratch and checked for competitive results.The Spacetimeformer implementation was originally based on the Informer release.Source: §B.2 · p. 14

Embedding and data preparation

  • Data use temporal train/validation/test splits or established dataset splits, with z-score normalization as the default when prior schemes are unavailable.Calendar variables are separately scaled, and time variables considered prone to overfitting are discarded.Source: §B.2 · p. 14
  • Each variable’s value at each timestep becomes its own token, with time, variable, position, and given-value embeddings combined into a Transformer-dimensional representation.The embedding scheme can accommodate variables sampled at different intervals, though the experiments do not use that flexibility.Source: §B.2 · pp. 14–15

Architecture and training

  • The encoder and decoder use separate embeddings, and the architecture supports configurable normalization, dropout, local attention, and efficient attention implementations.The default attention implementation is ReLU Performer; BatchNorm is used in place of LayerNorm in the reported setup.Source: §B.2 · p. 15

Metrics and training

  • Metrics average error over the forecast horizon and variables; missing values are discarded, and multiple reported timesteps are averaged for comparison with related work.Training uses early stopping and learning-rate reductions based on validation performance.Source: §B.2 · p. 15

Compute and code

  • Most experiments ran on one to four 12GB GPUs, while later results sometimes used A100 80GB GPUs; the longest reported runs took about five hours.Long-sequence attention is mainly constrained by GPU memory, and training commands are provided in lieu of a written hyperparameter listing.Source: §B.2 · p. 15

B.3 Toy Experiments

Toy experiments contrast temporal attention with spatiotemporal modeling on shifted-copy and dependent-sine tasks. Spatiotemporal attention reproduces shifted sequences far more accurately, while the full method performs best across the sine-task metrics.

Experimental setups

  • The shifted-copy setup uses binary masks with variable-specific temporal shifts, and the dependent-sine task uses 2,000 daily timesteps, 20 sequences, and 128/32 context/target lengths.The final quarter of the sine data is held out for testing.Source: §B.3 · p. 16

Shifted Copy Task

  • Standard temporal attention produces blurry or missing outputs for lower variables in the eight-variable shifted-copy task.The figure caption attributes the blurry copy to variables sharing attention graphs across columns.Source: §B.3 · p. 16, Figure 8 · p. 16
  • About 17× lower MAE and near-perfect copies result when the shifted-copy task uses spatiotemporal attention instead of a standard Transformer.Flattening context and target tokens gives variables at each timestep independent attention mechanisms; Figure 10 depicts the learned relationships.Source: §B.3 · p. 16

Dependent Sine Waves

  • On the dependent-sine task, the full method gives the most accurate predictions across all three metrics among the reported ablations.The temporal embedding compromises across timesteps when variables have distinct frequencies; the standard full-attention variant is well approximated by Performer.Source: §B.3 · p. 16, Table 8 · p. 16

B.4 Distributional Shift and Time Series Transformers

The section examines how distribution shift affects time-series forecasting and whether normalization helps. Input normalization improves results in the reported benchmarks, but can fail when predictions depend on magnitude that normalization removes.

Attention and shifted copies

  • The attention visualizations contrast temporal attention’s limited heads with spatiotemporal attention’s ability to represent N² relationships per head and copy shifted inputs accurately.The figures describe temporal attention running out of heads and the spatiotemporal output reproducing the shifted input.Source: Figure 9 · p. 17, Figure 10 · p. 17, Figure 11 · p. 17

Distributional shift

  • On ETTm1, the tiny LinearAR model nearly matches ETSFormer, while the comparison with NY-TX Weather shows LinearAR performs worse than Transformer baselines on Weather.ETTm1 has a much larger train/test magnitude gap than NY-TX Weather.Source: §B.4 · p. 17
  • Input normalization improves all evaluated models on ETTm1 and Weather, while seasonal decomposition performs comparably to instance normalization on Weather.The Spacetimeformer results in Table 2 use both seasonal decomposition and reversible instance normalization.Source: §B.4 · p. 17
  • The paper links Transformer test-set inaccuracy to distribution shift and hypothesizes that linear models are less affected because their outputs more directly combine input magnitudes.The section notes that apparent overfitting showed few signs of improvement with additional data or smaller, highly regularized architectures.Source: §B.4 · pp. 16–17

Scale-dependent patterns

  • RevIN-style normalization reduces the out-of-distribution d generalization gap to nearly zero for LSTM, temporal-attention, and spatiotemporal-attention models in the sine experiment.The paper also reports that RevIN is unable to generalize when the curve shifts between context and target.Source: §B.4 · p. 18
  • RevIN can fail when predictions depend on magnitude, because normalizing the input discards the magnitude information needed for accurate predictions.The paper notes that real-world series may change behavior with overall magnitude and calls for more work on robustness to small, non-stationary datasets.Source: §B.4 · p. 18, Figure 12 · p. 18

B.5 Ablations

The ablations show that attention and variable-identity information matter differently across temporal and spatial tasks. The future-directions section proposes extending the approach to asynchronous, semi-independent series using each series’ full history as context.

B.5 Ablations

  • Variable-identity classification rises above 99% within a few hundred updates; it stays stable on NY-TX weather but declines on Metr-LA as forecasting accuracy increases.Source: §B.5 · p. 18
  • For spatial traffic forecasting, global spatiotemporal attention is key to state-of-the-art performance, while local attention alone remains competitive.Source: §B.5 · p. 18
  • Removing global attention and variable embeddings together produces the least accurate predictions; the authors interpret this as local attention adapting to node embeddings.Source: §B.5 · pp. 18–19

B.6 Future Directions in Multi-series Generalization

  • For semi-independent series on nonoverlapping time intervals, the proposed setup uses all prior observations as context and future values as targets for in-context forecasting.Source: §B.6 · p. 19
  • Varying samples per series require padding and masking; the implementation uses vanilla attention with arbitrary masking and shifted-window attention to reduce GPU memory use.Source: §B.6 · p. 19
  • Preliminary M4 and Wikipedia results are described as potentially competitive with ML-based leaderboard scores; released routines also cover Monash, M4, and Wikipedia.Source: §B.6 · p. 19

Related papers

Loading 2109.12218v3…