Spatial-Temporal-Decoupled Masked Pre-training for Spatiotemporal Forecasting

1 Introduction

• Weekday/weekend traffic profiles and sensor-specific flows illustrate temporal and spatial heterogeneity that becomes mixed across long records and many sensors.The introduction describes distinct peak patterns by day type and contrasting flows across sensors.Source: §1 · p. 1
• Short input horizons can leave models with fragmented heterogeneity, producing spatiotemporal mirages where similar histories have different futures or dissimilar histories have similar futures.The introduction illustrates both patterns with traffic sensors.Source: §1 · p. 2
• STD-MAE pre-trains on spatiotemporal data to learn clear, complete heterogeneity that can be integrated into downstream baselines to address mirages.The framework uses masked reconstruction to learn context-rich representations.Source: §1 · p. 2
• The framework is designed to enhance downstream predictors of arbitrary architectures without changing their original structures.Source: §1 · p. 2
• Its decoupled masking strategy captures long-range context across spatial and temporal dimensions.Source: §1 · p. 2
• Evaluations on six benchmarks with typical backbones report quantitative gains on baselines and qualitative evidence of meaningful long-range spatiotemporal patterns.The benchmarks are PEMS03, PEMS04, PEMS07, PEMS08, METR-LA, and PEMS-BAY.Source: §1 · p. 2

2.1 Spatiotemporal Forecasting

• Joint GCN-temporal models and transformers address spatial correlations and spatiotemporal relations, but end-to-end models still focus on short-term input that limits complete dependency capture.Earlier approaches included traditional time-series models, RNNs [Chung et al., 2014], and CNNs [Oord et al., 2016].Source: §2.1 · p. 2

2.2 Masked Pre-training

• Masked reconstruction is used for representation learning in NLP and computer vision, while time-series pre-training methods often remain channel-independent or neglect spatial pre-training.STD-MAE masks spatial and temporal dimensions separately to capture long-range heterogeneity.Source: §2.2 · p. 2

3 Problem Definition

• Given T past time steps, forecasting maps the sequence of per-step N-by-C node and channel matrices to the future bT time steps; C=1 in the datasets.Source: §3 · pp. 2–3

4.1 Spatial-Temporal Masked Pre-training

• Separate random masks hide sensor series or time steps, letting each reconstruction learn spatial or temporal heterogeneity from visible context.Source: §4.1 · p. 3
• Paired spatial and temporal autoencoders use attention along their respective dimensions, while patching addresses the computational challenges of long inputs.Source: §4.1 · p. 3
• Visible patches enter the encoders, and lightweight decoders reconstruct masked inputs using a loss computed on masked parts.Source: §4.1 · pp. 3–4

4.2 Downstream Spatiotemporal Forecasting

• Long-range inputs pass through the pretrained encoders while a downstream predictor processes its usual short input; projected spatial and temporal representations are added to its hidden state.Source: §4.2 · p. 4
• The augmented state combines predictor-derived and long-range spatial and temporal representations, which the paper says can enhance downstream performance.Source: §4.2 · p. 4
• Generality is also tested with DCRNN, MTGNN, STID, and STAEformer alongside GWNet.Source: §4.2 · p. 4

5.1 Experimental Setup

• Six benchmarks use five-minute sampling; PEMS datasets use 6:2:2 train/validation/test splits, while METR-LA and PEMS-BAY use 7:1:2 splits and 12-step prediction windows.Pre-training inputs range from 864 to 2016 time steps across the datasets.Source: §5.1 · p. 5, Table 1 · p. 5

5.3 Ablation Study

• On PEMS03 and PEMS07, decoupled masking outperforms ablated variants; spatial-only and temporal-only masking still improve over the original model.The authors describe mixed spatial-temporal masking as a trivial reconstruction task that learns less rich semantics.Source: Figure 3 · p. 5, §5.3 · p. 5

5.2 Overall Performance

• STD-MAE outperforms baselines by a significant margin on all evaluation metrics across the reported datasets.The results are reported in Tables 2–4.Source: §5.2 · p. 5, Table 2 · p. 6, Table 3 · p. 6

5.3 Ablation Study

• Across five downstream predictors, STD-MAE augmentation yields consistent, substantial gains, supporting representation transfer across predictor architectures.The tested predictors include GCN+RNN, GCN+TCN, Linear, and Transformer backbones.Source: §5.3 · p. 6, Table 5 · p. 7

5.4 Hyper-parameter Study

• A masking ratio of 0.25 yields the lowest error across datasets, while three-day pre-training performs best on two of four datasets and the optimal duration varies by dataset.The masking-ratio study tests 0.25, 0.5, and 0.75; the pre-training-length study tests one day, three days, and one week.Source: §5.4 · pp. 6–7, Figure 4 · p. 7, Table 6 · p. 7

5.5 Efficiency Test

• Despite using two decoupled autoencoders, STD-MAE is reported as more efficient than non-decoupled pre-training, especially on datasets with many sensors.The supplied passage does not provide a complete complexity expression.Source: §5.5 · p. 7

5.6 Case Study

• Pre-training reconstruction recovers temporal periodicity and trends and restores fully masked sensors from spatially related sensor context.These examples are presented as evidence of learned temporal patterns and spatial correlations.Source: Figure 5 · p. 7, §5.6 · p. 7
• In two spatiotemporal-mirage cases, STD-MAE improves GWNet’s ability to distinguish mirages that GWNet alone handles poorly.The figure description attributes the improvement to STD-MAE’s pre-trained component.Source: §5.6 · p. 7, Figure 6 · p. 8, Figure 1c

• Experiments on six benchmark datasets demonstrate STD-MAE’s superiority for spatiotemporal forecasting.Source: §6 · p. 8
• STD-MAE uses spatial-temporal-decoupled masked pre-training to model spatiotemporal data heterogeneity.Source: §6 · p. 8
• STD-MAE’s hidden representations are used to boost downstream spatiotemporal predictors.Source: §6 · p. 8