PaperlayerView exact revision on arXiv

Source-linked AI summary

VaryST: Generalizable Spatio-Temporal Forecasting Beyond the Closed-World Setting

Shengnan Guo, Jinjun Cheng, Xiaowei Mao, Jianhao Zhang, Haochen Lv, Yang Jiang, YiHeng Huang, Yan Lin, Youfang Lin, Huaiyu Wan

arXiv:2610.19813v1cs.LGcs.AI

TL;DR

Spatio-temporal forecasting models are often coupled to training conditions even as temporal distributions, spatial domains, and observation availability change. VaryST dynamically constructs predictive context from long-term observations, and experiments report strong accuracy and generalization across these changes.

  • Problem

    Many existing forecasting models depend on training-environment node representations, predefined support sets, or consistently available observations, while deployment conditions can change across time, space, and availability.

    Source: §1 · p. 1

  • Method

    VaryST uses long-term observation-derived node representations, retrieves functional support nodes dynamically, organizes target-specific context, and masks highly relied-upon observations during training.

    Source: §1 · p. 2

  • Results

    Across six real-world datasets and seventeen baselines, VaryST shows strong in-domain forecasting, zero-/few-shot generalization across temporal and spatial changes, and robustness to unseen observation conditions.

    Source: §1 · p. 3

  • Takeaways & Limitations

    VaryST is a compact model trained from scratch that dynamically constructs predictive context for forecasting across changing deployment conditions.

    Source: §6 · p. 10

Abstract

from arXiv · show

Spatio-temporal forecasting is fundamental to many real-world systems, yet deployment environments are inherently varying: temporal distributions may evolve, spatial domains may change, and observation availability may shift from training conditions. Existing models are often tightly coupled to their training environments through node-specific representations, predefined spatial support sets, and reliance on consistently available observations, limiting generalization to changing environments. To this end, we propose VaryST, a compact spatio-temporal forecasting model trained from scratch that achieves strong generalization by dynamically constructing predictive context from available long-term observations. To accommodate temporal changes, VaryST derives node representations from long-term histories, capturing current behavior in the context of longer-term patterns. To adapt to spatial changes, it dynamically selects functionally relevant support nodes beyond predefined neighborhoods and integrates their information with recent and long-term target states and global context. To handle varying observation availability, VaryST selectively masks historical patches on which the model relies most during training, encouraging it to exploit alternative temporal and spatial dependencies for accurate forecasting when key observations become unavailable. Extensive experiments on six real-world datasets against seventeen baselines show that VaryST achieves strong in-domain forecasting accuracy, substantially improves zero- and few-shot generalization under spatial and temporal changes, and remains robust to varying observation availability.

1 INTRODUCTION

Spatio-temporal forecasting models often assume stable temporal distributions, spatial domains, and observation availability, despite changing deployment conditions. VaryST dynamically constructs predictive context from available long-term observations; experiments report strong in-domain forecasting and generalization across these changes.

1 INTRODUCTION

  • Existing methods may depend on node-specific or graph-specific parameters, predefined support nodes, and observations consistently available during training, coupling forecasts to the training environment.These dependencies constrain transfer to unseen nodes or networks and can make forecasting difficult when relied-upon observations are unavailable.Source: §1 · p. 2
  • Long-term histories distinguish persistent behavior from transient fluctuations and provide alternative cues when recent observations are incomplete, supporting context reconstruction as deployment conditions change.The paper frames changing temporal distributions, spatial domains, and observation availability as distinct deployment challenges.Source: §1 · pp. 1–2
  • VaryST combines long-term observation-derived representations, dynamic functional support retrieval, hierarchical contextual decoding, and importance-guided observation perturbation to construct predictive context.The representations characterize nodes without fixed identities; retrieval finds relevant support beyond predefined neighborhoods; decoding organizes target states, supports, and global context; perturbation masks highly relied-upon observations during training.Source: §1 · pp. 2–3
  • Across six real-world datasets and seventeen baselines, VaryST shows strong in-domain forecasting, zero-/few-shot generalization across temporal and spatial changes, and robustness to unseen observation conditions.The supplied results state these outcomes but do not give comparative metric values.Source: §1 · p. 3

2 RELATED WORK

Prior spatio-temporal forecasting methods model temporal and spatial dependencies and often achieve strong in-domain performance, but some are tied to training-network parameters. Long-context and efficient modeling work motivates VaryST’s use of long-term observations to guide spatial support and predictive evidence organization.

2.1 SPATIO-TEMPORAL FORECASTING

  • Some forecasting models rely on node-specific embeddings or adjacency parameters tied to the training network, limiting direct application to unseen nodes or networks.The section also reviews recurrent, graph-based, and attention-based approaches, alongside pretrained and OOD forecasting work.Source: §2.1 · p. 3
  • OOD forecasting studies address particular spatial and temporal deployment challenges, whereas this paper considers spatial domains, temporal distributions, and observation availability together.The cited examples are STONE and STOP.Source: §2.1 · p. 3

2.2 LONG-CONTEXT SPATIO-TEMPORAL MODELING

  • Long-context and efficient spatio-temporal studies motivate efficient historical modeling, while VaryST uses long-term observations to guide spatial support selection and predictive evidence organization.The reviewed work includes extended temporal dependencies, historical-pattern retrieval, and separate pretraining of historical representations.Source: §2.2 · p. 3

3 PRELIMINARIES

The paper defines spatio-temporal observations over a graph, with masks indicating which observations are available. Its deployment setting varies across temporal distributions, spatial graphs, or observation masks; evaluation considers zero-shot, few-shot, and unseen-availability conditions.

3 PRELIMINARIES

  • Spatio-temporal observations X_t are defined over graph G = (V, A), and a binary mask M_t indicates whether each node-feature observation is available.The mask value is 1 when the corresponding observation is available.Source: §3 · p. 3
  • VaryST’s architecture comprises long-term observation-derived representations, dynamic functional support retrieval, and target-centric hierarchical contextual decoding.The figure caption names these three components.Source: Figure 2 · p. 4
  • Deployment environments vary through changes in temporal distributions, spatial graphs, or observation-mask distributions, including shifts in missing rates or patterns.The definitions include evolving distributions, graph changes such as added or removed nodes and relations, and transfer to another network.Source: §3 · p. 4
  • Generalization is evaluated with zero-shot forecasting and few-shot adaptation for spatial and temporal changes, and without retraining for unseen missing rates and patterns.The forecasting task uses past observations and availability masks to predict future steps.Source: §3 · p. 4

4 METHODOLOGY

VaryST constructs predictive context from long-term observations, combining dynamic node representations and retrieved functional supports with target and global context. Importance-guided perturbation encourages reliance on alternative evidence when observations are unavailable.

4 METHODOLOGY

  • Long-term observation-derived representations, dynamic support retrieval, hierarchical decoding, and importance-guided perturbation form VaryST’s approach to varying deployment environments.The framework is trained from scratch and dynamically selects and organizes predictive evidence.Source: §4 · p. 4

4.1 LONG-TERM OBSERVATION-DERIVED REPRESENTATIONS

  • Each node’s available long-term observations, interpolated values, temporal differences, and availability mask are jointly encoded into a dynamic representation.Its final hidden state summarizes preceding historical patches; observation-derived representations can be constructed for unseen nodes and updated as regimes change.Source: §4.1 · pp. 4–5

4.2 DYNAMIC FUNCTIONAL SUPPORT NODE RETRIEVAL

  • VaryST ranks candidate nodes by RBF affinity between dynamic representations and retrieves the top K as functionally relevant supports.The affinity reflects current functional compatibility rather than predefined spatial proximity; retrieved sets can adapt across regimes, domains, and observation availability.Source: §4.2 · p. 5

4.3 TARGET-CENTRIC HIERARCHICAL CONTEXTUAL DECODING

  • The target-centric decoder combines global context, retrieved supports, and recent and long-term target states, with hop-aware encoding applied to support tokens.Target-relative hop encoding can be reused on unseen graphs because it represents relations rather than node identity or absolute graph position.Source: §4.3 · pp. 5–6

4.4 TRAINING WITH IMPORTANCE-GUIDED OBSERVATION PERTURBATION

  • IGOP masks historical patches with the highest attention-derived importance scores, then trains forecasting on the perturbed history using the original target.A gradient-free probe constructs the mask; no reconstruction loss is imposed, and forecasting supervision encourages use of remaining temporal and spatial evidence.Source: §4.4 · pp. 6–7

5 EXPERIMENTS

Across standard forecasting, temporal and spatial transfer, and varying observation availability, VaryST performs strongly against the evaluated baselines. The experiments also examine design ablations, hyperparameters, and inference efficiency.

5.1 EXPERIMENTAL SETUP

  • The benchmarks cover in-domain and cross-network traffic forecasting, missingness, non-traffic PM2.5, and long-term temporal shift; evaluation forecasts 12 steps from chronological splits.Baselines include classical, pretrained LLM-based, and extended-history models; specialized comparisons address temporal transfer, spatial transfer, and missing observations.Source: §5.1 · p. 7, §5 · p. 7

5.2 STANDARD FORECASTING PERFORMANCE (RQ1)

  • VaryST achieves the best results on four PEMS datasets, reducing MAE by 3.16%–4.39%, RMSE by 1.71%–5.72%, and MAPE by 0.67%–8.61% over the strongest baselines.On PM2.5, it achieves the best MAE and MAPE alongside a competitive RMSE.Source: Table 2 · p. 8, §5.2 · p. 8, Table 3 · p. 8

5.3 TEMPORAL GENERALIZATION (RQ2)

  • With training on 2011 data and no updates, VaryST outperforms STOP across 2012–2017, reducing MAE by 16.22%–21.97%, RMSE by 15.34%–21.27%, and MAPE by 15.43%–18.08%.The evaluation tests zero-shot transfer across evolving temporal distributions.Source: §5.3 · p. 8, Figure 3 · p. 9

5.5 ROBUSTNESS TO OBSERVATIONS AVAILABILITY(RQ4)

  • On PEMS04, zero-shot VaryST degrades 3.68%/5.30%/8.89% (RMSE/MAE/MAPE), versus 52.36%/63.86%/91.33% for STD-PLM; with 5%–10% target data, it surpasses full-data in-domain baselines.The study evaluates cross-network transfer across four PEMS benchmarks under zero-shot and few-shot settings.Source: §5.5 · p. 9, §5.3 · p. 9

5.6 ABLATION STUDIES (RQ5)

  • IGOP trained at a 30% missing rate reduces MAE by 0.39 at a 70% test missing rate on PEMS08 without retraining.The ablation also reports that removing long-term history increases PEMS03 MAE from 13.95 to 15.95.Source: §5.6 · p. 10
  • Extending history from 36 to 2016 steps reduces MAE by 8.8% on PEMS03 and 12.8% on PEMS08; increasing support count from 16 to 64 lowers PEMS03 MAE by 2.9%.The study evaluates context lengths of 36–4032 and support counts of 16–128.Source: §5.6 · p. 10

5.7 EFFICIENCY ANALYSIS (RQ7)

  • On PEMS07, VaryST achieves the lowest MAE (18.21), reducing error by 3.8% with faster inference than FaST† and using substantially less memory than STD-PLM.The decoder processes K + 3 tokens per target after long-context encoding compresses each node’s history into one representation.Source: Figure 4 · p. 9, §5.7 · p. 10

6 CONCLUSION

VaryST dynamically constructs predictive contexts for generalizable spatio-temporal forecasting. The paper reports strong accuracy and generalization across the evaluated shifts and observation availability settings.

6 CONCLUSION

  • VaryST addresses generalizable spatio-temporal forecasting by dynamically constructing predictive contexts.Source: §6 · p. 10
  • Trained from scratch, VaryST demonstrates superior in-domain accuracy and robust zero- and few-shot generalization across spatial and temporal shifts.Source: §6 · p. 10
  • VaryST also demonstrates resilience to varying observation availability.Source: §6 · p. 10

REPRODUCIBILITY STATEMENT

The statement identifies public benchmarks and says experiment scripts and configurations will be released upon publication. The appendix roadmap describes supplementary methods, setups, and results.

REPRODUCIBILITY STATEMENT

APPENDIX

  • The appendix roadmap includes training and inference pseudocode, component complexity analysis, experimental details, and comprehensive results.Source: APPENDIX · p. 13, §A.1 · p. 13

A.2 COMPLEXITY ANALYSIS

VaryST keeps gradient-carrying temporal encoding and decoding linear in node count at fixed context length; the only super-linear step is parameter-free support retrieval. Graph-dependent quantities are computed once per graph, outside training.

Temporal encoding

  • With U fixed at 56 in the experiments, temporal encoding is linear in B, N, and T.Per window-to-node pair, encoding includes unit projection, self-attention over U tokens, readout attention, and output projection.Source: §A.2 · pp. 13–14

Support Node retrieval

  • Support retrieval costs O(BN^2D), the only step super-linear in N, and is parameter-free and outside the gradient graph.Candidate keys are detached and top-K indices are gather indices; gradients flow through retrieved node values.Source: §A.2 · p. 14

Context assembly and decoding

  • At fixed sequence length ℓ=K+O(1), decoding processes each target instance independently of N, with attention cost O(ℓ^2D + ℓD^2) per layer.The hop-bucket key rotation adds a constant factor of 10 to attention-score computation.Source: §A.2 · pp. 13–14
  • Graph-dependent Laplacian eigenvectors, hop buckets, and the hop matrix are computed once per graph, outside the training loop.Source: §A.2 · p. 14

Where the quadratic term lives

  • All gradient-carrying computation scales linearly with N at fixed ℓ, and per-instance attention uses an ℓ^2 rather than N^2 footprint.Unlike a graph Transformer, the quadratic term occurs once in retrieval rather than in every backbone layer and backward pass.Source: §A.2 · p. 14

B.1 DATASET STATISTICS

The experiments span traffic and air-quality benchmarks, with a detailed year-by-year protocol for PEMSD3-Stream. The appendix also specifies shared settings, metrics, compute, transfer splits, and baseline input handling.

B.1 DATASET STATISTICS

  • PEMSD3-Stream uses PeMS District 3 flow from 2011–2017, with yearly graphs growing from 655 to 871 stations as sensors join.Training and validation use 2011; each later year is tested zero-shot using the 2011 checkpoint.Source: §B.1 · p. 14, Table 4 · p. 14

B.2 HYPERPARAMETERS

  • The default configuration is shared across PEMS experiments, while PM2.5 uses a 168-step history and 12-step patches.Context length, patch length, and retrieved support-node count are studied for sensitivity.Source: §B.2 · p. 15, Table 5 · p. 15

B.3 METRICS

  • MAE, RMSE, and MAPE are averaged over 12 horizons at observed positions, with lower values indicating better performance.MAPE is expressed in percent.Source: §B.3 · pp. 15–16

B.4.1 HARDWARE AND ENVIRONMENT

  • Experiments use one NVIDIA RTX A4000 GPU, with training taking 2.7–4.4 hours on traffic benchmarks and 44 minutes on PM2.5.Peak memory is 0.9–2.9 GB for traffic and below 0.1 GB for PM2.5.Source: §B.4.1 · p. 16

B.4.2 DATA SPLITTING AND LEAKAGE PREVENTION

  • Chronological 6:2:2 splits prevent time shuffling; few-shot budgets retain evenly spaced full input-target windows and evaluation covers the whole test span.Zero-shot transfer uses no target updates; VaryST carries no node identity, only relative hop-distance and Laplacian-eigenvector priors.Source: §B.4.2 · p. 16

B.4.3 BASELINE ADAPTATION AND MISSING-INPUT PROTOCOL

  • Adapted baselines receive VaryST-length inputs without architectural changes, while missing observations are linearly interpolated for baselines and masked natively for VaryST.PatchSTG uses synthesized virtual coordinates because PEMS lacks sensor latitude and longitude metadata.Source: §B.4.3 · p. 16

C.1 FULL TEMPORAL GENERALIZATION RESULTS

VaryST generalizes across held-out years and cross-dataset transfers, with strong reported results in both temporal and spatial evaluations. Its few-shot transfer performance also compares favorably with a baseline using a larger target-data budget.

C.1 FULL TEMPORAL GENERALIZATION RESULTS

  • 13.01 to 10.49: VaryST reduces six-year average MAE versus STOP by 19.4% and attains the best MAE, RMSE, and MAPE in every year.Models train on 2011 and evaluate zero-shot on 2012–2017 as new sensors join; metrics use observed positions averaged over 12 horizons.Source: §C.1 · p. 16, Table 6 · p. 17, Table 7 · p. 17

C.2 FULL SPATIAL GENERALIZATION RESULTS

  • VaryST achieves the lowest MAE, RMSE, and MAPE across all 12 zero-shot transfer directions among four PEMS datasets.Its maximum MAE increase over the corresponding in-domain baseline is 9.9%; PEMS07→PEMS03 reaches MAE 13.78 versus 13.95 in-domain.Source: Table 8 · p. 17, §C.2 · p. 17
  • Across all 12 transfer directions, 1% target-data fine-tuning of VaryST outperforms STD-PLM adapted with 10% target data.Increasing VaryST's target-data budget to 10% yields at most a 3.7% additional MAE reduction.Source: §C.2 · p. 17, Table 9 · p. 18

C.3 FULL MISSING-OBSERVATION RESULTS

VaryST performs strongly under missing observations across same-rate, unseen-rate, and cross-pattern evaluations. Ablations also identify structural attention and global-state completion as important sources of in-domain accuracy.

C.3 FULL MISSING-OBSERVATION RESULTS

  • VaryST attains the lowest metric in all 48 reported same-rate missing-observation cells.The evaluation covers point and spatiotemporal block missing at 10%–70% on PEMS04 and PEMS08.Source: §C.3 · p. 19, Table 10 · p. 19
  • Under unseen-rate transfer, VaryST ranks first across all 24 point-missing cells, but its relative degradation is larger than the baselines'.At 70% missing, its MAE is 0.63 and 0.35 below the runner-up on PEMS04 and PEMS08; its same-rate gap reaches +0.58 and +0.53, versus a baseline maximum of 0.16. Under block missing, it leads through 50%; at 70% it has the best MAE and MAPE on PEMS04, but on PEMS08 it is second on MAE and last on RMSE on both datasets.Source: §C.3 · p. 20, Table 11 · p. 20
  • For cross-pattern transfer, VaryST has the lowest MAE in three of four directions and the lowest MAPE in all four.Its RMSE is second in both RM→CM directions by at most 0.08; STID leads MAE and RMSE in the CM→RM direction on PEMS08.Source: §C.3 · pp. 20–21, Table 12 · p. 20

C.4 FULL ABLATION RESULTS

  • Removing any component raises PEMS03 MAE and RMSE, with structural attention and global-state completion producing the largest reported MAE costs.Removing hop-aware structural attention costs 0.47 MAE on PEMS03, while removing the global token costs 0.33 MAE on PEMS08; the full model has the best MAPE10 on both datasets.Source: §C.4 · p. 21, Table 13 · p. 21

C.5 FULL HYPERPARAMETER STUDY RESULTS

Context length has the strongest influence in the hyperparameter study, with gains that diminish beyond 288 steps. Support-node count and patch length are less sensitive choices.

C.5 FULL HYPERPARAMETER STUDY RESULTS

  • Increasing context length from 36 to 288 steps lowers MAE by 1.04 on PEMS03 and 1.18 on PEMS08; longer contexts yield diminishing gains.At the one-week default of 2016 steps, MAEs are 13.95 and 12.89, respectively; extending to 4032 steps brings no PEMS08 gain and degrades PEMS03 MAE by 0.45.Source: §C.5 · p. 21, Table 14 · p. 22
  • With 16 support nodes, MAE remains within 0.42 of the optimum on both datasets, while increasing to 128 gives no consistent benefit.Changing patch length from 12 to 96 steps shifts MAE by at most 0.35 on either dataset.Source: §C.5 · p. 21, Table 14 · p. 22

Related papers

Loading 2610.19813v1…