PaperlayerView exact revision on arXiv

Source-linked AI summary

STCFormer: Adaptive Spatio-Temporal Modeling with Dynamic Cluster Transformer for Station-based Weather Forecasting

Rongwen Li, Haixin Xie, Mingyang Wang, Hongwu Liu, Kun Fang, Changjian Chen, Zhuo Tang, Kenli Li

arXiv:2610.00377v1cs.LGstat.ML

TL;DR

Station-based weather forecasting must model complex station dependencies, but shared window-level groupings can miss local changes and cluster-only interactions can lose global context. STCFormer dynamically groups stations within temporal patches and combines local attention with global regional summaries. Across eight tasks, it achieves the lowest 24-hour MSE on every task and ranks first or second in 47 of 48 comparisons.

  • Problem

    Shared window-level groupings may obscure changing station relationships, cluster-only interactions may miss global context, and the theoretical advantages of selective interactions remain insufficiently explained.

    Source: §1 · p. 2

  • Method

    STCFormer dynamically groups stations by local evolution within temporal patches and combines cluster-local attention with global attention over regional summaries.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    STCFormer achieves the lowest 24-hour MSE on all eight tasks and ranks first or second in 47 of 48 metric–task–horizon comparisons.

    Source: §4 · p. 10

  • Takeaways & Limitations

    Ablations and case studies support the complementary roles of locally adaptive grouping and local–global interactions in the evaluated weather-forecasting tasks.

    Source: §1 · p. 2, §4 · p. 10

    The sensitivity bound is derived for a fixed hard partition and bounded feature and weight norms, so it does not directly establish the bound for changing partitions.

    Source: §2.4 · p. 6

Abstract

from arXiv · show

Station-based weather forecasting supports daily life and economic activity, yet accurate forecasts require modeling complex spatial dependencies among stations. Recent clustering-based selective modeling offers a promising alternative to dense inter-station interactions. However, a grouping shared across an observation window may obscure local changes in station relationships, while intra-cluster interactions alone may miss important global context. The theoretical advantages of selective interactions over dense connectivity also remain insufficiently understood. We therefore propose STCFormer, an adaptive spatio-temporal Transformer that dynamically groups stations according to their local evolution within each temporal patch. Its Cluster-Guided Attention Block combines fine-grained local attention within clusters and global attention over regional state summaries, allowing each station to access information beyond its own cluster. We further show that a derived Lipschitz upper bound for cluster-conditioned local attention is no larger than its fully connected counterpart, explaining a potential robustness benefit and motivating the design of InfoLoss. Experiments on three real-world weather datasets spanning eight temperature and wind forecasting tasks show that STCFormer achieves the lowest 24-hour mean squared error on all eight tasks and ranks first or second in 47 of 48 comparisons across metrics and forecasting horizons. Ablations and case studies further confirm the benefits of locally adaptive grouping and complementary local-global interactions. Our code can be obtained at https://github.com/hnu-vis/STCFormer.

1 INTRODUCTION

Station-level forecasting must capture complex spatial and temporal dependencies shaped by local environments and atmospheric processes. STCFormer addresses changing station relationships with locally adaptive grouping, local–global attention, and a theoretical analysis of selective interactions.

1 INTRODUCTION

  • Accurate station-level forecasting must capture complex spatio-temporal dependencies shaped by local environments and atmospheric processes.The introduction frames this as a key challenge for weather forecasting.Source: §1 · p. 1
  • Clustering-based selective modeling can reduce the need for dense station-pair interactions, but existing groupings may miss evolving relationships and cross-region context.A shared window-level grouping can blend changing local ties, while restricting interactions to cluster members may lose useful information.Source: §1 · pp. 1–2
  • The introduction identifies insufficient theoretical support for clustering-based selective interactions and the need to explain their potential advantages over dense interactions.It also identifies potential robustness benefits as insufficiently explained.Source: §1 · p. 2
  • STCFormer dynamically groups stations from their evolution within each temporal patch and combines cluster-local attention with global attention over regional summaries.This design aims to adapt to local relationship changes while retaining access to global context.Source: §1 · p. 2
  • The paper compares a derived Lipschitz upper bound for cluster-conditioned local attention with the fully connected bound and connects the analysis to InfoLoss.The bound is no larger than its fully connected counterpart, and the connection to mutual information motivates the regularizer.Source: §1 · p. 2
  • Experiments across three real-world datasets and eight forecasting tasks, alongside ablations and case studies, assess the model and its design choices.The introduction describes the results as competitive and says the studies validate locally adaptive grouping and complementary local–global interactions.Source: §1 · p. 2

2 STCFORMER

STCFormer forecasts from joint spatio-temporal patch tokens using temporally contextualized state features and locally adaptive station groups. CGAB combines within-group attention with global attention over regional summaries, while sensitivity analysis motivates InfoLoss.

2.3 CLUSTER-GUIDED ATTENTION BLOCK (CGAB)

  • STCFormer's Cluster-Guided Attention Block combines dynamic station clustering, local attention within clusters, and global attention over regional summaries.Each station can access context beyond its own group.Source: §2 · p. 3, §2.3 · p. 4, Figure 2 · p. 4

2.1 SPATIO-TEMPORAL PATCH EMBEDDING

  • Joint spatio-temporal patches retain local spatial context by grouping observations across station and temporal axes, with nearby stations placed at nearby indices.Stations are latitude-banded and ordered by longitude in alternating directions across adjacent bands.Source: §2.1 · p. 3

2.2 DUAL-VIEW TEMPORAL REPRESENTATION

  • The state view contextualizes features across the observation window, while the evolution view concatenates each token with its first-order temporal difference for patch-specific grouping.The first evolution-view difference is set to zero.Source: §2.2 · p. 3

2.3.1 DYNAMIC STATION CLUSTERING

  • At each temporal patch, an EM-style procedure updates shared learnable prototypes using local evolution, then recomputes assignments so stations can change groups.Hard assignments route local attention; soft assignments support InfoLoss and regional aggregation.Source: §2.3.1 · pp. 4–5

2.3.2 CLUSTER-GUIDED LOCAL–GLOBAL ATTENTION

  • Hard-routed local attention models dependencies among stations with similar recent evolution, and global station-to-region attention supplies context from all regional content centers.Soft assignments retain graded membership in regional summaries, although local attention uses hard cluster boundaries.Source: §2.3.2 · p. 5

2.4 LIPSCHITZ BOUNDS AND MUTUAL INFORMATION REGULARIZATION

  • The derived local-attention Lipschitz upper bound is no larger than the dense bound, and is strictly tighter when m(H) < n and v, β > 0.The maximum cluster size controls the local-attention certificate.Source: §2.4 · pp. 5–6
  • InfoLoss maximizes within-patch station–cluster mutual information; increasing it tightens or preserves a probabilistic certificate on the largest sampled group at fixed n, K, v, β, and δ.The resulting group-size certificate connects the training objective to the local-attention bound; δ is an analysis parameter, not a training hyperparameter.Source: §2.4 · p. 6

3 EXPERIMENTS

Across three weather datasets and eight forecasting tasks, STCFormer leads the 24-hour MSE comparison and ranks first or second in 47 of 48 metric–task–horizon comparisons. Ablations, sensitivity tests, efficiency results, and a clustering case study examine its components and behavior.

3.1 EXPERIMENTAL SETUP

  • The evaluation covers French, Hunan, and Global weather datasets, eight temperature and wind tasks, and 15 spatio-temporal and multivariate forecasting baselines.All datasets use a 6:2:2 train/validation/test split and training-set Z-score normalization; experiments repeat three times and report mean and standard deviation.Source: §3.1 · p. 7

3.2 MAIN RESULTS

  • STCFormer achieves the lowest mean 24-hour MSE on all eight tasks and ranks first in 42 and second in five of 48 metric–task–horizon comparisons.It outperforms the clustering-based methods DUET and CCM and the dynamic interaction-selection method TimeFilter on every main-table task.Source: §3.2 · p. 7

3.3 ABLATION STUDIES

  • The full model has the lowest mean MSE on all eight tasks; window-shared grouping raises average MSE by 2.9%, full attention by 3.6%, and fixed geographic grouping by 3.3%.Removing either local or global attention worsens all tasks; on Global wind, MSE rises from 3.584 to 3.760 without global attention and 3.811 without local attention. InfoLoss removal has task-dependent effects, including Hunan temperature rising from 4.743 to 5.119 and Global temperature changing from 6.917 to 6.930.Source: §3.3 · p. 8, Table 2 · p. 8

3.4.1 HYPERPARAMETER SENSITIVITY

  • Across French, Hunan, and Global temperature tasks, the best InfoLoss weight is 0.1 on French and Hunan and 0.2 on Global, with nearby cluster ratios remaining competitive.The best cluster ratio is 0.1 on French and Hunan and 0.05 on Global; French favors dmodel = 256, while Hunan and Global favor 512, with small gains beyond 256.Source: §3.4.1 · pp. 8–9, Figure 3 · p. 8

3.4.2 EFFICIENCY ANALYSIS

  • On Hunan temperature, STCFormer reports the lowest MSE (4.743) and memory usage (7.429 GiB), taking 46.372 ± 0.095 s/epoch.Compared with Crossformer, it reduces training time by 26.0% and memory by 35.4%; PatchTST is faster at 36.913 s/epoch but has higher MSE (5.509) and memory usage (7.930 GiB).Source: §3.4.2 · p. 10, Figure 4 · p. 10

3.4.3 DYNAMIC CLUSTERING CASE STUDY

  • On French-temperature test windows, dynamic grouping achieves the highest median coherence (0.385) and lowest forecasting MSE (6.291), versus Fixed (0.290; 6.516) and Random (0.244; 6.620).Fixed grouping also outperforms Random, while Dynamic’s additional gain is consistent with capturing transient couplings obscured by a shared partition.Source: §3.4.3 · p. 10

4 CONCLUSION

STCFormer targets changing station dependencies while retaining access to global context, combining locally adaptive grouping with local and regional attention. The paper concludes that results across three datasets support locally adaptive, selective spatial modeling for station-based weather forecasting.

4 CONCLUSION

  • STCFormer combines locally updated station groups with attention over all regional content centers, addressing the limits of window-shared assignments and exclusively within-group interactions.The conclusion reports the lowest 24-hour MSE on all eight tasks and first- or second-place rankings in 47 of 48 comparisons.Source: §4 · p. 10, A RELATED WORK · p. 15

A RELATED WORK

  • Prior work spans graph and physical-structure spatio-temporal methods, channel-independent or channel-mixing MTS methods, and clustering or patch-specific interaction selection.STCFormer’s stated focus is grouping stations from local evolution at each temporal patch while retaining information from all groups.Source: A RELATED WORK · p. 15

B EFFICIENT IMPLEMENTATION OF INTRA-CLUSTER LOCAL ATTENTION

The implementation packs cluster-local attention and recomputes station assignments at every temporal patch and encoder layer. A logarithmic membership bias preserves hard routing in the forward pass while allowing assignment gradients in the backward pass.

B EFFICIENT IMPLEMENTATION OF INTRA-CLUSTER LOCAL ATTENTION

  • Assignments are recomputed at every temporal patch and encoder layer, and packed local attention avoids an explicit N′ × N′ mask.The hard partition defines each token’s local attention neighborhood.Source: B EFFICIENT IMPLEMENTATION OF… · p. 15
  • Selected memberships make the logarithmic key bias zero for hard forward routing, while its backward pass propagates gradients through the Gumbel–Softmax relaxation to R.Inference instead uses deterministic argmax routing.Source: B EFFICIENT IMPLEMENTATION OF… · p. 15

C COMPUTATIONAL COMPLEXITY ANALYSIS

The analysis gives per-module operation and memory costs under stated fixed-width assumptions, then compares selected operators and bounds the sensitivity of fixed-partition local attention. Its derived local-attention upper bound depends on the largest cluster and is strictly below the dense bound when clusters are smaller than the full token set.

C.1 MODULE-WISE COMPUTATION

  • Per block, temporal attention costs O(nm^2D), clustering and station-to-region attention cost O(mnKD), InfoLoss costs O(mnK), and the prediction head also depends on the horizon.The head’s two projections cost O(nmDDh + nDhOLs) for one target variable.Source: §C.1 · p. 16

C.2 OVERALL COST AND MEMORY

  • Increasing K reduces balanced local-attention work but increases clustering, global-attention, and InfoLoss costs; when K is proportional to n, the full block remains quadratic in n.Memory estimates distinguish materialized attention scores from token features and assignments; fused kernels can avoid storing full score matrices.Source: §C.1 · p. 16, §C.2 · p. 17

C.3 COMPARISON WITH CURRENT BASELINES

  • The baseline table compares representative dependency-modeling operators at fixed widths, excluding embeddings, output heads, and training losses; matched measurements show a favorable accuracy–resource trade-off.The comparison covers operator costs rather than end-to-end forecasting costs.Source: §C.3 · p. 17, Table 3 · p. 18

D.1 NOTATION AND PRELIMINARIES

  • The sensitivity analysis fixes the realized partition and projection weights, using Frobenius norms for feature perturbations and spectral norms for weights.The analyzed domain bounds each token’s Euclidean norm by RX.Source: §D.1 · p. 18

D.2 PROOF OF THEOREM 2.1

  • When the largest cluster has size m(H) < n, the derived local-attention upper bound B(m(H)) is strictly below the dense bound B(n).These are upper bounds, not exact Lipschitz constants.Source: §D.2 · p. 20

D.3 PROOF OF THEOREM 2.2

The proof connects the information gap of soft assignments to bounds on sampled cluster size and then to the fixed-partition sensitivity bound. It also gives a deterministic argmax counterpart and discusses patching and shared-partition multihead extensions.

D.3 PROOF OF THEOREM 2.2

  • Theorem 2.2 relates the information gap log K − I(X; Y) to a probabilistic upper bound on the largest sampled group and, through Theorem 2.1, its sensitivity bound.At fixed n, K, v, β, and δ, increasing mutual information tightens or preserves the certificate.Source: §D.3 · pp. 20–21

D.4 DETERMINISTIC ARGMAX GROUPING AT INFERENCE

  • For row-wise argmax grouping, the separate deterministic size and sensitivity bounds require no sampling assumption or failure probability.The bound applies separately to every temporal patch.Source: §D.4 · p. 21

D.5 PATCHING, MULTIPLE HEADS, AND SIMULTANEOUS BOUNDS

  • Spatial patching reduces n and balanced groups reduce the maximum interacting set; simultaneous sampled bounds use a union bound, and the multihead extension assumes a shared partition.The local-attention bound alone does not bound normalization, feed-forward layers, regional attention, or the prediction head.Source: §D.5 · pp. 21–22

E PROPERTIES OF INFOLOSS

InfoLoss’s properties link soft cluster balance and assignment confidence to a mutual-information gap. That gap is connected to bounds on the largest group and on fixed-partition sensitivity, with assignments evaluated separately at each temporal patch.

E PROPERTIES OF INFOLOSS

  • Maximizing marginal entropy encourages balanced soft cluster masses and discourages collapse into a few clusters.The marginal entropy equals log K minus the KL divergence from the uniform cluster distribution.Source: E PROPERTIES OF INFOLOSS · p. 22
  • Minimizing conditional entropy favors confident, near-deterministic token-to-cluster assignments.Together with evolution-driven centroids, this produces clear and input-adaptive regional memberships.Source: E PROPERTIES OF INFOLOSS · p. 22
  • The nonnegative information gap combines conditional entropy and deviation from uniform cluster masses; a small gap promotes confidence and balance and bounds the largest sampled group.The resulting group-size bound connects to fixed-partition sensitivity; per-patch, label-permutation-invariant loss supports locally adaptive regrouping.Source: E PROPERTIES OF INFOLOSS · p. 22

F.1 DATASETS AND PREPROCESSING

The appendix documents the datasets, baselines, and shared evaluation setup used for STCFormer’s experiments. It also specifies model configurations and metric calculations.

F.1 DATASETS AND PREPROCESSING

  • The processed data comprise eight forecasting tasks across French, Hunan, and Global weather datasets, with station observations split chronologically 6:2:2 and Z-score normalized using training statistics.French has 234 stations; Hunan and Global have 3,570 and 3,850, respectively.Source: §F.1 · pp. 22–23, Table 4 · p. 22

F.2 BASELINES

  • The comparison uses 15 baselines: six spatio-temporal methods and nine multivariate time-series methods spanning graph-based, station-specific, and temporal approaches.CCM uses DLinear as its forecasting backbone and learns cluster-specific shared parameters.Source: §F.2 · p. 23

F.3 IMPLEMENTATION AND HYPERPARAMETERS

  • Experiments use 48-hour inputs to forecast 24 or 72 hours, with Adam, MSE training loss, up to 50 epochs, and early stopping after five unimproved epochs.Reported results average three runs; additional metrics and 72-hour results are reported in Appendix G.Source: §F.3 · p. 23
  • STCFormer uses three layers, four attention heads, and dataset-specific temporal and spatial patch sizes; InfoLoss has weight λ = 0.1 across datasets.SEDI uses a fixed training-period 95th-percentile threshold pooled across stations; its pooled hit and false-alarm rates are computed per run.Source: §F.3 · p. 23

G ADDITIONAL FORECASTING RESULTS

STCFormer performs consistently across the additional forecasting metrics and horizons. Across 48 comparisons, it ranks first or second in all but one.

G ADDITIONAL FORECASTING RESULTS

H ADDITIONAL ABLATION RESULTS

Additional experiments assess ablations, external model comparisons, hyperparameters, efficiency, attention design, and spatial patch construction. Across these settings, results favor STCFormer and show that choices such as look-back length and patch construction can affect performance.

H ADDITIONAL ABLATION RESULTS

I.1 COMPARISON WITH TIME SERIES FOUNDATION MODELS

  • STCFormer obtains lower mean MSE on all eight tasks than the evaluated fine-tuned time-series foundation models, with reductions of 1.1%–5.6% against the strongest foundation-model baseline per task.Time-MoE is strongest on six tasks; Chronos-Bolt-Tiny is strongest on Hunan temperature and U-wind.Source: §I.1 · p. 27

I.2 COMPARISON WITH GRID-BASED PREDICTIVE MODELS

  • STCFormer achieves the lowest mean MSE on all eight tasks against the evaluated grid-based models; on Global temperature and wind, MSE falls from 20.139 and 6.574 to 6.917 and 3.584.The authors note that station–grid interpolation can introduce conversion error and argue for direct forecasting on irregular station networks.Source: §I.2 · pp. 27–28

I.3.1 LOOK-BACK WINDOW LENGTH

  • For 72-hour French temperature forecasting, STCFormer’s lowest MSE is 11.245 with a 48-hour look-back; longer histories do not consistently improve performance.CCM performs best at 96- and 168-hour look-backs.Source: §I.3.1 · p. 28, Figure 6 · p. 28

I.3.2 SPATIO-TEMPORAL PATCH SIZES

  • Joint patch-size sweeps show dataset-dependent optima: French favors (Lt, Ls) = (6, 2), while Hunan reaches 4.742 at (2, 10) and 4.743 at (4, 7).With Lt = 4, increasing Ls from 1 to 2 on French reduces MSE from 6.553 to 6.291; increasing it from 1 to 7 on Hunan reduces MSE from 5.163 to 4.743.Source: §I.3.2 · p. 28, Figure 7 · p. 29

I.4 ADDITIONAL EFFICIENCY ANALYSIS

  • On French temperature, STCFormer records MSE 6.291 at 15.306 ± 0.003 s/epoch and 0.378 GiB, while training 18.1× faster than DCRNN.It uses 28.3% less training time and 86.8% less memory than TimeFilter; PatchTST and TimeXer train faster, while DUET uses less memory.Source: §I.4 · p. 29, Figure 8 · p. 29

I.4.1 COMPARISON WITH FULL ATTENTION

  • Compared with full attention, CGAB lowers MSE on French and Global temperature, with epoch-time increases of 22.7% and 11.4%, respectively.On the larger Global network, peak memory falls 11.1%, from 10.023 to 8.913 GiB.Source: §I.4.1 · p. 30, Table 13 · p. 30

I.5 SENSITIVITY TO SPATIAL PATCH CONSTRUCTION

  • All three geographic patch constructions achieve lower mean MSE than random grouping on both datasets; Hilbert ordering is best on French (6.152), while serpentine ordering is best on Hunan (4.743).Serpentine ordering reduces mean MSE versus random grouping by 1.4% on French and 4.8% on Hunan.Source: §I.5 · p. 30

I.6 NUMERICAL DIAGNOSTICS FOR THE THEORETICAL ANALYSIS

Numerical diagnostics on French temperature examine cluster-size effects, InfoLoss, and perturbation amplification. Prediction visualizations illustrate forecasts across three dataset-variable settings.

I.6 NUMERICAL DIAGNOSTICS FOR THE THEORETICAL ANALYSIS

  • The mean diagnostic ratio B(q)/B(n) is 0.4076 with InfoLoss and 0.5337 without it, and a smaller largest cluster yields a smaller fixed-partition diagnostic.At q = n, the diagnostic recovers the dense value.Source: §I.6 · p. 30, Figure 9 · p. 31
  • With InfoLoss, mean station–cluster mutual information is higher while largest cluster size, cluster-size variation, and empty-cluster rate are lower; the mean local-to-dense diagnostic ratio falls by 0.1261.The authors describe these findings as consistent with the theoretical chain linking more informative, balanced assignments to smaller clusters and a tighter fixed-partition diagnostic.Source: §I.6 · p. 30, Table 15 · p. 31
  • Maximum sampled amplification rises from 2.6266 for Balanced to 3.8256 for Dense partitions; Learned partitions reach 2.7020, below size-matched Random at 2.9542.The perturbation comparison freezes partitions and disables dropout; it covers 64 test windows and reports the largest sampled amplification for each family.Source: §I.6 · p. 31, Figure 9 · p. 31

I.7 PREDICTION VISUALIZATIONS

  • Across the illustrated French temperature, Hunan temperature, and Global wind cases, STCFormer tracks troughs, regime changes, daily fluctuations, and rising wind trends more faithfully than the cited baseline forecasts.The visualizations use 48 hours of observations to predict 72 hours at the same station and forecast window across methods.Source: §I.7 · p. 31, Figure 10 · p. 32, Figure 11 · p. 33

Related papers

Loading 2610.00377v1…