Beyond Static Positional Encoding: Bayesian Time-aware Attention for Non-Stationary Long-term Time Series Forecasting

• Changing trends, seasonality, and distributions can alter which historical observations inform future predictions, challenging Transformers to model dependencies that vary across input windows.Source: §1 · p. 1
• Normalization may leave different temporal dependence structures among windows with similar marginal statistics, while specialized architectures can add complexity and limit applicability across Transformer backbones.Source: §1 · pp. 1–2
• The paper interprets position-aware attention as Bayesian inference and derives a population-level ELBO, showing static positional encoding can be suboptimal when temporal dependencies change.Source: §1 · p. 2
• BTA uses an MoE router to combine complementary temporal kernels into input-conditioned, head-specific positional biases.Source: §1 · p. 2

Experiments

• BTA consistently improves non-stationary LTSF across conventional Transformers and time-series foundation models while maintaining computational efficiency.The paper presents BTA as a plug-and-play module that learns input-conditioned, head-specific temporal biases.Source: §1 · pp. 2–3

• Non-stationary forecasting methods either reduce distributional variation in representations or modify forecasting mechanisms to capture evolving distributions and temporal patterns.Source: §2 · p. 3
• Many positional methods use predefined or globally learned structures shared across samples; BTA instead mixes probabilistic temporal kernels into an input-conditioned relative attention bias.Its MoE router dynamically combines kernels to adapt the bias to heterogeneous periodicities and temporal scales.Source: §2 · p. 3

3.1 Preliminaries and Notations

• The analysis models a zero-mean weakly stationary Gaussian process whose covariance is determined by the autocovariance at each lag.Source: §3.1 · p. 4
• Without positional encoding, self-attention is permutation-equivariant and cannot distinguish temporal positions from content alone.The simplified forecasting Transformer retains attention, positional encoding, aggregation, and linear prediction for tractable analysis.Source: §3.1 · p. 4

3.2 Bayesian Explanations for Transformer in non-stationary LTSF

• Position-aware attention is interpreted as a posterior over selected key positions, with content-based attention as the prior and positional encoding as a temporal-evidence likelihood.Source: §3.2 · p. 5
• For weakly stationary Gaussian forecasting, the population-level ELBO lower-bounds expected log predictive likelihood and includes a divergence between the PE-induced posterior and the temporal-evidence prior.Source: §3.2 · p. 5
• The non-stationary analysis assumes each forecasting window is weakly stationary conditional on its regime, making the per-regime characterization tractable.Source: §3.2 · p. 5
• When regimes have incompatible optima, a static PE shared across regimes can be strictly suboptimal relative to a regime-specific choice, motivating input-adaptive positional evidence.The paper illustrates this with alternating smooth and periodic Gaussian-process regimes, where BTA has a higher estimated ELBO than evaluated PE baselines.Source: §3.2 · pp. 5–6

4 Methodology

• BTA combines a probabilistic mixture of candidate temporal patterns with an MoE router that produces input- and head-specific weights without explicit regime labels.The resulting temporal-evidence likelihood enters attention logits as an adaptive positional bias.Source: §4 · p. 7

4.1 Probabilistic Mixture of Temporal Kernels

• The kernel bank mixes component likelihoods whose time-aware log-kernels encode distinct patterns such as locality, periodicity, or long-range dependence.The router selects their input- and head-specific contributions, allowing the candidate set to represent complementary temporal structures.Source: §4.1 · p. 7

4.2 MoE Router

• The router aggregates content-based attention scores by relative lag, combines the lag profile with its Fourier-spectrum magnitude, and uses an MLP to generate mixture weights.Top-k masking gives unselected kernels zero weight; the resulting bias is added to attention logits before softmax, with only the router learning to combine the fixed kernels.Source: §4.2 · pp. 7–8

5.1 Experimental Setup

• The evaluation spans eight multivariate benchmarks and tests BTA with four Transformer backbones and ten LTSF baselines.The benchmarks cover 7–862 variables and diverse domains and temporal resolutions.Source: §5.1 · p. 8

5.2 Main Results

• Across four datasets, BTA reduces average MSE by 9.70%, 10.35%, and 5.70% for Informer, PatchTST, and Timer-XL, respectively.The corresponding MAE reductions are 5.99%, 6.84%, and 2.68%.Source: §5.2 · pp. 8–9, Table 1 · p. 9
• TimeBridge+BTA reduces dataset-averaged MSE by 2.39% and MAE by 1.89%, with best MSE on 7/8 datasets and best MAE on 5.The comparison covers eight datasets and ten reference algorithms.Source: Table 2 · p. 9, §5.2 · p. 9

5.3 Case Study

• In two ETTm1 windows, weekly-pattern weights favored periodic kernels (0.75), while local dependence favored RBF kernels (0.66); BTA cut OT-channel MSE about 37% in both.The reported OT-channel MSEs were 0.219 → 0.139 and 0.970 → 0.608.Source: Figure 2 · p. 10, §5.3 · p. 10

5.4 Representation Analysis

• Across ETTm1 test samples and attention heads, longer horizons shift kernel weights toward broader local and periodic patterns.Shorter horizons emphasize relatively local dependencies.Source: §5.4 · p. 10, Figure 3 · p. 11

5.5 Hyperparameter Analysis

• The preferred Top-k varies by dataset (3 for ETTm1 and Solar, 1 for ETTh1), as does the best router dimension (192, 160, and 96, respectively).The study reports that router capacity is non-monotonic and selects dr by validation performance.Source: §5.5 · p. 11, Figure 4 · p. 11

5.6 Ablation Study

• Ablations show gains from adding local kernels, combining local and periodic kernels, and replacing static weights with an MoE router using temporal evidence.Adding the Fourier spectrum of temporal evidence produced the largest reported gains, 5.65% and 6.55%.Source: §5.6 · pp. 11–12, Table 3 · p. 12

6 Conclusions

• BTA adaptively combines complementary temporal kernels through an input- and head-specific mixture-of-experts router.The paper presents BTA as a plug-and-play mechanism.Source: §6 · p. 12
• The analysis shows that static positional encoding can be suboptimal when temporal dependence varies across input windows.The paper derives a population-level ELBO by interpreting position-aware self-attention as Bayesian inference over attended positions.Source: §6 · p. 12
• Across multiple Transformer backbones and eight LTSF benchmarks, BTA consistently improves forecasting accuracy and remains competitive with recent non-stationarity methods.Source: §6 · p. 12

Disclosure of AI Use

• The authors report using ChatGPT-5.6 Sol only to polish manuscript language and readability, with authors reviewing the edits and taking responsibility for the final manuscript.They state that generative AI was not used to develop theory, design or implement BTA, conduct experiments, process data, interpret results, or translate the manuscript.Source: Disclosure of AI Use · pp. 12–13

A The Simplified Transformer Architecture

• In the simplified Transformer, positional encoding affects the forecasting operator through its influence on the attention distribution.The analysis omits layer normalization, feed-forward sublayers, and residual connections; adding an attention-block residual replaces the attention matrix with that matrix plus the identity without changing the effective linear-operator representation.Source: A The Simplified Transformer… · p. 18

B Proof of Proposition 1

• Position-aware self-attention is Bayesian reweighting: scaled dot-product similarity supplies a content-based prior, and positional encoding supplies the temporal-evidence likelihood.The proof establishes this interpretation by matching the normalized logits under its candidate-specific temporal-evidence assumption.Source: B Proof of Proposition 1 · p. 19

C Probabilistic Interpretations of Positional Encoding

• Positional encodings encode distinct temporal-evidence assumptions; unlike distance-based biases, RoPE’s evidence depends jointly on relative lag and query–key content.The accompanying table summarizes representative methods’ probabilistic interpretations; RoPE can therefore assign different evidence at the same lag for different query–key pairs.Source: C Probabilistic Interpretations of… · pp. 19, 21, Table 4 · p. 20

D Proof of Theorem 2

• The proof of Theorem 2 proceeds from a sample-level ELBO to a future-free PE-induced variational family and then averages the bound over the forecasting population.The variational family depends on historical input at evaluation, even though its parameters may be learned using forecasting errors involving future targets.Source: D Proof of Theorem 2 · p. 21, §D.2 · p. 22, §D.3 · p. 22

D.1 Step 1: Sample-level ELBO

• The sample-level variational lower bound applies without assuming a Gaussian forecasting likelihood or weak stationarity.It is derived for a forecasting sample with latent temporal evidence using a variational distribution supported within the prior’s support.Source: §D.1 · pp. 21–22

D.4 Step 4: Covariance-based Population Form

• For static positional encoding, the population ELBO has a covariance-based form involving historical covariance and future–history cross-covariance.The derivation preserves the dependence of the PE-induced posterior on historical input rather than replacing it with a population marginal.Source: §D.4 · p. 24

D.5 Step 5: Prior and Posterior of Temporal Evidence

• The latent temporal-evidence prior is moment-matched to autocovariance statistics, which are invariant across realizations under a fixed weakly stationary process but may vary across non-stationary regimes.The analysis derives the evidence statistics from the underlying Gaussian process’s autocovariance function.Source: §D.5 · p. 25
• Static positional encoding induces a posterior shared across input windows, whereas adaptive positional encoding yields a window-specific posterior.For static encoding, the shared posterior lets the forecasting moments factor into population covariance matrices.Source: §D.5 · pp. 26–27
• The population variational formulation lower-bounds expected forecasting log-likelihood, with the Gaussian-process input determining the temporal-evidence prior and positional encoding determining the posterior through lag-profile statistics.The posterior captures first- and second-order statistics of lag-wise positional profiles.Source: §D.5 · p. 26

E Proof of Corollary 3

• Under regime-conditioned non-stationarity, the population objective is a weighted combination of regime-specific ELBOs and is strict when no static positional encoding sequence can approach all regime-specific optima.Static encoding shares one temporal-evidence posterior across regimes even when their GP-derived priors and covariance structures differ, while input-adaptive encoding can vary temporal evidence across windows.Source: E Proof of Corollary 3 · pp. 27–28

F.2 Simplified Transformer for Forecasting

• In the simplified Transformer, BTA generates an input-conditioned positional bias by combining RBF and periodic experts initialized for the synthetic setting.The model uses a single-layer, single-head Transformer; the four compared positional encodings are RPE, ALiBi, RoPE, and BTA.Source: §F.2 · p. 29

F.1 Binary-Regime Synthetic Dataset

• The dataset alternates between smooth RBF and periodic Gaussian-process regimes, with each 64-step input forecasting the next four observations.Each trajectory draws one regime and retains it throughout; the periodic regime has a 24-step cycle, and regime labels are withheld from models.Source: Figure 5 · p. 28, §F.1 · p. 28, §F.3 · p. 29

F.3 Synthetic Validation of the Population-Level ELBO

• The ELBO estimate samples temporal evidence while holding trained parameters and observed content scores fixed, and combines forecasting fit with agreement to the regime-specific GP prior.The evaluation uses 16 samples per test window and a shared Gaussian observation variance of 1.Source: §F.3 · p. 31
• BTA achieves both the lowest forecasting MSE and highest estimated ELBO on the binary-regime synthetic test set.Its expected log likelihood is higher and posterior–prior KL lower than those of the baseline positional-encoding methods; the values estimate the input-dependent perturbation objective, not the static covariance-reduced formula.Source: §F.3 · p. 31, Table 6 · p. 31

G Data Description

• The benchmark protocol uses chronological train, validation, and test splits across eight multivariate datasets and horizons of 96, 192, 336, and 720.Preprocessing parameters are estimated on training data and applied unchanged to validation and test sets.Source: G Data Description · p. 32, Table 7 · p. 32

H Transformer-based backbone architectures; I LTSF reference methods

• The experiments test BTA across Informer, PatchTST, TimeBridge, and Timer-XL, and compare it with non-stationary-specific and general-purpose LTSF baselines.The four backbones span different forecasting architectures; the baseline set includes five methods targeting non-stationarity and five widely used LTSF models.Source: H Transformer-based backbone… · p. 33, I LTSF reference methods · p. 33

J.1 Backbone Integration; J.2 Temporal Kernel Bank

• BTA adds an adaptive temporal bias to native attention logits while leaving the backbone's other components unchanged.Its fixed kernel bank combines RBF and periodic patterns; the same construction applies across point-wise and patch-based models.Source: §J.1 · p. 34, §J.2 · pp. 34–35

J.3 MoE Router Configuration; J.5 Training and Optimization

• For each input and head, the router aggregates attention scores by lag, adds Fourier information, and selects a sparse mixture of candidate kernels.A one-hidden-layer MLP outputs weights over five RBF and six periodic experts using Top-k selection; kernels stay fixed while the router and backbone are optimized jointly.Source: §J.3 · pp. 35–36, §J.5 · p. 36

K Main Experiment Appendix

• 95 of 96 backbone–dataset–horizon–metric settings improve with BTA; the exception is Informer's ETTm1 MAE at H = 192, which changes by −0.49%.Average MSE reductions across horizons are 5.79–14.05% for Informer, 4.34–17.51% for PatchTST, and 1.74–11.80% for Timer-XL.Source: K Main Experiment Appendix · pp. 36, 38, Table 10 · p. 37
• Across three backbones, BTA generally has lower mean MSE than vanilla attention, with limited variation across random initializations.Table 11 reports mean ± sample standard deviation across three random seeds.Source: Table 11 · p. 38, K Main Experiment Appendix · p. 38
• TimeBridge+BTA reduces overall average MSE by 2.39% and MAE by 1.89% across eight datasets, with the lowest MSE in 29 of 32 horizon-level settings.It also achieves the lowest MAE in 23 horizon-level settings.Source: K Main Experiment Appendix · p. 38, Table 12 · p. 39

L More case studies

• On Solar, BTA selects broader periodic and long-range local kernels for irregular peak amplitudes, but narrower periodic kernels for more regular peaks.The two windows show MSE reductions of 58% (22.48 → 9.41) and 34% (1.46 → 0.96), respectively.Source: L More case studies · p. 38, Figure 8 · p. 40

M More representation analysis on Solar

• On Weather, a recent level shift favors long-range local kernels, whereas recurring structure favors a 3-day periodic kernel.The level-shift window assigns 0.69 total weight to local RBF kernels; the recurring window assigns 0.68 to the 3-day periodic kernel.Source: Figure 9 · p. 41
• For abrupt-change ETTm1 inputs, BTA favors short-range local kernels, while a stable window favors broader local and periodic dependencies.The corresponding MSEs fall from 0.019 to 0.012 and from 0.031 to 0.019.Source: Figure 10 · p. 41
• As Solar's horizon increases, BTA shifts toward broader local dependencies, with the ℓ = 1.25-day RBF kernel dominant at H = 720.The allocation among periodic kernels also shifts toward broader 3-day patterns at longer horizons.Source: M More representation analysis on Solar · p. 40, Figure 11 · p. 42

N Additional Hyperparameter Analysis

• Solar's lowest displayed MSE occurs at k = 3, while other settings increase MSE by up to 8.20%; ETTh1 varies by less than 0.04%.The ETTh1 k = 1 and Solar k = 3 runs differ in batch size and patch-downsampling layers, respectively, so these are descriptive rather than controlled Top-k ablations.Source: N Additional Hyperparameter Analysis · p. 42
• The lowest tested router dimension differs by dataset: ETTh1 reaches its minimum at dr = 192, while Solar reaches its minimum at dr = 96.Across tested dimensions, MSE increases by at most 1.23% on ETTh1 and 9.30% on Solar; together with ETTm1 results, the findings suggest moderate capacity is generally sufficient.Source: N Additional Hyperparameter Analysis · p. 42

O Efficiency Analysis

• On ETTh1 and ETTm1, TimeBridge+BTA lowers average MSE from 0.399 to 0.394 (1.3%) and 0.348 to 0.337 (3.2%), respectively.It adds approximately 0.3 M FLOPs (0.10%) and 14 K trainable parameters (1.1%) on ETTh1, and 0.5 M FLOPs (1.00%) and 22 K parameters (6.0%) on ETTm1.Source: O Efficiency Analysis · pp. 42, 44, Table 13 · p. 43
• On Weather, BTA further improves the TimeBridge backbone while adding small computational cost.Source: O Efficiency Analysis · p. 44
• At ETTh1 horizon H = 720, BTA increases FLOPs by 0.10% and trainable parameters by 0.44%, with absolute overhead approximately constant across horizons.Source: O Efficiency Analysis · p. 44
• On ETTh1, DUET has a higher average MSE than TimeBridge+BTA (0.404 versus 0.394) despite using approximately 2.2–4.6× as many parameters.This comparison does not isolate model capacity; a capacity-matched ablation would be needed for that causal attribution.Source: O Efficiency Analysis · p. 44

P More Ablation Study

• Adding local and periodic kernels improves vanilla self-attention, and replacing static kernel weights with input-adaptive MoE routing yields further gains.Both MoE variants use local and periodic kernels; TE denotes the temporal-evidence profile, and Fourier denotes its Fourier spectrum.Source: P More Ablation Study · p. 44
• On Weather, adding the Fourier spectrum of the TE profile yields the best overall performance, reducing average MSE by 14.05% and MAE by 7.73%.Source: Table 14 · p. 44, P More Ablation Study · p. 44
• The Weather ablation progressively adds local kernels, periodic kernels, MoE routing based on the TE profile, and its Fourier spectrum to vanilla self-attention.Source: Table 14 · p. 44