PaperlayerView exact revision on arXiv

Source-linked AI summary

Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting

Yong Liu, Haixu Wu, Jianmin Wang, Mingsheng Long

arXiv:2205.14415v4cs.LGeess.SP

TL;DR

Transformers face reduced forecasting reliability on non-stationary real-world series, while conventional stationarization can erase useful temporal information. The paper introduces Nonstationary Transformers, combining Series Stationarization with De-stationary Attention, and reports large improvements across mainstream Transformers and six benchmarks.

  • Problem

    Non-stationary series change statistical properties over time, while stationarization can produce over-stationarization that removes distinctions needed to model eventful temporal dependencies.

  • Method

    Nonstationary Transformers combine parameter-free Series Stationarization with De-stationary Attention that restores intrinsic non-stationary information from original series statistics.

  • Results

    Nonstationary Transformers consistently boost four mainstream Transformers and achieve state-of-the-art performance on six real-world benchmarks.

  • Takeaways & Limitations

    The framework jointly leverages the predictability of stationarized series and temporal dependencies from original non-stationary data.

  • Takeaways & Limitations

    The analysis assumes linear embedding and feed-forward layers and shared variance across series variables.

Abstract

from arXiv · show

Transformers have shown great power in time series forecasting due to their global-range modeling ability. However, their performance can degenerate terribly on non-stationary real-world data in which the joint distribution changes over time. Previous studies primarily adopt stationarization to attenuate the non-stationarity of original series for better predictability. But the stationarized series deprived of inherent non-stationarity can be less instructive for real-world bursty events forecasting. This problem, termed over-stationarization in this paper, leads Transformers to generate indistinguishable temporal attentions for different series and impedes the predictive capability of deep models. To tackle the dilemma between series predictability and model capability, we propose Non-stationary Transformers as a generic framework with two interdependent modules: Series Stationarization and De-stationary Attention. Concretely, Series Stationarization unifies the statistics of each input and converts the output with restored statistics for better predictability. To address the over-stationarization problem, De-stationary Attention is devised to recover the intrinsic non-stationary information into temporal dependencies by approximating distinguishable attentions learned from raw series. Our Non-stationary Transformers framework consistently boosts mainstream Transformers by a large margin, which reduces MSE by 49.43% on Transformer, 47.34% on Informer, and 46.89% on Reformer, making them the state-of-the-art in time series forecasting. Code is available at this repository: https://github.com/thuml/Nonstationary_Transformers.

1 Introduction

Transformers struggle with non-stationary time series, while stationarization improves predictability but can erase useful temporal distinctions. Nonstationary Transformers address this trade-off through Series Stationarization and De-stationary Attention.

  • Non-stationary data changes its statistical properties and joint distribution over time, making forecasting and deep-model generalization difficult.
  • Stationarization can stabilize distributions and improve predictability, but may remove inherent non-stationary information useful for forecasting eventful temporal dependencies.
  • Over-stationarization makes Transformers produce indistinguishable attentions for different series, limiting predictive ability and potentially causing large deviations from ground truth.
  • Series Stationarization unifies each series’ key statistics without extra parameters, while De-stationary Attention re-incorporates non-stationarity from the original series.
  • Nonstationary Transformers boost four mainstream Transformers and achieve state-of-the-art performance on six real-world benchmarks.

2 Related Work

Related work spans recurrent and Transformer-based forecasting models, alongside statistical and deep-learning approaches that address non-stationarity through stationarization.

  • RNN-based models support autoregressive sequence modeling but can suffer from difficulty modeling long-term dependencies.
  • Transformer variants reduce self-attention complexity or add forecasting-specific mechanisms, including dominant-query selection, locality-sensitive hashing, decomposition, and hierarchical attention.
  • ARIMA stationarizes time series through differencing, while deep forecasting methods widely use stationarization to address distribution changes.

3 Non-stationary Transformers

Non-stationary Transformers balance predictability and model capability by stationarizing inputs and restoring non-stationary information in attention. The framework wraps base Transformers with Series Stationarization and De-stationary Attention to preserve temporal dependencies from raw series.

  • Series Stationarization: Series Stationarization attenuates non-stationarity by normalizing each input series and de-normalizing the predicted output.The two-stage transformation gives base models stable-distribution inputs while restoring the original statistics for forecasting.
  • Series Stationarization: Normalization decreases distributional discrepancy among input series, making the model input more stable and easier to generalize.The method transforms each series through translation and scaling, then applies the corresponding statistics to the output.
  • Over-stationarization: Series Stationarization alone can map distinct series to the same stationarized input, producing identical attention and losing temporal dependencies entangled with non-stationarity.This over-stationarization can yield over-stationary and uneventful outputs.
  • De-stationary Attention: De-stationary Attention approximates attention learned from unstationarized series by restoring non-stationary information removed during stationarization.It replaces the original attention mechanism and rescales temporal dependency weights with learned factors τ and ∆.
  • De-stationary Attention: An MLP learns de-stationary factors τ and ∆ from the mean and standard deviation of the unstationarized input, while stationarized values provide the attention values.The resulting mechanism uses both stationarized-series representations and raw-series statistics to retain predictability and temporal dependencies.
  • Overall architecture: The framework uses a standard Encoder-Decoder architecture, wrapping the base model at input and output while replacing Self-Attention with De-stationary Attention.The encoder extracts information from past observations, and the decoder aggregates it to refine predictions.

4 Experiments

Experiments evaluate the framework across six real-world benchmarks, forecasting settings, Transformer variants, and ablations. The framework consistently improves forecasting, especially on highly non-stationary data, while De-stationary Attention restores non-stationary behavior suppressed by stationarization.

  • Main Results: 17% MSE reduction (0.509 →0.421) on Exchange and 25% (2.669 →2.010) on ILI were achieved at prediction length 336 against previous state-of-the-art results.These gains occur on datasets characterized by high non-stationarity.
  • Framework Generality: 49.43% promotion on Transformer, 47.34% on Informer, 46.89% on Reformer and 10.57% on Autoformer was achieved by applying the framework.The improvements are averaged across prediction lengths, with hardly any additional parameters or computation.
  • Ablation Study: Series Stationarization improves generalization on out-of-distribution data, while De-stationary Attention preserves detailed series variation needed for accurate real-world forecasting.The ablation compares vanilla Transformer, stationarization alone, and the complete framework on ETTm2 predictions.
  • Model Analysis: Relative stationarity ∈[97%, 103%] for De-stationary Attention outputs is close to ground truth, unlike the unexpectedly stationary outputs from stationarization-only models.The discrepancy becomes more significant as the degree of series stationarity increases.
  • Model Analysis: Re-incorporating non-stationarity is necessary when inputs are stationarized, and Stat + DeAttn achieves the best result in most cases (77%).The ablation compares feed-forward-layer reintegration with attention-based reintegration.

5 Conclusion

The paper frames time series forecasting through stationarity and proposes jointly improving series predictability and model predictive capability. Experiments on six real-world benchmarks support the framework’s generality and performance, while future work targets a more model-agnostic solution.

  • Conclusion: The framework increases series stationarity while renovating the internal mechanism to re-incorporate non-stationary information.This design addresses the over-stationarization issue while improving data predictability and model predictive capability simultaneously.
  • Conclusion: Experiments show great generality and performance on six real-world benchmarks, with derivations and ablations supporting the effectiveness of each component.The conclusion summarizes the framework-level empirical validation.
  • Conclusion: Future work will explore a more model-agnostic solution for the over-stationarization problem.The paper identifies this as its stated future direction.

A Proof of De-stationary Attention

The proof analyzes how stationarization changes self-attention and motivates De-stationary Attention, which restores non-stationary information through learned scaling and shifting factors.

  • Self-Attention computes row-wise Softmax over scaled query-key similarities and values, with sequence length S and feature dimension d_k.The queries, keys, and values have shape R^{S×d_k}.
  • The analysis assumes time-point-wise linear embedding and feed-forward layers, with equal variance across input variables.Under these assumptions, the normalized input can be simplified using a scalar standard deviation.
  • Stationarization normalizes each time point as x′_i = (x_i − µ_x)/σ_x, removing mean and scale information from the model input.The resulting queries and keys are therefore learned from stationarized rather than raw series.
  • Because Softmax is invariant to row-wise translations, the proof derives an equivalent attention expression across Transformer layers under its assumptions.The induction extends the first-layer analysis through repeated attention, feed-forward, and residual blocks.
  • De-stationary Attention learns scaling τ and shifting ∆ factors from µ_x, σ_x, and raw x to approximate attention from the original series.It combines stationarized queries and keys with recovered non-stationary information while retaining stationarized values.
  • Projector dimension dim is tested at 64, 128, and 256, with larger dimensions favored for the more non-stationary Exchange and ILI datasets.The comparison uses input-36-predict-48 for ILI and input-96-predict-336 for other datasets.

C.1 Multivariable Forecasting Results

On ETT benchmarks, Non-stationary Transformers improve forecasting across prediction horizons and outperform additional baseline models in most comparisons.

  • 11.5% average MSE reduction is achieved over previous state-of-the-art deep forecasting models on ETT datasets.The ETT benchmarks include hourly ETTh1 and ETTh2 and 15-minutely ETTm1.
  • 4.4% MSE reduction is reported on ETTh1, 3.5% on ETTh2, and 26.7% on ETTm1 for input-96-predict-336 forecasting.The corresponding MSE pairs are 0.615 → 0.588, 0.572 → 0.552, and 0.675 → 0.495.
  • 83% of additional comparisons are won against FEDformer, LSSL, GRU, and other baseline models.The comparison includes a concurrent Transformer model and non-Transformer baselines.
  • The framework consistently promotes six Transformer variants, including concurrent ETSformer and FEDformer.The variants are Transformer, Informer, Reformer, Autoformer, ETSformer, and FEDformer.

C.3 Comparison with Stationarization Methods

Compared with stationarization baselines, the framework improves Transformer-family forecasting, especially on highly non-stationary datasets, while adding few parameters.

  • 11.6% relative promotion is reported for Reformer on ETTm2 and 4.4% on ILI under predict-336 forecasting.The corresponding MSE pairs are 0.632 → 0.559 and 2.834 → 2.770.
  • Qualitative ETTm1 forecasts show vanilla Transformer scale and level errors, while Series Stationarization can produce stationary, uneventful predictions.The framework is presented as addressing this loss of intrinsic non-stationarity across prediction horizons 96, 192, 336, and 720.
  • The framework significantly boosts forecasting performance with hardly any additional parameters.The comparison explicitly tracks parameter increment alongside performance promotion.
  • Series Stationarization has no additional learnable parameters and makes distributions more stable by attenuating non-stationarity.Lower ADF test statistics after normalization support its effectiveness as a normalization design.

D.2 Ablation of De-stationary Factors

Ablation results show that De-stationary Attention performs best when both its scaling and shifting factors are used together.

  • Forecasting performance degrades when only τ or only ∆ is used, validating the complete pair of de-stationary factors.The ablation compares only τ, only ∆, and τ with ∆ on Exchange and Electricity.
  • Without τ, MSE worsens from 0.196 to 0.212 and from 0.441 to 0.550 under predict-336 settings.The reported degradation is especially pronounced when the scaling factor is removed.

E Non-stationary Transformers: Experimental Details

The framework is evaluated with controlled Transformer comparisons, specified architectures, and explicit stationarization and de-stationary-attention procedures. It achieves state-of-the-art performance across five real-world forecasting applications.

  • Evaluation setup: The study compares Transformers with and without the framework using the same training strategy, with framework hyperparameters selected on validation data.The framework’s only hyperparameters come from the projector design that learns de-stationary factors.
  • Evaluation setup: The evaluated models use two encoder layers, one decoder layer, and feature dimension d_k=512, with Informer’s embedding and one-step generation strategy.The setup includes Transformer, Informer, Reformer, Autoformer, ETSformer, and FEDformer.
  • Framework procedures: Series Stationarization returns a stationarized input together with the original statistics, while de-normalization restores those statistics to the predicted series.The procedures operate on input past series and predicted output series, respectively.
  • Framework procedures: De-stationary Attention is implemented as a dedicated algorithm that produces the attention output used within the overall Non-stationary Transformers architecture.The overall architecture takes past time series, input length, prediction length, and variable count as requirements.
  • Results: The proposed method achieves state-of-the-art performance in five real-world applications by using attention to capture temporal dependencies associated with inherent non-stationarity.The paper positions this approach as going beyond methods that only stationarize the time series.

G Limitation

The framework’s De-stationary Attention is derived from vanilla Self-Attention, and its current scope does not cover advanced attention mechanisms or non-Transformer forecasting models.

  • Method scope: De-stationary Attention may not be optimal for advanced attention mechanisms because it is deduced from vanilla Self-Attention.The paper identifies improving this aspect as an open direction.
  • Model scope: The proposed framework is currently limited to Transformer-based models, although over-stationarization may also occur in other deep forecasting models.The paper calls for a more model-agnostic solution.
  • Stationarization scope: The framework has not yet re-incorporated non-stationarity for other classical stationarization methods such as differencing and quantile methods.The paper identifies these methods as another area for future work.
Loading 2205.14415v4…