PaperlayerView exact revision on arXiv

Source-linked AI summary

Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting

Haixu Wu, Jiehui Xu, Jianmin Wang, Mingsheng Long

arXiv:2106.13008v5cs.LGcs.AI

TL;DR

Long-term time-series forecasting must capture intricate future patterns while extending predictions far into the future. Autoformer addresses this with progressive decomposition and series-level Auto-Correlation, achieving a 38% averaged MSE reduction across reported multivariate settings.

  • Problem

    Long-term time-series forecasting requires reliable modeling of intricate future patterns for applications such as energy, traffic, economics, weather, and disease forecasting.

  • Method

    Autoformer embeds progressive series decomposition within a Transformer architecture and replaces point-wise self-attention with efficient series-level Auto-Correlation.

  • Results

    38% averaged MSE reduction was achieved across the reported multivariate long-term forecasting settings, with consistent state-of-the-art performance across benchmarks and prediction lengths.

  • Takeaways & Limitations

    Autoformer provides a long-term forecasting architecture with O(L log L) complexity and state-of-the-art performance on extensive real-world datasets.

Abstract

from arXiv · show

Extending the forecasting time is a critical demand for real applications, such as extreme weather early warning and long-term energy consumption planning. This paper studies the long-term forecasting problem of time series. Prior Transformer-based models adopt various self-attention mechanisms to discover the long-range dependencies. However, intricate temporal patterns of the long-term future prohibit the model from finding reliable dependencies. Also, Transformers have to adopt the sparse versions of point-wise self-attentions for long series efficiency, resulting in the information utilization bottleneck. Going beyond Transformers, we design Autoformer as a novel decomposition architecture with an Auto-Correlation mechanism. We break with the pre-processing convention of series decomposition and renovate it as a basic inner block of deep models. This design empowers Autoformer with progressive decomposition capacities for complex time series. Further, inspired by the stochastic process theory, we design the Auto-Correlation mechanism based on the series periodicity, which conducts the dependencies discovery and representation aggregation at the sub-series level. Auto-Correlation outperforms self-attention in both efficiency and accuracy. In long-term forecasting, Autoformer yields state-of-the-art accuracy, with a 38% relative improvement on six benchmarks, covering five practical applications: energy, traffic, economics, weather and disease. Code is available at this repository: \url{https://github.com/thuml/Autoformer}.

1 Introduction

The introduction motivates long-term time-series forecasting as important yet difficult because future dependencies are obscured by entangled patterns and self-attention is computationally prohibitive. Autoformer addresses these challenges through progressive decomposition and series-level Auto-Correlation.

  • Motivation: Long-term forecasting supports energy, traffic, economics, weather, and disease applications requiring far-future planning and early warning.The paper studies forecasts characterized by large predicted-series length.
  • Challenges: Long-term forecasting is challenging because entangled temporal patterns obscure reliable dependency discovery, while self-attention has quadratic sequence-length complexity.Canonical Transformers are therefore computationally prohibitive for long-term forecasting.
  • Motivation: Standard decomposition extracts predictable components but is restricted to preprocessing past series, limiting its capabilities and overlooking future interactions among components.The future is unknown during preprocessing, motivating decomposition within the forecasting model.
  • Autoformer: Autoformer replaces the Transformer with a decomposition forecasting architecture that embeds decomposition blocks as inner operators.It retains residual and encoder-decoder structures while progressively separating long-term trends from predicted hidden variables.
  • Autoformer: Autoformer introduces series-level Auto-Correlation for dependency discovery and information aggregation, extending beyond self-attention while improving computational efficiency and information utilization.The mechanism operates at the series level rather than through previous self-attention mechanisms.

2 Related Work

Related work spans classical statistical methods, recurrent neural networks, and Transformer-based forecasting models. Existing efficient Transformers sparsify point-wise attention, while Autoformer instead uses periodicity-based series-wise connections; decomposition has traditionally been applied as preprocessing.

  • Classical and neural forecasting: Classical forecasting methods include ARIMA, which differences non-stationary processes, filtering methods, and recurrent neural networks for temporal dependencies.These approaches represent established statistical and neural-network directions in time series forecasting.
  • Transformer-based forecasting: Transformers use self-attention for sequential data, but its memory and time costs scale quadratically with sequence length L.This computational burden makes vanilla self-attention prohibitive for long-term time series forecasting.
  • Transformer-based forecasting: LogTrans and Informer improve efficiency through sparse self-attention variants, with Informer achieving O(L log L) complexity.These methods retain the vanilla Transformer framework while modifying self-attention sparsity.
  • Transformer-based forecasting: Unlike sparse Transformers’ point-wise dependencies and aggregation, Auto-Correlation exploits inherent time-series periodicity to provide series-wise connections.The distinction is between point-wise attention mechanisms and dependencies discovered at the series level.
  • Time-series decomposition: Time-series decomposition separates signals into more predictable pattern components and is typically used as preprocessing before forecasting future series.Examples mentioned include Prophet’s trend-seasonality decomposition and N-BEATS.

3 Autoformer

Autoformer renovates the Transformer into a deep decomposition architecture that progressively separates temporal components and uses Auto-Correlation to discover and aggregate period-based sub-series dependencies. Its encoder emphasizes seasonal modeling, while its decoder refines seasonal predictions and accumulates trend-cyclical components.

  • Architecture: Autoformer renovates Transformer into a deep decomposition architecture with inner series decomposition, Auto-Correlation, and encoder-decoder components.The architecture addresses intricate temporal patterns and computation-efficiency and information-utilization bottlenecks in long-term forecasting.
  • Series decomposition: The series decomposition block separates trend-cyclical and seasonal parts as an inner model operation, enabling progressive decomposition despite unknown future values.The trend-cyclical component reflects long-term progression, while the seasonal component reflects series seasonality.
  • Encoder: The encoder focuses on seasonal modeling, retaining past seasonal information as cross information for decoder refinement.Encoder layers repeatedly apply series decomposition and Auto-Correlation, which replaces self-attention seamlessly.
  • Decoder: The decoder combines stacked Auto-Correlation for seasonal components with trend-cyclical accumulation, progressively extracting latent trend information during refinement.Each decoder layer includes inner and encoder-decoder Auto-Correlation to refine predictions and use past seasonal information.
  • Auto-Correlation: Auto-Correlation discovers period-based dependencies through series autocorrelation, rolls similar sub-series into aligned phase positions, and aggregates them using normalized confidences.This series-wise aggregation differs from the point-wise dot-product aggregation used in self-attention.

4 Experiments

Autoformer is evaluated across six real-world benchmarks and multiple long-term horizons, achieving consistent state-of-the-art results in multivariate and univariate forecasting. Ablations and analyses attribute these gains to progressive decomposition and Auto-Correlation, which improve dependency learning, interpretability, and efficiency.

  • Multivariate results: Autoformer achieves consistent state-of-the-art performance across all multivariate benchmarks and prediction lengths.For input-96-predict-336, it reduces MSE by 74% on ETT, 18% on Electricity, 61% on Exchange, 15% on Traffic, and 21% on Weather.
  • Univariate results: Autoformer remains state-of-the-art in univariate long-term forecasting, reducing MSE by 14% on ETT and 17% on Exchange under input-96-predict-336.The reductions are 0.180→0.145 on ETT and 0.611→0.508 on Exchange.
  • Decomposition architecture: Progressive decomposition consistently improves other models, particularly at longer prediction lengths, while outperforming pre-processing decomposition.Pre-decomposing can even hurt performance because it neglects interactions among components during the long-term future.
  • Time series decomposition: Progressive decomposition helps Autoformer capture increasing trends, seasonal peaks, and troughs by progressively refining trend-cyclical and seasonal components.Without decomposition, the model misses the increasing trend and seasonal peaks.
  • Complex seasonality modeling: Learned periodic lags capture real-world seasonalities, including monthly, quarterly, yearly, daily, and weekly cycles, enabling human-interpretable predictions.Traffic lags align with 24-hour and 168-hour intervals, matching daily and weekly periods.
  • Efficiency analysis: Autoformer has O(L log L) memory and time complexity and achieves better efficiency on long-term sequences than self-attention-based models.The comparison measures running memory and time during training.

5 Conclusions

The paper addresses long-term time-series forecasting by proposing Autoformer, which embeds progressive series decomposition within the model and introduces an efficient Auto-Correlation mechanism.

  • 5 Conclusions: Autoformer embeds series decomposition as an inner operator that progressively aggregates the long-term trend from intermediate predictions.This design targets intricate temporal patterns that hinder reliable dependency learning.
  • 5 Conclusions: The model also introduces an efficient Auto-Correlation mechanism for long-term time-series forecasting.

A Full Benchmark on the ETT Datasets … C.2 Past Information Utilization

Across ETT benchmarks, Autoformer substantially improves forecasting accuracy, while its hyper-parameter and input-design studies show that periodicity, input history, and decoder context govern the performance–efficiency trade-off. The model benefits from more decoder past information but uses a reduced context to balance memory cost.

  • A Full Benchmark on the ETT Datasets: The benchmark covers hourly ETTh1 and ETTh2 and 15-minute ETTm1 and ETTm2 datasets.Autoformer uses a fixed input length of 96 in the benchmark.
  • A Full Benchmark on the ETT Datasets: Autoformer reduces MSE by 55% on ETTh1, 80% on ETTh2, 40% on ETTm1, and 66% on ETTm2 versus previous best results.These improvements yield a 60% average MSE reduction over previous state-of-the-art across the four ETT datasets.
  • B Hyper-Parameter Sensitivity: The robust hyper-parameter c is selected between 1 and 3 to trade off Auto-Correlation performance and efficiency.Datasets with obvious periodicity, including ETT and Traffic, tend to favor larger c, whereas larger c can introduce noise on the non-periodic ILI dataset.
  • C.1 Input Length Selection: The relationship between input length and performance is dataset-specific, so model input should be selected according to data characteristics.For the periodic ETT dataset, input length 96 is sufficient to provide enough information.
  • C.2 Past Information Utilization: Autoformer attaches I_2 past information to the decoder placeholder to provide recent history to the decoder.The study evaluates decoder inputs with no, half, and full past information under input-96-predict-336 on ETT.
  • C.2 Past Information Utilization: More decoder past information improves performance but increases memory cost, so Autoformer uses decoder input I_2 + O to balance accuracy and efficiency.Here, I_2 + O, I + O, and O denote half, full, and no past information, respectively.

D Ablation of Decomposition Architecture

The ablation tests Autoformer’s progressive decomposition against separate prediction with established preprocessing decompositions. Progressive decomposition consistently performs better, particularly for long-term forecasting, despite the comparator using a model twice as large.

  • Architecture ablation: Progressive decomposition consistently outperforms separate prediction, especially in long-term forecasting.The comparison uses mature decomposition algorithms for the separate-prediction setting, yet the proposed architecture remains superior.
  • Architecture ablation: The separate-prediction baseline applies decomposition as preprocessing and forecasts seasonal and trend-cyclical parts separately.Its result is the sum of two separately forecast components, using two Transformers.
  • Architecture ablation: The ablation is conducted on the ETT dataset with input length 96 and prediction horizons of 96, 192, 336, or 720.The separate-prediction backbone is the canonical Transformer.

E Supplementary of Main Results · E.1 Multivariate Showcases

Autoformer achieves the best qualitative forecasting performance among the compared models on the ETT test set. Its predictions accurately capture both periodicity and long-term variation across multiple forecasting horizons.

  • E.1 Multivariate Showcases: Autoformer gives the best performance among the different models in qualitative comparisons of the ETT test set.The comparison plots the last forecasting dimension.
  • E.1 Multivariate Showcases: Autoformer accurately predicts periodicity and long-term variation in the ETT forecasting cases.This observation accompanies the qualitative comparison across Figures 8, 9, 10, and 11.
  • E.1 Multivariate Showcases: Under the input-96-predict-96 setting, Figure 8 compares model predictions with the ground truth.The first 96 points are the input, with blue denoting ground truth and orange denoting model prediction.
  • E.1 Multivariate Showcases: Under the input-96-predict-192 setting, Figure 9 presents prediction cases from the ETT dataset.The figure extends the forecast horizon to 192 points.
  • E.1 Multivariate Showcases: Under the input-96-predict-336 setting, Figure 10 presents prediction cases from the ETT dataset.The figure evaluates a 336-point forecasting horizon.
  • E.1 Multivariate Showcases: Under the input-96-predict-720 setting, Figure 11 presents prediction cases from the ETT dataset.The figure evaluates a 720-point forecasting horizon.

E.2 Performance on Data without Obvious Periodicity · E.3 Univariate Forecasting Showcases

Autoformer remains effective on aperiodic economic data, where rapid fluctuations make prediction difficult, and produces accurate, smooth forecasts in univariate showcases. Its predictions capture future-horizon periodicity and remain closer to ground truth than several baselines.

  • E.2 Performance on Data without Obvious Periodicity: Autoformer achieves the best performance among six datasets, including the aperiodic Exchange dataset.Exchange series fluctuate rapidly and lack obvious periodicity, making prediction extremely difficult.
  • E.2 Performance on Data without Obvious Periodicity: Autoformer’s strong Exchange-dataset performance demonstrates effectiveness despite the absence of obvious economic periodicity.
  • E.2 Performance on Data without Obvious Periodicity: The Exchange dataset showcase evaluates multivariate forecasts with a 96-step input and 192-step prediction horizon.
  • E.3 Univariate Forecasting Showcases: Autoformer gives the most accurate prediction in the univariate showcase.
  • E.3 Univariate Forecasting Showcases: Compared with Informer, Autoformer precisely captures periods in the future horizon.
  • E.3 Univariate Forecasting Showcases: Autoformer predicts the center area better than LogTrans and produces smoother, more ground-truth-like series than Reformer.
  • E.3 Univariate Forecasting Showcases: DeepAR’s predictions become increasingly fluctuating and over-smoothed as prediction length increases.
  • E.3 Univariate Forecasting Showcases: The ETT showcase uses a 96-step input and 720-step prediction horizon in a univariate setting.

E.4 Main Results with Standard Deviations · F COVID-19: Case Study

The experiments were repeated three times to assess robustness, with standard deviations reported separately in Table 10. The paper also evaluates Autoformer on normalized daily COVID-19 data from two anonymous European countries using chronological train/validation/test splits.

  • E.4 Main Results with Standard Deviations: Each experiment was repeated three times to obtain more robust results.The main text omits standard deviations because of page limits.
  • E.4 Main Results with Standard Deviations: Table 10 reports the experiments’ standard deviations.These fluctuations are presented for different prediction lengths in multivariate forecasting.
  • F COVID-19: Case Study: The COVID-19 case study uses daily confirmed-death and recovered-patient data collected from January 22, 2020, to May 20, 2021.The dataset covers multiple countries.
  • F COVID-19: Case Study: The study selects two anonymous European countries for the COVID-19 experiments.
  • F COVID-19: Case Study: The COVID-19 data are split chronologically into training, validation, and test sets with a 7:1:2 ratio.The data are also normalized.
  • F COVID-19: Case Study: Table 10 evaluates multivariate forecasting under different prediction lengths, using input length 36 for ILI and 96 for other datasets.Lower MSE or MAE indicates better performance.

F.1 Quantitative Results … G.1 Model Design

On COVID-19 forecasting with limited data and short inputs, Autoformer maintains state-of-the-art accuracy and captures long-term epidemic patterns. Its speedup implementation makes Auto-Correlation more memory-access friendly while preserving O(L log L) complexity.

  • F.1 Quantitative Results: Autoformer maintains state-of-the-art accuracy for COVID-19 forecasts with limited data and short inputs.It predicts the next week, half month, and full month using prediction lengths 1, 2.1, and 4.3 times the input length.
  • F.1 Quantitative Results: The COVID-19 evaluation uses an input length of 7 and prediction lengths of 7, 15, and 30, with lower MSE or MAE indicating better predictions.These horizons represent a week, half month, and month, respectively.
  • F.2 Showcases: Autoformer accurately predicts early peaks and troughs and nearly exact long-term values in COVID-19 forecasts.The paper emphasizes extreme-value and long-term-trend forecasting for epidemic prevention and control.
  • G.1 Model Design: The implementation provides pseudo-code for Autoformer and Auto-Correlation, along with tensor shapes and hyper-parameter settings.All experiments in the paper use the efficiency-oriented speedup version.
  • G.1 Model Design: The speedup version makes Auto-Correlation more memory-access friendly by adapting a batch-normalization-style procedure.It separates training and inference and reduces channel and head dimensions because deep-representation channels are equivalent under the linear layer.
  • G.1 Model Design: Auto-Correlation aggregates ⌊c × log L⌋ delayed length-L series with O(L log L) complexity in both standard and speedup versions.The speedup version is faster because its memory access is more friendly.

G.2 Experiment Details · H Broader Impact

The experiments standardize Transformer baselines under matched shallow architectures and generation settings, while the broader-impact discussion reports applications, academic contributions, robustness, and limitations. Autoformer is presented as useful across five forecasting domains, but potentially weaker when temporal coherence is absent.

  • G.2 Experiment Details: G.2 Experiment Details: All compared Transformer models use two encoder layers and one decoder layer for fair performance and efficiency comparisons.The models include Informer, Reformer, LogTrans, and canonical Transformer.
  • G.2 Experiment Details: G.2 Experiment Details: The compared models adopt Informer’s embedding method and one-step generation strategy, while Autoformer omits position embedding because series-wise aggregation supplies sequential information.
  • H Broader Impact: H Broader Impact: Autoformer achieves consistent state-of-the-art performance in five applications: energy, traffic, economics, weather, and disease.The paper also provides a COVID-19 dataset case study.
  • H Broader Impact: H Broader Impact: The work contributes a deep decomposition architecture with a novel Auto-Correlation mechanism inspired by classic time series analysis and stochastic process theory.Code is available at the Autoformer repository.
  • H Broader Impact: H Broader Impact: Extensive experiments find no exceptional failure cases, and Autoformer remains robust on the Exchange dataset despite its lack of obvious periodicity.Progressive inner decomposition can expose deeply hidden periodicity.
  • H Broader Impact: H Broader Impact: Autoformer and other models may degenerate on random data or data with extremely weak temporal coherence.
  • H Broader Impact: H Broader Impact: The authors state that the work focuses on a scientific problem and therefore presents no potential ethical risk.
Loading 2106.13008v5…