PaperlayerView exact revision on arXiv

Source-linked AI summary

ETSformer: Exponential Smoothing Transformers for Time-series Forecasting

Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, Steven Hoi

arXiv:2202.01381v2cs.LG

TL;DR

Traditional Transformers do not fully exploit time-series structure, limiting decomposition, interpretability, and long-term forecasting effectiveness and efficiency. ETSformer redesigns the architecture with exponential-smoothing-inspired decomposition, Exponential Smoothing Attention, and Frequency Attention. It achieves state-of-the-art benchmark performance, ranking best on most multivariate and univariate settings.

  • Problem

    Traditional Transformers lack decomposition and interpretability and are not sufficiently effective or efficient for long-term time-series forecasting.

  • Method

    ETSformer decomposes representations into level, growth, and seasonal components and replaces vanilla self-attention with ESA and FA tailored to recency and seasonal patterns.

  • Results

    ETSformer achieves the best MSE on 35 of 40 multivariate and 17 of 23 univariate benchmark settings.

  • Takeaways & Limitations

    The architecture provides interpretable component forecasts while maintaining O(L log L) complexity for its proposed attention mechanisms.

  • Takeaways & Limitations

    A straightforward ESA implementation constructs an attention matrix and has O(L^2) computational complexity.

Abstract

from arXiv · show

Transformers have been actively studied for time-series forecasting in recent years. While often showing promising results in various scenarios, traditional Transformers are not designed to fully exploit the characteristics of time-series data and thus suffer some fundamental limitations, e.g., they generally lack of decomposition capability and interpretability, and are neither effective nor efficient for long-term forecasting. In this paper, we propose ETSFormer, a novel time-series Transformer architecture, which exploits the principle of exponential smoothing in improving Transformers for time-series forecasting. In particular, inspired by the classical exponential smoothing methods in time-series forecasting, we propose the novel exponential smoothing attention (ESA) and frequency attention (FA) to replace the self-attention mechanism in vanilla Transformers, thus improving both accuracy and efficiency. Based on these, we redesign the Transformer architecture with modular decomposition blocks such that it can learn to decompose the time-series data into interpretable time-series components such as level, growth and seasonality. Extensive experiments on various time-series benchmarks validate the efficacy and advantages of the proposed method. Code is available at https://github.com/salesforce/ETSformer.

1 Introduction

Transformer-based forecasting faces difficulty extracting salient temporal patterns from noisy, non-stationary series, while vanilla attention is poorly matched to recency and seasonality. ETSformer addresses these issues with decomposition and time-series-specific attention mechanisms.

  • Noisy, non-stationary time series make salient temporal-pattern extraction and accurate long-term forecasting challenging for Transformer models.
  • Content-based dot-product attention does not explicitly prioritize recent observations, even though temporal dependence generally weakens with time.
  • Vanilla attention lacks an in-built prior for extracting fixed-period seasonal patterns that are important for forecasting.
  • ETSformer performs layer-wise level, growth, and seasonal decomposition while using residual learning to model latent temporal patterns and their dependencies.
  • Its Exponential Smoothing Attention uses relative time lag, while Frequency Attention selects dominant Fourier bases; both achieve O(L log L) complexity.

2 Related Work

Prior work applies Transformers and alternative attention or mixing mechanisms to time-series forecasting, often targeting sparse structure or lower computational complexity. ETSformer instead emphasizes time-series characteristics through exponential-smoothing-inspired decomposition and attention.

  • Transformer-based forecasting methods adapt the architecture with local context, sparse attention, distillation, or sparse normalization to address time-series forecasting.
  • Alternative attention studies show that learned, local, fixed positional, or Fourier-based mixing can compete with vanilla self-attention in other sequence tasks.
  • ETSformer’s ESA and FA avoid pairwise query-key dependence while incorporating recency and seasonal-pattern structure specific to time-series data.

3 Preliminaries and Background

Classical exponential smoothing decomposes a series into seasonal and trend components, with trend further divided into level and growth. Holt-Winters forecasts combine these components, optionally damping growth for more robust multi-step prediction.

  • The forecasting task maps a multivariate lookback window of length L to future values over a horizon H by minimizing a loss function.
  • Additive Holt-Winters decomposes a time series into seasonal and trend components, with trend further divided into level and growth.
  • The Holt-Winters forecast combines the latest estimated level, horizon-scaled growth, and a seasonal factor.
  • Level, growth, and seasonal estimates are updated through weighted averages controlled by smoothing parameters α, β, and γ.
  • Damped-trend forecasting scales accumulated growth by φ, recovering the vanilla forecast when φ = 1 and approaching an asymptote when 0 < φ < 1.

4 ETSformer

ETSformer redesigns the Transformer for time-series forecasting around exponential smoothing, frequency-based seasonal extraction, and interpretable level, growth, and seasonal decomposition. Its encoder-decoder architecture combines these components into efficient multi-step forecasts.

  • Architecture: ETSformer progressively extracts level, growth, and seasonal representations through stacked encoder layers and composes them in the final forecast.Residual learning supports cascaded extraction of latent growth and seasonal patterns before level smoothing and decoder composition.
  • Input embedding: ETSformer maps raw lookback signals into latent space with a temporal convolution and does not require manually designed dynamic time-dependent covariates.The paper attributes this reduced feature-engineering requirement to Frequency Attention's ability to uncover seasonal patterns automatically.
  • Decoder: The decoder generates H-step forecasts from level, growth, and seasonal components using a Level Stack and N Growth + Seasonal stacks.The Level Stack repeats the final lookback level, while growth damping supports robust multi-step trend forecasts.
  • Attention mechanisms: ESA assigns stronger weights to recent observations using learnable exponential decay based on relative time lag rather than input content.The mechanism is non-adaptive with respect to input content, while its smoothing parameter and initial state are learnable.
  • Attention mechanisms: FA uses Fourier transformation to select dominant seasonal patterns and extrapolates the resulting seasonality representation across the forecast horizon.An inverse DFT returns the selected seasonal pattern to the time domain, with complexity reported as O(L log L) for small K.
  • Efficiency: A straightforward ESA implementation has O(L^2) complexity, whereas the paper describes an efficient algorithm exploiting the structured smoothing-attention matrix.The efficient implementation is presented as a way to avoid the quadratic cost of explicitly constructing the attention matrix.

5 Experiments

ETSformer is evaluated on six real-world datasets through benchmark, ablation, and interpretability experiments. It achieves strong forecasting performance while producing decomposed components that track interpretable patterns.

  • Experimental Setup: The evaluation covers six real-world datasets across multivariate and univariate long-sequence forecasting settings, with benchmark, ablation, and interpretability studies.Computational-efficiency analysis is reported separately in Appendix H.
  • Results: 35 out of 40 multivariate and 17 out of 23 univariate settings achieve the best MSE among the evaluated methods.ETSformer is also within the top two on all 40 multivariate settings and 21 of 23 univariate settings.
  • Results: On Exchange, ETSformer improves average MSE over forecast horizons by 39.8% versus the best baseline, despite the dataset lacking obvious periodic patterns.The result is presented as evidence of strong trend forecasting capabilities.
  • Ablation Study: Removing level, growth, or seasonality worsens forecasts, while replacing multi-head exponential smoothing attention with vanilla multi-head attention is less effective.The ablation identifies level estimation as particularly critical.
  • Interpretability: On synthetic data, ETSformer’s decomposed level, trend, and seasonal forecasts closely track ground-truth patterns, whereas Autoformer struggles to separate trend from seasonality.The interpretability experiment uses synthetic data because real-world ground-truth decompositions are unavailable.

6 Conclusion

ETSformer combines exponential-smoothing-inspired attention with latent level, growth, and seasonal representations for time-series forecasting. The architecture achieves state-of-the-art benchmark performance with O(L log L) complexity.

  • Conclusion: ETSformer learns level, growth, and seasonal latent representations and their complex dependencies in a Transformer-based forecasting architecture.Its design is inspired by classical exponential smoothing methods and Transformer approaches.
  • Conclusion: Exponential Smoothing Attention and Frequency Attention replace vanilla self-attention to model time-series structure more effectively and efficiently.The paper presents these mechanisms as the core attention innovations.
  • Conclusion: ETSformer achieves the best MSE in 35 of 40 multivariate and 17 of 23 univariate forecasting settings.These are the reported benchmark totals across datasets and settings.
  • Conclusion: The efficient implementation of exponential smoothing attention uses FFT-based convolution to achieve O(L log L) complexity for lookback-window length L.The lower-triangular smoothing matrix enables convolution-based multiplication, including the initial-state contribution.

A.4 Further Details on ESA Implementation

The ESA implementation accelerates exponential smoothing by exploiting the structure of its smoothing matrix. FFT-based convolution replaces naive matrix multiplication while the DFT represents signals in the frequency domain.

  • ESA Implementation: The naive ESA implementation constructs the exponential smoothing attention matrix AES and performs full matrix-vector multiplication.This approach is contrasted with the efficient implementation.
  • ESA Implementation: Algorithm 3 computes the convolution efficiently with FFT operations, reducing the complexity to O(L log L).The implementation uses zero-padding and real-input forward and inverse Fourier transforms.
  • ESA Implementation: Because AES is lower triangular apart from its first column, its matrix-vector multiplication can be expressed as convolution.This structural equivalence enables the efficient implementation.
  • Discrete Fourier Transform: The DFT maps a real-valued length-N signal to Fourier coefficients in the frequency domain, retaining the first floor(N/2)+1 coefficients by conjugate symmetry.The inverse DFT maps the frequency representation back to the time domain.

C Implementation Details

ETSformer uses shared training settings with grid search over selected forecasting hyperparameters. Training also applies data augmentation, dropout, Adam optimization, and scheduled learning rates.

  • Hyperparameters: The hyperparameter search varies the number of frequencies K, lookback-window size, and learning rate, selecting the best validation MSE averaged over three runs.The lookback-window search range is tied to the forecast horizon for each dataset.
  • Hyperparameters: The listed hyperparameters include model dimensions, encoder and decoder configuration, attention heads, kernel size, frequencies, lookback window, and learning rate.Table 3 is identified as the source of the ETSformer hyperparameter values.
  • Optimization: Training uses Adam with β1 = 0.9, β2 = 0.999, ϵ = 1e-08, batch size 32, and 15 total epochs on an Nvidia A100 GPU.Learning rate warmup lasts three epochs and is followed by cosine annealing.
  • Regularization: Scale, shift, and jitter augmentations are applied in that order, each activating with probability 0.5.Scale and shift use a random scalar, while jitter adds independent Gaussian noise at each time step.
  • Regularization: Dropout with rate p = 0.2 is applied across embeddings, attention blocks, feedforward layers, attention weights, and damping weights.The model applies dropout at multiple points throughout the architecture.

D Datasets

The benchmark datasets cover electricity, exchange rates, traffic, weather, and influenza-like illness across different sampling frequencies and domains.

  • ETTm records transformer load and oil temperature every 15 minutes, with ETTh providing an hourly-aggregated variant.
  • ECL contains hourly electricity consumption from 321 household clients over two years, originally collected at 15-minute intervals.
  • Exchange tracks daily exchange rates for eight countries from 1990 to 2016.
  • Traffic is an hourly dataset of road occupancy rates on San Francisco Bay Area freeways.
  • Weather measures 21 meteorological indicators every 10 minutes throughout 2020.
  • ILI records the weekly ratio of influenza-like illness patients to total patients in the United States from 2002 to 2021.

E Synthetic Dataset

The synthetic dataset combines a nonlinear saturating trend with complex periodic seasonality for 192-step lookback windows and 48-step forecasts.

  • Each synthetic instance uses a lookback window of 192 and a forecast horizon of 48.
  • The trend follows a nonlinear saturating pattern with β0 = −0.2 and β1 = 192.
  • Seasonality is formed by summing sinusoids with frequencies f1 = 1/10 and f2 = 1/13, each having amplitude 0.15.

F Univariate Forecasting Benchmark

The univariate benchmark reports forecasting results across horizons, while the computational-efficiency analysis distinguishes multivariate and univariate settings and evaluates scaling with lookback and forecast horizon.

  • Univariate benchmark: Univariate forecasting results are reported over various forecast horizons, with best and second-best results visually distinguished.
  • Computational efficiency: Figure 5 includes separate multivariate and univariate benchmark views for computational-efficiency analysis.
  • Computational efficiency: With horizon fixed at 48, lookback-window plots are evaluated; with lookback fixed at 48, forecast-horizon plots are evaluated.
  • Computational efficiency: ETSformer maintains competitive efficiency with quasilinear-complexity Transformers and superior efficiency as forecast horizon increases.
Loading 2202.01381v2…