Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting

Haoyi Zhou; Shanghang Zhang; Jieqi Peng; Shuai Zhang; Jianxin Li; Hui Xiong; Wancai Zhang

Source-linked AI summary

Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, Wancai Zhang

arXiv:2012.07436v3Dec 14, 2020cs.LGcs.AIcs.IR

TL;DR

LSTF requires long-range dependency modeling while avoiding Transformer’s quadratic computation and memory costs and stepwise decoding limitations. Informer combines ProbSparse attention, self-attention distilling, and a generative decoder to address these issues. Experiments on real-world data demonstrate improved prediction capacity and effectiveness for LSTF.

Problem
LSTF requires precise long-range dependency alignment, but Transformer faces quadratic attention costs, stacked-layer memory bottlenecks, and slow dynamic decoding for long outputs.
Method
Informer combines ProbSparse self-attention, self-attention distilling, and a generative-style decoder for efficient long-sequence forecasting.
Results
Informer significantly outperforms existing methods and demonstrates effectiveness for enhancing prediction capacity in LSTF.
Takeaways & Limitations
Informer provides a solution for predicting long time-series sequences while addressing efficiency and encoder-decoder limitations.
Takeaways & Limitations
The paper reports reduced multivariate performance relative to univariate results, attributing the phenomenon to anisotropy in feature-dimension prediction capacity and leaving it for future work.

Abstract

from arXiv · show

Many real-world applications require the prediction of long sequence time-series, such as electricity consumption planning. Long sequence time-series forecasting (LSTF) demands a high prediction capacity of the model, which is the ability to capture precise long-range dependency coupling between output and input efficiently. Recent studies have shown the potential of Transformer to increase the prediction capacity. However, there are several severe issues with Transformer that prevent it from being directly applicable to LSTF, including quadratic time complexity, high memory usage, and inherent limitation of the encoder-decoder architecture. To address these issues, we design an efficient transformer-based model for LSTF, named Informer, with three distinctive characteristics: (i) a $ProbSparse$ self-attention mechanism, which achieves $O(L \log L)$ in time complexity and memory usage, and has comparable performance on sequences' dependency alignment. (ii) the self-attention distilling highlights dominating attention by halving cascading layer input, and efficiently handles extreme long input sequences. (iii) the generative style decoder, while conceptually simple, predicts the long time-series sequences at one forward operation rather than a step-by-step way, which drastically improves the inference speed of long-sequence predictions. Extensive experiments on four large-scale datasets demonstrate that Informer significantly outperforms existing methods and provides a new solution to the LSTF problem.

1 Introduction

Long-sequence time-series forecasting matters across real-world domains but strains methods built for short horizons. Informer addresses Transformer’s efficiency and architecture limitations with sparse attention, distilling, and generative decoding.

Motivation: LSTF supports long-horizon forecasting in domains including sensor monitoring, energy management, finance, and disease propagation analysis.Existing methods are mostly designed for short-term settings such as predicting 48 points or fewer.
Motivation: 480 points over 20 days exposes a substantial performance gap beyond 48 points, where LSTM MSE becomes unsatisfactory and inference speed drops sharply.The example forecasts hourly transformer-station temperature from 12 points to 480 points.
Challenges: Transformer self-attention offers short signal paths for long-range dependencies but requires O(L^2) computation and memory, violating efficient long-sequence processing requirements.The LSTF challenge combines long-range alignment with efficient operations on long inputs and outputs.
Contribution: The paper investigates self-attention sparsity and redesigned network components to improve Transformer efficiency and prediction capacity for LSTF.These investigations are presented as the paper’s response to the three identified Transformer limitations.
Approach: Informer replaces canonical self-attention with ProbSparse attention, uses distilling to reduce network size, and decodes long outputs in a generative style.The overview describes massive long inputs, attention distilling, robust layer replicas, zero-padded targets, and one-operation output prediction.

2 Preliminary

The preliminary section defines LSTF as rolling-window sequence prediction with longer outputs and potentially multivariate features. It also describes the encoder-decoder setup and its step-by-step dynamic decoding process.

LSTF Definition: Under rolling forecasting with a fixed-size window, LSTF maps an input sequence X_t to a corresponding output sequence Y_t.The supplied definition introduces input and output sequences at time t.
LSTF Definition: LSTF emphasizes longer output lengths than prior sequence-prediction work and allows feature dimension d_y ≥ 1 rather than restricting the task to univariate series.The formulation distinguishes the longer target horizon from earlier work and permits multiple output features.
Encoder-Decoder Architecture: In the encoder-decoder architecture, the encoder maps input representations X_t to hidden states H_t, and the decoder generates output representations Y_t from H_t.The hidden representation is a sequence of states indexed over its length.
Encoder-Decoder Architecture: Dynamic decoding predicts sequentially: each decoder state h_(k+1) depends on the previous state h_k and outputs from the kth step.The process then predicts the (k + 1)th sequence output.
Input Representation: The input representation combines global positional context and local temporal context to strengthen representation of time-series inputs.The supplied passage notes that implementation details are deferred to an appendix.

3 Methodology

Informer’s methodology combines sparse attention selection, encoder distilling, and generative decoding to address long-sequence forecasting constraints. Its encoder uses ProbSparse attention and progressively downsampled feature maps, while its decoder avoids autoregressive inference.

Efficient Self-attention Mechanism: The sparsity measurement identifies queries whose attention distributions are more diverse and therefore more likely to contain dominant dot-product pairs.
Efficient Self-attention Mechanism: ProbSparse self-attention selects the Top-u dominant queries using a sparsity measurement instead of computing canonical attention for every query.The method sets u = c · ln LQ and uses sparse query-key pairs for each attention head.
Efficient Self-attention Mechanism: When query and key lengths are both L, ProbSparse self-attention has O(L ln L) time and space complexity.
Encoder: Self-attention distilling privileges dominant features, applies convolution and ELU operations, then max-pools with stride 2 to halve the sequence dimension after each layer.The resulting total memory usage is stated as O((2 −ϵ)L log L).
Encoder: The encoder builds replicated stacks with progressively halved inputs, decreases distilling layers across stacks, and concatenates their outputs into the final hidden representation.The main stack receives the whole input sequence, while subsequent stacks receive half slices and repeat the pattern.
Decoder: The decoder uses generative inference with a target-length zero placeholder, masked ProbSparse attention, and a fully connected layer for the final output.Masking sets future dot products to −∞, preventing each position from attending to coming positions and avoiding autoregressive decoding.

4 Experiment

Experiments evaluate Informer on four datasets across univariate and multivariate LSTF settings, progressively increasing prediction horizons. Informer generally outperforms competing methods, while ablations and runtime studies support the contributions of ProbSparse attention, distilling, and generative decoding.

Experimental Setup: Experiments use four datasets and progressively longer prediction horizons for univariate and multivariate LSTF evaluation.The datasets include ETT, ECL, and Weather; evaluation covers multiple forecasting cases and uses a single Nvidia V100 GPU.
Univariate Forecasting: Informer significantly improves inference performance across all datasets, with prediction error rising smoothly and slowly as the horizon grows.In univariate forecasting, Informer also exceeds its canonical self-attention variant in winning counts, 32>12.
Univariate Forecasting: 26.8%, 52.4%, and 60.1% MSE decreases at horizons 168, 336, and 720, respectively, compared with LSTMa.Informer also outperforms ARIMA, Prophet, and DeepAR, with average MSE decreases of 49.3%, 61.1%, and 65.1% at those horizons.
Multivariate Forecasting: 26.6%, 28.2%, and 34.3% average MSE decreases at horizons 168, 336, and 720, respectively, compared with LSTMa and LSTnet in multivariate forecasting.The paper reports that multivariate gains are smaller than univariate gains and attributes this to anisotropy of feature dimensions’ prediction capacity.
Granularity Consideration: Informer outperforms other baselines across different granularity levels when aligned ETTm1 minute-level and ETTh1 hour-level sequences are compared.The compared sequence lengths are {96, 288, 672} for ETTm1 and {24, 48, 168} for ETTh1.
Sensitivity and Ablations: The selected layer combination joining L and L/4 is the most robust strategy, while ProbSparse performance increases slightly with sampling factor and then stabilizes.The study sets the sampling factor c = 5 in practice; removing distilling reaches OOM with inputs longer than 720, and generative decoding remains robust as offsets increase.
Computation Efficiency: Informer achieves the best training efficiency among Transformer-based methods and is much faster during testing because of generative style decoding.The reported runtime behavior aligns with the theoretical time-complexity and memory-usage comparisons.

5 Conclusion

The paper proposes Informer for long-sequence time-series forecasting, combining efficient attention, distilling, and a generative decoder to address vanilla Transformer limitations.

Informer uses ProbSparse self-attention and distilling to address vanilla Transformer’s quadratic time complexity and memory usage.
Its generative decoder alleviates the traditional encoder-decoder architecture’s limitation for long-sequence prediction.
Experiments on real-world data demonstrate Informer’s effectiveness for enhancing prediction capacity in LSTF.

Appendices

The appendices review forecasting and attention methods, define the long-sequence input problem, and describe Informer’s time-aware input representation.

Time-series Forecasting: Existing time-series forecasting methods include classical models and deep learning-based methods.
Long sequence input problem: The long sequence time-series input problem arises when methods truncate, summarize, or sample very long inputs, potentially losing valuable data.
Long sequence input problem: LSTF differs from long-input methods because it requires modeling long-range dependencies between outputs and inputs.
Transformer-based time-series model: Transformer-based time-series models had difficulty with LSTF when using vanilla Transformer architectures.
Input representation: Informer’s input representation combines scalar projection, local positional information, and global time-stamp embeddings.

Appendix C The long tail distribution in self-attention feature map

A canonical Transformer’s self-attention scores on ETTh1 exhibit a long-tail distribution in which a few query-key pairs dominate attention.

The attention scores form a long-tail distribution, with a few dot-product pairs contributing most of the attention.
The remaining dot-product pairs can be ignored under the observed attention distribution.
A 4-layer canonical Transformer trained on ETTh1 was used to inspect the self-attention feature-map distribution.

Appendix D Details of the proof

The proof analyzes the query-sparsity measurement by bounding its behavior through convexity and the largest query-key inner product.

The proof relaxes discrete keys into continuous d-dimensional vectors to analyze the query-sparsity measurement M(q_i, K).
For a fixed query, the log-sum-exp component is convex, making the measurement convex in the key vectors.
The measurement reaches its minimum ln L_K when all key vectors are equal.
Selecting the largest query-key inner product provides the upper-bound term used in the proof.
Combining the derived inequalities establishes Lemma 1, including the case where the key and query sets coincide.

Proof of Proposition 1

The proof reduces the proposition to comparing sums and differences of lognormal variables. Under conditions on centered maxima and variances, the ordering holds with high probability, with an approximation supported by a top-variable simulation condition.

Reformulation: The proposition is reformulated using centered attention-score arrays, where max(A1) − mean(A1) and Var(A1) determine the comparison with A2.This yields the equivalent conclusion ES(A1) > ES(A2) with high probability when both inequalities hold.
Probability comparison: The approximate normal distribution links larger variance differences to a higher probability that ES(A1) exceeds ES(A2).The argument notes positive correlation between the probability and Var(A1) − Var(A2), while the associated mean–variance correlation is not significant in the illustrative experiment.
Proposition conditions: When both queries lie near the maximum M(qi, K), the centered-maximum and variance conditions imply M(q1, K) > M(q2, K) with high probability.The same ordering is expressed equivalently as ES(A1) > ES(A2).
Lognormal approximation: The proof models the relevant quantities through Gaussian increments and reduces the analysis to approximating sums and differences of lognormal variables.The cited approximation lacks a general closed-form PDF for sums of lognormals but is treated as accurate in typical cases.
Approximation boundary: The approximation requires σ̃^2_±(t − t0) to be sufficiently small, and the difference of lognormals remains difficult to solve exactly.The illustrative simulation sets t − t0 = 1 without loss of generality.
Top-variable selection: Selecting approximately the top ⌊1/4Lk⌋ variables preserves the relevant variation while increasing expectations, yielding over 99% probability for the ordering in the specified interval.This is the simulation condition used for the actual experiment and is illustrated by the black curve in Figure 8.

Details of the experiments

The experiments specify Informer’s network components, preprocessing, search spaces, and decoding procedures across several long-sequence forecasting datasets. Informer and LogTrans use non-dynamic decoding, while recurrent baselines decode dynamically.

Model configuration: Informer’s detailed network configuration is summarized in Table 7, including ProbSparse self-attention and standard residual, feed-forward, and dropout components.The implementation uses d=32, n=16, a feed-forward inner dimension of 2048, and dropout p = 0.1.
Evaluation protocol: Experiments preserve 10% validation data, average results over five random train/validation time shifts, and standardize every dataset to mean 0 and standard deviation 1.These procedures are applied across the reported experiments.
Search spaces: Input-length candidates range from 24 to 720 for ETTh1, ETTh2, Weather, and Electricity, and from 24 to 672 for ETTm.The exact candidate sets differ between the ETTm and other datasets.
Visual comparison: Figure 9 compares prediction and ground-truth slices for eight models on the ETTm dataset using red prediction curves and blue ground-truth curves.The figure evaluates Informer, Informer†, LogTrans, Reformer, DeepAR, LSTMa, ARIMA, and Prophet.
Search spaces: Informer uses a searched encoder depth with decoder depth fixed at two layers, while recurrent baselines tune hidden-state dimensions and LSTnet-specific recurrent, convolutional, and skip settings.Informer’s encoder depth is selected from {6, 4, 3, 2}.
Decoding: RNN-based methods dynamically decode prediction windows with left shifting, whereas Informer-series and LogTrans use non-dynamic decoding.The distinction concerns how prediction windows are generated during inference.

Appendix F Extra experimental results

The extra experimental results compare eight forecasting models on ETTm prediction slices. Informer and Informer† are reported to produce the strongest visual results among the compared methods.

Extra results: Informer and Informer† show significantly better results than the other models in the ETTm prediction slice comparison.LogTrans and Reformer are described as acceptable, while other baselines capture trends or changing points with different strengths.
Experimental environment: The experiments run on Nvidia Tesla V100 SXM2 GPUs with 32GB memory and a dual Xeon Gold 6148 CPU system with 384GB RAM.The hardware configuration also includes two 240GB M.2 SSDs.