PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Pattern Attention for Multivariate Time Series Forecasting

Shun-Yao Shih, Fan-Keng Sun, Hung-yi Lee

arXiv:1809.04206v3cs.LGcs.CLstat.ML

TL;DR

Multivariate time-series forecasting must model complex dependencies across variables and time, while typical attention can miss temporal patterns spanning multiple steps. The paper uses CNN filters to extract such patterns and attention to select relevant series, achieving state-of-the-art results across multiple real-world tasks with one exception.

  • Problem

    MTS forecasting involves complex interdependencies across time steps and series, while typical attention focuses on individual time steps and misses multi-step temporal patterns.

  • Method

    Temporal pattern attention applies CNN filters to hidden-state rows to extract temporal patterns and learns attention weights that select relevant time series.

  • Results

    The proposed model achieves state-of-the-art results across multiple real-world datasets, with one exception.

  • Takeaways & Limitations

    The approach learns interdependencies among multiple variables across previous times and series, while filter visualizations provide interpretable evidence of the learned patterns.

  • Takeaways & Limitations

    The forecasting setup assumes that observations before a fixed input window contain no useful information for prediction.

Abstract

from arXiv · show

Forecasting multivariate time series data, such as prediction of electricity consumption, solar power production, and polyphonic piano pieces, has numerous valuable applications. However, complex and non-linear interdependencies between time steps and series complicate the task. To obtain accurate prediction, it is crucial to model long-term dependency in time series data, which can be achieved to some good extent by recurrent neural network (RNN) with attention mechanism. Typical attention mechanism reviews the information at each previous time step and selects the relevant information to help generate the outputs, but it fails to capture the temporal patterns across multiple time steps. In this paper, we propose to use a set of filters to extract time-invariant temporal patterns, which is similar to transforming time series data into its "frequency domain". Then we proposed a novel attention mechanism to select relevant time series, and use its "frequency domain" information for forecasting. We applied the proposed model on several real-world tasks and achieved state-of-the-art performance in all of them with only one exception.

1 Introduction

Multivariate time-series forecasting requires modeling complex dependencies across variables and long time spans. The paper proposes temporal pattern attention, which extracts multi-step patterns and selects relevant series, achieving strong results across diverse datasets.

  • Multivariate time-series forecasting involves complex dynamic interdependencies among variables observed over discrete time steps.
  • RNNs are commonly used for MTS forecasting, but they struggle with long-term dependencies such as yearly patterns in daily sequences.
  • Temporal pattern attention selects relevant time series rather than only relevant time steps, targeting time-invariant patterns across multiple steps.
  • A CNN extracts temporal-pattern information from each individual variable, addressing periodic patterns that typical attention may miss.
  • Toy examples show the mechanism can extract temporal patterns and focus on different time steps for different time series.
  • The proposed attention mechanism achieves state-of-the-art results across multiple real-world datasets, including periodic, partially linear, non-periodic, and non-linear tasks.

2 Related Work

Prior forecasting methods range from classical linear models to deep neural networks, but many have limited multivariate or long-term modeling capacity. LSTNet addresses large-scale MTS forecasting, while the proposed mechanism learns patterns without manually specified periodicity.

  • ARIMA, SVR, and VAR primarily address linear forecasting, while many earlier approaches target univariate or bivariate series and do not capture broad nonlinear MTS dependencies.
  • LSTNet was designed for MTS forecasting with up to hundreds of time series and combines CNNs, recurrent modules, attention, and autoregression.
  • LSTNet’s recurrent-skip layer requires manually tuning its skip length to match the data period, whereas the proposed approach learns periodic patterns automatically.
  • The proposed attention mechanism selects relevant time series, unlike typical attention in LSTNet-Attn, which selects relevant hidden states.

3 Preliminaries

The preliminaries describe recurrent hidden states and typical attention for extracting context from prior states. Typical attention scores previous states against the current state and combines them into a context vector for prediction.

  • Recurrent Neural Networks: An RNN applies a recurrent function to an input sequence and computes a hidden state h_t at each time step.
  • Recurrent Neural Networks: LSTM cells are a widely used RNN variant with recurrent equations involving input, forget, output, and cell-state variables.
  • Typical Attention Mechanism: Typical attention extracts a context vector from previous hidden states as a weighted sum of information relevant to the current time step.
  • Typical Attention Mechanism: A scoring function measures relevance between each previous hidden state and the current hidden state to determine attention weights.

4 Temporal Pattern Attention

Typical attention can miss useful variables and temporal patterns in multivariate forecasting because it averages hidden-state information across time. Temporal pattern attention instead applies CNN filters to RNN hidden-state rows, selects relevant variables, and forms a context vector containing multi-step temporal information.

  • Typical attention selects information relevant to the current step but can retain noisy variables and miss temporal patterns spanning multiple steps.
  • The proposed mechanism attends to row vectors of previous RNN hidden states, allowing attention weights to select variables useful for forecasting.
  • CNN filters are applied to hidden-state row vectors to extract signal patterns across multiple time steps before attention scoring.
  • The attention weight α_i is computed with a sigmoid scoring function based on the filtered row representation and the current hidden state.
  • Sigmoid rather than softmax permits multiple variables to receive useful forecasting weight simultaneously.
  • The context vector v_t is a weighted sum of filtered hidden-state rows and is integrated with h_t to produce the final prediction.

5 Analysis of Proposed Attention on Toy Examples

Toy examples isolate the effects of temporal patterns and interdependencies on attention mechanisms. The proposed CNN-based attention is designed to select information across different time lags and exploit interdependencies between series.

  • The analysis compares attention mechanisms on two artificially constructed toy-example types to examine traditional-attention failure and interdependency effects.
  • The first toy type contains mutually independent sine-wave series with different periods.
  • The second toy type introduces interdependencies by mixing the time series, with D denoting the number of series.
  • The experiments use window size w = 64, horizon ∆ = 1, 200 training epochs, and toy-example dimensions including D = 1 and D = 6.
  • Traditional attention can fail because different series require looking back different numbers of time steps, whereas the proposed attention selects CNN-extracted features across multiple steps.
  • When D ≫ 1, LSTM with Luong attention performs poorly, while the larger hidden size of attention-free LSTM helps it outperform that baseline.
  • With interdependencies, proposed-attention loss decreases while Luong-attention and attention-free LSTM losses remain unchanged, indicating effective use of interdependencies.

6 Experiments and Analysis

The experiments evaluate the proposed attention on typical multivariate time-series and polyphonic-music datasets. These datasets span continuous and binary observations, real interdependencies, periodic patterns, and more complex non-repetitive sequences.

  • The evaluation uses two dissimilar dataset groups: typical multivariate time series and polyphonic music.
  • Datasets: Typical datasets include Solar Energy, Traffic, Electricity, and Exchange Rate, covering solar production, road occupancy, client electricity consumption, and international exchange rates.
  • Datasets: These real-world datasets contain linear and nonlinear interdependencies, while Solar Energy, Traffic, and Electricity exhibit strong daily or weekly periodic patterns.
  • Datasets: Polyphonic music datasets are described as more complicated because they lack apparent linearity and repetitive patterns.
  • Music datasets: Music forecasting represents played notes as 1 and rests as 0, using four bars of 16 beats to predict pitch activation at the next time step.
  • Dataset statistics: MuseData and LPD-5-Cleansed contain variable-length time series because the lengths of their musical pieces vary.

6.2 Methods for Comparison

The study compares the proposed model with traditional forecasting methods, deep MTS baselines, and LSTM-based baselines suited to polyphonic music. Experimental settings standardize model capacity and specify dataset-specific optimization choices.

  • Typical MTS: Typical MTS comparisons include AR, LRidge, LSVR, GP, SETAR, LSTNet-Skip, and LSTNet-Attn.
  • Typical MTS: AR, LRidge, LSVR, GP, and SETAR are traditional baselines, whereas LSTNet-Skip and LSTNet-Attn are deep-neural-network methods.
  • Polyphonic music: For polyphonic music, LSTM and LSTM with Luong attention serve as baselines because traditional methods and LSTNet are ill-suited to the datasets.
  • Polyphonic music: Luong attention scores hidden states using f(h_i, h_t) = (h_i)^⊤W h_t, with W ∈ R^{m×m}.
  • Experimental settings: All experiments use LSTM units and 32 CNN filters, with an autoregression component added for typical MTS datasets.
  • Experimental settings: Typical MTS models use grid searches over window size and hidden units, while learning rates and normalization choices vary by dataset and method.
  • Experimental settings: Polyphonic-music models use three RNN layers, approximately 5 · 10^6 trainable parameters, Adam, a 10^-5 learning rate, and cross-entropy loss.

6.4 Evaluation Metrics

The paper evaluates typical multivariate time-series forecasts with RAE, RSE, and CORR, while polyphonic music models use validation loss, precision, recall, and F1. Table 2 reports mean and standard deviation for the proposed model across ten runs, with other results sourced from LSTNet.

  • Typical MTS datasets are evaluated using RAE, RSE, and CORR.RAE and RSE are normalized error measures, where lower is better; higher CORR is better.
  • Table 2 reports mean and standard deviation for the proposed model over ten runs.The remaining numerical results are referenced from the LSTNet paper.
  • Figure 5 compares the proposed model with LSTNet-Skip on Traffic at a 3-hour forecasting horizon.The comparison focuses on prediction behavior after the peak and in the valley.
  • Figure 6 displays validation loss across training epochs for MuseData and LPD-5-Cleansed.
  • Polyphonic music models are compared using validation loss, precision, recall, and F1 score.Validation loss is negative log-likelihood.

6.5 Results on Typical MTS Datasets

The proposed model consistently outperforms competing methods across typical MTS datasets, including both periodic and non-periodic data. Its evaluation also extends to polyphonic music, where the proposed attention learns more effectively and achieves better reported classification metrics than the compared attention mechanism.

  • Results on Typical MTS Datasets: The proposed model outperforms almost all other methods across typical MTS datasets, horizons, and metrics.The evaluation covers datasets ranging from the 534 KB Exchange Rate dataset to the 172 MB Solar Energy dataset.
  • Results on Typical MTS Datasets: The model shows especially strong performance on Traffic and Electricity, which contain the largest numbers of time series.
  • Results on Typical MTS Datasets: The proposed model achieves state-of-the-art performance on both periodic and non-periodic MTS datasets.
  • Results on Polyphonic Music Datasets: On polyphonic music datasets, the proposed model learns more effectively than LSTM throughout training and yields better precision, recall, and F1 scores.

6.7 Analysis of CNN Filters

The analysis examines whether learned CNN filters capture recurring temporal patterns in multivariate data and how attention and activation choices affect forecasting. The filters align with dominant dataset frequencies, while ablations show that removing components can reduce performance.

  • The average CNN-filter frequencies often peak at the same periods as the Traffic dataset, suggesting the filters resemble DFT bases.Shared peaks occur at 24-, 12-, 8-, and 6-hour periods.
  • Different CNN filters specialize in long-term or short-term temporal patterns, including 24-hour and 8-hour periods.
  • The ablation compares attention over positions, attention over filters, and models without CNN filters, alongside sigmoid and softmax activations.The study evaluates these choices on Solar Energy, Traffic, Electricity, and MuseData.
  • On MuseData, sigmoid activation with attention over filter values at each position performs best, while removing proposed components lowers performance.The negative log-likelihood increases from 0.04882 with sigmoid to 0.04923 with softmax, and reaches 0.4979 without CNN filters.
  • Softmax provides no significant improvement over the proposed model on Solar Energy, Traffic, and Electricity, likely because their strong periodicity makes few basic patterns sufficient.

7 Conclusions

The paper proposes temporal pattern attention for multivariate time-series forecasting to model dependencies across variables, time steps, and series. Experiments support its effectiveness, while filter visualizations make the motivation more interpretable.

  • Temporal pattern attention removes a limitation of typical attention mechanisms for multivariate time-series forecasting.
  • Feature-wise attention models interdependencies among multiple variables within the same time step and across previous times and series.
  • Toy and real-world experiments support the proposed model, which achieves state-of-the-art results, while filter visualizations clarify its motivation.
Loading 1809.04206v3…