PaperlayerView exact revision on arXiv

Source-linked AI summary

Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks

Guokun Lai, Wei-Cheng Chang, Yiming Yang, Hanxiao Liu

arXiv:1703.07015v3cs.LG

TL;DR

Multivariate forecasting must handle dependencies across variables and mixtures of short- and long-term recurring patterns. LSTNet combines convolutional, recurrent, skip-recurrent, and autoregressive components for this setting, and it improves benchmark performance, especially at long horizons, while remaining weaker on Exchange-Rate data.

  • Problem

    Multivariate time series often mix short- and long-term recurring patterns, creating a challenge for forecasting models that must capture dependencies across variables.

  • Method

    LSTNet combines convolutional and recurrent components for local and long-term dependencies, a Recurrent-skip structure for very long-term patterns, and an autoregressive component for changing scales.

  • Results

    LSTNet models consistently outperform state-of-the-art baselines on periodic datasets, including RSE improvements over RNN-GRU of 9.2%, 11.7%, and 22.2% at horizon 24 on Solar-Energy, Traffic, and Electricity.

  • Takeaways & Limitations

    The results provide empirical evidence that LSTNet models long- and short-term dependencies effectively when repetitive patterns occur, while remaining comparable with strong baselines otherwise.

  • Takeaways & Limitations

    The skip length p is manually tuned on validation data, and the convolutional layer treats all variable dimensions equally despite potentially useful attribute information.

Abstract

from arXiv · show

Multivariate time series forecasting is an important machine learning problem across many domains, including predictions of solar plant energy output, electricity consumption, and traffic jam situation. Temporal data arise in these real-world applications often involves a mixture of long-term and short-term patterns, for which traditional approaches such as Autoregressive models and Gaussian Process may fail. In this paper, we proposed a novel deep learning framework, namely Long- and Short-term Time-series network (LSTNet), to address this open challenge. LSTNet uses the Convolution Neural Network (CNN) and the Recurrent Neural Network (RNN) to extract short-term local dependency patterns among variables and to discover long-term patterns for time series trends. Furthermore, we leverage traditional autoregressive model to tackle the scale insensitive problem of the neural network model. In our evaluation on real-world data with complex mixtures of repetitive patterns, LSTNet achieved significant performance improvements over that of several state-of-the-art baseline methods. All the data and experiment codes are available online.

1 INTRODUCTION

Multivariate forecasting must capture dependencies across variables and both short- and long-term recurring patterns. LSTNet combines convolutional, recurrent, and autoregressive components to address these patterns and scale changes.

  • Motivation: Real-world multivariate series combine short-term and long-term repetitions, such as daily and weekly traffic patterns.Solar-energy data similarly contain day–night and seasonal trends alongside weather-related short-term effects.
  • Motivation: Forecasting requires leveraging dynamic dependencies among multiple variables while predicting future observations from historical signals.Applications include traffic, stock-market, and solar-energy forecasting.
  • Proposed framework: LSTNet uses convolutional layers to discover local cross-variable patterns and recurrent layers to capture complex long-term dependencies.Its Recurrent-skip structure targets very long-term dependencies by exploiting periodicity.
  • Proposed framework: LSTNet adds a parallel autoregressive linear component to improve robustness when input scales change non-periodically.The autoregressive component complements the nonlinear neural-network part.
  • Evaluation: The framework is evaluated against strong baselines on real-world seasonal time-series datasets.The paper reports consistent improvements over traditional linear models and GRU recurrent networks.

2 RELATED BACKGROUND

Prior forecasting approaches include autoregressive, regression, and Gaussian-process models. These methods offer useful structure or flexibility but face limitations in multivariate scale, dependency modeling, or computational cost.

  • Autoregressive methods: ARIMA generalizes several classical time-series models but is rarely used for high-dimensional multivariate forecasting because of high computational cost.Its variants include autoregression, moving average, and autoregressive moving average models.
  • Autoregressive methods: VAR extends autoregression to multiple variables and is widely used for its simplicity, but it ignores dependencies between output variables.The passage also notes that VAR capacity grows with the number of variables.
  • Regression methods: Regression-based approaches apply different losses and regularization terms to time-series forecasting, including support vector regression and ridge regression.These methods treat forecasting as regression with time-varying parameters.
  • Gaussian Processes: Gaussian Processes model distributions over continuous-domain functions and can capture complex dynamical phenomena through Bayesian formulations.Their power comes with high computational complexity; a straightforward multivariate implementation has cubic complexity in observations.

3 FRAMEWORK

LSTNet formulates multivariate forecasting as rolling prediction from fully observed historical signals. Its architecture combines convolutional processing for short-term and local dependencies with recurrent processing for long-term patterns.

  • Problem Formulation: Given historical multivariate signals, the task predicts future signals at a chosen forecasting horizon in a rolling fashion.The input at time T is the matrix X_T containing observations y_1 through y_T.
  • Problem Formulation: Forecasting horizons depend on the application, ranging from hours or a day for traffic to seconds or minutes for stock-market data.The horizon is selected according to environmental demands.
  • Architecture overview: LSTNet is designed for multivariate series containing mixtures of long- and short-term patterns.The architecture is introduced as a deep-learning framework for this forecasting setting.
  • Convolutional Component: Its convolutional layer uses filters spanning all variables to extract short-term temporal patterns and local dependencies among variables.The convolution operates without pooling, and the resulting representation has one time-aligned output per filter.

3.3 Recurrent Component

The recurrent component processes convolutional features over time using a GRU-based recurrent layer. It uses ReLU for hidden updates, which the authors report improves gradient backpropagation reliability.

  • Recurrent Component: The convolutional output is fed into the recurrent component, which uses a gated recurrent unit to process temporal representations.The recurrent layer also feeds the Recurrent-skip component described elsewhere in the architecture.
  • Recurrent Component: The recurrent layer outputs the hidden state at each time stamp while using ReLU as its hidden-update activation.The authors empirically found ReLU more reliable than tanh because gradients are easier to backpropagate.

3.4 Recurrent-skip Component

The recurrent-skip component extends information flow across long temporal spans by connecting recurrent states separated by a skip length p, helping model periodic dependencies that standard recurrent units struggle to capture.

  • 3.4 Recurrent-skip Component: Standard GRU and LSTM units can fail to capture very long-term correlations because of gradient vanishing.The paper motivates recurrent skip-connections for dependencies spanning periods such as 24 hours.
  • 3.4 Recurrent-skip Component: Skip-links connect the current hidden cell with earlier hidden cells to extend the temporal span of information flow.The recurrent-skip layer receives the convolutional layer output and skips p hidden cells.
  • 3.4 Recurrent-skip Component: The skip length p can be set from known periodicities, such as p = 24 for hourly electricity-consumption and traffic-usage datasets.For datasets without clear periodicity, p must be tuned.
  • 3.4 Recurrent-skip Component: A dense layer combines the recurrent hidden state at time t with p recurrent-skip hidden states from t − p + 1 through t.The resulting dense-layer output contributes to the neural-network prediction.

3.5 Temporal Attention Layer

The temporal attention layer provides an alternative to a predefined recurrent-skip length by learning weighted combinations of hidden representations across the input window.

  • 3.5 Temporal Attention Layer: Attention addresses recurrent-skip limitations when seasonality is absent or the period changes dynamically over time.The recurrent-skip layer requires a predefined hyper-parameter p, which is unfavorable in these settings.
  • 3.5 Temporal Attention Layer: At time t, the attention mechanism calculates weights α_t over hidden representations at each input-window position.These weights are learned as a weighted combination of hidden representations.
  • 3.5 Temporal Attention Layer: The attention score can use dot product, cosine similarity, or a parameterized multilayer perceptron.The hidden representations are stacked column-wise in the matrix H_t.
  • 3.5 Temporal Attention Layer: The layer concatenates the weighted context vector c_t = H_tα_t with the last hidden representation in the input window.This concatenation forms the temporal-attention output.

3.6 Autoregressive Component

The autoregressive component supplements LSTNet’s nonlinear neural-network prediction with a linear model to address sensitivity to changing input scales.

  • 3.6 Autoregressive Component: Neural-network outputs can be insensitive to input scale because the convolutional and recurrent components are nonlinear.The paper links this issue to reduced forecasting accuracy when input-signal scale changes non-periodically.
  • 3.6 Autoregressive Component: LSTNet decomposes its final prediction into a neural-network part and a linear autoregressive part.The autoregressive coefficients are shared across all dimensions.
  • 3.6 Autoregressive Component: The autoregressive model uses an input window of size q_ar and coefficients W_ar with bias b_ar.The model applies the same linear parameters to every dimension.
  • 3.6 Autoregressive Component: The integrated output is LSTNet’s final prediction at time stamp t.The notation Ŷ_t denotes this combined prediction.

3.7 Objective function

The paper considers squared and absolute loss objectives for training, with validation selecting between them; absolute loss is motivated by robustness to anomalies.

  • 3.7 Objective function: The default forecasting objective uses squared error over training timestamps, with model parameters Θ and forecast horizon h.The formulation uses the Frobenius norm to measure prediction error.
  • 3.7 Objective function: Linear Ridge is equivalent to a vector autoregressive model with ridge regularization under square loss.The paper contrasts it with Linear SVR, whose objective function differs.
  • 3.7 Objective function: Linear SVR constrains each training prediction error using slack variables ξ_t,i and an ϵ tolerance.C and ϵ are hyper-parameters of the objective.
  • 3.7 Objective function: With ϵ = 0, the Linear SVR objective reduces to absolute loss, which is incorporated into LSTNet as an alternative.The alternative is motivated by Linear SVR’s performance on certain datasets.
  • 3.7 Objective function: Validation data determine whether LSTNet uses square loss or absolute loss.The paper states that absolute loss is more robust to anomalies in real time-series data.

3.8 Optimization Strategy

LSTNet reformulates forecasting as supervised regression by mapping a tunable historical window to a future target, then optimizes the resulting examples with gradient-based methods.

  • A tunable window q converts observations into input sequences X_t={y_t−q+1,…,y_t} paired with future targets Y_{t+h}.
  • The resulting forecasting problem is a regression task over feature-value pairs {X_t,Y_{t+h}}.
  • Stochastic Gradient Descent or Adam can optimize the model from these supervised examples.

4 EVALUATION

The evaluation compares LSTNet variants with eight representative baselines across four benchmark datasets and multiple forecasting horizons. LSTNet performs best when data contain repetitive short- and long-term patterns, while its autoregressive component supports robustness to scale changes and large horizons.

  • 4 EVALUATION: Nine methods were evaluated on four benchmark datasets for multivariate time-series forecasting.The comparison includes autoregressive, Gaussian-process, regression, recurrent, and LSTNet variants.
  • 4.3 Data: Traffic, Solar-Energy, and Electricity exhibit repetitive patterns, including daily and weekly cycles in Traffic and Electricity, whereas Exchange-Rate lacks clear long-term repetition.
  • 4.5 Main Results: 17 bold-faced results belong to LSTNet-Skip and 7 to LSTNet-Attn, compared with 0–3 for each remaining method.Table 2 evaluates RSE and CORR across horizons from 3 to 24 hours, 30 to 240 minutes, or 3 to 24 days depending on the dataset.
  • 4.5 Main Results: LSTNet improves over RNN-GRU by 9.2%, 11.7%, and 22.2% in RSE on Solar-Energy, Traffic, and Electricity at horizon 24.
  • 4.5 Main Results: LSTNet is slightly worse than AR and LRidge on Exchange-Rate, where repetitive long-term patterns are not evident.
  • 4.6 Ablation Study: Removing the AR component causes the largest performance drops on most datasets, while removing CNN or skip components hurts some datasets but not all.
  • 4.6 Ablation Study: In simulation, LSTNet follows test-set scale changes better than RNN-GRU, whose neural-network model cannot track the changing scale.

5 CONCLUSION

LSTNet combines convolutional, recurrent, and autoregressive components for multivariate time-series forecasting, capturing short- and long-term repeating patterns while supporting robust prediction. The paper identifies automatic skip-length selection and richer variable attributes as directions for extending the model.

  • LSTNet combines convolutional and recurrent networks with an autoregressive component for multivariate time-series forecasting.
  • Empirical analysis shows that LSTNet captures short- and long-term repeating patterns and combines linear and nonlinear models for robust prediction.
  • The skip-recurrent layer’s length p is manually tuned, motivating automatic selection according to the data.
  • The convolutional layer treats variable dimensions equally, leaving integration of richer attribute information as an open challenge.
Loading 1703.07015v3…