PaperlayerView exact revision on arXiv

Source-linked AI summary

Think Globally, Act Locally: A Deep Neural Network Approach to High-Dimensional Time Series Forecasting

Rajat Sen, Hsiang-Fu Yu, Inderjit Dhillon

arXiv:1905.03806v2stat.MLcs.LG

TL;DR

High-dimensional forecasting must handle millions of correlated series, yet existing deep models often rely on local histories and normalization for diverse scales. The paper proposes DeepGLO, combining TCN-regularized global matrix factorization with local temporal modeling and covariates. DeepGLO is reported to outperform benchmark methods, including on four real-world datasets.

  • Problem

    Millions of correlated time series require forecasting methods that exploit global patterns while handling wide scale variation, whereas existing deep models often use local histories and require normalization.

  • Method

    DeepGLO combines a TCN-regularized global matrix factorization model with a local temporal convolution network using series values and covariates.

  • Results

    DeepGLO outperformed other benchmarks on four real-world datasets and ranked among the top two models across most reported categories and metrics.

  • Takeaways & Limitations

    The proposed hybrid approach supports global and local forecasting while enabling training on diverse-scale time series without a priori normalization.

Abstract

from arXiv · show

Forecasting high-dimensional time series plays a crucial role in many applications such as demand forecasting and financial predictions. Modern datasets can have millions of correlated time-series that evolve together, i.e they are extremely high dimensional (one dimension for each individual time-series). There is a need for exploiting global patterns and coupling them with local calibration for better prediction. However, most recent deep learning approaches in the literature are one-dimensional, i.e, even though they are trained on the whole dataset, during prediction, the future forecast for a single dimension mainly depends on past values from the same dimension. In this paper, we seek to correct this deficiency and propose DeepGLO, a deep forecasting model which thinks globally and acts locally. In particular, DeepGLO is a hybrid model that combines a global matrix factorization model regularized by a temporal convolution network, along with another temporal network that can capture local properties of each time-series and associated covariates. Our model can be trained effectively on high-dimensional but diverse time series, where different time series can have vastly different scales, without a priori normalization or rescaling. Empirical results demonstrate that DeepGLO can outperform state-of-the-art approaches; for example, we see more than 25% improvement in WAPE over other methods on a public dataset that contains more than 100K-dimensional time series.

1 Introduction

High-dimensional forecasting involves millions of correlated series, while existing methods struggle with scale variation and fail to use global patterns during prediction. The paper proposes DeepGLO to combine global and local information while supporting reliable training across diverse scales.

  • Modern datasets may contain millions of correlated time series, represented as an n×t matrix for forecasting future demands.Applications include retail, finance, traffic, and weather forecasting.
  • Traditional methods require individual training and cannot readily scale to millions of series or exploit shared temporal patterns.
  • Wide scale variation makes deep models difficult to train, often requiring series-specific normalization whose choices affect accuracy.
  • Deep forecasting models can train on entire datasets but commonly use only each series’ local history during prediction.
  • DeepGLO combines global and local patterns during training and prediction and aims to train reliably without prior normalization.
  • The paper contributes LeveledInit, TCN-regularized matrix factorization, and a hybrid DeepGLO forecaster evaluated against benchmarks on four real-world datasets.

2 Related Work

Related work spans traditional forecasting, deep sequence models, and temporally regularized matrix factorization. The paper extends these directions by combining nonlinear global modeling with local temporal convolution.

  • Traditional forecasting research includes AR, ARIMA, exponential smoothing, Box-Jenkins methods, and linear state-space models.
  • Recent deep-learning approaches use LSTMs, state-space components, history encoders, multi-horizon decoders, and architectures such as LSTNet.
  • Temporal matrix factorization has been applied to speech denoising and spatio-temporal forecasting, with TRMF providing AR-based temporal regularization.
  • This paper extends TRMF toward nonlinear temporal regularization with a temporal convolution network and combines global factorization with local modeling.

3 Problem Setting

The task is to forecast future values for many correlated time series from their observed histories and optional covariates. The data are represented using matrices and tensors, with low-rank temporal structure used for forecasting.

  • The paper uses matrices, tensors, submatrices, subvectors, and Frobenius or spectral norms to formalize the data and models.
  • A dataset contains raw series in Y=[Y^(tr) Y^(te)], where n series are observed for t training points and forecast over a τ-step test window.
  • Optional covariates form a tensor Z∈R^(n×r×(t+τ)) and may encode global features or series-specific information.
  • The forecasting task predicts Y^(te) using the training series Y^(tr) and available covariates Z.
  • Forecast quality is evaluated with normalized absolute deviation, also called WAPE, alongside other metrics such as squared loss.
  • Figure 1 illustrates both a TCN mapping lagged inputs to one-shifted outputs and a matrix-factorization approach for time-series forecasting.
  • The matrix-factorization formulation decomposes training data into low-rank factors F and X^(tr), then forecasts X^(te) to obtain predictions FX^(te).

4 LeveledInit: Handling Diverse Scales with TCN

LeveledInit initializes temporal convolution networks to address widely varying series scales without prior normalization. It makes the initial prediction an average over the TCN’s look-back window and remains competitive with a deeper variant.

  • LeveledInit is designed to train TCNs on high-dimensional series with diverse scales without a priori normalization.
  • Temporal Convolution: A TCN uses dilated one-dimensional convolutions whose dynamic range is l′=1+2(k−1)2^(d−1), with past values and covariates producing one-step forecasts.
  • LeveledInit Scheme: LeveledInit initializes the network to approximately predict the average of a window of past time points, allowing training to learn variations around that level.
  • LeveledInit Scheme: With zero biases, weights 1/k, and ReLU activations, the initialized network predicts the empirical mean of the preceding window.
  • LeveledInit Scheme: For k=2, the prediction averages the past l time points, where l is the TCN dynamic range; the scheme extends to covariates and multiple channels.
  • Empirical behavior: LeveledInit can be trained reliably without a priori normalization on real-world datasets, including configurations with filter size k≠2.
  • Empirical behavior: LeveledInit matched the performance of the more sophisticated Deep Leveled Network variant.

5 DeepGLO: A Deep Global Local Forecaster

DeepGLO combines a global low-rank temporal model with a local temporal network, allowing forecasts to use dataset-wide patterns alongside each series’ history and covariates.

  • 5.1 Global: Temporal Convolution Network regularized Matrix Factorization: DeepGLO represents original series as linear combinations of k basis series, where k ≪ n, to capture global temporal patterns.The basis series are forecastable with a temporal convolution network.
  • 5.1 Global: Temporal Convolution Network regularized Matrix Factorization: TCN-MF regularizes low-rank factors by encouraging factorized temporal values to match predictions from a temporal convolution network.The regularization uses squared loss between factor values and network predictions from preceding time steps.
  • 5.1 Global: Temporal Convolution Network regularized Matrix Factorization: The global factors and temporal network are trained alternately, with mini-batch SGD updating factors and then training the network on the factor matrix.LeveledInit initializes the temporal network before alternating optimization begins.
  • 5.2 Combining the Global Model with Local Features: The model forecasts basis series first, reconstructs original-series predictions through Ŷ(te) = F X̂(te), and supports rolling or multi-step prediction without retraining.Algorithm 4 trains the global component, creates global prediction covariates, and then trains the local network.
  • 5.2 Combining the Global Model with Local Features: DeepGLO’s local TCN receives original time-series points, original covariates, and global-model predictions as inputs.Its input size is r + 2: one channel for raw history, r for original covariates, and one for global predictions.

6 Empirical Results

DeepGLO was evaluated on four real-world rolling-forecasting datasets under normalized and unnormalized training settings. It ranked among the top two models in nearly all reported categories and exceeded 25% WAPE improvement over other models on wiki.

  • 6 Empirical Results: The evaluation covers four real-world datasets using rolling prediction tasks and compares DeepGLO with local, global, recurrent, classical, and factorization baselines.The datasets include electricity, traffic, wiki, and PeMSD7(M).
  • 6 Empirical Results: Models are evaluated under both normalized and unnormalized training, with all errors reported on the original data scale.The settings measure how scaling affects forecasting accuracy.
  • 6 Empirical Results: DeepGLO ranks among the top two models in almost all categories across WAPE, MAPE, and SMAPE under both scaling settings.The comparison is reported for the first three datasets.
  • 6 Empirical Results: More than 25% improvement in WAPE over all other models is reported for DeepGLO on the wiki dataset.The local TCN performs best there with greater than 30% improvement, while DeepGLO is close behind.

A Deep Leveled Network

DLN is a leveling architecture that avoids a priori normalization by separating forecasts into a rolling-mean component and a residual component. Its two temporal convolution blocks are trained concurrently and used recursively for multi-step prediction.

  • Motivation: DLN targets diverse-scale time-series datasets without requiring a priori normalization.The motivation is that scale diversity can make deep models difficult to train and normalization choices can affect predictive performance.
  • Architecture: The leveling component predicts the rolling mean of the next w future points from the past.The residual component predicts variations relative to that mean.
  • Architecture: DLN combines the leveling and residual temporal convolution outputs to produce forecasts.The two blocks use the same dynamic range or look-back l and are trained concurrently.
  • Training: Training uses mini-batch stochastic gradient updates, with the leveling component held fixed during the residual-update step.Both networks are trained using the loss defined by the paper’s evaluation metric.
  • Prediction: Multi-step prediction recursively feeds each forecast back into the network to generate later time steps.The procedure repeats τ times to obtain a τ-step-ahead forecast.

B Rolling Prediction without Retraining

After training, TCN-MF predicts latent basis-series values and reconstructs original-series forecasts through the factor matrix. Rolling predictions update latent values without retraining the model.

  • Prediction: TCN-MF first predicts the basis time-series in X(tr) and then reconstructs original-series forecasts as Ŷ(te) = F X̂(te).The reconstruction applies the learned factor matrix F to the predicted latent series.
  • Rolling prediction: For rolling prediction, newly observed values are used to estimate the next latent segment through an optimization over M.The displayed objective minimizes the global loss for the newly observed interval.
  • Rolling prediction: Unlike TRMF, the proposed rolling procedure does not require retraining from scratch to incorporate newly observed values.The passage states that good performance is achieved without retraining.

C More Experimental Details

The paper indicates that further experimental details cover the rolling-prediction settings, evaluation metrics, and model hyperparameters.

  • Experimental details: The experiments section provides exact rolling-prediction settings for each dataset.
  • Experimental details: It also specifies the evaluation metrics and model hyperparameters.

C.1 Rolling Prediction

Rolling prediction evaluates forecasts in sequential future windows as additional observations are revealed. The final score aggregates predictions across all test windows using the paper’s loss metric.

  • Rolling procedure: The rolling task predicts future windows in batches as more data becomes available.At each step, the model receives data through the previous window before forecasting the next one.
  • Rolling procedure: After each forecast, the actual values for that window are revealed before the process continues.The sequence repeats across i = 1, 2, ..., nw test windows.
  • Evaluation: The final performance measure is the loss over all predicted test windows.The loss compares the forecasts and actual values across the interval from t0 + 1 through tnw.
  • Dataset settings: Traffic and electricity use τ = 24 with w = 7, while wiki uses τ = 14 with w = 4.Their initial training periods are t0 = 10392, 25968, and 747, respectively.

C.2 Loss Metrics

This section defines the loss metrics used to evaluate predictions, with actual values Y and predictions Ŷ arranged as n′×t′ matrices.

  • The evaluation uses WAPE, MAPE, and SMAPE as loss metrics.Y denotes actual values, while Ŷ denotes corresponding predictions.
  • WAPE is defined as Weighted Absolute Percent Error.
  • MAPE is defined as Mean Absolute Percent Error.
  • The metrics use Z0 to account for nonzero actual values in the evaluation.The supplied definitions specify Z0 through indicators for |Yij| > 0.
  • SMAPE is defined as Symmetric Mean Absolute Percent Error.

C.3 Model Parameters and Settings

The experiments compare multiple forecasting models under specified network, rank, lag, growth, and training settings, with DeepGLO using dataset-specific architectures and ranks.

  • All models are trained with early stopping using a specified tenacity or patience setting.
  • DeepGLO uses dataset-specific TX and TY architectures, α = 0.2, λT = 0.2, and ranks 64, 64, 256, and 64 across four datasets.The four ranks correspond respectively to electricity, traffic, wiki, and PeMSD7(M).
  • Local TCN, Local DLN, SVD+Leveled, and TCN use five 32-filter layers followed by one output layer, with kernel size 7 where specified.
  • TRMF uses ranks 60, 60, and 1024 for electricity, traffic, and wiki, with traffic and electricity lags covering the last day and same day in the previous week.
  • LSTM uses parameters (45, 3), DeepAR uses GluonTS default parameters, and Prophet uses logistic growth with automatic parameter selection.Prophet is parallelized over 32 cores.
  • The reported STGCN values on PeMSD7(M) replicate the exact values from the cited prior work.
Loading 1905.03806v2…