PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Factors for Forecasting

Yuyang Wang, Alex Smola, Danielle C. Maddix, Jan Gasthaus, Dean Foster, Tim Januschowski

arXiv:1905.12417v1stat.MLcs.LG

TL;DR

Large collections of related time series require forecasts that scale while capturing complex dependencies and calibrated uncertainty. The paper proposes Deep Factor Models, combining a global deep component with local probabilistic models, and reports improved data efficiency, accuracy, variability, and runtime in experiments.

  • Problem

    Forecasting large collections of dependent series requires scalable models that learn complex patterns while retaining classical models’ data efficiency and uncertainty characterization.

  • Method

    Deep Factor Models combine a global deep neural backbone with local probabilistic models, representing each series as global patterns plus local random effects.

  • Results

    The model outperforms alternatives in accuracy and variability in most cases, while Figure 2 reports improved data efficiency, lower variance, and better runtime scaling than a standard RNN.

  • Takeaways & Limitations

    The global-local framework provides a scalable way to combine expressive deep forecasting with probabilistic local uncertainty and is presented as achieving state-of-the-art performance.

  • Takeaways & Limitations

    A related survival-recommender model primarily handles point processes and does not capture local uncertainty accurately, unlike the proposed models.

Abstract

from arXiv · show

Producing probabilistic forecasts for large collections of similar and/or dependent time series is a practically relevant and challenging task. Classical time series models fail to capture complex patterns in the data, and multivariate techniques struggle to scale to large problem sizes. Their reliance on strong structural assumptions makes them data-efficient, and allows them to provide uncertainty estimates. The converse is true for models based on deep neural networks, which can learn complex patterns and dependencies given enough data. In this paper, we propose a hybrid model that incorporates the benefits of both approaches. Our new method is data-driven and scalable via a latent, global, deep component. It also handles uncertainty through a local classical model. We provide both theoretical and empirical evidence for the soundness of our approach through a necessary and sufficient decomposition of exchangeable time series into a global and a local part. Our experiments demonstrate the advantages of our model both in term of data efficiency, accuracy and computational complexity.

1. Introduction

Large-scale probabilistic forecasting must combine learning complex dependencies across many related series with the data efficiency and uncertainty estimates of classical models. Deep Factor Models address this through a scalable global-local framework using a deep global component and local probabilistic models.

  • Motivation: Forecasts support operational decisions in retail, computing, energy, and workforce scheduling.Applications include stocking and replenishment, capacity planning, power generation, and workload scheduling.
  • Existing approaches: Classical methods often rely on simple models, manual feature engineering, and expert model design for individual or small groups of series.
  • Existing approaches: Deep neural models learn complex patterns from large related datasets but struggle with calibrated uncertainty, sample efficiency, and type (2) dependencies.Type (2) dependencies are indirect relationships through shared latent causes.
  • Research challenge: The paper targets models that borrow statistical strength across diverse dependent series while combining classical data efficiency and uncertainty characterization with neural expressiveness.
  • Proposed framework: Deep Factor Models use a global deep component for scalable nonlinear pattern extraction and local probabilistic models for individual random effects and uncertainty.The framework systematically combines deep neural networks with probabilistic models such as state-space models and Gaussian processes.

2. Exchangeable Series

The paper characterizes exchangeable time series through a global latent series and conditionally independent local series, then uses this decomposition to motivate tractable global-local modeling choices. The framework also extends to hierarchical exchangeability, while related special cases and scope boundaries are identified.

  • Exchangeability: Exchangeable time series have distributions invariant to permutations of series, covering examples such as similar-product sales, securities prices, and electricity use.
  • Characterization: Theorem 1 decomposes an exchangeable collection into a global time series and local series that are conditionally independent given the global latent series.The proof uses de Finetti’s theorem and causal chain-rule decompositions.
  • Hierarchical exchangeability: Theorem 2 extends the characterization to tree-wise exchangeable series through corresponding hierarchical latent-variable models.This is useful for product hierarchies, where sales events within related categories may be exchangeable.
  • Special cases and scope: The presentation omits time series beginning at different times without loss of generality, stating that the approach covers those cases.
  • Modeling choices: Practical models approximate past histories with tractable statistics, use local SSMs or GPs, and may include observed covariates alongside the time-series signal.
  • Modeling choices: A dynamic global latent variable can make inference costly because local and global variables must interact during prediction, motivating explicit use of past local observations.
  • Special cases and scope: The framework covers prior global-local special cases, while a related survival-recommender model does not accurately capture local uncertainty and primarily models point processes.

3. Deep Factor Models with Random Effects

Deep Factor Models with Random Effects represent each time series as a global deep component plus a stochastic local model, with inference designed for large collections. The framework includes several local probabilistic models and shows improved data and computational efficiency over a standard RNN.

  • 3.1. Model structure: The framework models each series using global RNN factors and a random effect drawn from a series-specific probabilistic model.The observation model can use Gaussian, Poisson, or Negative Binomial likelihoods, with covariates supplied to the global and local components.
  • 3.1.1. Global effects (common patterns): The global effects are deterministic linear combinations of deep factors, avoiding costly inference that depends on all time series.The factors act as dynamic principal components that drive shared dynamics across series.
  • Empirical comparison: The proposed fixed-effect formulation improves data and computational efficiency and has less variance than a comparable standard RNN on the electricity dataset.Figure 2 compares generalization error, test-set MAPE, and running time across training-set sizes over 10 runs.
  • 3.1.2. Random effects (local fluctuations): DF-RNN, DF-LDS, and DF-GP instantiate the local random-effects component with a noise RNN, innovation state-space model, or Gaussian process.DF-LDS represents level, trend, and seasonality through a latent state, while DF-GP uses series-specific kernel hyperparameters.
  • 3.2. Inference and learning: The model jointly estimates global RNN parameters, embeddings, and local-model hyperparameters using maximum likelihood and marginal-likelihood computations.For non-Gaussian likelihoods, the exact marginal likelihood is intractable, so the method uses variational inference and a stochastic lower bound.
  • 3.2. Inference and learning: A structured variational approximation enables per-time-series inference, parallelism, and scaling to collections of millions of time series.The approximation combines a recognition network for latent function values with the exact conditional posterior of the local probabilistic model.

4. Related Work and Discussions

Deep Factors situates global-local deep probabilistic forecasting among hybrid neural and state-space approaches. It provides a systematic framework that combines deep neural networks with state-space models and Gaussian processes while avoiding some costly global inference.

  • Hybrid neural-probabilistic models: Prior hybrid models combine RNNs with state-space models by extending likelihoods or making state transitions nonlinear.Examples include nonlinear transitions through multilayer perceptrons or RNN-specified transition matrices.
  • Hybrid neural-probabilistic models: Deep Markov Models, VRNNs, and related approaches can require expensive inference at the global level through recognition networks.Deep Factors instead restrict global effects to deterministic states.
  • Comparison with Deep State Models: Deep State Models retain linear Gaussian transitions so that efficient Kalman filtering and smoothing remain applicable.Deep Factors additionally avoid sampling latent states in the ELBO, eliminating variance from the corresponding Monte Carlo estimate.
  • Gaussian-process connections: The framework offers an alternative way to use Gaussian processes more efficiently by placing them in a global-local forecasting model.This differs from Deep Gaussian Processes, which stack GPs over latent variables to obtain more expressive mappings.
  • Framework scope: Deep Factors provide a systematic framework that incorporates state-space models and Gaussian processes with deep neural networks.The framework can interpolate between purely local and purely global models and subsume related forecasting methods under particular choices.

5. Experiments

Experiments evaluate Deep Factor models on synthetic and real-world forecasting tasks, emphasizing recognition networks, latent-factor recovery, scalability, and short-horizon accuracy under limited data. The proposed approach shows improved data efficiency, accuracy, variability, and forecasting flexibility in the reported comparisons.

  • Model Understanding and Exploration: On electricity, the fixed-effect formulation improves data and computational efficiency and has lower variance than a standard RNN forecaster.The comparison uses roughly equal parameter counts.
  • Model Understanding and Exploration: RNN recognition networks outperform MLPs for reconstructing and forecasting sequential non-Gaussian observations in the variational LDS experiment.The experiment uses Poisson observations and evaluates reconstructed intensity and predictive distributions.
  • Model Understanding and Exploration: The method identifies the true latent global factors even under Poisson noise, measured by distances between the subspaces they span.This tests recovery of global factors under Gaussian and non-Gaussian noise.
  • Real-world datasets: Across electricity, traffic, NYC taxi, and Uber datasets, experiments compare Deep Factor with DeepAR, MQ-RNN, and Prophet using quantile loss.Reported metrics include P50QL (MAPE) and P90QL.
  • Real-world datasets: With one week of training data, the model outperforms competing methods in accuracy and variability in most reported short-term and near-term cases.Short-term forecasts use a 72-hour horizon, while near-term forecasts use 24 hours.
  • Real-world datasets: Deep Factor can forecast beyond the training-series length and adapt to changing forecasting scenarios without retraining for a longer decoder horizon.The cited comparison identifies this as a limitation of DeepAR and MQ-RNN, not of the proposed method.

6. Conclusion

The paper concludes that Deep Factor Models with Random Effects provide a scalable global-local framework for collections of related time series, supported by a characterization of exchangeable time series. It reports state-of-the-art performance and identifies probabilistic uncertainty methods for non-probabilistic deep models as future work.

  • Conclusion: The proposed global-local framework scales to collections of related time series and is accompanied by a unique characterization of exchangeable time series.The conclusion presents this characterization as a main contribution.
  • Conclusion: The framework obtains state-of-the-art performance while combining a general modeling approach with practical scalability.The conclusion describes the framework as general, powerful, and practically relevant.
  • Conclusion: Future work includes evaluating variational dropout and Deep Ensemble methods for uncertainty in non-probabilistic deep neural networks.The suggested models include RNNForecaster-style architectures.

A. Deep Factor Models with Random Effects

The appendix describes DF-RNN as a global-factor model with a local stochastic component and details how its noise process and inference are handled. It also explains that the local process can induce correlated latent noise despite i.i.d. innovations.

  • Model notation: The global-factor function maps d-dimensional inputs to K global factors, which are used by the model's fixed-effect component.The notation is introduced as g_t(·): IR^d → IR^K.
  • DF-RNN local model: DF-RNN uses a zero-mean i.i.d. Gaussian local model whose time-varying variance is generated by a noise RNN.The local variance is denoted σ²_i,t.
  • Inference: For non-Gaussian likelihoods, inference is required for latent innovation terms in the latent function u and is performed using Algorithm 1.The variational lower bound is computable through a Gaussian marginal likelihood in the described setting.
  • Inference: Although the local noise is i.i.d., the latent RNN process produces correlated noise, making DF-RNN a type of deep Gaussian latent-variable model.The covariance structure includes latent-process contributions.

A.2. DF-LDS: LDS as the local model

DF-LDS uses a level-trend linear dynamical system as its stochastic local model, with variational inference over latent states and a non-Gaussian emission extension.

  • DF-LDS: The model assumes random effects evolve through a linear state transition with Gaussian innovations.The random-effect transition is h_i,t = F_i,t h_i,t−1 + q_i,t ε_i,t, with ε_i,t ∼ N(0, 1).
  • DF-LDS: DF-LDS uses a level-trend ISSM with damping for the local stochastic dynamics.δ_i and γ_i control damping, while α_i and β_i determine level and trend innovation strength.
  • Variational LDS: With no global factors, DF-LDS recovers a state-space model with a non-Gaussian likelihood, yielding the proposed Variational LDS approach.The appendix identifies the resulting Poisson special case as the popular Poisson LDS.
  • Inference: Inference approximates p(h, u|z) with q_φ(u|z)p(h|u), using a recognition network for the first factor and the conditional posterior for the second.Training follows Algorithm 1 without global factors.
  • Inference: A bidirectional LSTM encoder uses past and future information during training, analogous to a backward message in the Kalman smoother.The recognition network is used only during training to better approximate the posterior of the latent function.

A.3. DF-GP: Gaussian Process as the local model

DF-GP models local random effects with a zero-mean Gaussian process and applies the proposed variational inference approach to non-Gaussian likelihoods.

  • DF-GP: DF-GP assumes each local random effect is drawn from a zero-mean Gaussian process.The latent function combines a global component f_i,t with a local GP term r_i(x_i,t).
  • DF-GP: The latent function is formed as u_i,t = f_i,t + r_i,t and emitted through an arbitrary observation likelihood p(·|u_i,t).The local GP contribution depends on the input x_i,t.
  • Inference: The proposed inference algorithm provides approximate inference for Gaussian processes with non-Gaussian likelihoods.Comparison with Laplace approximation, expectation propagation, and variational inference is left for future work.

B. Detailed Experimental Analysis

The appendix reports additional synthetic and empirical results for the proposed models.

  • The appendix provides further results in both synthetic and empirical cases.

B.1. Synthetic Experiments

The synthetic experiment tests whether the algorithm can recover global factors added to data with local variational-LDS dynamics, under Gaussian and Poisson observations.

  • Synthetic setup: Global factors are generated from Fourier series of different orders and combined across series with uniformly sampled coefficients.The local model and observations come from the variational-LDS example.
  • Recovery results: The algorithm does not precisely recover global factors under Gaussian LDS noise, but the true and estimated factor subspaces are reasonably close.
  • Recovery results: Under Poisson observations, the method still roughly recovers the global factors.
  • Figures: Figure B.1 compares true solid global factors with estimated dashed factors for synthetic data with Gaussian noise.

B.2. Empirical Studies

The empirical studies compare Deep Factor variants with classical and neural baselines across electricity and other forecasting settings. DF-RNN is selected for broader comparison because it offers similar accuracy to DF-LDS with computational advantages, while tuned DeepAR can reach comparable accuracy.

  • The experiments evaluate Deep Factor models using datasets summarized in Table B.1 and RMSE results for 72-hour and 24-hour horizons.Table B.3 reports short-term and near-term scenarios with one week of training data.
  • DF-RNN and DF-LDS achieved similar accuracy against level-trend ISSM on electricity over 50 trials, with DF-RNN slightly preferred.DF-RNN was chosen for broader comparisons because of its simplicity and computational advantages.
  • With hyperparameter tuning and a larger model, DeepAR achieved 0.118 MAPE and 0.042 P90 Loss on electricity for horizon 72, on par with the reported Deep Factor results.The passage states that Deep Factor achieves such accuracy with fewer parameters.
  • Figures B.3 and B.4 visualize normalized P90QL, P50QL, MAPE, and log(1 + RMSE) results for forecast horizons 72 and 24.Purple denotes the proposed method in Figure B.3.
  • Figure B.5 presents example forecasts from the model in cold-start and regular settings across uber, taxi, traffic, and electricity.The taxi cold-start example fails to produce a reasonable forecast, possibly because of insufficient latitude and longitude information.
Loading 1905.12417v1…