PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Latent Auto-Encoder: A Method for Probabilistic Multivariate Time Series Forecasting

Nam Nguyen, Brian Quanz

arXiv:2101.10460v1cs.LGcs.AIcs.NE

TL;DR

Multivariate forecasting must model many related series, yet existing approaches can miss cross-series effects or impose limits on latent factorization. TLAE combines nonlinear encoding and decoding with an end-to-end temporal latent model and probabilistic sampling. The paper reports superior performance to other state-of-the-art methods on several common time-series datasets.

  • Problem

    Multivariate forecasts require modeling related series, but multi-task univariate methods cannot capture cross-series effects or use information from noisy or sparse related series.

    Source: §1 · p. 1

  • Method

    TLAE uses neural encoders and decoders with an end-to-end temporal latent forecast model, injecting latent noise to model joint predictive distributions.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    TLAE achieves superior performance compared with other state-of-the-art methods on several common time-series datasets.

    Source: §5 · p. 7

  • Takeaways & Limitations

    The latent-space formulation models complex predictive distributions through a nonlinear decoder while combining flexible autoencoding with latent temporal dynamics.

    Source: §5 · p. 7

    The Wiki experiments used relatively few epochs, and further hyperparameter settings for that dataset remain future work.

    Source: §6.2 · p. 11

Abstract

from arXiv · show

Probabilistic forecasting of high dimensional multivariate time series is a notoriously challenging task, both in terms of computational burden and distribution modeling. Most previous work either makes simple distribution assumptions or abandons modeling cross-series correlations. A promising line of work exploits scalable matrix factorization for latent-space forecasting, but is limited to linear embeddings, unable to model distributions, and not trainable end-to-end when using deep learning forecasting. We introduce a novel temporal latent auto-encoder method which enables nonlinear factorization of multivariate time series, learned end-to-end with a temporal deep learning latent space forecast model. By imposing a probabilistic latent space model, complex distributions of the input series are modeled via the decoder. Extensive experiments demonstrate that our model achieves state-of-the-art performance on many popular multivariate datasets, with gains sometimes as high as $50\%$ for several standard metrics.

1 Introduction

Multivariate forecasting must handle large collections of related series, but common approaches either struggle to scale or miss cross-series effects. TLAE extends latent factorization with nonlinear, end-to-end temporal modeling and probabilistic outputs.

1 Introduction

  • Multivariate forecasting covers large numbers of related series, such as billions of product-and-location sales series in retail.The passage also describes forecasting as predicting future time-series values and notes its use across industries.Source: §1 · p. 1
  • Multi-task univariate models cannot capture cross-series effects or use information from related series when individual series are noisy or sparse.Retail products may affect one another, while a single product-store series may have only one sale per month.Source: §1 · pp. 1–2
  • Linear factorization scales forecasting through a low-dimensional latent space while retaining cross-series relationships, but existing approaches have several limitations.TRMF uses linear autoregressive latent dynamics; DeepGLO adds a latent TCN through alternating optimization.Source: §1 · p. 2
  • TLAE replaces linear mappings with neural encoders and decoders, and trains them end-to-end with a temporal latent model and reconstruction and forecast objectives.Temporal models can let embeddings evolve over time.Source: §1 · p. 2
  • Noise injected into the latent space before forecast decoding lets TLAE model joint predictive distributions through transformed latent samples.The latent forecast mean is not constrained as in a VAE.Source: §1 · p. 2

2 Related Work

Related work spans classical forecasting, shared deep models, and approaches to multivariate and probabilistic prediction. The paper distinguishes TLAE from these methods by its factorization of cross-series relationships and its latent-space temporal modeling.

2 Related Work

  • Classical forecasting methods include VAR, ARIMA, and ETS, while deep forecasting has historically focused mostly on univariate models (Zhang, Patuwo, and Hu 1998; Benidis et al. 2020).The paper compares representative classical methods in its experiments.Source: §2 · p. 2
  • DeepAR scales series by their averages and shares an RNN across them (Salinas, Flunkert, and Gasthaus 2019), but shared multi-task univariate models do not model dependencies between series.LSTNet combines a linear AR model with convolutional and recurrent layers.Source: §2 · p. 2
  • TADA, DA-RNN, and GeoMAN use encoder-decoder forecasting with exogenous variables, while Cirstea et al. use autoencoders for individual-series denoising rather than cross-series probabilistic factorization.These approaches are described as not modeling cross-series relationships or probabilistic outcomes in the way TLAE does.Source: §2 · p. 2
  • Probabilistic alternatives include a low-rank Gaussian copula (Salinas et al. 2019), a deep factor generative model, and normalizing flows (Rasul et al. 2020), each with stated flexibility or scalability limits.The flow model requires as many latent dimensions as input dimensions, while the deep factor model uses linear combinations of global series.Source: §2 · pp. 2–3
  • VRNN-style methods apply recurrent models in input space and propagate multi-step predictions through the full model, unlike the paper's low-dimensional latent-space approach.The cited discussion also notes that noise at every time step may make VRNNs struggle with long-term dependencies.Source: §2 · p. 3

3 Problem Setup and Methodology

The section formulates high-dimensional multivariate forecasting and develops a nonlinear encoder–temporal-model–decoder architecture. Its probabilistic extension forecasts in latent space and decodes samples into input-space predictions.

3 Problem Setup and Methodology

  • The task is to forecast τ future values of high-dimensional series from their history, including the conditional distribution of those future vectors.The method turns multi-step forecasting into rolling prediction, feeding each prediction forward to produce the next.Source: §3 · p. 3

3.1 Point prediction

  • TLAE encodes multivariate series into a lower-dimensional latent space, forecasts latent states with a temporal model, and decodes them to reconstruct or predict the input series.The encoder and decoder are nonlinear, and the latent variables are parameterized by networks for end-to-end training.Source: §3.1 · p. 4

3.2 Probabilistic prediction

  • For probabilistic forecasting, TLAE imposes a conditional Gaussian model on latent states and decodes sampled latent predictions into potentially complex input-space distributions.The model fixes latent variance to one and uses a reparameterization trick to back-propagate through samples.Source: §3.2 · p. 5

4 Experiments

The experiments report strong point-forecasting and probabilistic results for TLAE, then examine how batch size, latent regularization, and embedding dimension affect prediction performance.

4.1 Point estimation

  • TLAE outperforms other global modeling and factorization methods in 8/9 dataset-metric combinations; gains on traffic and electricity reach 50%.The comparison covers WAPE, MAPE, and SMAPE; the reported scores average three runs.Source: §4.1 · p. 6, Table 1 · p. 7
  • Traffic and electricity use latent dimensions of 16 and 32, respectively, versus 64 in Sen, Yu, and Dhillon (2019), which the authors say indicates a more compact representation.The model also does not need retraining during testing.Source: §4.1 · p. 6

4.2 Probabilistic estimation

  • TLAE outperforms competing methods in 7/10 probabilistic dataset-metric combinations, with significant gains on Solar, Traffic, and Taxi.Table 2 compares CRPS-Sum and MSE; lower scores indicate better performance.Source: §4.2 · p. 6, Table 2 · p. 7

4.3 Hyper parameter sensitivity & ablation study

  • All three metrics improve as batch size increases in the tested range, supporting a balance between directly encoded and LSTM-predicted latent subsequences.The experiment varies batch size from L + 1 to 3L with L set to 194.Source: §4.3 · p. 6
  • With batch size fixed at 2L, the best tested regularization range is λ = 1e-4 to 1e-2; very small λ yields poor performance.The authors report that larger λ shifts dominance toward the latent constraint rather than reconstruction.Source: §4.3 · p. 6
  • Increasing latent dimension slightly improves performance before it stabilizes, while linear encoder-decoder ablations perform worse than nonlinear TLAE but better than DeepGLO.Dimensions tested were 2, 4, 8, 16, and 32; the dimension experiment used 200 versus 1000 training epochs.Source: §4.3 · p. 7

5 Conclusion

TLAE combines nonlinear autoencoder mapping with LSTM-modeled latent temporal dynamics for high-dimensional multivariate forecasting. Experiments report superior performance to other state-of-the-art methods on several common datasets, with future work proposed on temporal models, tensor inputs, and local modeling.

5 Conclusion

  • TLAE combines flexible nonlinear autoencoder mapping with LSTM-modeled latent dynamics, enabling end-to-end training and complex predictive distributions through a nonlinear decoder.The formulation models the distribution in the latent embedding.Source: §5 · p. 7
  • Experiments show superior performance to other state-of-the-art methods on several common time-series datasets.Future directions include temporal models in the autoencoder, 3D tensor inputs, and combinations with local modeling.Source: §5 · p. 7

6 Supplementary Material

The supplement collects dataset and training details, complexity and runtime analysis, ablations, additional comparisons, visualizations, and a tensor-input extension. It documents dataset statistics and forecasting setups, including the use of smaller electricity and wiki datasets for probabilistic comparisons.

6 Supplementary Material

  • The supplement covers complexity and runtime, nonlinear-factorization ablations, additional comparisons, univariate-method comparisons, predictive visualizations, and tensor inputs for exogenous features.It also includes evaluation metrics, dataset details, and network architecture and learning procedures.Source: §6 · p. 10

6.1 Dataset details

  • Electricity and wiki use smaller datasets for probabilistic experiments, following prior probabilistic-method setups; larger versions support deterministic forecasting.The authors state that matching prior setups enables direct comparison and that prior probabilistic methods lacked scalability to larger data.Source: §6.1 · p. 10
  • The supplement describes rolling-window evaluation, with test windows excluded from training and prediction inputs.It also provides dataset descriptions for traffic, electricity, wiki, solar, and taxi series.Source: §6.1 · p. 10

6.2 Network architectures and learning

The supplement reports that TLAE scales favorably by operating its temporal model in a low-dimensional latent space, and provides experimental architecture, training, and runtime details. It also notes that broader hyperparameter search could improve reported results.

6.2 Network architectures and learning

  • Experiments use feed-forward encoder and decoder layers with ReLU activations, plus a four-layer LSTM with 32 hidden units per layer.The decoder's layer-size sequence reverses the encoder's; the supplement also describes validation-based architecture selection and training settings.Source: §6.2 · p. 11
  • 0.106/0.118/0.092 WAPE/MAPE/SMAPE were reported for traffic after 2,000 epochs, with sufficiently more accurate performance than Table 1.The authors suggest additional epochs might improve results for some datasets.Source: §6.2 · p. 11
  • The experiments use λ = 0.5 for deterministic models and λ = 0.005 for probabilistic models to balance reconstruction and regularization loss magnitudes.The input reconstruction loss is ℓ1, while latent regularization uses ℓ2 for deterministic models and negative log likelihood for probabilistic models.Source: §6.2 · p. 11
  • The authors did not conduct a thorough architecture and hyperparameter search, and state that further tuning may improve performance.They note that wiki used relatively few epochs and identify more wiki hyperparameter settings as future work.Source: §6.2 · pp. 11–12

6.3 Model complexity and run times

  • O((nd + bd)L) complexity is linear in input dimension when b, d, and L are much smaller than n, versus n^2 parameters for direct input-domain distribution modeling.The temporal model runs in the lower-dimensional latent space.Source: §6.3 · p. 12
  • 115,084-series large wiki training took only marginally longer per epoch than small wiki with 2,000 series, indicating scalability.The reported runtimes used PyTorch and eight CPUs; the authors expected GPUs to reduce them substantially.Source: §6.3 · p. 12, Table 5 · p. 12

6.4 Benefit of nonlinear factorization - Ablation Study

The ablation compares linear and nonlinear encoders on traffic and reports significantly better prediction accuracy for the nonlinear encoder at the same hidden dimension. The authors note that these ablation scores exceed their main-paper results because the later runs used more batches and GPUs.

6.4 Benefit of nonlinear factorization - Ablation Study

  • Significantly better prediction accuracy was achieved by the ReLU encoder than by the linear encoder at the same hidden dimension.Table 6 compares linear and nonlinear encoders using WAPE, MAPE, and SMAPE, where lower scores are better.Source: Table 6 · p. 12, §6.4 · p. 12
  • The ablation targets nonlinear embedding, which the authors identify as missing from TRMF and DeepGLO global factorization approaches.The comparison uses the traffic dataset.Source: §6.4 · p. 12
  • The ablation's metric scores were somewhat better than the main-paper results because it used more batches and GPUs in later experiments.The authors say the comparison's point is the nonlinear encoder's advantage under the same training setup and architecture.Source: §6.4 · p. 12

6.5 Additional comparisons, metrics, and result details

TLAE outperforms Gaussian-process methods on CRPS for solar, traffic, and taxi, and is comparable on electricity and wiki. It also shows lower MSE on almost all datasets and more consistent predictions across runs.

6.5 Additional comparisons, metrics, and result details

  • TLAE achieves better CRPS than GP methods on solar, traffic, and taxi, and comparable CRPS on electricity and wiki.The comparison reports additional CRPS and MSE tables with standard deviations over multiple runs.Source: §6.5 · pp. 12–13
  • TLAE outperforms GP methods on MSE for almost all datasets.MSE predictions use the latent Gaussian distribution’s sample mean.Source: §6.5 · pp. 12–13
  • On taxi data, GP models show very high prediction variance, while TLAE’s predictions are more consistent across runs.The GP’s MSE variance across runs is much higher than its mean; TLAE’s is described as consistent.Source: §6.5 · p. 13

6.6 Comparison to univariate methods

Against univariate forecasting methods, TLAE achieves quantile accuracy comparable to the state-of-the-art Transformer while modeling the full distribution. The comparison uses quantile loss for compatibility with prior work, though it evaluates only two quantiles.

6.6 Comparison to univariate methods

  • The comparison uses ρ-quantile loss to match prior work, although it evaluates only two quantiles rather than the full predictive distribution.TLAE is trained to model the entire distribution, unlike methods targeted at specific quantiles.Source: §6.6 · p. 13
  • TLAE obtains scores comparable to the state-of-the-art univariate Transformer on the reported quantile metrics.The comparison reports ρ-quantile metrics for ρ = 0.5 and ρ = 0.9.Source: §6.6 · p. 13
  • The compared DeepAR, DeepState, and Transformer models use external features that TLAE does not include.The authors say those features likely have a strong influence on overall performance and describe TLAE as complementary to univariate methods.Source: §6.6 · p. 13

6.7 Visualization - of losses, predictions, and predictive distributions

The visualizations show training convergence, latent variables capturing global trends, and forecasts that represent local variation and changing uncertainty. Joint prediction samples also exhibit nonlinear and multimodal relationships between series.

Visualization of losses

Visualization of latent variables and predictions

  • Latent-variable plots show shared global repeating patterns, while forecasts capture both global trends and local variation in individual series.The prediction displays include 90% intervals, with actual series and predictions shown in blue and orange.Source: §6.7 · p. 13

Visualization of predictive distributions

  • Output predictive distributions vary across time points and series despite a fixed latent-space variance.The paper attributes this capability to decoding through a nonlinear network from a latent state that progresses over time.Source: §6.7 · p. 16

Visualization of joint predictive distributions

  • Joint prediction samples show some strongly correlated series pairs, while most plotted pairs exhibit multimodal distributions and nonlinear relationships.The authors say the captured marginal joint distributions are more complex than a joint Gaussian.Source: §6.7 · p. 16

6.8 Extension to tensor data for incorporating exogenous variables

The extension represents series with associated exogenous features as a three-dimensional tensor, while keeping the latent space unchanged. It also describes modified prediction losses and options for supplying or forecasting future features in multi-step prediction.

6.8 Extension to tensor data for incorporating exogenous variables

  • Exogenous features can be represented with each series in a three-dimensional tensor, extending the two-dimensional series-by-time input.The extension is discussed but was not experimentally evaluated.Source: §6.8 · p. 16
  • The encoder can map the tensor to the same d-dimensional latent space, with flattening or CNN-style architectures proposed for encoding its series-feature relationships.Temporal convolutions or recurrent structures are also proposed for incorporating prior time points.Source: §6.8 · p. 16

Updated losses

  • For input-space prediction loss, masking can exclude future exogenous-feature elements when those features need not be forecast.The latent prediction and reconstruction losses remain unchanged.Source: Updated losses · pp. 16–17

Multistep prediction

  • For multi-step prediction, future exogenous features may be supplied when known, forecast separately, or predicted by the model itself.A multi-time-step output alternative requires separate models for different forecast horizons.Source: Multistep prediction · p. 17

6.9 Evaluation metrics

Section 6.9 describes deterministic and probabilistic forecast metrics, including how CRPS measures predictive-distribution fit and how it is evaluated from samples.

Deterministic metrics

  • WAPE, MAPE, SMAPE, and MSE are used as deterministic forecasting metrics.For probabilistic forecasts, the predicted distribution's mean is used as the prediction for these metrics.Source: Deterministic metrics · p. 17

Probabilistic metrics

  • CRPS measures the mean discrepancy between the predictive distribution and an observation by integrating the squared difference between their CDFs.The observation CDF is represented by a Heaviside function; Figure 19 illustrates the discrepancy being compared.Source: Probabilistic metrics · p. 17, Figure 19 · p. 30
  • Because the predictive distribution and CDF lack a closed form, CRPS is estimated from 1,000 predictive samples using 20 quantiles.The calculation uses the alternative pinball-loss definition, which approximates the integral through sampled quantiles.Source: Probabilistic metrics · p. 17
  • CRPS-sum evaluates joint effects by computing CRPS on the sum of observed target values and statistics on summed predictive samples.Source: Probabilistic metrics · p. 18
  • The ρ-quantile loss scores an empirical predicted quantile against the observed value using an indicator for whether the prediction is at or below the observation.Source: Probabilistic metrics · p. 18

Related papers

Loading 2101.10460v1…