Source-linked AI summary
A Multi-Horizon Quantile Recurrent Forecaster
Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, Dhruv Madeka
TL;DR
Large-scale forecasting must handle multiple related series, covariates, long horizons, shifting seasonality, planned events, and predictive uncertainty. The paper proposes MQ-R(C)NN, a sequence-to-sequence framework for multi-horizon quantile forecasting, and reports consistently best accuracy across horizons in Amazon demand experiments.
Problem
Large-scale forecasting requires multi-step probabilistic predictions across related series with dynamic, static, and future covariates, long dependencies, and nonlinear relationships.
Method
MQ-R(C)NN combines sequence-to-sequence neural networks, quantile regression, and direct multi-horizon forecasting with forking-sequences training.
Results
MQ RNN achieved consistently the best accuracy across all horizons in the Amazon demand experiment, while quantile-based models stood out for P90QL.
Takeaways & Limitations
The framework supports probabilistic large-scale forecasting with temporal and static covariates, related series, shifting seasonality, planned events, and cold-starts.
Takeaways & Limitations
The paper does not discuss explicit multivariate forecasting or modeling the joint distribution of horizons.
Abstract
from arXiv · showhide
We propose a framework for general probabilistic multi-step time series regression. Specifically, we exploit the expressiveness and temporal nature of Sequence-to-Sequence Neural Networks (e.g. recurrent and convolutional structures), the nonparametric nature of Quantile Regression and the efficiency of Direct Multi-Horizon Forecasting. A new training scheme, *forking-sequences*, is designed for sequential nets to boost stability and performance. We show that the approach accommodates both temporal and static covariates, learning across multiple related series, shifting seasonality, future planned event spikes and cold-starts in real life large-scale forecasting. The performance of the framework is demonstrated in an application to predict the future demand of items sold on Amazon.com, and in a public probabilistic forecasting competition to predict electricity price and load.
1. Introduction
The paper addresses complex large-scale forecasting settings requiring multi-step probabilistic predictions across related series with diverse covariates. It proposes MQ-R(C)NN, a Seq2Seq multi-horizon quantile framework with forking-sequences training and support for future information, cross-series learning, and cold starts.
- Motivation: Real-world forecasting involves related series, dynamic and static covariates, known future events, long-term dependencies, and shifting seasonal patterns.These complexities extend beyond conventional single-series next-step forecasting.
- Motivation: Recursive multi-step forecasting feeds predictions back as inputs, potentially compounding errors and relying on autoregressive or Markovian assumptions.This contrasts with directly modeling multiple future horizons.
- Motivation: Probabilistic forecasting is needed when decisions depend on asymmetric over- and under-prediction costs rather than only the conditional mean.Quantile regression provides distribution-free probabilistic forecasts with sharp prediction intervals.
- Proposed framework: MQ-R(C)NN combines Seq2Seq recurrent or convolutional networks with quantile regression and direct multi-horizon forecasting for large-scale time series regression.Each series is treated as a sample in a shared model, enabling cross-series learning and cold-start forecasting for items with limited history.
- Contributions: Forking-sequences trains on all forecast-creation time points in one pass, improving the stability and performance of encoder-decoder recurrent networks or ConvNets.The framework also accommodates known future information, including shifting seasonality and events producing large spikes and dips.
2. Related Work
Prior work applied recurrent and convolutional neural networks to time-series point forecasting, while probabilistic approaches used parametric distribution outputs and likelihood training. Neural quantile forecasting addressed financial and stock-price prediction but did not use sequential networks.
- Neural point forecasting: RNNs and CNNs were applied to time-series point forecasting across load, univariate and multivariate, and financial applications.The cited studies included comparative RNN evaluation for short-term load forecasting, attention-based Seq2Seq models, and dilated CNNs.
- Probabilistic forecasting: DeepAR used an encoder-decoder architecture to directly output Negative Binomial parameters for probabilistic forecasting.Related approaches predicted Gaussian parameters, extending a strategy dating back to Bishop, 1994.
- Probabilistic forecasting: DeepAR was trained by maximizing likelihood with Teacher Forcing, feeding ground truth recursively during training.At prediction time, the model was fed a sample.
- Quantile forecasting: Neural quantile forecasting was used for financial returns and stock-price prediction, including calibrated Value-at-Risk from GARCH innovations.These approaches fed previously estimated quantiles or processed another model’s innovations, but neither used sequential nets.
3. Method
MQ-RNN combines quantile-loss training with a Seq2Seq architecture whose shared local decoders produce horizon-specific quantiles from future-aligned contexts. Forking-sequences training, target masking, and encoder extensions improve efficiency, stability, boundary learning, and long-dependency modeling.
- Quantile loss: Quantile Regression trains the model by minimizing total Quantile Loss across forecast creation times, horizons, and selected quantiles.At q = 0.5, Quantile Loss equals Mean Absolute Error and is minimized by the predictive median; components can receive different weights.
- MQ-RNN architecture: MQ-RNN uses an LSTM encoder, a global MLP for horizon-specific and shared contexts, and a shared local MLP that outputs quantiles for each horizon.The local MLP combines each horizon’s future input with the two global contexts, while sharing parameters across horizons.
- MQ-RNN architecture: The local MLP aligns future seasonality and events and enables sharp spiky forecasts, while recursive prediction is unnecessary and expensive.The horizon-specific context carries awareness of temporal distance between the forecast creation time and each horizon.
- Forking-sequences training: Forking-sequences places shared-parameter decoders at encoder time points, letting each time series serve as one training sample without data augmentation.Jointly updating correlated forecast-creation-time tasks stabilizes optimization and reduces training time.
- Target masking: Target masking preserves learning from available short-horizon targets near the training boundary when complete multi-horizon targets are unavailable.Error terms after the observed target boundary are masked while shared parameters continue learning from partial targets.
- Encoder extensions: Encoder extensions address long dependencies: NARX-style recurrent inputs improve over vanilla LSTM, while lagged series features perform even better in experiments.The lagged-feature approach feeds past series values alongside the current value into the recurrent layer, effectively creating input skip-connections.
4. Application
The framework is applied to large-scale Amazon demand forecasting and GEFCom2014 electricity price and load forecasting. MQ-RNN consistently performs best across Amazon forecast horizons and achieves competition-winning results with limited tuning.
- Amazon demand forecasting: MQ-RNN efficiently forecasts millions of time series at industrial scale, demonstrated first on weekly demand from around 60,000 Amazon products across categories.The dataset covers US marketplace products from 2012 to 2017, with data before 2016 used for training.
- Amazon demand forecasting: Across all Amazon forecast horizons, MQ-RNN with the proposed structural improvements achieves the best accuracy, while the cut variant performs worse.The cut model also has a more volatile training-loss curve that flattens early.
- GEFCom2014 forecasting: The framework is applied to probabilistic, multi-horizon electricity price and load forecasting tasks from GEFCom2014, using future-horizon information where available.MQ-RNN predicts five quantiles and linearly interpolates them to obtain all 99 evaluated quantiles.
- GEFCom2014 forecasting: MQ-RNN would have achieved 1st place in GEFCom2014 electricity price forecasting without intensive tuning.The price task forecasts 24 hours ahead using load forecasts and calendar-based features.
- GEFCom2014 forecasting: 7.43 average quantile loss was achieved for electricity load forecasting, compared with 7.45, 7.51, and 7.83 for the top three competitors.The task required hourly load-distribution forecasts 744 hours into the future; the encoder operated at daily grain while the decoder remained hourly.
5. Conclusion
The paper presents a general probabilistic time series regression framework whose novel components contribute to performance over state-of-the-art methods, with relevance to large-scale forecasting and encoder-decoder architectures. Explicit multivariate forecasting and joint horizon-distribution modeling remain extensions for future work.
- The framework’s novel components each contribute to final performance over state-of-the-art methods.
- The findings inform practical large-scale forecasting applications and encoder-decoder-style deep learning architectures.
- Explicit multivariate forecasting and modeling the joint distribution of horizons are identified as extensions not discussed in this work.