Source-linked AI summary
Deep and Confident Prediction for Time Series at Uber
Lingxue Zhu, Nikolay Laptev
TL;DR
Reliable uncertainty estimation is needed for time-series forecasting used in applications such as special-event trip prediction and anomaly detection, but classical models are difficult to tune, scale, and extend with exogenous variables. The paper proposes an end-to-end Bayesian neural approach using MC dropout and model-misspecification uncertainty, and applies it to Uber forecasting and anomaly detection. Experiments report improved prediction accuracy, large-scale anomaly-detection performance, and 95% uncertainty coverage.
Problem
Time-series forecasts need reliable uncertainty estimates, while classical models are difficult to tune, scale, and incorporate exogenous variables into.
Method
The paper uses an end-to-end Bayesian neural model with MC dropout and an encoder-decoder to estimate model, inherent-noise, and model-misspecification uncertainty.
Results
The model improves prediction accuracy on completed-trip data, achieves 100% recall and 80.95% precision on an annotated anomaly set, and provides 95% uncertainty coverage.
Takeaways & Limitations
The framework supports generic, scalable uncertainty estimation for neural forecasts and large-scale anomaly detection at Uber.
Takeaways & Limitations
The dropout probability is not selected adaptively because doing so requires modifying the training phase, although the estimates are robust within a reasonable range.
Abstract
from arXiv · showhide
Reliable uncertainty estimation for time series prediction is critical in many fields, including physics, biology, and manufacturing. At Uber, probabilistic time series forecasting is used for robust prediction of number of trips during special events, driver incentive allocation, as well as real-time anomaly detection across millions of metrics. Classical time series models are often used in conjunction with a probabilistic formulation for uncertainty estimation. However, such models are hard to tune, scale, and add exogenous variables to. Motivated by the recent resurgence of Long Short Term Memory networks, we propose a novel end-to-end Bayesian deep model that provides time series prediction along with uncertainty estimation. We provide detailed experiments of the proposed solution on completed trips data, and successfully apply it to large-scale time series anomaly detection at Uber.
1. Introduction
The paper addresses unreliable uncertainty estimates in time-series forecasting, especially when external factors drive high variance. It proposes an end-to-end Bayesian neural model that estimates multiple uncertainty sources and supports large-scale anomaly detection.
- 1. Introduction: High-variance events make accurate forecasting and uncertainty estimation difficult because weather, population growth, and driver incentives affect predictions.Classical models also make exogenous variables difficult to incorporate and require manual tuning of model and uncertainty parameters.
- 1. Introduction: LSTM models offer end-to-end modeling, easier incorporation of exogenous variables, and automatic feature extraction for complex nonlinear interactions.Prior work reported that neural forecasting can outperform classical methods for long, interdependent time series.
- 1. Introduction: Neural-network uncertainty estimation remains open, and missing uncertainty information can produce high false anomaly rates during holidays.Uncertainty helps assess how much to trust a forecast, particularly for anomaly detection.
- 1. Introduction: The proposed end-to-end Bayesian neural model quantifies prediction uncertainty and applies it to large-scale time-series anomaly detection.The framework is presented as generic and scalable for deep prediction models.
- 1. Introduction: The framework distinguishes model uncertainty, inherent noise, and model misspecification, with misspecification uncertainty identified as previously overlooked.An encoder-decoder is proposed to incorporate misspecification uncertainty, which concerns testing samples from a different population than the training set.
- 1. Introduction: At Uber, Bayesian neural networks with uncertainty estimation are used for anomaly detection at scale to improve performance.The paper reports this as a real-world use case motivated by the forecasting problem.
2. Related Works
Related work contrasts classical forecasting and Bayesian inference approaches with a practical need for scalable uncertainty estimation that preserves existing neural-network architectures. The paper adopts MC dropout as a generic route to time-series prediction and anomaly detection.
- 2. Related Works: Classical time-series models require manual tuning and may suffer from dimensionality problems or frequent retraining when incorporating exogenous variables.These constraints motivate alternatives for high-dimensional forecasting settings.
- 2. Related Works: LSTM-based modeling supports end-to-end learning, exogenous variables, automatic feature extraction, and nonlinear interactions for complex extreme events.These capabilities have been associated with improved modeling of long, interdependent time series.
- 2.2. Bayesian Neural Networks: Existing neural-network uncertainty methods often modify the loss, training algorithm, or architecture and may introduce additional parameters that hinder scalability.An out-of-the-box method applicable to previously trained models is preferred in practice.
- 2.2. Bayesian Neural Networks: MC dropout applies stochastic dropout after hidden layers, treating outputs as approximate samples from the posterior predictive distribution.The resulting sample variance estimates model uncertainty without changing the existing model architecture.
- 2.2. Bayesian Neural Networks: The paper extends MC dropout to time-series prediction and large-scale anomaly detection because it is generic, easy to implement, and directly applicable to existing neural networks.The authors describe this as an application of the framework to real-world problems at large scale.
3. Method
The method estimates uncertainty around a neural-network forecast and uses the resulting prediction interval for downstream decisions such as anomaly alerts. Its framework is designed to account for multiple uncertainty sources and generalize across neural-network architectures.
- 3. Method: Given a trained neural network and a new sample, the method produces a forecast and estimates its prediction standard error.The standard error is used to construct an approximate prediction interval.
- 3. Method: The prediction interval uses an upper α/2 standard-Normal quantile to define the uncertainty bounds.The supplied method text identifies zα/2 as that quantile.
- 3. Method: In anomaly detection, an alert fires when the observed value falls outside the constructed 95% interval.Underestimating the prediction standard error can lead to high false-positive rates.
- 3. Method: The uncertainty-estimation algorithm accounts for three different prediction-uncertainty sources and can be generalized to any neural-network architecture.The paper separately presents the algorithm and its neural-network design for time-series prediction.
3.1. Prediction Uncertainty
The paper decomposes predictive uncertainty into model uncertainty, inherent noise, and model misspecification, estimating these components with Bayesian inference, MC dropout, an encoder-decoder, and validation residuals.
- Uncertainty decomposition: Prediction variance reflects uncertainty in model parameters and noise in the data-generating process, but anomaly detection also requires accounting for model misspecification.The misspecification term addresses test samples whose patterns differ substantially from training data.
- Model uncertainty: MC dropout approximates model uncertainty by repeatedly applying stochastic dropout and using the sample variance of the resulting predictions.The method is applied to the neural network during inference without requiring exact posterior inference.
- Model misspecification: The encoder-decoder captures representative time-series features so encoding quality can signal when test samples differ from training patterns.The encoder produces a latent embedding, while distances or reconstruction quality indicate potential distributional mismatch.
- Inherent noise: Inherent noise is estimated from residual sum of squares on an independent held-out validation set and combined with MC-dropout uncertainty.The final inference procedure combines these estimates into predictive uncertainty and prediction intervals.
- Estimator behavior: The validation-based noise estimator is asymptotically unbiased and conservatively overestimates noise in finite samples.Its training-data variance component decreases as the training sample size increases and approaches zero as N → ∞.
3.2. Model Design
The model first learns time-series embeddings with an LSTM encoder-decoder, then uses those embeddings and optional external features in a prediction network for forecasting.
- Architecture: The architecture has an encoder-decoder component for learning inherent time-series patterns and a prediction network for forecasting.The encoder-decoder is trained before the prediction model.
- Encoder-decoder: A two-layer LSTM encoder-decoder is pretrained to produce useful predictive embeddings and capture unusual inputs in embedded space.The decoder reconstructs future timestamps from the encoder state with guidance from recent observations.
- Prediction network: The prediction network uses the encoder’s learned embedding and concatenates available external features before forecasting one or more future timestamps.The prediction network is implemented as a multi-layer perceptron.
- Inference: Inference uses only the encoder and prediction network, while MC dropout estimates uncertainty across both components.The resulting uncertainty combines model and misspecification uncertainty with inherent noise in the prediction interval.
- Inference settings: The main inference hyperparameters are dropout probability p and iteration count B, with a few hundred iterations usually sufficient for stable uncertainty estimates.The authors report relative stability across a range of dropout probabilities.
4. Evaluation
The model improves trip-forecast accuracy across eight cities, calibrates predictive uncertainty, and supports interpretable uncertainty analysis and large-scale anomaly detection at Uber.
- Prediction performance.: The evaluation uses four years of daily trips across eight U.S. and Canadian cities, with three years for training, four months for validation, and eight months for testing.Samples use 28 preceding days to forecast the next day, with log and trend transformations.
- Prediction performance.: 26% average SMAPE improvement over eight cities comes from the vanilla LSTM, while the proposed model adds another 36% improvement through encoder-decoder modeling and holiday features.The comparisons use Last-Day, QRF, LSTM, and the proposed model.
- Prediction performance.: The proposed model produces accurate predictions on regular days and during holiday seasons in the San Francisco testing example.Figure 2 shows true values, predictions, and a 95% prediction band; exact values are anonymized.
- Uncertainty estimation.: Adding encoder dropout raises empirical coverage from 78% to 90%, while including inherent noise centers the final 95% predictive intervals around 95% coverage.The compared configurations are PredNet, Enc+Pred, and Enc+Pred+Noise.
- Uncertainty estimation.: New Year’s Eve has significantly higher estimated predictive uncertainty than the other six U.S. holidays.The authors relate this pattern to New Year’s Eve being the most difficult holiday to predict.
- Embedding features and anomaly detection.: The encoder embedding separates weekdays and weekends, while the framework achieves 100% recall and 80.95% precision on 100 manually annotated metrics.Across all metrics, precision improves by 4% over the previous ad-hoc solution; inference adds only a small computational overhead within ten milliseconds per metric.
5. Conclusion
The paper presents a scalable neural architecture that estimates uncertainty for forecasts and applies it to holiday analysis and anomaly detection at Uber. The authors report 95% uncertainty coverage and a 4% precision improvement for anomaly detection, while identifying neural-network debugging as future work.
- Conclusion: The framework combines MC dropout and a model-misspecification distribution to provide scalable uncertainty estimation with 95% uncertainty coverage.It is applicable to any neural network without modifying the underlying architecture.
- Conclusion: The uncertainty estimate identifies New Year’s Eve as the most uncertain special event and improves anomaly-detection precision during high-uncertainty events by 4%.The precision improvement is reported as substantial at Uber’s scale of tracked metrics.
- Conclusion: Future work will use uncertainty information for neural-network debugging during periods of high error.