PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond expectation: Deep joint mean and quantile regression for spatio-temporal problems

Filipe Rodrigues, Francisco C. Pereira

arXiv:1808.08798v1stat.MLcs.LGcs.NE

TL;DR

Deep spatio-temporal models often focus on conditional expectations, although many decisions require calibrated uncertainty estimates. This paper jointly predicts the conditional mean and multiple quantiles with a multi-output deep-learning approach. On transportation datasets, it reports fewer quantile crossings, stronger quantile predictions, and improved mean predictions.

  • Problem

    Deep-learning approaches to spatio-temporal problems often focus on conditional expectations and lack calibrated uncertainty estimates needed for decisions.

    Source: §I · p. 1

  • Method

    A multi-output deep neural network jointly models the conditional expectation and multiple conditional quantiles using convolutional LSTM layers.

    Source: §I · p. 2, §IV · p. 3

  • Results

    The approach addresses quantile crossings, outperforms state-of-the-art quantile regression methods on quantile predictions, and can improve mean predictions.

    Source: §I · pp. 1–2

  • Takeaways & Limitations

    Jointly predicting means and quantiles provides a fuller predictive-density description with little computational overhead and can capture heteroscedastic properties.

    Source: §VI · pp. 10–11

Abstract

from arXiv · show

Spatio-temporal problems are ubiquitous and of vital importance in many research fields. Despite the potential already demonstrated by deep learning methods in modeling spatio-temporal data, typical approaches tend to focus solely on conditional expectations of the output variables being modeled. In this paper, we propose a multi-output multi-quantile deep learning approach for jointly modeling several conditional quantiles together with the conditional expectation as a way to provide a more complete "picture" of the predictive density in spatio-temporal problems. Using two large-scale datasets from the transportation domain, we empirically demonstrate that, by approaching the quantile regression problem from a multi-task learning perspective, it is possible to solve the embarrassing quantile crossings problem, while simultaneously significantly outperforming state-of-the-art quantile regression methods. Moreover, we show that jointly modeling the mean and several conditional quantiles not only provides a rich description about the predictive density that can capture heteroscedastic properties at a neglectable computational overhead, but also leads to improved predictions of the conditional expectation due to the extra information and a regularization effect induced by the added quantiles.

I. INTRODUCTION

Deep spatio-temporal models often predict conditional means without calibrated uncertainty, although decisions in applications such as transportation require fuller forecasts. The paper jointly predicts means and multiple quantiles, reporting reduced quantile crossings, stronger quantile predictions, and improved mean predictions.

I. INTRODUCTION

  • Deep learning spatio-temporal models often focus on conditional expectations, while decisions such as mobility planning need calibrated uncertainty estimates.The introduction also cites risk assessment, energy regulation, and product sales as settings where confidence in predictions matters.Source: §I · p. 1
  • The evaluation focuses on taxi demand and traffic speed, while the authors state that the approach also extends to univariate time-series forecasting.Source: §I · pp. 1–2
  • The proposed multi-output architecture uses convolutional LSTM layers to jointly predict the mean and several quantiles across spatial locations.Joint modeling aims to provide a more complete description of the predictive density.Source: §I · pp. 1–2
  • The approach produces better quantiles than state-of-the-art quantile regression and other uncertainty methods, including Monte Carlo dropout, in taxi-demand and traffic-speed forecasting.The experiments use two large-scale transportation datasets.Source: §I · pp. 1–2
  • A multi-task perspective can address quantile crossings by enforcing consistency among jointly estimated quantiles.The paper presents related quantile tasks as a way to use their shared structure.Source: §I · pp. 1–2
  • Adding quantile losses can provide extra information and a regularization effect that leads to better mean predictions.The paper reports this effect empirically.Source: §I · p. 2

II. LITERATURE REVIEW

The literature review contrasts distributional assumptions in heteroscedastic models with quantile regression’s flexible description of predictive distributions. It also highlights quantile crossings and sparse data at the extremes as challenges.

II. LITERATURE REVIEW

  • Heteroscedastic models commonly retain a Gaussian predictive distribution, while variance regression can use additional predictors relevant to uncertainty.The review notes that real phenomena may have unobserved components that produce non-constant variance, skew, multimodality, or long tails.Source: §II · p. 2
  • Estimating many quantiles can approximate a predictive distribution without imposing a parametric density, but independently estimated quantiles may cross.Prior approaches address crossings with constraints; one cited RKHS model guarantees their absence under homoscedasticity assumptions on individual quantile functions.Source: §II · p. 2
  • Extreme quantiles face sparse data, making models unstable and difficult to compare; consequently, wide prediction intervals remain an open question in the literature.The review identifies quantiles below 5% and above 95% as examples of extremes.Source: §II · p. 2
  • The review distinguishes methods for constructing prediction intervals from approaches that model uncertainty in network weights, including Monte Carlo dropout.Monte Carlo dropout is used as an experimental baseline, while this paper targets the predictive distribution directly.Source: §II · pp. 2–3

III. PROBLEM FORMULATION

The problem formulation predicts future values across a spatial grid using spatio-temporal observations. It proposes jointly estimating conditional means and multiple quantiles to describe the predictive density and reduce quantile crossings.

III. PROBLEM FORMULATION

  • Spatio-temporal forecasting uses observations on an M × N grid over time to predict future values at each grid location.Source: §III · p. 3
  • Conditional quantiles complement the conditional expectation when a fuller predictive-density description is needed, particularly for questions about distribution tails.Source: §III · p. 3
  • Lower and upper quantiles define a prediction interval whose probability level is α = τu − τl.Source: §III · p. 3
  • Rather than fit J separate functions, the proposed multi-task formulation jointly predicts the expectation and multiple quantiles, whose latent spatio-temporal features are expected to be similar.The paper states that this joint approach can reduce quantile crossings and prediction error.Source: §III · p. 3

IV. PROPOSED APPROACH

The proposed approach uses stacked ConvLSTM layers to build shared spatial-temporal representations and jointly predict the conditional mean and multiple quantiles. Shared parameters and quantile-specific tilted losses connect the tasks, addressing quantile crossings and potentially improving mean forecasts.

IV. PROPOSED APPROACH

  • ConvLSTM represents cell and hidden states as spatial tensors and uses convolutions so each grid cell can incorporate local-neighbor information across space and time.The paper notes that its simplified ConvLSTM omits connections between gates and the cell state.Source: §IV · pp. 3–4
  • Stacked ConvLSTM layers transform recent observations into a shared latent representation, which the output layer maps to a mean and multiple quantile forecasts for each grid cell.The output layer uses weights and biases shared among grid cells to produce the 1 + J outputs.Source: §IV · p. 4
  • The model fits the conditional mean with summed squared error and fits each conditional quantile by minimizing the corresponding tilted loss.For a given quantile, the tilted loss weights residuals asymmetrically according to their sign and τ.Source: §IV · p. 5
  • Hard parameter sharing across the mean and quantile tasks makes the additional quantile losses act as a regularizer that can reduce overfitting and improve mean forecasts.The shared latent structures and predictors are described as a multi-task learning approach.Source: §IV · pp. 4–5
  • The shared ConvLSTM representation and hard-shared output layer are used to address quantile crossings without adding explicit crossing constraints.The approach limits the flexibility of independently estimated quantile models.Source: §IV · p. 5

V. EXPERIMENTS

The experiments include comparisons of the proposed approach with other methods, with a Motorcycle dataset example contrasting independent models and joint quantile prediction. The supplied passages identify a results table for that dataset but do not report its values.

V. EXPERIMENTS

  • The experiments are described as comparing the proposed approach with several state-of-the-art methods across different datasets.The paper emphasizes spatio-temporal data in its experiments.Source: §V · p. 5
  • Figure 5 compares independent models with the joint quantiles model on the Motorcycle dataset, showing 5%, 20%, 80%, and 95% quantiles for 60% and 90% prediction intervals.Source: Figure 5 · p. 6
  • Table I is identified as reporting results for the Motorcycle dataset.Source: Table I · p. 6

A. Motorcycle dataset

On the motorcycle dataset, DeepJMQR jointly predicts the mean and four conditional quantiles, reducing quantile crossings and improving mean predictions compared with independent models. Joint quantile learning also shows a regularization effect in the training and test losses.

A. Motorcycle dataset

  • The motorcycle dataset contains 133 acceleration-versus-time measurements and is used to examine heteroscedastic noise and compare DeepJMQR with independent quantile models.A two-hidden-layer network uses 50 tanh units followed by 10 linear units; outputs are either one unit or J + 1 units for independent or joint modeling.Source: A. Motorcycle dataset · p. 6
  • The comparison uses a 2/3 training and 1/3 evaluation split, four quantiles (5%, 20%, 80%, 95%), and 60% and 90% prediction intervals.Quantile quality is evaluated by summed test tilted loss, crossing counts, and crossing loss; mean predictions use MAE and RMSE.Source: A. Motorcycle dataset · p. 6
  • DeepJMQR significantly reduces quantile crossing compared with independent models, while joint modeling also reduces mean MAE and RMSE relative to a single independent mean model.The authors attribute the crossing reduction to multi-task learning enforcing consistency among quantiles.Source: A. Motorcycle dataset · pp. 6–7
  • With longer training, independent models’ test loss begins to rise while train loss continues to fall; DeepJMQR is reported as robust to overfitting.The authors attribute this regularization effect to modeling a fuller predictive distribution with the same underlying network structure.Source: A. Motorcycle dataset · p. 7, Figure 6 · p. 7

B. Taxi demand in NYC

In the NYC taxi-demand experiment, DeepJMQR produced accurate quantiles with very few crossings and improved conditional-mean predictions. Its ConvLSTM-based intervals also achieved target coverage with narrower bounds.

B. Taxi demand in NYC

  • The experiment used 1.1 billion NYC taxi trips around Manhattan from January through June 2016 to forecast pickups one hour ahead across a 12x12 grid.The data were split into three training months, one validation month, and two test months.Source: B. Taxi demand in NYC · p. 7, Figure 7 · p. 7
  • DeepJMQR’s ConvLSTM quantiles had the lowest test tilted loss, while joint modeling kept crossings practically zero versus almost 300,000 average cases for Indep. DL.MCdropout quantiles never crossed by construction, but their tilted loss was significantly higher than DeepJMQR’s.Source: B. Taxi demand in NYC · p. 8, Table II · p. 8
  • Mean predictions improved significantly over other approaches, with MAE and RMSE p-values of 1.17 × 10−5 and 2.81 × 10−4, respectively.The passage attributes this improvement to jointly modeling the mean and multiple conditional quantiles.Source: B. Taxi demand in NYC · p. 8
  • ConvLSTM-based approaches achieved target prediction-interval coverage with narrower bounds; ICP measures the fraction inside an interval, while MIL measures its average length.The authors note that tighter bounds at the same coverage can mean fewer taxis need to be on call in an area.Source: B. Taxi demand in NYC · pp. 8–9, Table III · p. 8

C. Traffic speeds in Copenhagen

On Copenhagen traffic-speed forecasting, DeepJMQR produced accurate quantiles without test-set crossings and improved mean predictions with little added training time. Its quantiles also capture changing uncertainty across day and night.

C. Traffic speeds in Copenhagen

  • The study used six months of crowdsourced Google traffic speeds from nine Copenhagen road segments, with 5-minute observations, to forecast the next 5-minute interval.The data covered January to June 2015 and were split into three training months, one validation month, and two test months.Source: C. Traffic speeds in Copenhagen · p. 9
  • DeepJMQR had by far the lowest test tilted loss and eliminated quantile crossings, with zero test cases; average tilted loss rose only from 55.25 to 55.33.The authors describe a trade-off between tilted loss and crossing control when multi-task learning constrains quantile flexibility.Source: C. Traffic speeds in Copenhagen · pp. 9–10, Table IV · p. 10
  • DeepJMQR significantly reduced mean-prediction MAE and RMSE versus all other approaches, with only a small increase in training time: 28.19 versus 27.69 minutes for a mean-only model.The reported p-values versus Indep. DL were 2.38×10−18 for MAE and 1.49×10−30 for RMSE.Source: C. Traffic speeds in Copenhagen · p. 10
  • Deep learning and vector-valued RKHS approaches produced the tightest 90% and 80% intervals, while JQR (RKHS) had coverage below the respective targets.The authors caution that lower-than-target coverage can be problematic.Source: C. Traffic speeds in Copenhagen · p. 10, Table V · p. 10
  • DeepJMQR captured heteroscedasticity that linear quantile regression missed, producing narrower daytime intervals and wider nighttime intervals when crowdsourced speed data were less reliable.The passage links lower nighttime data quality to fewer crowdsourced samples and higher speed variances.Source: C. Traffic speeds in Copenhagen · pp. 9–10

VI. CONCLUSION

DeepJMQR jointly models conditional quantiles and the conditional expectation to describe predictive densities, and significantly outperforms state-of-the-art quantile-regression methods and other mean-prediction methods.

VI. CONCLUSION

  • The proposed multi-output, multi-quantile approach jointly models conditional quantiles with the conditional expectation to describe predictive density.Source: §VI · p. 10
  • Jointly predicting the mean and multiple quantiles addresses quantile crossings and reduces forecasting error versus a similar mean-only network.The authors attribute the reduction to extra quantile information and a regularization effect; the added predictive-density description has almost no computational overhead.Source: §VI · p. 11
  • DeepJMQR significantly outperforms state-of-the-art quantile-regression methods and other methods for predicting the mean.Source: §VI · p. 11

Related papers

Loading 1808.08798v1…