Source-linked AI summary
Uncertainty Estimation with Deep Learning for Rainfall-Runoff Modelling
Daniel Klotz, Frederik Kratzert, Martin Gauch, Alden Keefe Sampson, Günter Klambauer, Sepp Hochreiter, Grey Nearing
TL;DR
The paper addresses limited uncertainty estimation in deep learning rainfall–runoff studies and the lack of standardized benchmarks. It develops a benchmarking procedure with four deep learning baselines and finds that deep learning can produce reliable, precise distributional predictions, while noting important scope limitations.
Problem
Deep learning rainfall–runoff studies commonly omit uncertainty estimates, while standardized open procedures for comparing uncertainty-estimation methods are lacking.
Method
The paper develops data-driven uncertainty-estimation benchmarks and evaluates four deep learning baselines, including three mixture-distribution approaches and Monte Carlo Dropout.
Results
Deep learning distributional predictions were reliable and precise, with particularly good reliability and sharpness reported for one approach.
Takeaways & Limitations
Deep learning uncertainty-estimation baselines appear promising for rainfall–runoff modeling and provide a public data-driven benchmark for future work.
Takeaways & Limitations
Open datasets can enable over-fitting on test data, and some uncertainty-estimation approaches are not completely general for future benchmarking studies.
Abstract
from arXiv · showhide
Deep Learning is becoming an increasingly important way to produce accurate hydrological predictions across a wide range of spatial and temporal scales. Uncertainty estimations are critical for actionable hydrological forecasting, and while standardized community benchmarks are becoming an increasingly important part of hydrological model development and research, similar tools for benchmarking uncertainty estimation are lacking. We establish an uncertainty estimation benchmarking procedure and present four Deep Learning baselines, out of which three are based on Mixture Density Networks and one is based on Monte Carlo dropout. Additionally, we provide a post-hoc model analysis to put forward some qualitative understanding of the resulting models. Most importantly however, we show that accurate, precise, and reliable uncertainty estimation can be achieved with Deep Learning.
1 Introduction
Rainfall–runoff uncertainty estimation lacks a prevailing distributional method and standardized open benchmarks. This paper motivates deep learning approaches that intrinsically model distributions and introduces benchmark baselines for comparison.
- Existing uncertainty methods: Most rainfall–runoff methods augment deterministic models with separate uncertainty strategies rather than directly estimating predictive uncertainty.Classical approaches include ensemble-based, Bayesian or pseudo-Bayesian, post-processing, and stochastic-model methods.
- Existing uncertainty methods: Assigning sampling distributions a priori can produce degenerate distributions, causing well-known errors and biases in uncertainty estimation.This limitation is described for stochastic approaches that require prior distributional assumptions.
- Deep learning motivation: Deep learning representations can support distribution estimation while accounting for strongly nonlinear interactions between model inputs and outputs.This avoids relying on deterministic predictions that must subsequently be sampled, perturbed, or inverted.
- Study objectives and methods: The paper benchmarks four deep learning approaches for distributional rainfall–runoff prediction: three neural-network mixture methods and Monte Carlo Dropout.The approaches adapt LSTM rainfall–runoff models to produce distributional predictions.
- Benchmarking gap: The study addresses the lack of standardized, open procedures for comparative uncertainty-estimation studies by developing benchmarks on a public large-sample dataset.The benchmark is intended to provide a baseline for future studies, while uncertainty estimation remains important because hydrological predictions should account for inherent uncertainty.
2 Data and Methods
The study establishes a benchmarking framework for deep-learning uncertainty estimation using data, metrics, baselines, and post-hoc model examination. It uses CAMELS data and evaluates predictive distributions through reliability and resolution measures, while recognizing limitations of aggregate probability plots and some metrics.
- Benchmarking framework: The benchmarking framework combines data, metrics, and baselines with post-hoc examination of model properties.The post-hoc analysis assesses whether model properties correspond to hydrological intuitions and demonstrates procedures for checking them.
- Data: CAMELS provides daily meteorological forcings, streamflow, and basin attributes for basins spanning diverse sizes and geological, ecological, and climatic conditions.The experiments use 531 basins selected from the original CAMELS collection.
- Reliability: Reliability is assessed with probability plots, where consistency is indicated by observations following a uniform distribution and the resulting plot approaching the 1:1 line.The evaluation uses 10% binning thresholds and interprets deviations from the line as under-confidence or over-confidence.
- Evaluation limitations: The study notes that probability plots aggregate across time and basins, while some resolution metrics are difficult to interpret or are not fully general for future benchmarks.Proper scoring rules such as likelihoods are suggested, but likelihoods lack an absolute scale for direct interpretation.
- Resolution: Resolution metrics complement reliability because infinitely many models can produce perfect probability plots, including models that ignore inputs or produce delta distributions at observations.Resolution concerns the sharpness of predictive distributions and is preferred only when predictions remain reliable.
- Baselines: The benchmark includes three Mixture Density Network approaches and Monte Carlo Dropout as data-driven baselines.The approaches are intended to provide a useful set of baselines for future benchmarking, with one design removing the need to pre-specify mixture components.
3 Results
CMAL generally provided the most reliable uncertainty estimates, while UMAL and MCD produced sharper but overconfident distributions at high flows. Asymmetric mixture models also performed strongly for point prediction and revealed hydrologically interpretable mixture-component behavior.
- Reliability: The mixture-density approaches performed better than MCD in the probability-plot benchmark, with CMAL and UMAL better centered than GMM and MCD.The outer probability thresholds showed evaluation-induced bias because deviations are bounded to [0,1].
- Reliability: CMAL had the best overall reliability performance and stayed close to the 1:1 line across flow regimes.Most methods were under-confident at low flows, whereas UMAL and MCD underestimated uncertainty at higher flows.
- Resolution: UMAL and MCD produced the sharpest distributions, but their narrowness accompanied overconfidence at high flow volumes.GMM and CMAL produced less sharp distributions, with GMM showing under-confidently wide distributions.
- Benchmark interpretation: The benchmark exposed a reliability–resolution trade-off because sharper distributions can become overconfident while wider distributions can become under-confident.The models are trained with log-likelihood, which requires balancing these properties.
- Accuracy: CMAL and UMAL performed best among uncertainty approaches for single-point prediction, and UMAL provided the best point estimates.Despite using distributional predictions, both asymmetric mixture models outperformed the single-point-optimized model on some metrics.
- Model analysis: CMAL learned three mixture components corresponding to rising limbs, falling limbs, and low-flow periods without explicit hydrological conceptualization.The components respectively emphasized peak runoff, throughflow, and low-flow behavior along the hydrograph.
4 Conclusions and Outlook
The study establishes a public benchmark for deep-learning uncertainty estimation and finds that distributional predictions can be reliable, precise, and hydrologically interpretable. It also identifies important limitations in current metrics, datasets, and post-hoc analyses, motivating stronger future benchmarks.
- Benchmarking: The benchmark combines data, metrics, baselines, and diagnostic analysis to support comparative uncertainty-estimation studies.The authors intend the public, data-driven scheme to enable more formal and community-minded benchmarking.
- Findings: Deep-learning uncertainty estimation produced dynamic distributions that responded to inputs and generally increased nonlinearly with streamflow.The predictions changed with precipitation and other dynamic inputs, broadly matching hydrological intuition.
- Findings: The MCD approach gave the worst uncertainty estimates, plausibly because its Gaussian assumption is difficult to maintain across low- and high-flow situations.MCD and the MDN approaches also target different uncertainty types: epistemic and aleatoric uncertainty, respectively.
- Findings: Asymmetric distributions in UMAL and CMAL had advantages over GMM for reliability and single-point accuracy, with CMAL performing well in reliability and sharpness.CMAL also produced strong single-point estimates.
- Limitations: The study cautions that likelihood-based aleatoric estimates can be over-confident, although this could not be diagnosed empirically in the present inquiry.The authors attribute this uncertainty partly to limits of the investigation rather than concluding that over-confidence is absent.
- Limitations: Open test datasets may permit over-fitting, and future benchmarks should withhold test data to enable more rigorous evaluation.The authors identify this as a downside of relying on current open datasets.
- Limitations: The probability plot does not capture precision, consistency, or event-specific properties, so more powerful metrics and stronger baselines remain necessary.The current metric may also be difficult for future models to improve upon.
- Post-hoc analysis: Post-hoc analysis linked predicted probabilities to hydrological behavior, including different mixture components activating on rising and falling hydrograph limbs.The analysis examined uncertainty models at hydrologically relevant points and found reliable, precise distributions with strong single-point estimates.
A1 General Setup
Table A provides the general setup used for hyperparameter search and model training.
- Table A defines the general setup for the hyperparameter search.
- The setup also specifies the framework used for model training.
- Hyperparameter search and model training are covered together in the reported setup.
A2 Noise Regularization
The paper applies relative additive noise as a smoothness regularizer, motivated by evidence that noise regularization produces smoother density estimates closer to the true conditional density.
- Plain maximum likelihood estimation can overfit, producing spiky distributions that generalize poorly beyond training data.
- Noise regularization produces smoother density estimates closer to the true conditional density.
- The experiments use relative additive noise as a first-order approximation to contamination expected in hydrological time series.
- The regularization operation applies noise to dynamic or static inputs or observed runoff, with Gaussian noise having mean zero and standard deviation σ.
A3 Search
The study searches across six hyperparameters using a staged procedure that balances computational resources with search depth, then refines the best models.
- Six hyperparameters are searched to balance computational resources and search depth.
- The first search step informally identifies sensible general presets.
- The second step trains models across hidden size, noise, number of densities, and dropout rate combinations.
- The best resulting model is refined by searching batch size and learning rate settings.
A4 Results
The paper reports the outcomes of hyperparameter selection in a dedicated results table and gives the resulting parameterization separately.
- The hyperparameter search results are summarized in Table A3.
- The results section presents the selected parameterization as the output of hyperparameter search.
- Table A3 reports the resulting parameterization from the hyperparameter search.
Appendix B: Baselines
Mixture Density Networks model runoff as conditional distributions whose parameters are predicted from the input by a neural network. The appendix introduces Gaussian and asymmetric-Laplacian mixtures, with CMAL offering greater asymmetry but requiring more parameters.
- Gaussian Mixture Model: Gaussian mixture models use a neural network to predict mixture weights, means, and standard deviations as distributional outputs.The resulting mixture represents runoff conditionally on precipitation, temperature, and preceding time steps.
- Gaussian Mixture Model: The Gaussian-mixture parameters are functions of the input, and training minimizes the negative log-likelihood of the estimated conditional distribution.
- Countable Mixture of Asymmetric Laplacians: Countable mixtures of asymmetric Laplacian distributions use ALDs as mixture components and estimate their parameters and weights with a neural network.The ALD uses an asymmetry parameter, location parameter, and scale parameter.
- Countable Mixture of Asymmetric Laplacians: CMAL components intrinsically represent asymmetric distributions, making them more expressive than Gaussian mixtures but more difficult to handle because they require more parameters.
B3 Uncountable Mixture of Asymmetric Laplacians
UMAL extends CMAL by implicitly approximating a mixture of asymmetric Laplacians through sampled asymmetry parameters, fixed equal weights, and stochastic realizations. This reduces the number of parameters that must be estimated while retaining a learned conditional distribution.
- Model construction: UMAL samples the asymmetry parameter, supplies it to the model and loss, fixes mixture weights at αk = 1/K, and sums realizations stochastically.
- Parameterization: 39: UMAL requires considerably fewer estimated parameters than GMM or CMAL because the network estimates only location and scale.
- Distribution approximation: The learned distribution can be approximated after training using a selectable number of discrete samples.
- Training: UMAL training minimizes the negative log-likelihood of the training data under the estimated distribution.
B4 Monte Carlo Dropout
Monte Carlo Dropout estimates uncertainty by averaging predictions from dropout-induced subnetworks rather than predicting a heterogeneous mixture distribution. It is trained with mean squared error and provides epistemic uncertainty under a Gaussian-form assumption.
- Mechanism: Monte Carlo Dropout samples Bernoulli dropout masks to form subnetworks whose predictions are averaged over the dropout rate.The mask, network weights, and neural network define each subnetwork realization.
- Training: MCD is trained by minimizing mean squared error, making it substantially different from the Mixture Density Network approaches.
- Uncertainty representation: MCD estimates epistemic uncertainty but does not provide a heterogeneous, multimodal estimate because it assumes a Gaussian form.
- Caveat: Prior work has observed that MCD can underestimate epistemic uncertainty.