Ranked Right, Sized Wrong: Ensemble Spread Fails Silently at Streamflow Extremes

1 Introduction

• As climate change increases hydroclimatic extremes, forecasting systems will increasingly encounter conditions poorly represented in historical records.The paper notes that point predictions alone do not indicate how trustworthy they are, particularly outside the model’s historical experience.Source: §1 · p. 1
• Ensemble means provide point predictions, while between-member spread is commonly used as a signal of epistemic uncertainty.This uncertainty is distinct from aleatoric uncertainty, which is inherent to observations and treated separately.Source: §1 · pp. 1–2
• Random-initialization ensemble spread is not guaranteed to match predictive-error magnitude, even within the training regime.The authors expect spread to increase with error while remaining informative about which predictions are most likely to be erroneous.Source: §1 · p. 2
• The paper asks whether raw ensemble spread remains reliable as flows become underrepresented, testing both its scaling to error and its ranking of relative error.The question matters because realized error is unavailable at deployment, when spread is the uncertainty signal.Source: §1 · p. 2

2 Methods

• The baseline is a regional LSTM rainfall–runoff model for 269 unregulated Western Canadian catchments, tested on 2013–2022 data after chronological training and validation.Its median test NSE is 0.75; the training period defines the fixed reference flow distribution.Source: §2 · p. 2
• For each test catchment-day, the mean of 10 independently trained models is the point prediction, and their between-member variance is the spread signal.The analysis evaluates dispersion relative to realized error and ranking by realized error.Source: §2 · p. 2
• The under-dispersion factor compares residual dispersion after removing stratum mean with corrected pooled ensemble spread, using 1 as a null baseline under stated exchangeability assumptions.Observation error and variability shared across ensemble members contribute to residual dispersion but are not represented in ensemble spread.Source: §2 · p. 2
• The study stratifies by observed-flow percentiles and repeats the analysis by ensemble-mean flow, then regresses log residual SD on log spread across deciles 10–90 and holds out the top decile.A slope of 1 indicates proportional scaling; values above 1 indicate error grows faster than spread.Source: §2 · pp. 2–3
• Test flows are grouped by exceedance above catchment-specific 1980–2005 thresholds, including flows above the historical maximum.For ranking, distribution-free AUSE compares progressive removal by spread or ensemble mean with oracle and random ordering; lower AUSE indicates better ranking, not correct magnitude.Source: §2 · p. 3

3 Results and Discussion

• UDF rises from 2.1× to 4.3× across ensemble-mean flow deciles, with a superlinear log-UDF versus log-flow relationship that persists after conditioning reduces bias.The corresponding figure shows UDF by observed-discharge and ensemble-mean deciles, with a fitted log-log relationship and confidence intervals.Source: §3 · p. 3, Figure 1 · p. 3
• Ensemble spread approaches oracle error ranking on the full test set and remains substantially better than random ordering in the highest observed-flow decile.Ordering by ensemble mean yields slightly higher AUSE than spread for both the full test set and highest-flow decile; the figure compares sparsification curves and catchment-level AUSE versus UDF.Source: Figure 2 · p. 3, §3 · p. 4
• Under-dispersion occurs in nearly every catchment despite generally strong ranking, consistent with shared errors remaining invisible to member disagreement.The paper identifies systematic forcing error as one example of error shared across ensemble members.Source: §3 · p. 4
• UDF rises from 5.65× above the training-period 90th percentile to 17.93× for flows exceeding the historical maximum.The table reports UDF against each catchment’s fixed 1980–2005 training flow distribution; extreme observations may also have greater rating-curve uncertainty.Source: §3 · p. 4, Table 1 · p. 4

4 Conclusion

• The random-initialization LSTM ensemble consistently underestimates prediction-error magnitude, worsening at extreme flows, while retaining near-oracle AUSE ranking that only modestly outperforms the ensemble mean.The paper concludes that successful error ranking can coexist with substantial underestimation of error magnitude.Source: §4 · p. 4
• At deployment, spread is available but the streamflow residual is not, so under-dispersion is silent and worsens as flows become more extreme and underrepresented.The conclusion states that ranking alone is insufficient and that correction should be conditioned on flow, for example through conditional post-processing or distributional heads.Source: §4 · p. 4

A.1 Study Region

• The study focuses on Western Canada, particularly British Columbia and Alberta, a region characterized by large hydroclimatic variability and a dense gauge network.The passage also notes well-documented recent extreme events in the region.Source: §A.1 · p. 7

A.2 Data Sources and Preprocessing

• Streamflow data cover natural, unregulated gauges active during 1980–2022, subject to a limit of 50% missing observations in any calendar year.Daily mean streamflow targets and basin polygons came from the HYDAT database.Source: §A.2 · p. 7
• Hourly ERA5-Land temperature and precipitation were spatially averaged over each catchment to derive daily precipitation and maximum and minimum temperature.The reanalysis data have a 0.1° × 0.1° resolution.Source: §A.2 · p. 7
• Catchment attributes include area, elevation, elevation range, slope, and glacier extent, while predictors were standardized using 1980–2005 training-period statistics.Topographic measures came from Terrain Tiles; glacier extent came from the Randolph Glacier Inventory Version 6.Source: §A.2 · pp. 7–8

A.3 Dataset Partitions

• Forcing-data subsets begin exactly one year before their corresponding streamflow subsets to supply the 365-day context window for the first target prediction.The passage identifies the partitioning rule but does not provide the partition dates.Source: Table 3 · p. 8

A.4 Model Architecture

• A single 256-unit LSTM layer processes 365 days of meteorological inputs, with static catchment attributes concatenated at each timestep.A fully connected output layer produces the daily streamflow prediction for each input sequence.Source: §A.4 · p. 8

A.5 Training Procedure

• The network minimizes basin-averaged NSE* loss, which normalizes squared residuals by each basin’s observed-flow variability; missing observations are masked during loss computation.Training uses Adam with learning rate 0.001, batch size 512, dropout 0.4, early stopping, and ten independent random seeds to form an M = 10 ensemble.Source: §A.5 · p. 8

B.1 Under-Dispersion Factor (UDF)

• The UDF compares residual dispersion after removing stratum-mean bias with corrected pooled between-member spread, using within-catchment deciles of observed flow or ensemble mean.The finite-ensemble factor corrects the ratio so its exchangeability null is 1; pooled variances are used before taking roots.Source: §B.1 · pp. 8–9

B.2 Finite-Ensemble Correction and Exchangeability Null

• The exchangeability null treats the observation as an additional ensemble member, but shared training-procedure error is invisible to member disagreement.A corrected factor above 1 can therefore reflect residual components not represented by between-member spread.Source: §B.2 · p. 9

B.3 Conditioning on ˆµ and Interpretation of the Null

• Conditioning on observed flow mechanically inflates mean residuals, whereas conditioning on ensemble mean gives a conditional-bias measure; top-decile bias falls from +1.304 to +0.25 mm day−1.Observation error contributes to residual dispersion and may grow with flow, but cannot by itself explain the rapid UDF growth; the factor’s growth, rather than its absolute magnitude, is the diagnostic focus.Source: §B.3 · p. 10

B.4 Superlinear Scaling of Error with Spread

• β = 1.30 (95% CI [1.20, 1.41]), indicating that residual dispersion scales superlinearly with corrected spread.The fit uses the nine lowest ensemble-mean deciles, holding out the top decile as an out-of-sample check.Source: §B.4 · p. 10
• No constant spread multiplier calibrates the band across the fitted range; calibration must be conditioned on flow.For rescaled spread, the calibration ratio remains increasing in spread when β exceeds 1.Source: §B.4 · p. 10

C.1 Nash-Sutcliffe Efficiency (NSE)

• NSE compares residual variance with observed-flow variance; 1.0 indicates perfect agreement, while 0 means performance matches predicting the long-term observational mean.Source: §C.1 · p. 10

C.2 Root Mean Squared Error (RMSE)

• RMSE measures error magnitude in streamflow units and disproportionately penalizes larger errors; 0 indicates perfect accuracy, with larger values reflecting greater average deviation.Source: §C.2 · p. 11

C.3 Area Under Sparsification Error (AUSE)

• AUSE integrates the gap between sparsification and oracle error curves; 0 indicates perfect uncertainty ordering, and larger values indicate poorer ranking.Source: §C.3 · p. 11

C.4 Spearman’s Rho

• Spearman’s ρ measures monotonic association using the rank order of AUSE and UDF across catchments, ranging from −1.0 to 1.0.A value of 1.0 indicates perfect increasing association, 0 no monotonic correlation, and −1.0 perfect decreasing association.Source: §C.4 · p. 11

C.5 Log-log slope for Simple Linear Regression

• The log-log slope β1 measures how residual standard deviation scales with ensemble spread: 1 indicates proportional scaling, while values above 1 indicate error grows faster.The slope is estimated by ordinary least squares on log-transformed variables.Source: §C.5 · p. 11

Figure 1

• UDF is 5.76 [4.48, 6.90] when conditioned on observed flow and 4.27 [3.50, 4.99] when conditioned on ensemble mean prediction in the top decile.The table also reports conditional bias and mean residual by decile of each conditioning variable.Source: Table 4 · p. 12, Figure 1 · p. 12

Figure 2

• The sparsification table compares spread ordering by ensemble variance with flow ordering by ensemble mean prediction across 91 grid points, sampled at every tenth point.Each curve is normalized by its set’s full-sample RMSE; this normalization does not affect AUSE.Source: Table 5 · p. 12, Figure 2 · p. 12
• Across 269 catchments, per-catchment UDF exceeds 1 in every case, while AUSE and UDF ranks have Spearman ρ = −0.23 (p = 0.00014).The table bins catchments into deciles of per-catchment AUSE; sorting by ranking quality does not sort them by spread scaling.Source: Table 6 · p. 13

• All study data, code, and preprocessed data used to reproduce the analyses, figures, and tables are publicly available.Source: E Code and Data Availability · p. 13