Source-linked AI summary
Enhancing streamflow forecast and extracting insights using long-short term memory networks with data integration at continental scales
Dapeng Feng, Kuai Fang, Chaopeng Shen
TL;DR
The paper addresses whether recent streamflow observations can be integrated effectively with LSTM forecasts. It tests data integration using direct lagged inputs or a CNN unit and reports improved continental-scale forecasting, with benefits linked to flow autocorrelation but persistent difficulty for flashy arid basins.
Problem
The paper asks whether LSTM models with data integration outperform standard LSTM forecasts when using recent streamflow observations.
Method
The study uses LSTM models with data integration, accepting recent lagged observations directly or through a convolutional neural network unit.
Results
0.86 was the continental-scale median Nash-Sutcliffe Efficiency value achieved by the data-integrated forecasts.
Takeaways & Limitations
Data integration was most beneficial where streamflow had high autocorrelation, improving forecasts in settings with baseflow or dynamic surface-water storage.
Takeaways & Limitations
Data integration could not elevate forecasts for high-aridity basins with one-day flashy streamflow peaks.
Abstract
from arXiv · showhide
Recent observations with varied schedules and types (moving average, snapshot, or regularly spaced) can help to improve streamflow forecasts, but it is challenging to integrate them effectively. Based on a long short-term memory (LSTM) streamflow model, we tested multiple versions of a flexible procedure we call data integration (DI) to leverage recent discharge measurements to improve forecasts. DI accepts lagged inputs either directly or through a convolutional neural network (CNN) unit. DI ubiquitously elevated streamflow forecast performance to unseen levels, reaching a record continental-scale median Nash-Sutcliffe Efficiency coefficient value of 0.86. Integrating moving-average discharge, discharge from the last few days, or even average discharge from the previous calendar month could all improve daily forecasts. Directly using lagged observations as inputs was comparable in performance to using the CNN unit. Importantly, we obtained valuable insights regarding hydrologic processes impacting LSTM and DI performance. Before applying DI, the base LSTM model worked well in mountainous or snow-dominated regions, but less well in regions with low discharge volumes (due to either low precipitation or high precipitation-energy synchronicity) and large inter-annual storage variability. DI was most beneficial in regions with high flow autocorrelation: it greatly reduced baseflow bias in groundwater-dominated western basins and also improved peak prediction for basins with dynamical surface water storage, such as the Prairie Potholes or Great Lakes regions. However, even DI cannot elevate high-aridity basins with one-day flash peaks. Despite this limitation, there is much promise for a deep-learning-based forecast paradigm due to its performance, automation, efficiency, and flexibility.
1. Introduction
Streamflow forecasting needs to incorporate recent observations across varied schedules and types, while existing deep-learning studies have rarely addressed transferable continental-scale integration. This work proposes a flexible LSTM-based data-integration procedure to absorb such observations and derive hydrologic insights.
- Flooding is the biggest weather-related killer in the United States, while droughts cause an average $6B of annual US damage.
- Data assimilation updates model states or structures using relationships between observed variables and process-model states, but requires covariance modeling and often bias correction.Traditional assimilation can also steer models away from trajectories caused by forcing errors, inadequate structures, or parameters.
- Deep-learning streamflow studies have largely focused on individual basins, leaving continental-scale transferable forecasting and regional hydrologic insight underexplored.Earlier LSTM applications showed promise, but locally trained models without basin descriptors cannot learn from large-scale regional patterns.
- The study tests whether a uniform LSTM with data integration can outperform conventional approaches at CONUS scale and reveal landscape-dependent hydrologic processes.The proposed DI procedure accepts lagged observations directly or through a CNN unit, automatically integrating recent discharge measurements.
- Recent observations have varied schedules, latencies, and forms, yet it remains unclear whether LSTM can effectively use discharge averages, snapshots, and irregular intervals.Traditional assimilation of these data types entails expert choices for assimilation schemes, transformations, and bias correction.
- Data integration is distinguished from traditional data assimilation as a flexible procedure for integrating recent observations into deep-learning forecasts at continental scales.The authors use the term DI to reference this procedure and differentiate it from heavily calibrated traditional DA schemes.
2. Data and Methods
The study uses CAMELS basin attributes, meteorological forcing, and streamflow observations to train an LSTM-based forecasting model. Data integration supplements the base model with recent observations, either directly or through optional dimensional reduction.
- Dataset: The study uses CAMELS data from 671 minimally disturbed CONUS catchments, with daily observations mostly spanning 1980–2014.A 531-basin subset without watershed-boundary or area concerns is also reported for comparison.
- Dataset: The basin characterization includes slope, soil depth, aridity, snow fraction, precipitation seasonality, baseflow index, streamflow ACF(1), and TWSA variability.Baseflow index represents the fraction of outflow attributed to baseflow, while γ compares inter-annual with intra-annual storage variability.
- LSTM model: The base projection model is an LSTM that learns from sequential data and predicts discharge from 365-day sequences of meteorological and static inputs.LSTM gates regulate information entering, being forgotten by, and leaving the memory cell.
- Data integration: Data integration adds observed discharges before the prediction time to the original inputs, either directly or after dimensional reduction through an optional CNN unit.Most experiments directly used historical observations without the optional operator R.
- Dataset: Model inputs include six meteorological features and 17 static features describing topography, climate, land cover, soil, and geology.Mean climate attributes were excluded to avoid increasing overfitting risk.
- Data integration: Forecast models integrate discharge observations at every time step while using only observations available before that time.The implementation supports different historical-observation forms and latencies, including directly supplied inputs and CNN-transformed segments.
3. Results & Discussions
The projection LSTM performed competitively across the CONUS, while data integration (DI) broadly improved forecast skill, bias, and flow-regime representation. Benefits varied by basin characteristics, observation type, and lag, with especially strong gains in autocorrelated systems but persistent difficulty for some low-flow and flash-flood basins.
- Overall performance: The projection LSTM achieved a median NSE of 0.73 across 671 basins, exceeding the previously reported value of 0.64.
- Limitations: The projection LSTM showed substantial low-flow bias, while high-aridity basins with one-day flash peaks remained difficult because daily precipitation lacks minute-level rainfall intensity.
- Overall performance: DI increased median NSE to 0.86 with one-day-lag discharge and 0.80 with three-day-lag discharge; 96% of CONUS basins benefited.
- Spatial patterns: DI improved Great Plains NSE from below 0.4 to 0.7–0.95 and raised mid-latitude western-state NSE from 0.5–0.7 to 0.9.
- Lag effects: DI benefits were larger for low-flow than high-flow regimes and remained evident at a 30-day lag, although benefits became less clear at lags of about 60 days or longer.
- Hydrologic interpretation: DI reduced baseflow bias in groundwater-dominated basins and improved peak locations and magnitudes in basins with dynamic surface-water storage.In one Prairie Pothole basin, DI(1) increased NSE from 0.43 to 0.92 by improving peak performance.
- Observation formats: Using all previous daily observations outperformed moving averages, which outperformed regularly spaced snapshots; single-day data with the same lag was least useful.
4. Conclusions
The study shows that LSTM-based data integration can flexibly use recent streamflow observations to improve continental-scale forecasts, especially where flow is strongly autocorrelated. DI substantially improves baseflow and peak predictions, but arid basins with flashy peaks remain difficult.
- 4. Conclusions: The base projection LSTM performed well in mountainous and snow-dominated basins but retained issues with baseflow bias, storage variability, aridity, transfers, and complex hydrogeology.Even DI could not improve southern Texas, an arid region with flashy peaks and no baseflow.
- 4. Conclusions: DI improved continental-scale streamflow forecasts, reaching an unprecedented national-scale NSE of 0.86.The value came from the ensemble mean discharge of DI(1).
- 4. Conclusions: DI produced widespread, spatially varying gains, with the largest improvements in basins showing strong flow autocorrelation.These gains suggest strong storage-surface-water connections or surface-water retention.
- 4. Conclusions: DI improved both baseflow and peak predictions, particularly in regions with varying surface-water connectivity such as the Prairie Potholes and Great Lakes.The method also outperformed simpler statistical models for both peak-flow and baseflow portions of streamflow.
- 4. Conclusions: The flexible LSTM integrated lagged, multi-day, moving-average, and regularly spaced observations, and each source improved performance over the base projection LSTM.The improvement was not attributable simply to longer forcing memory, because assimilating precipitation provided no performance gain.
- 4. Conclusions: Directly integrating 1-day-lag observations performed comparably to the more complex CNN-LSTM architecture.The authors relate this to evidence that structural LSTM modifications often fail to produce performance gains when sequential information is unchanged.