PaperlayerView exact revision on arXiv

Source-linked AI summary

The Proper Care and Feeding of CAMELS: How Limited Training Data Affects Streamflow Prediction

Martin Gauch, Juliane Mai, Jimmy Lin

arXiv:1911.07249v3cs.LGstat.ML

TL;DR

The paper investigates how scarce training data affect streamflow prediction models, including the roles of geographic diversity, training duration, and input sequence length. It evaluates XGBoost and EA-LSTM models on CAMELS data and finds similar performance on small datasets but an advantage for EA-LSTMs with more training data.

  • Problem

    The study addresses how limited training period length, basin diversity, and input sequence length affect tree- and LSTM-based streamflow prediction when historical records are scarce.

  • Method

    The authors compare XGBoost and EA-LSTM models on CAMELS data while varying training years, training basins, and input sequence lengths.

  • Results

    With limited training data, XGBoost and EA-LSTM have statistically similar NSE distributions; with more data, EA-LSTMs significantly outperform XGBoost.

  • Takeaways & Limitations

    Adding training data from more years and basins improves prediction accuracy, while the best input sequence length increases with larger training sets.

  • Takeaways & Limitations

    The study focuses on EA-LSTM rather than standard LSTM and limits XGBoost sequence lengths to 100 days because longer flattened inputs make training challenging.

Abstract

from arXiv · show

Accurate streamflow prediction largely relies on historical meteorological records and streamflow measurements. For many regions, however, such data are only scarcely available. Facing this problem, many studies simply trained their machine learning models on the region's available data, leaving possible repercussions of this strategy unclear. In this study, we evaluate the sensitivity of tree- and LSTM-based models to limited training data, both in terms of geographic diversity and different time spans. We feed the models meteorological observations disseminated with the CAMELS dataset, and individually restrict the training period length, number of training basins, and input sequence length. We quantify how additional training data improve predictions and how many previous days of forcings we should feed the models to obtain best predictions for each training set size. Further, our findings show that tree- and LSTM-based models provide similarly accurate predictions on small datasets, while LSTMs are superior given more training data.

1. Introduction

The study examines how limited training data affect tree- and LSTM-based streamflow models, focusing on training duration, basin diversity, and input sequence length. This addresses data scarcity in regions where historical streamflow and meteorological records are limited.

  • Limited historical streamflow and meteorological records make accurate streamflow prediction especially important for water management and flood forecasting.
  • Tree-based models may perform well with limited data, whereas neural networks generally require more training data for accurate predictions.
  • The study compares tree-based models with Long Short-Term Memory (LSTM)-based neural networks under varying amounts of training data.LSTMs process time series through internal memory states, whereas tree-based models use flat input vectors.
  • The study also asks which input sequence length produces the best predictions for each model and whether that choice depends on training set size.
  • The central questions concern how training period length and basin count affect prediction quality for tree- and LSTM-based models.

2. Data and Methods

The study evaluates XGBoost and EA-LSTM streamflow predictions under controlled reductions in training-period length, basin diversity, and input sequence length using CAMELS data. It compares model performance across these training-data dimensions and selects the best sequence length for each training configuration.

  • Data: CAMELS provides daily streamflow measurements for 671 U.S. basins, of which the study uses 531 after excluding basins with large area discrepancies.The experiments replicate the setup and data usage of Kratzert et al. (2019).
  • Experimental design: Input sequence length k determines which previous forcings are used to predict streamflow q_t, with EA-LSTMs tested on 10, 30, 100, 270, and 365 days.For a prediction on day t, the model receives forcings from the previous k days.
  • Experimental design: Training-data size is varied independently through training-period length, hydro-geo-climatic diversity, and input sequence length.The study treats the first two as the amount of historical data and basin coverage, while sequence length controls forcings supplied for each prediction.
  • Data: The models use 27 static catchment attributes and five normalized Maurer meteorological variables as inputs.The attributes describe climate, vegetation, soil, and topography.
  • Experimental design: The experiments combine three training periods—three, six, or nine years—with basin sets of 13, 26, 53, 265, or 531 basins.For basin sets below 531, five random selections are evaluated using the same selections across comparable setups.
  • Evaluation: For each training-period and basin-set combination, the EA-LSTM and XGBoost sequence length is selected by the best median NSE, and performance is summarized with NSE distributions and their cumulative-distribution area.Larger cumulative-distribution area A indicates worse overall performance, while the ideal value is zero.

3. Results

Optimal input sequence lengths vary with training-set size and model: XGBoost generally favors 30 days, while EA-LSTM benefits from longer sequences. Across experiments, more training data improves prediction quality, with EA-LSTM increasingly outperforming XGBoost as data increase.

  • 3.1. Input Sequence Length: EA-LSTM’s best input sequence length increases with training-set size, reaching 365 days in the three largest configurations.XGBoost instead consistently performs better with 30-day than 100-day sequences.
  • 3.2.1. Analysis of Prediction Accuracy Distributions: Both models’ area under the NSE distribution curve decreases as training data increase, from 0.64 and 0.58 to 0.38 and 0.30 for XGBoost and EA-LSTM, respectively.Lower area indicates better overall performance.
  • 3.2.1. Analysis of Prediction Accuracy Distributions: EA-LSTM outperforms mHM with 53 basins, while XGBoost reaches roughly mHM-level accuracy only in the largest configuration.With very limited data, the models’ NSE distributions do not differ significantly, although EA-LSTM has a slightly better median NSE.
  • 3.2.1. Analysis of Prediction Accuracy Distributions: The models’ NSE distributions differ significantly for three-to-six-year conditions with at least 26 basins and for all nine-year conditions.The stated threshold is p<0.01/15 following the Kolmogorov–Smirnov test.
  • 3.2.2. Analysis of Median Prediction Accuracy: EA-LSTM generally achieves higher median NSE than XGBoost, and its advantage increases as the number of training samples grows.Median NSE rises strongly with training-set size up to about 24 × 10^4 samples, after which improvements become smaller.
  • 3.2.2. Analysis of Median Prediction Accuracy: Both models benefit from longer training periods and larger basin subsets, but longer periods appear more effective than more basins at equal training-set size.This comparison is based on relatively few examples.

4. Discussion and Future Work

The study finds that larger and more diverse training sets improve streamflow prediction, while model performance also depends on matching input sequence length to training-set size. LSTMs become preferable with more data, whereas adaptive sequence lengths make their accuracy comparable to XGBoost on small datasets.

  • Input sequence length: Adaptive input sequence lengths make EA-LSTM accuracy comparable to XGBoost on small datasets.The authors contrast this with an earlier study version that used fixed sequence lengths.
  • Training period length and number of basins: Longer training periods and additional basins both increase prediction accuracy, supporting spatial generalization beyond individually trained basin models.The authors link this result to learning relationships between catchment characteristics and streamflow patterns rather than merely fitting the training basins.
  • Future work: Cross-basin knowledge transfer could improve predictions in data-poor regions by using training data from similar, data-rich basins.The authors identify this as a step toward machine-learning prediction in ungauged basins and propose transfer learning as future work.
  • Model comparison: LSTMs are recommended over XGBoost because they outperform it in almost all experiments despite requiring more careful sequence-length tuning.The recommendation is based on the models’ ability to ingest time-series data directly and their experimental accuracy.
  • Model comparison: Figure 3 compares basin-level NSE heatmaps for the smallest three-year, 13-basin configuration and the largest nine-year, 531-basin configuration.The figure provides a basin-level view of model behavior under contrasting training-data scales.

A. Static Basin Attributes

The study uses 27 static basin attributes from the CAMELS dataset to train its models, matching the attribute set used by Kratzert et al. (2019).

  • Static basin attributes: The models use 27 static basin attributes from the CAMELS dataset.These attributes are listed in Table 3.
  • Static basin attributes: The selected attributes are the same as those used by Kratzert et al. (2019).

B. Training Procedures

The training procedures combine EA-LSTM optimization details with sequence-length-specific XGBoost tuning and early stopping. XGBoost hyperparameters are calibrated on progressively larger training configurations for longer input sequences.

  • EA-LSTM training: EA-LSTM models are trained for 30 epochs with a 256-neuron hidden layer, dropout rate 0.4, and scheduled learning-rate reductions.Training uses batches of 256 samples and learning rates of 0.001, 0.0005, and 0.0001 across successive epochs.
  • XGBoost training: XGBoost hyperparameters are tuned separately for each sequence length using three-fold cross-validated random searches.The procedure includes searches for model and regularization parameters.
  • XGBoost training: XGBoost calibration uses three years and 13 basins for sequence length 10, six years and 53 basins for length 30, and nine years and 265 basins for length 100.
Loading 1911.07249v3…