A trans-disciplinary review of deep learning research for water resources scientists

1. Motivations

• Deep learning has achieved major predictive advances, repeatedly outperforming earlier machine-learning and statistical methods while generalizing to new instances.Its momentum accelerated after 2012, when deep-learning systems began winning machine-learning contests.Source: §1 · p. 3
• DL-powered AI is transforming scientific disciplines and industries because it delivers substantial performance gains across image recognition, speech, sequential data, and other applications.Examples include medical diagnosis, autonomous vehicles, speech recognition, digital assistants, and traditional industries.Source: §1 · p. 3
• Water sciences face interconnected societal and environmental processes, rapidly expanding heterogeneous datasets, and difficult information-extraction demands that conventional approaches handle laboriously.Relevant systems include biogeochemistry, climate, vegetation, soils, human dynamics, satellite observations, unmanned vehicles, radar, and infrastructure sensors.Source: §1 · p. 4
• Unrepresented processes and complex linkages can reduce model fidelity, while data-driven mechanisms could enable rapid hypothesis testing when relationships are difficult to expose or implement mechanistically.Examples include lateral groundwater flow, preferential flow, ecosystem demographics, rooting depth, and root hydraulics.Source: §1 · pp. 4–5
• Hydrologic scaling, subgrid heterogeneity, equifinality, and expensive hypothesis exploration remain longstanding challenges for process-based and multi-model approaches.Hyper-resolution simulation is prohibitively expensive, and equifinality includes competing parameters and model structures that produce similar outcomes.Source: §1 · p. 5
• The review argues that DL can extract abstract information, support interdisciplinary modeling, address scaling, build uniform parameterizations, generate scenarios, and reveal organizing principles through network interpretation.The paper is designed as a launchpad covering technical foundations, cross-disciplinary scientific progress, DL interpretation, and directions for water research.Source: §1 · p. 6

2.1. Basic machine learning terminology

• Supervised learning predicts categorical or continuous targets, whereas unsupervised learning discovers hidden structure for tasks such as dimensional reduction and feature extraction.Artificial neural networks learn weighted transformations across layers by minimizing a loss function, typically through backpropagation; depth denotes the number of layers.Source: §2.1 · pp. 7–8

2.2. Deep vs. non-deep machine learning

• Deep learning uses large, multi-layer networks to automatically extract increasingly abstract features from raw data, enabling transfer learning and richer representations of complex functions.Domain expertise remains necessary for framing questions, selecting inputs and architectures, and interpreting results.Source: §2.2 · pp. 8–9
• In a precipitation-retrieval example, an SDAE using raw satellite images removed 98% of bias and reduced variance by 31% relative to the earlier system.The example illustrates how deep learning can outperform approaches relying on manually engineered features; a deep model also won the 2012 Merck challenge without feature engineering.Source: §2.2 · p. 9

2.3. Popular deep learning network architectures

• CNNs and LSTMs are widely adopted architectures for image and sequence tasks, while SDAEs and DBNs provide foundational approaches for learned representations and probabilistic modeling.The section also contrasts these architectures with earlier multilayer networks and introduces hardware and software support elsewhere in the technical background.Source: §2 · p. 7, §2.3 · p. 11

2.3.1. Autoencoders and stacked denoising autoencoders (SDAEs)

• Autoencoders compress inputs through an information bottleneck to retain crucial structure, while denoising variants improve robustness by corrupting inputs during training.SDAEs stack denoising autoencoders and greedily train progressively narrower layers to capture increasingly abstract representations.Source: §2.3.1 · pp. 11–12

2.3.2. Convolutional Neural Network (CNN)

• CNNs use local convolution and pooling to capture hierarchical spatial features, and shared filters reduce parameters relative to fully connected networks.Their local connectivity also supports localization of input-output relationships and makes CNNs easier to train than vanilla multilayer perceptrons.Source: §2.3.2 · pp. 12–13

2.3.3. Long Short-Term Memory (LSTM)

• LSTMs address recurrent-network vanishing gradients with gates and memory cells that regulate what information is remembered, forgotten, and exposed over long sequences.Simple recurrent networks repeatedly multiply earlier states by weights, causing exponentially small training signals and extremely slow training.Source: §2.3.3 · p. 13

2.3.4. Deep Belief Network (DBN)

• DBNs stack Restricted Boltzmann Machines to reconstruct input distributions probabilistically through learned representations, using greedy layer-by-layer training.Unlike recurrent networks, DBNs do not maintain system memory or specifically model time dependencies.Source: §2.3.4 · pp. 13–14

2.4. Regularization (model complexity penalization) techniques

• Regularization reduces network complexity and overfitting, which otherwise causes models to fit noise and lose accuracy on unseen data.The section presents norm regularization, early stopping, and dropout as effective responses to the large degrees of freedom in deep networks.Source: §2.4 · p. 14
• Norm regularization penalizes large parameter norms, retaining all predictors while assigning lower weights to less important ones.The regularization parameter λ controls the penalty on coefficient magnitude, preserving potentially useful contributions from many factors.Source: §2.4 · pp. 14–15
• Early stopping halts training before full optimization, commonly when validation error begins increasing relative to earlier epochs.Its criteria often use a separate validation dataset and a smoothness condition.Source: §2.4 · p. 15
• Dropout randomly blocks connections, improving robustness by averaging reduced-order subnetworks and limiting coadaptation among neuron weights.Each randomly generated mask creates a different subnetwork, while blocked connections prevent weights from jointly adjusting to fit targets.Source: §2.4 · p. 15
• For recurrent networks, naïve dropout on recurrent links amplifies noise and harms memory, so effective variants restrict or time-share recurrent masks.Zaremba et al. (2015) applied dropout only to forward connections; Gal and Ghahramani (2016) and Semeniuta et al. (2016) used recurrent masks that remain constant across time steps, while FSKY17 combined these approaches.Source: §2.4 · pp. 15–16

2.5. Generative Adversarial Networks (GAN)

• GANs model complex, multimodal and conditional distributions without explicit distributional assumptions, generating samples that preserve spatiotemporal dependencies.This avoids the overly smooth outputs that Gaussian losses can produce for multimodal data, although GAN training requires delicate procedures (Goodfellow, 2016).Source: §2.5 · pp. 16–17
• GANs have supported image generation from text, style transfer, image-based age evolution, video prediction, super-resolution, and more robust deep-network training.These applications helped make GANs a major driver of advances and opened a sub-discipline in AI research (Sriram et al., 2017; J.-Y. Zhu et al., 2017; Antipov et al., 2017; Mathieu et al., 2015; Ledig et al., 2016; Ororbia et al., 2016).Source: §2.5 · p. 16
• GANs simultaneously train a generator to produce convincing instances and a detector to distinguish generated outputs from real data in a minimax game.The detector’s feedback helps the generator improve; at equilibrium, the generator has learned the data-generating distributions and their dependence structures.Source: §2.5 · pp. 17–18

2.6. Hardware innovations and software support

• GPU advances and parallel computation made modern DL plausible, with GPUs offering thousands of times more matrix-computing power than CPUs and ASICs emerging for training and inference.Training commonly uses GPUs or specialized hardware, whereas inference may use CPUs, GPUs, or FPGAs.Source: §2.6 · p. 18
• Open-source frameworks including TensorFlow and PyTorch support DL development, while ONNX seeks to make networks transferable across frameworks.TensorFlow has a longer history and larger community, whereas PyTorch is described as easier to learn and experiment with and benefits from cuDNN optimization.Source: §2.6 · p. 18

2.7. Understanding the generalization power of DL

• Why massive deep networks generalize well remains debated despite explanations involving big data, regularization, hierarchical architectures, and test-set constraints.Neural networks can approximate almost any function, but the mechanisms behind their generalization power remain unsettled.Source: §2.7 · p. 18
• Evidence conflicts over whether deep networks mainly memorize data or learn generalizable structure: Zhang et al. (2016a) showed memorization of noise, whereas Arpit et al. (2017) found that networks prioritize simple patterns.Regularization may reduce training-set memorization without harming test generalization, and DL has also been linked to renormalization-group feature extraction across length scales (Mehta & Schwab, 2014).Source: §2.7 · p. 19

3.1. DL applications in sciences

• The review frames cross-disciplinary adoption as a latecomer advantage for water sciences, which can learn suitable problem types, architectures, customizations, and software practices from other fields.The reviewed applications emphasize DL’s relevance to image-like and large scientific datasets rather than reproducing every study’s technical details.Source: §3.1 · p. 19

3.1.1. In physical sciences

• Physical sciences use DL to learn features directly from large datasets, improving particle detection, astronomical information extraction, genomic prediction, protein-structure prediction, and molecular-property modeling.Examples include an 8% improvement over earlier particle-detection approaches, millions-fold-faster astronomical parameter extraction at similar accuracy, and chemistry models using raw data, multitask learning, or atom-like structures.Source: §3.1.1 · pp. 19–20

3.1.2. In Geosciences: Remote Sensing, Global Change, and Climate

• Geoscientific DL extracts spatial and temporal information from imagery, detects hazards and climate extremes, improves resolution, and learns realistic urban-growth distributions.A CNN detected extreme climate events with 89%-99% accuracy, while a GAN trained on images of 30,000 cities reproduced realistic urban-mass patterns without externally imposed constraints.Source: §3.1.2 · pp. 21–22

3.1.3. In hydrology

• Hydrologic DL applications span image-based information extraction, dynamic modeling, and complex-distribution generation, generally surpassing traditional statistical methods but remaining sparsely interpreted.Reported applications include precipitation retrieval, soil-moisture and flow prediction, continental-scale modeling, and geological-media simulation.Source: §3.1.3 · pp. 22–25

3.2. The budding area of AI interpretation and its application in sciences

• DL interpretation treats networks as gray boxes and organizes posterior analysis into decision or feature backpropagation, reduced-order approximation, and visualization.These approaches aim both to increase confidence in predictions and to expose linkages that human assumptions may overlook.Source: §3.2 · p. 25

3.2.1. Local approximation by interpretable reduced-order models

• Local interpretable models explain a deep network near one input by perturbing nearby examples and fitting a simpler model, such as lasso regression, to approximate its outputs.For hydrology, synthetic network outputs could be regressed on soil, slope, and landcover to identify physiographic factors associated with different soil-moisture responses.Source: §3.2.1 · p. 26

3.2.2. Attributing decisions and features to input features

• Attribution methods assign network decisions or hidden-layer features to input fractions, offering a route to identify cloud patterns responsible for accurate precipitation-extreme estimates.The proposed hydrologic use is to visualize which cloud patterns or patches contributed to the prediction.Source: §3.2.2 · p. 26

I. Backpropagation of decisions

• These schemes can reveal the image region that most decisively supports a classification, such as identifying pixels distinguishing a boat from alternative themes (Montavon et al., 2017).Source: §I · p. 26
• Samek et al. (2017) backpropagate CNN decisions layer by layer using recorded neuron activations and contributions to produce an input-pixel importance heatmap.The network performs a forward pass, records activations, and then propagates the output activation backward through the network.Source: §I · pp. 26–27
• Decision backpropagation is especially suited to CNNs because they preserve spatial locality in intermediate hidden layers, whereas applying it to recurrent networks is more difficult.Source: §I · p. 27

II. Reverse-engineering representations stored in hidden cells.

• Deconvolutional networks reverse CNN operations to map hidden features back to input pixels, highlighting the pixels responsible for constructing features despite non-invertible pooling.Zeiler & Fergus (2014) recorded max-pooling locations and restored maxima at those positions during deconvolution.Source: §II · p. 27
• Guided inversion reconstructs images from hidden representations and shows that network layers retain photographically accurate information with varying geometric and photometric invariance.Mahendran and Vedaldi (2015) used activation maximization with a guide image and gradient ascent, while noting that inversion is non-unique and can be difficult to interpret.Source: §II · pp. 27–28

3.2.3. Visualization tools

• Visualization tools use neural-network weights and activations to help interpret deep models and are naturally related to correlation-based analysis in hydrology.These techniques are presented as a potential first choice for interpreting DL models created for hydrology.Source: §3.2.3 · p. 28

I. Building prototypes through activation maximization.

• Activation maximization synthesizes class prototypes by maximizing output activations, while expert-image regularization constrains the result toward realistic inputs (Simonyan et al., 2013; Mahendran & Vedaldi, 2015; Nguyen et al., 2016).Interactive visualization combining activations, deconvolved images, and prototypes showed that some CNN layers learn localized representations such as faces, text, and flowers despite lacking explicit recognition objectives (Yosinski et al., 2015; Zeiler & Fergus, 2014).Source: §I · p. 28
• In drought-related plant-disease models, prototypes could reveal typical infected-plant patterns and the plant and soil-moisture conditions associated with disease.Such prototypes may help humans interpret image-based disease signals and environmental conditions in hydrology.Source: §I · p. 29

II. Visualization tools for RNNs

• RNN visualization is complicated by many-to-many relationships between sequences and neurons, but can help identify forcing inputs associated with responses in dynamical water models.For example, visualization could link increased vegetation leaf area index to unusually rapid groundwater decline.Source: §II · p. 29
• LSTM activation visualizations revealed interpretable neurons that count sentence length or respond to quotations, parentheses, and punctuation, supporting diagnosis and network design.These neurons appear to support abstract concepts and long-range information retention.Source: §II · p. 29
• Context-aware RNN visualization combines expected memory responses, co-clustering, and sequence information-flow views to show how inputs assume different roles across contexts.Expected responses and co-clustering may help water-related dynamical models reveal possible causal relationships, whereas sequence visualization is language-model specific.Source: §II · pp. 29–30

3.2.4. Interpretative DL studies in sciences

• Interpretive DL studies use activation visualization, input-activation correlations, domain-specific network design, relevance backpropagation, and physical knowledge to extract scientific information.These methods treat deep networks as knowledge-extraction tools rather than only predictive models.Source: §3.2.4 · p. 30
• Input-activation correlations identify which input components trained networks use, while activation-feature analyses can reveal discriminative signals beyond established physical explanations.Examples include frequency-sensitive pathology detection and jet-image features that remained after known physics-based discriminators were removed; analogous analyses could examine temperature or precipitation signals in water demand.Source: §3.2.4 · pp. 30–31
• Physics-emulating networks can expose emergent organizing structures by conditioning elementary neurons with observational data, an approach termed “inverse emergence.”Unlike conventional hydrologic calibration, backpropagation can condition process representations as well as parameters, allowing complex functions beyond explicitly prescribed formulations.Source: §3.2.4 · pp. 31–32

• DL can model interdisciplinary human–water processes when mathematical formulations, boundary conditions, and reliable parameter values are unavailable, provided sufficient data exist.Applications include land management, irrigation demand, water saving, ecosystem responses, and urban water flows; temporal models may learn collective memory and behavioral swings.Source: §4 · pp. 32–33
• DL can convert raw and unconventional water data into high-level information, improving extraction from images and sequences, monitoring infrastructure, and prediction of crop productivity.CNNs and LSTMs could identify flood inundation, water levels, irrigation, precipitation, vegetation stress, and other observable states, with transfer learning and more uniform training.Source: §4 · p. 33
• DL can test whether new information sources and seemingly unrelated variables improve prediction, uncovering linkages among landcover, climate, soil, geomorphology, and hydrologic responses.Its expressive power integrates big data, engineers features, and can operate with lower statistical thresholds than conventional data-mining approaches.Source: §4 · pp. 33–34
• DL may address hydrologic scaling, equifinality, and regionalized parameterization by learning high-resolution effects, accelerating simulation, and predicting effective parameters from raw data.A global or continental-scale model could use diverse datasets, avoid regional customizations, reduce overfitting to regional patterns, and provide more data for training.Source: §4 · p. 34
• GANs can generate spatially structured weather, vegetation, and soil realizations, while supporting inverse hydrology when consequences are observed but driving forces are unknown.They can respect observed constraints, avoid extensive error-model specification, and explore multimodal distributions relevant to uncertainty quantification.Source: §4 · pp. 34–35
• DL may turn improved prediction into scientific discovery by exposing limits of model-free forecasting and revealing elementary processes that produce hydrologic and human-hydrology emergent dynamics.Backpropagation can condition parameters and physical processes using big data, while emulator networks may represent land gridcells or water-consumption communities.Source: §4 · p. 35

GAN training limitations

• GAN training can be unstable because generator–discriminator convergence is difficult, mode collapse can occur, and an overly strong discriminator may leave little gradient information.GANs may therefore require tricks, significant experimentation, and computer science expertise.Source: §5 · pp. 35–36

Adversarial vulnerability

• Small, unperceivable input perturbations can cause large prediction changes, making adversarial attacks a security threat, although their extent in regression and scientific applications remains little studied.Adversarial examples may be incorporated into training to improve model robustness.Source: §5 · p. 36

Model selection

• DL should not be applied non-discriminatively because random forests, Gaussian processes, CART, and SVMs can be powerful, robust, uncertainty-aware, or easier to interpret for suitable problems.Earlier-generation methods are especially valuable with limited or homogeneous data and where expertise or features are well established.Source: §5 · p. 36

Concluding remarks

• Deep learning’s self-engineered features and ability to capture high-dimensional, multimodal distributions make it a potentially disruptive force with vast opportunities in water sciences.Progress in multiscale representations, overfitting, and efficient computing enabled rapid growth in DL applications.Source: §6 · pp. 36–37
• Deep learning can improve prediction, processing efficiency, and data discoverability while also enabling exploratory discoveries that expand scientific knowledge.Its two broad functions are predictive modeling with reduced human involvement and exploratory mining of observations that can prompt new theories and experiments.Source: §6 · pp. 36–37
• CNNs and LSTMs are strong starting architectures for image and sequential data, while SDAEs suit non-image-like organization and transfer learning can substantially improve performance when applicable.Inverse emergence requires architectures designed from the ground up, whereas multi-task learning generally provides incremental gains.Source: §6 · p. 37
• AI neuroscience offers water resources major opportunities to interpret deep networks, but water applications have scarcely explored DL interpretability.Correlation-based analyses provide accessible first-order screening, while holistic tools such as reduced-order models can better capture distributed representations.Source: §6 · pp. 37–38
• Relevance backpropagation can identify inputs explaining spatial events such as precipitation retrieval, but it may be difficult to apply to recurrent networks.Emulator networks could help uncover mechanisms of emergence, though constructing viable and efficient models requires substantial expertise and collaboration with computer scientists.Source: §6 · p. 38

Tree-based methods

• CART provided interpretable predictor hierarchies for hydrologic classification and causal hypotheses, but lower-level splits were unstable and statistically inefficient.Applications included catchment classification, vegetation variability, and streamflow–water-storage relationships.Source: Appendix A · p. 38
• Random forests improved predictive power by aggregating classification trees and were used to estimate CONUS soil properties at 30-m resolution, but were difficult to interpret.Chaney et al. (2016) combined random forests with environmental covariates for this application.Source: Appendix A · p. 39

Support vector methods

• SVMs and support vector regression were robust across diverse hydrologic predictions, downscaling, runoff forecasting, and groundwater-model bias correction, though most studies used only a few sites.SVM applications included reservoir inflow, soil moisture, streamflow, evapotranspiration, lake and groundwater levels, flood stage, and sediment yield.Source: Appendix A · p. 39

Genetic programming

• Genetic programming generated explicit mathematical formulas for hydrologic mappings, ranging from simple regression-like equations to highly complex expressions when bloat control was insufficient.Applications included rainfall-runoff, groundwater levels, saturated hydraulic conductivity, evapotranspiration, flood routing, and streamflow forecasting; computational intensity limited its recent use.Source: Appendix A · pp. 39–40

Artificial neural networks

• ANNs supported extensive hydrologic applications, model-input preparation, satellite retrieval, and model-error correction, including a reported 95% reduction in annual land-surface-model error.Network geometry and activation functions affected training speed and generalization, while few studies interpreted knowledge extracted by ANNs.Source: Appendix A · p. 40

Gaussian processes

• Gaussian process regression provided posterior uncertainty estimates for streamflow, precipitation, and soil-moisture predictions, but faced computational, expressive-power, likelihood, and large-dataset scaling challenges.Sparse Gaussian processes also supported information-theoretic separation of forcing, structural, and parameter uncertainty through best achievable performance and mutual information.Source: Appendix A · pp. 40–41