PaperlayerView exact revision on arXiv

Source-linked AI summary

A trans-disciplinary review of deep learning research for water resources scientists

Chaopeng Shen

arXiv:1712.02162v3stat.MLcs.LG

TL;DR

Water sciences face challenges involving interdisciplinary processes, data discoverability, scaling, and regionalization. This review provides a technical and cross-disciplinary synthesis of deep learning for water scientists, including predictive and interpretive uses. It finds that DL has addressed data challenges across scientific fields and offers promising tools for water research, while requiring large datasets and careful comparison with simpler methods.

  • Problem

    Water sciences need approaches for interdisciplinary systems, rapidly expanding and difficult-to-discover data, hydrologic scaling, equifinality, and parameter regionalization.

  • Method

    The paper combines a concise technical overview of DL with a trans-disciplinary review of scientific and water-related applications and interpretation methods.

  • Results

    DL applications across physical and geoscientific disciplines address data challenges, improve efficiency, support scientific insight, and perform especially well for image-like and sequential data.

  • Takeaways & Limitations

    DL can serve water scientists both as a modeling approach for complex data and as an exploratory tool for interpreting learned features and emergent patterns.

  • Takeaways & Limitations

    DL requires large sample sizes, while simpler models with stronger structural assumptions can be better constrained when data are limited.

Abstract

from arXiv · show

Deep learning (DL), a new-generation of artificial neural network research, has transformed industries, daily lives and various scientific disciplines in recent years. DL represents significant progress in the ability of neural networks to automatically engineer problem-relevant features and capture highly complex data distributions. I argue that DL can help address several major new and old challenges facing research in water sciences such as inter-disciplinarity, data discoverability, hydrologic scaling, equifinality, and needs for parameter regionalization. This review paper is intended to provide water resources scientists and hydrologists in particular with a simple technical overview, trans-disciplinary progress update, and a source of inspiration about the relevance of DL to water. The review reveals that various physical and geoscientific disciplines have utilized DL to address data challenges, improve efficiency, and gain scientific insights. DL is especially suited for information extraction from image-like data and sequential data. Techniques and experiences presented in other disciplines are of high relevance to water research. Meanwhile, less noticed is that DL may also serve as a scientific exploratory tool. A new area termed 'AI neuroscience,' where scientists interpret the decision process of deep networks and derive insights, has been born. This budding sub-discipline has demonstrated methods including correlation-based analysis, inversion of network-extracted features, reduced-order approximations by interpretable models, and attribution of network decisions to inputs. Moreover, DL can also use data to condition neurons that mimic problem-specific fundamental organizing units, thus revealing emergent behaviors of these units. Vast opportunities exist for DL to propel advances in water sciences.

1. Motivations

Water sciences face expanding interdisciplinarity, data discoverability, scaling, equifinality, and parameter-regionalization challenges. The review presents deep learning as a potential framework for addressing these challenges while offering technical orientation and cross-disciplinary examples.

  • Water research increasingly must model interactions among hydrology, biogeochemistry, climate, vegetation, soils, and human dynamics.
  • The growing volume and diversity of satellite, sensor, and other observations makes extracting hidden, abstract information difficult with conventional approaches.
  • Scale-dependent process representations, unresolved heterogeneity, and equifinality remain longstanding hydrologic modeling problems.
  • Regionalized datasets reduce the number of observations available to condition models and can weaken their statistical power.
  • The review argues that DL can support interdisciplinary modeling, information extraction, scaling, uniform parameterization, scenario generation, and scientific exploration.
  • The paper provides water scientists with a concise technical overview, a cross-disciplinary progress update, and an entry point for exploring DL.

2. Deep learning basics

This section introduces machine learning and explains how deep networks differ from earlier methods. It emphasizes automatic feature engineering, scalability to complex data, and representative scientific applications.

  • Machine learning includes supervised learning for predicting observed targets and unsupervised learning for discovering structure without labeled targets.
  • Artificial neural networks learn transformations through connected neurons by minimizing a loss function, commonly using backpropagation to update weights.
  • Deep learning generally uses large, multilayer networks operating directly on big, raw data.
  • Deep networks automatically extract cascades of abstract features, reducing reliance on manually engineered inputs and enabling transfer learning.
  • The SDAE precipitation example removed 98% of bias and reduced variance by 31% relative to the referenced operational system.
  • Deep networks can model complex functions and spatiotemporal dependencies from large datasets, whereas earlier methods often require expert-designed features and may scale less effectively.

2.3. Popular deep learning network architectures

The section surveys common deep-learning architectures and their structural purposes. CNNs specialize in local spatial patterns, while LSTMs preserve sequential information through memory mechanisms.

  • CNNs are widely adopted for image tasks, whereas LSTMs are designed for sequence-learning tasks such as time-series modeling.
  • MLPs require many parameters, do not enforce spatial structure, and are not the modern architecture of choice for many complex tasks.
  • Autoencoders reconstruct inputs through an information bottleneck, while denoising autoencoders add noise to improve robustness and reduce overfitting.
  • SDAEs stack denoising autoencoders so deeper layers capture increasingly abstract representations as network width shrinks.
  • CNN convolution and pooling operate on local neighborhoods, enabling hierarchical extraction of geometric and spatial patterns.
  • LSTMs learn directly from time series and use memory cells with gates that control what information is remembered, retained, and output.
  • DBNs stack restricted Boltzmann machines but lack recurrent memory, so previous inputs and states do not influence future predictions.

2.4. Regularization (model complexity penalization) techniques

Regularization controls model complexity to reduce overfitting while preserving predictive information. The section reviews norm penalties, early stopping, dropout, and recurrent-network-specific dropout choices.

  • Regularization adds penalties or training constraints that reduce complexity and help prevent overfitting to noise or peculiarities in training data.
  • AIC favors simpler models by trading off fit against parameter count, but discarding predictors can sacrifice information.
  • Norm-based regularization penalizes large coefficients while retaining all predictors and assigning lower weights to less important parameters.
  • Early stopping halts training before maximum performance when validation error begins increasing relative to earlier epochs.
  • Dropout randomly blocks connections, creating reduced-order subnetworks whose ensemble-like predictions improve robustness.
  • Blind dropout on recurrent links can amplify noise and cause memory loss, so recurrent LSTMs apply dropout only to forward connections.

2.5. Generative Adversarial Networks (GAN)

GANs use two competing deep networks to learn complex, multi-modal data distributions and generate realistic new instances. Their flexibility has enabled applications across image, video, style-transfer, and geoscientific problems, although training is delicate.

  • Strengths and applications: GAN applications include sentence-conditioned image generation, painting-style transfer, image age transformation, video prediction, super-resolution, and geoscientific inversion and urbanization projection.These applications illustrate GANs’ ability to generate or transform instances that resemble training data.
  • Strengths and applications: GANs capture complex, multi-modal, and conditional distributions without explicit distributional assumptions.This flexibility is also accompanied by delicate training procedures.
  • Architecture and training: GANs simultaneously train a generator and detector in competition: the generator produces instances, while the detector distinguishes generated from real data.The networks play a minimax game in which the generator tries to confuse the detector and the detector tries to make correct decisions.
  • Architecture and training: The GAN value function combines detector judgments of real data with judgments of generator outputs from random noise.The detector and generator have separate parameters, and the two expectation terms reward correct classification of real and counterfeit instances.
  • Architecture and training: At equilibrium, the detector cannot reliably distinguish generated instances from real ones, indicating that the generator has learned data-generating distributions and dependence structures.This outcome describes the intended endpoint of alternating optimization, in which each network is trained while the other is held fixed.

2.6. Hardware innovations and software support

Deep learning became practical through major advances in parallel-computation hardware, especially GPUs, and is supported by evolving open-source software platforms. The section also situates deep learning within ongoing debates about generalization and memorization.

  • Hardware innovations: GPU advances and the recognition of GPUs as efficient parallel-computation hardware were central to deep learning’s practical emergence.Deep learning concepts predated widespread adoption, but popularity increased after GPUs became suitable for the task around 2010.
  • Software support: TensorFlow and PyTorch provide open-source environments with access to low-level operators and loss functions for building deep learning models.TensorFlow is Google-supported and Python-based, while PyTorch is the Python version associated with Torch.
  • Generalization debates: Deep-network generalization remains debated despite explanations involving big data, regularization, hierarchical architectures, and test-set constraints.The discussion reflects uncertainty about why very large networks generalize so well.
  • Generalization debates: Empirical studies show that deep networks can memorize noise, while other work finds they tend to learn simple patterns before memorizing data.These contrasting findings complicate explanations that attribute generalization solely to regularization.

3. Trans-disciplinary applications of DL and its interpretation

Across physical sciences, remote sensing, climate science, and hydrology, deep learning has addressed large or complex data problems through feature learning, transfer learning, and specialized architectures. Reported applications include faster information extraction, improved simulations and retrievals, hydrologic forecasting, and more realistic geological-media generation, while interpretation and small-data justification remain concerns.

  • Cross-disciplinary applications: Deep learning has helped scientific disciplines handle large datasets, improve efficiency, and extract information by learning features rather than relying solely on manual feature engineering.Examples include abnormal-particle detection in high-energy physics and automatic parameter extraction from telescope images.
  • Physical sciences: In astronomy, CNNs extracted galactic parameters from telescope images at accuracy similar to complex approaches while operating millions of times faster.The passage also reports applications to gravitational-wave artifacts and galaxy morphology, with mixed results in some cases.
  • Physical sciences: Computational biology and chemistry used deep learning to learn from genomic or raw-level data, with multi-task learning and shared representations emerging as recurring strategies.DeepBind was used to predict protein interactions from genomic sequences, while chemistry studies reported raw-data models performing on par with or better than predefined-feature models.
  • Remote sensing and climate: Remote sensing uses CNNs to extract geometric, textural, and spatial information from raw images, often outperforming methods based on spectral signatures or handcrafted features.Transfer learning, data augmentation, and architecture adaptation help address limited data quantities.
  • Remote sensing and climate: Remote-sensing deep learning has been applied to wildfire and landslide detection, but one landslide study asserted preprocessing was necessary without directly showing supporting results.This is a concrete example of an application accompanied by an evidentiary limitation.
  • Remote sensing and climate: Climate applications focus on detecting extreme events and addressing resolution challenges through CNNs, dynamic convolutional layers, and super-resolution downscaling.These methods target image-based event identification, dynamic weather prediction, and higher-resolution climate variables.
  • Remote sensing and climate: A GAN trained on images of 30,000 cities reproduced realistic urban-mass concentrations and spreads without externally imposed constraints.The model learned constraints such as avoiding construction on water from the training data itself.
  • Hydrology: Hydrologic studies reported gains in precipitation retrieval, time-series prediction, continental soil-moisture modeling, and geological-media simulation.Examples include 33%-43% corrections on false-alarm pixels versus PERSIANN-CCS, LSTM reproduction of SMAP soil-moisture dynamics, and faster GAN exploration of posterior geological distributions.

I. Backpropagation of decisions

Decision-interpretation methods trace network outputs back to influential inputs or reconstruct inputs from hidden representations. These tools support model diagnosis and interpretation, especially for image-based CNNs.

  • I. Backpropagation of decisions: Decision backpropagation traces a CNN’s output backward through recorded neuron contributions to produce an input heatmap.The heatmap highlights pixel importance for the network’s decision.
  • I. Backpropagation of decisions: CNN decision-backpropagation is especially suitable when spatial locality persists through intermediate hidden layers.The approach is more difficult for recurrent neural networks.
  • I. Backpropagation of decisions: Deconvolutional networks reverse CNN operations to disaggregate hidden features into the input space, using recorded maximum-pooling locations.The reconstruction assigns the maximum value to the recorded location and zeros elsewhere.
  • I. Backpropagation of decisions: Activation maximization synthesizes input images whose hidden representations match a target while maximizing the output activation for a class.Gradient ascent solves the maximization problem, and reconstructed layers can be photographically accurate.
  • I. Backpropagation of decisions: Prototype synthesis creates the most typical input pattern for a class, helping diagnose model errors and examine class differences.The paper suggests applying this idea to typical drought-related plant-disease patterns.

II. Visualization tools for RNNs

Visualization methods expose how recurrent and other deep-network units respond to inputs, contexts, and learned features. These approaches extend interpretation from individual activations to broader scientific knowledge extraction.

  • II. Visualization tools for RNNs: RNN visualization is more complicated than CNN visualization because inputs can activate every neuron and hidden neurons can respond to multiple sequences.This creates many-to-many relationships between sequences and neuron responses.
  • II. Visualization tools for RNNs: LSTM visualization revealed interpretable neurons that count sentence length or respond to quotation marks, parentheses, and punctuation.These findings show that individual recurrent units can exhibit recognizable response functions.
  • II. Visualization tools for RNNs: Parallel-coordinate analysis lets users select time windows and search for hidden states with contrasting activation patterns across periods.Selected hidden states can then be matched to input words and other related words.
  • II. Visualization tools for RNNs: Because identical rainfall can produce different runoff under different antecedent soil moisture, neuron responses must be interpreted in context.Ming et al.’s system addresses this by averaging responses, co-clustering neurons and inputs, and examining context.
  • II. Visualization tools for RNNs: Interpretive deep-learning studies combine activation visualization, correlation analysis, domain-specific network design, and relevance backpropagation.These methods use neural networks as tools for extracting scientific knowledge.
  • II. Visualization tools for RNNs: Correlating neuron activations with input signals can identify information sources the model learned as important, rather than merely observed-variable associations.Applications include locating discriminating image pixels and features beyond known physical laws.

4. Tackling water resources challenges with the help of DL

The paper proposes deep learning as a flexible approach for extracting information, modeling interdisciplinary and temporal relationships, and exploring hydrologic mechanisms. It also highlights opportunities for scaling, generative modeling, and emergent-behavior analysis.

  • 4. Tackling water resources challenges with the help of DL: DL can model interdisciplinary processes when mathematical formulations are incomplete but sufficient data exist, including human-water interactions.Its flexible representations can capture system-wide response patterns and temporal dependence.
  • 4. Tackling water resources challenges with the help of DL: DL can extract hydrologic and ecosystem states from remote-sensing images and sequential monitoring data with automated feature engineering.Candidate outputs include flood inundation, water levels, irrigation amounts, precipitation, and vegetation stress.
  • 4. Tackling water resources challenges with the help of DL: DL-based models can test whether new information sources and seemingly unrelated variables contain useful relationships for hydrologic prediction.Examples include linkages among landcover, climate, soil, geomorphology, Karst geology, and bedrock outcrops.
  • 4. Tackling water resources challenges with the help of DL: DL may help address hydrologic scaling and equifinality by learning effects of resolution from high-resolution simulations.The paper notes that hydrologic networks require customization for multiscale, heterogeneous processes.
  • 4. Tackling water resources challenges with the help of DL: GANs can generate weather, vegetation, and soil features while learning covariate relationships and preserving high-order spatial structure.The paper connects these capabilities to hydrologists’ existing use of stochastic generators.
  • 4. Tackling water resources challenges with the help of DL: Model-free learning predicted chaotic attractors and Lyapunov exponents for 8 Lyapunov times, extending previous methods by 8 times.The example illustrates how improved predictive accuracy can support qualitative changes in scientific understanding.
  • 4. Tackling water resources challenges with the help of DL: Networks designed around elementary units could use data to infer hydrologic functions and explain emergent dynamics at land-gridcell or community scales.This proposed approach contrasts with purely observational or fundamental-law-based routes to emergent behavior.

5. Limitations and potential issues of DL

The paper identifies practical and scientific limits on deep-learning use, including unstable GAN training, adversarial vulnerability, data requirements, and cases where simpler models are preferable.

  • 5. Limitations and potential issues of DL: GAN training can be unstable because generator–discriminator convergence is difficult, mode collapse can occur, and an overly strong discriminator can remove learning gradients.These failure modes complicate reliable generative modeling.
  • 5. Limitations and potential issues of DL: GAN applications may require specialized tricks, substantial experimentation, and computer-science expertise.The paper presents this as a practical constraint on applying GANs in water sciences.
  • 5. Limitations and potential issues of DL: Small, imperceptible input perturbations can cause large prediction changes under adversarial attack, producing incorrect outcomes.Most studies concern classification, while the issue remains less examined for regression and scientific applications.
  • 5. Limitations and potential issues of DL: DL models require large sample sizes, and simpler models with stronger structural assumptions can be better constrained when data are limited.The paper cautions against applying DL indiscriminately, especially to small or relatively homogeneous datasets.

6. Concluding remarks

Deep learning offers water sciences both powerful predictive and processing capabilities and an exploratory tool for discovering features and mechanisms beyond current knowledge. The review synthesizes architectural guidance and interpretive approaches while noting practical limits in understanding deep-network decisions.

  • Concluding remarks: Deep learning supports accurate prediction, greater processing capability, and reduced human involvement, while also enabling exploratory data mining that can expand knowledge.The review identifies these as two broad DL functionalities across scientific applications.
  • Concluding remarks: CNNs and LSTMs are recommended starting architectures for image-like and sequential data, while SDAEs suit data unlike 2D or 3D images.The architectural choice depends on the organization of the data and the scientific objective.
  • Concluding remarks: AI neuroscience interprets deep-network decisions through correlation analysis, reduced-order models, relevance backpropagation, and specialized architectures.These approaches can support scientific understanding, although water-resources applications remain underdeveloped.
  • Concluding remarks: Correlation-based analysis is useful for early screening, but isolated hidden-unit activations can provide incomplete or inconclusive interpretations.Holistic tools such as reduced-order models may better reflect the network’s inner workings.
  • Concluding remarks: Relevance backpropagation can identify input features associated with decisions in spatial or image-like problems, whereas recurrent-network applications may be difficult.Constructing emulator networks to study emergence also requires substantial computer-science expertise and collaboration.

Appendix A. Some early machine learning methods and their hydrologic applications

Early machine-learning methods have long supported hydrologic prediction, classification, uncertainty analysis, and explicit formula discovery. Their applications offer trade-offs among interpretability, robustness, computational cost, and predictive suitability.

  • Early machine-learning methods: SVMs, CART, and ANNs have been extensively applied to hydrologic and land-surface predictions with moderate success.CART additionally represents hypotheses through hierarchical predictor thresholds.
  • Early machine-learning methods: Random forests combine predictions from ensembles of classification trees and can estimate soil properties at 30-m resolution for CONUS.The method is well suited to this application but is usually difficult to interpret.
  • Early machine-learning methods: SVM constructs separating hyperplanes or robust regression models using a subset of training data.Support vector regression ignores points too far from model predictions to improve robustness.
  • Early machine-learning methods: Genetic programming evolves executable symbolic formulas for input-output mappings and has been used in soil, evapotranspiration, flood-routing, and streamflow applications.A soil-water-retention application produced a very complex formula, and GP can be computationally intensive.
  • Early machine-learning methods: ANN applications in hydrology span rainfall-runoff modeling, streamflow, groundwater management, water quality, salinity, rainfall estimation, wave heights, and real-time forecasting.This breadth reflects a long history of neural-network use in hydrologic research.
  • Early machine-learning methods: Gaussian process regression models covariance-based smoothness and naturally provides posterior uncertainty estimates.Deep Gaussian processes extend this framework through cascades of hidden layers modeled with Gaussian processes.
  • Early machine-learning methods: Information-theoretic methods can separate input uncertainty from model structural error by estimating the best achievable performance of a model given forcings and benchmark observations.The approach is model-independent in its estimation of the performance bound.
Loading 1712.02162v3…