PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning of Subsurface Flow via Theory-guided Neural Network

Nanzhe Wang, Dongxiao Zhang, Haibin Chang, Heng Li

arXiv:1911.00103v1cs.LGstat.ML

TL;DR

The paper addresses limitations of data-driven neural networks for scientific problems, especially their data demands and reliability with scarce data. It proposes TgNN, which jointly trains on observations or simulations and theory-based constraints, and reports better predictability, reliability, and generalizability than ANN in tested subsurface-flow scenarios.

  • Problem

    Scientific neural-network applications can require large datasets, while available data may be scarce and predictions may violate governing scientific laws or practical theories.

  • Method

    TgNN trains a neural network with data mismatch together with PDE, physical, engineering-control, and expert-knowledge loss terms for subsurface flow with heterogeneous parameters.

  • Results

    TgNN achieves much better predictability, reliability, and generalizability than ANN models across the tested subsurface-flow scenarios.

  • Takeaways & Limitations

    The framework produces predictions that fit available data while adhering to physical and engineering constraints and supports future-response prediction under varied conditions.

  • Takeaways & Limitations

    The heterogeneous-parameter formulation assumes a separable exponential covariance model, and transfer learning initializes the initial condition from a pre-trained model at time step 20.

Abstract

from arXiv · show

Active researches are currently being performed to incorporate the wealth of scientific knowledge into data-driven approaches (e.g., neural networks) in order to improve the latter's effectiveness. In this study, the Theory-guided Neural Network (TgNN) is proposed for deep learning of subsurface flow. In the TgNN, as supervised learning, the neural network is trained with available observations or simulation data while being simultaneously guided by theory (e.g., governing equations, other physical constraints, engineering controls, and expert knowledge) of the underlying problem. The TgNN can achieve higher accuracy than the ordinary Artificial Neural Network (ANN) because the former provides physically feasible predictions and can be more readily generalized beyond the regimes covered with the training data. Furthermore, the TgNN model is proposed for subsurface flow with heterogeneous model parameters. Several numerical cases of two-dimensional transient saturated flow are introduced to test the performance of the TgNN. In the learning process, the loss function contains data mismatch, as well as PDE constraint, engineering control, and expert knowledge. After obtaining the parameters of the neural network by minimizing the loss function, a TgNN model is built that not only fits the data, but also adheres to physical/engineering constraints. Predicting the future response can be easily realized by the TgNN model. In addition, the TgNN model is tested in more complicated scenarios, such as prediction with changed boundary conditions, learning from noisy data or outliers, transfer learning, and engineering controls. Numerical results demonstrate that the TgNN model achieves much better predictability, reliability, and generalizability than ANN models due to the physical/engineering constraints in the former.

1. Introduction

Deep neural networks have broad scientific applications but often require abundant data and may be unreliable when data are scarce. The paper proposes TgNN, which incorporates theory and engineering knowledge into training and reports superior performance to conventional ANN across several subsurface-flow scenarios.

  • DNNs have been applied across AI and scientific fields, including biomedicine, economics, chemistry, and physics.
  • Limited data can reduce DNN reliability because scientific and engineering data collection is often time-consuming and expensive.
  • Theory-guided learning incorporates scientific knowledge into neural-network models through designs, loss functions, refinements, hybrid models, or theory-based augmentation.
  • TgNN transforms governing equations, physical constraints, engineering controls, and expert knowledge into loss-function regularization terms.
  • TgNN is applied to subsurface flow with heterogeneous model parameters and tested under changed boundaries, noisy or outlier data, and transfer learning.
  • TgNN achieves superior performance to conventional ANN in the tested scenarios.

2. Methodology

The methodology develops a theory-guided neural network that combines data-driven approximation with governing-equation, boundary/initial-condition, engineering, and expert-knowledge constraints. It also extends the formulation to heterogeneous subsurface parameters using random-field representations and numerical derivatives.

  • Deep neural network architecture: The deep neural network maps input variables to outputs through input, hidden, and output layers, with trainable weights, biases, and activation functions.Training minimizes a loss function, commonly based on mean squared error, using optimization algorithms such as stochastic gradient descent.
  • Theory-guided neural network: The TgNN approximates hydraulic head with a neural network and defines a residual measuring violation of the saturated-flow governing equation.Automatic differentiation computes the network-output derivatives needed for the residual.
  • Physical constraints: Boundary and initial conditions are expressed as residuals whose mean squared errors are added to the training objective.Residual collocation points can be randomly selected because they do not require labeled data.
  • Engineering and expert constraints: Engineering controls and expert knowledge impose inequality constraints, with ReLU-based penalties applied when those constraints are violated.The ReLU function returns zero for nonpositive inputs and the original input for positive inputs.
  • TgNN training objective: The total loss combines data, PDE, boundary-condition, initial-condition, engineering-control, and expert-knowledge terms weighted by hyper-parameters.The TgNN is trained by minimizing this composite loss with optimization algorithms such as SGD, Adagrad, or Adam.
  • Heterogeneous parameters: For heterogeneous hydraulic conductivity, the flow equation includes spatial conductivity derivatives, while network derivatives use automatic differentiation and conductivity derivatives generally use numerical differences.Karhunen–Loeve expansion parameterizes heterogeneous fields through eigenvalues, eigenfunctions, and random variables, with truncation determined by eigenvalue decay.

3. Case Studies

The case studies evaluate TgNN for two-dimensional transient saturated subsurface flow across heterogeneous conductivity fields and challenging prediction settings. TgNN predictions generally match reference solutions better than ANN predictions, including changed boundary conditions, noisy data, outliers, and transfer learning.

  • Case-study design: The experiments assess TgNN accuracy and robustness against ANN across several subsurface-flow cases and scenarios.The scenarios include heterogeneous conductivity fields, changed boundary conditions, noisy data, outliers, and transfer learning.
  • TgNN formulation: TgNN uses time and spatial position as inputs and hydraulic head as output, with data, PDE, boundary-condition, initial-condition, and expert-knowledge terms in the loss function.For the base case, expert knowledge constrains hydraulic head to the range 0 to 1, and the model is optimized using Adam.
  • Baseline predictions: Across four hydraulic conductivity fields, TgNN predictions match reference values well and are superior to ANN predictions.The reported relative L2 error is far lower for TgNN, while its R2 score is closer to 1 than ANN's.
  • Changed boundary conditions: TgNN handles changed boundary conditions by incorporating the new condition into the loss function, while ANN gives unsatisfactory predictions despite receiving the condition in training data.The changed condition raises the hydraulic head at x=1020 L to 2 L at time step 20; predictions are evaluated at time steps 30, 40, and 50.
  • Noise and outliers: TgNN is more robust than ANN to noisy data and outliers because physical and engineering constraints penalize violations during training.As outliers increase, accuracy worsens, especially at early time; the effect is reduced later, while ANN can be seriously degraded by outliers.
  • Transfer learning: 16.7%: transfer learning reduces training time while retaining a slight accuracy advantage over fully retrained models.The authors report that pre-trained parameters can transfer information to similar new systems, and efficiency gains may increase for more complicated problems.

L MSE MSE MSE

The engineering-control case adds penalties for violating a prescribed drawdown, allowing TgNN to model controlled pumping while exposing the limits of PDE-only constraints.

  • Engineering control: A drawdown violation triggers a penalty, and the constant-pumping-rate equation receives reduced loss weight when control is violated.This relaxes the strict pumping-rate constraint under engineering-control violations.
  • Prediction results: TgNN predicts the hydraulic-head distribution well at time steps 10 and 50 in the engineering-control case.The case evaluates spatial predictions under a governing equation containing a sink term.
  • Prediction results: With engineering control, TgNN produces a prediction similar to constant-pressure pumping at the well location.The comparison is made against TgNN without engineering control.
  • Prediction results: The hydraulic-head value can fall below the control value, despite the similar controlled-pumping prediction.The passage identifies this lower-than-control behavior as part of the comparison.
  • Implication: Engineering-control terms are necessary for practical engineering operations that cannot be achieved by incorporating only the PDE constraint.The conclusion links this necessity specifically to the engineering-control case.

4. Discussions and Conclusions

The paper presents TgNN as a physics- and engineering-guided framework for subsurface flow, including heterogeneous parameters, and reports superior performance across tested scenarios relative to ANN.

  • Conclusions: TgNN incorporates governing equations, boundary conditions, expert knowledge, and engineering controls into subsurface-flow learning.The framework combines data-driven learning with physical laws and engineering theory.
  • Conclusions: Karhunen–Loeve expansion parameterizes heterogeneous hydraulic-conductivity fields for TgNN.This enables the framework to address problems with heterogeneous model parameters more effectively.
  • Conclusions: TgNN is tested for future prediction, changed boundary conditions, noisy data or outliers, transfer learning, and engineering controls.These cases evaluate applicability across multiple subsurface-flow situations.
  • Conclusions: TgNN achieves far superior results to ANN across the tested scenarios.The reported advantages include better accuracy, generalization ability, and robustness.
  • Scope and limitations: The framework assumes early-time simulation or field data and known heterogeneous-field parameters.It remains a data-driven approach rather than a replacement for numerical solutions of the governing equations.

Appendix A

Appendix A develops analytical or semi-analytical eigenvalue and eigenfunction representations for exponential covariance fields, including a separable two-dimensional construction.

  • One-dimensional covariance: For a one-dimensional exponential covariance process, eigenvalues and eigenfunctions can be solved analytically or semi-analytically.The covariance uses variance and correlation length parameters for the random field.
  • One-dimensional covariance: The one-dimensional eigenfunctions use trigonometric forms whose frequencies are positive roots of a characteristic equation.The appendix gives the characteristic equation and associated eigenfunction expressions.
  • Two-dimensional construction: For separable two-dimensional covariance, eigenvalues and eigenfunctions are combined across the x and y directions.The appendix expresses the two-dimensional quantities through directional components.

Appendix B

Appendix B provides detailed prediction results for multiple scenarios in the case without a sink term.

  • Appendix B: Appendix B focuses on prediction results for the case without a sink term.Its scope is stated as detailed scenario-specific prediction results.
  • Appendix B: The appendix contains detailed prediction results across different scenarios.The passage does not specify the individual scenarios.
  • Appendix B: Appendix B supplements the main case analysis with additional prediction results.This follows directly from its stated purpose of providing detailed results.

B.1 Prediction with changed boundary conditions

Using the first 18 monitored time steps as training data, the models predict the following 32 steps for several hydraulic-conductivity fields. At time step 50, TgNN predictions match reference values well.

  • The experiment trains on hydraulic-head observations from the first 18 time steps and predicts the following 32.The training set contains 1000 extracted data points per monitored time step.
  • At time step 50, TgNN predictions match the reference hydraulic-head values well across the tested conductivity fields.The comparison is shown for hydraulic conductivity fields (b), (c), and (d).

B.2 Prediction with changed boundary conditions

The boundary condition changes at time step 20, after which no observations are available for training. Across time steps 30, 40, and 50, TgNN performs better than ANN.

  • The prescribed hydraulic head at the boundary rises from 0 L to 2 L at time step 20.
  • Training uses 1000 hydraulic-head data points per time step from the first 20 time steps, with no data after the boundary change.
  • TgNN obtains better prediction performance than ANN at time steps 30, 40, and 50 after the boundary condition changes.The predictions are presented for each of these future time steps.

B.3 Prediction of future response in the presence of data noise and outliers

The study evaluates future-response prediction with noisy and outlier-contaminated training data. TgNN is more robust than ANN, and its outlier-related prediction improves from time step 30 to time step 50.

  • Noise levels of 5%, 10%, and 20% and outlier levels of 5%, 7%, and 10% are added to monitored training data.
  • TgNN has better robustness than ANN when noise exists in the training data.The comparison uses predictions at time step 50 for different noise levels.
  • TgNN achieves better results than ANN for predictions with different amounts of outliers.Predictions are compared at time steps 30 and 50.
  • Outlier effects are reduced by incorporated scientific knowledge as time passes, making time-step-50 predictions better than time-step-30 predictions.

Appendix C

Appendix C examines a pumping well with a prescribed volumetric rate and a hydraulic-head control rule. Without engineering control, the well head exceeds the control value; TgNN with control enforces the prescribed-head operation.

  • The case models a pumping well at x=520 L and y=520 L with volumetric pumping rate Q=50 L^3/T.
  • When the well head reaches the control value 81 L, the well is operated to maintain the prescribed hydraulic head.The control is intended to avoid excessive pressure drawdown at the well.
  • Without engineering control, the hydraulic head at the well location exceeds the control value.Figure C.1 compares TgNN with and without engineering control at time steps 20, 30, 40, and 50.
Loading 1911.00103v1…