Deep UQ: Learning deep neural network surrogate models for high dimensional uncertainty quantification

Rohit Tripathy; Ilias Bilionis

Source-linked AI summary

Deep UQ: Learning deep neural network surrogate models for high dimensional uncertainty quantification

Rohit Tripathy, Ilias Bilionis

arXiv:1802.00850v1Feb 2, 2018physics.comp-phcs.LGstat.ML

TL;DR

High-dimensional UQ can require hundreds of thousands or millions of forward evaluations, while the curse of dimensionality complicates surrogate construction. The paper proposes DNN surrogate models and demonstrates effective prediction of PDE solutions for varied diffusion fields.

Problem
Monte Carlo methods for UQ may require hundreds of thousands or millions of forward evaluations, making them unsuitable for UQ tasks in high-dimensional settings.
Method
The paper proposes a systematic methodology for constructing DNN surrogate models for numerical simulators.
Results
The DNN surrogate effectively predicts the PDE solution, including for diffusion fields with lengthscales that are not specified here.
Takeaways & Limitations
The demonstrated methodology supports surrogate prediction of responses for input random fields and PDE solutions across varied diffusion-field settings.
Takeaways & Limitations
PCA is constrained to discovering only linear projections, while Monte Carlo convergence can require hundreds of thousands or millions of evaluations.

Abstract

from arXiv · show

State-of-the-art computer codes for simulating real physical systems are often characterized by a vast number of input parameters. Performing uncertainty quantification (UQ) tasks with Monte Carlo (MC) methods is almost always infeasible because of the need to perform hundreds of thousands or even millions of forward model evaluations in order to obtain convergent statistics. One, thus, tries to construct a cheap-to-evaluate surrogate model to replace the forward model solver. For systems with large numbers of input parameters, one has to deal with the curse of dimensionality - the exponential increase in the volume of the input space, as the number of parameters increases linearly. In this work, we demonstrate the use of deep neural networks (DNN) to construct surrogate models for numerical simulators. We parameterize the structure of the DNN in a manner that lends the DNN surrogate the interpretation of recovering a low dimensional nonlinear manifold. The model response is a parameterized nonlinear function of the low dimensional projections of the input. We think of this low dimensional manifold as a nonlinear generalization of the notion of the active subspace. Our approach is demonstrated with a problem on uncertainty propagation in a stochastic elliptic partial differential equation (SPDE) with uncertain diffusion coefficient. We deviate from traditional formulations of the SPDE problem by not imposing a specific covariance structure on the random diffusion coefficient. Instead, we attempt to solve a more challenging problem of learning a map between an arbitrary snapshot of the diffusion field and the response.

1. Introduction

High-dimensional uncertainty quantification is difficult because Monte Carlo requires many expensive simulator evaluations, while surrogate construction faces the curse of dimensionality. The paper proposes DNN surrogates that learn output-informed nonlinear low-dimensional structure for challenging stochastic PDE inputs.

Monte Carlo methods may require hundreds of thousands or millions of samples for convergent statistics, making them unsuitable for UQ with expensive computer codes.
Surrogate models replace expensive simulators with cheap-to-evaluate approximations that can then support UQ using Monte Carlo techniques.
The curse of dimensionality causes the required number of input-space evaluations to grow exponentially with the number of stochastic dimensions.A d-variate function may require O(10^d) locations to maintain comparable approximation accuracy.
Existing dimensionality-reduction methods have important limitations: sensitivity methods struggle with functional uncertainties, while PCA discovers only linear projections and can retain high intrinsic dimensionality.
The paper proposes DNN surrogate models that recover a low-dimensional nonlinear manifold and use model outputs alongside inputs.The methodology is based on MLPs and is presented as a nonlinear generalization of active subspaces.
The approach is demonstrated on an SPDE with uncertain diffusion, without imposing a specific covariance structure, including structurally dissimilar test input fields.

2. Methodology

The methodology formulates a physical simulator as an unknown function whose evaluations are used to construct a surrogate for uncertainty quantification. High-dimensional inputs make direct surrogate construction computationally infeasible without dimensionality reduction.

The simulator maps input parameters such as material properties and boundary conditions to a scalar quantity of interest.The function is generally unknown in closed form and must be queried at feasible input values; observations may be noisy.
The proposed methodology builds an approximation of the unknown simulator from finite queried data, allowing noisy output observations.The observed response is represented as y = f(ξ) + ϵ, with Gaussian noise in the stated model.
Inputs may number in the hundreds or thousands, making surrogate construction computationally infeasible without dimensionality reduction.The challenge arises when only a finite number of simulator evaluations is available.

2.1. Uncertainty propagation

The uncertainty propagation problem begins with uncertain simulator inputs represented by a probability distribution. The goal is to characterize resulting output statistics, including the mean and probability density.

Uncertain inputs are formalized using a suitable probability distribution.This distribution represents beliefs about the unknown input vector.
The objective is to characterize statistical properties of the output f(ξ).The stated examples include the output mean and probability density.
This task is formally called the uncertainty propagation problem.

2.2. Surrogate model structure

The surrogate projects high-dimensional inputs onto a nonlinear low-dimensional manifold and applies a nonlinear link function to predict the response. This structure generalizes active-subspace surrogates and is implemented with a deep neural network.

Monte Carlo estimation is computationally infeasible because convergence requires many forward-model evaluations.The surrogate instead relies on information obtained from carefully selected design points.
The surrogate is trained from finite simulator design points and their potentially noisy outputs.The dataset consists of input samples, corresponding responses, and an approximation ˆf of the true model f.
The proposed surrogate projects input data onto a nonlinear low-dimensional manifold before producing the response.Its structure is written as ˆf(ξ) = g(h(ξ)), where h is the projection function and g is the link function.
The link function is a generic nonlinear function of the projected inputs.The projected variable ζ belongs to a lower-dimensional space than the original input.
The construction generalizes the active-subspace surrogate by allowing nonlinear projection directions.The proposed structure is described as capturing directions that explain most variation in the model output.
The complete projection-and-link structure is posed as a deep neural network.The projection function is also recognized as the encoder portion of a neural-network autoencoder.

2.3. Structure of a feedforward Deep neural network

A feedforward deep neural network composes nonlinear transformations across hidden layers, with neurons applying linear transformations followed by element-wise nonlinearities. Network depth and layer widths control representational complexity, and this work uses the Swish activation function.

A deep neural network contains many hidden layers whose activations feed into subsequent layers.Intermediate layers are hidden layers, while input and output layer sizes are fixed by the task.
Increasing hidden-layer number and size allows the network to learn functions of arbitrary complexity.The input and output dimensions remain determined by the problem.
An autoencoder is a deep neural network used to recover a low-dimensional embedding of a high-dimensional space.
Each neuron performs a linear transformation followed by an element-wise nonlinear transformation.A neuron accepts one or more inputs and produces an output through these two operations.
Each hidden-layer activation is computed as z^(j) = σ(W^(j)z^(j−1) + b^(j)).The input is z^0 = ξ with d_0 = D; W^(j) and b^(j) are the layer weights and biases.
The work uses the Swish activation function in its neural networks.Swish is introduced as an activation function with a parameter that may be constant or learned from data.
The network architecture is fully specified collectively by its weights and biases.

2.4. Training a deep neural network

Training treats the surrogate as a parameterized function fitted from finite input-output observations by minimizing prediction mismatch.

For regression, the mismatch function is typically the mean squared error.
Because only a limited set of observations is available, training uses dataset examples rather than direct access to the underlying function.
The training dataset contains N examples D_i = (x_i, y_i), and optimization minimizes mismatch between f̂(x_i) and y_i.
In practice, each optimization iteration averages the loss over a randomly sampled mini-batch D_M contained in D.

2.5. Regularized loss function

The regularized loss combines data-fit terms with weight penalties to control DNN complexity and improve generalization to unseen data.

Under Gaussian noise, the unknown variance σ^2 represents discrepancies from discretization, model discrepancy, noise, and other error sources.
DNN training penalizes the misfit function with a regularizer because DNNs are prone to overfitting.
Regularization ensures that the DNN generalizes better to unseen data.
L1 penalties promote sparsity, whereas L2 penalties drive weights close to 0.
The elastic net mixes L1 and L2 regularizers, sharing one scaling parameter λ to reduce model-selection complexity.
Normed weight penalties shrink the feasible parameter region and correspond to Bayesian priors, with L1 and L2 linked to Laplace and Gaussian priors.

2.6. Gradient computation and optimization

Gradient-based optimization trains the DNN using exact backpropagation gradients and stochastic updates, with ADAM selected for the high-dimensional objective.

Backpropagation computes DNN gradients through recursive applications of the chain rule and is exact unlike finite differences.
SGD approximates the negative objective gradient by averaging gradients from independently sampled subsets of training examples.
SGD updates optimization variables through appropriately sized steps in the negative-gradient direction.
The authors use ADAM instead of vanilla SGD because SGD is not guaranteed to perform well for complex high-dimensional objectives.
ADAM estimates gradients and squared gradients with exponential moving averages, correcting initialization bias and using a small η to avoid division by zero.

2.7. Selecting network structure

The network structure is designed to encode high-dimensional inputs into a lower-dimensional space, then apply a nonlinear link function, while selecting only L and d by cross-validation.

Selecting network structure: The paper notes that rigorous rules for selecting these quantities do not exist, so suitable configurations typically require extensive experimentation.
Selecting network structure: The encoder h maps a D-dimensional input vector to dimension d, with d << D and d determined during model selection.
Selecting network structure: The hidden-layer widths in the encoder section decay exponentially across layers.
Selecting network structure: The link function g is a single-layer MLP whose hidden layer has width 300d and is intended to capture arbitrary nonlinearities.
Selecting network structure: Figure 3 visualizes the parameterized surrogate structure for L = 3 and d = 1.
Selecting network structure: Network-structure optimization is reduced to cross-validating two integer quantities, L and d, rather than separately selecting every layer count and size.

2.8. Combined global optimization and grid search for model selection

Model selection combines grid search over discrete DNN structure parameters with Bayesian global optimization of the continuous regularization parameter. Validation error is estimated through repeated training and evaluated using a Gaussian-process surrogate for the noisy objective.

Model-selection strategy: Model selection searches over discrete structure parameters L and d together with continuous weight decay λ.The method replaces intuition-guided manual search with a systematic combined grid-search and stochastic global-optimization procedure.
Full procedure: The full procedure splits data into training, validation, and test sets, optimizes λ for each grid structure, then selects the structure with minimum estimated validation error.The final DNN surrogate is trained after choosing the optimal structure and regularization parameter.
Bayesian optimization: For each grid location S, Bayesian global optimization seeks λ∗S that minimizes the expected validation error E[R(S, λ)].The validation objective is stochastic because network initialization and random dataset splitting induce variability.
Bayesian optimization: BGO iteratively updates a Gaussian-process response surface for R(λ; S) and selects new observations by maximizing an acquisition function, typically expected improvement.The procedure evaluates candidate hyperparameters, augments the observation set, and refits the GP surrogate.
Bayesian optimization: The GP’s predictive variance is filtered to remain robust to observational noise from random initializations and dataset partitions.This variance represents epistemic uncertainty associated with limited observations of the validation objective.

3. Numerical Example - Stochastic Elliptic Partial Differential Equation

The numerical example learns a DNN surrogate for a stochastic elliptic PDE with an uncertain spatial diffusion field. It removes fixed-lengthscale restrictions, trains on discretized field snapshots, and reports accurate predictions across tested diffusion-field realizations.

Problem formulation: The benchmark is a two-dimensional steady-state diffusion PDE on the unit square, solved numerically with a finite-volume method.The domain is discretized into 32 × 32 finite-volume cells, with the diffusion field as input and the PDE solution as output.
Problem formulation: The diffusion coefficient is modeled as a log-normal random field whose covariance encodes spatial regularity and directional correlation lengths.Unlike the traditional setup, the example removes restrictions requiring fixed correlation lengths.
Surrogate construction: The surrogate directly maps a discretized random diffusion field to the numerical PDE solution, rather than mapping truncated KLE coefficients to a response.Its target is a field-to-solution map evaluated at finite-volume cell centers.
Data generation: The dataset contains 6000 paired diffusion-field and solution samples, split evenly into 2000 training, 2000 validation, and 2000 test examples.Samples are generated from 60 lengthscale pairs with 100 realizations per pair.
Results: For test fields with lengthscales unlike those used in training, the surrogate retains accuracy similar to that obtained on Dtest.This supports prediction across diffusion-field lengthscales beyond the training combinations.

4. Conclusion

The paper proposes DNN surrogates for high-dimensional UQ by parameterizing the network to recover a nonlinear active subspace. Demonstrations on elliptic SPDEs show effective prediction across matched and some unmatched diffusion-field lengthscales, with extensions toward Bayesian and multifidelity UQ.

Method: The proposed DNN surrogate parameterization recovers a low-dimensional nonlinear manifold interpreted as a nonlinear active subspace.The architecture is described as an encoder composed with a one-layer MLP.
Demonstration: The method maps a diffusion-field snapshot to the solution of an elliptic SPDE without imposing a specific covariance structure on the diffusion coefficient.The demonstration concerns uncertainty propagation for an elliptic SPDE with uncertain diffusion coefficient.
Results: The DNN surrogate effectively predicts PDE solutions for diffusion fields with lengthscales that are not used for training.This conclusion is stated for the demonstrated SPDE problem.
Implications: The work is presented as an early step toward deep-learning surrogate models for high-dimensional UQ tasks with computationally expensive simulators.The proposed methodology is also positioned for Bayesian DNNs and multilevel or multifidelity UQ.
Implications: Multifidelity extensions could use fully convolutional networks and hierarchical representations to learn correlations between heterogeneous information sources.The cited passage describes training on data from simulators with varying fidelity levels.