Simulator-free Solution of High-Dimensional Stochastic Elliptic Partial Differential Equations using Deep Neural Networks

• Monte Carlo converges independently of stochastic dimension, but typically needs many samples, making propagation costly with sophisticated PDE solvers.Source: §1 · p. 3
• Traditional response surfaces have not scaled to high stochastic dimensions, where maintaining accuracy requires exponentially more forward evaluations—the curse of dimensionality.Source: §1 · p. 4
• KLE and PCA overestimate intrinsic dimensionality because they recover only linear input manifolds and do not use model-output information.Source: §1 · p. 5
• Even newer high-dimensional surrogate approaches retain a fundamental dependence on repeated forward PDE solver evaluations.Source: §1 · p. 6

Introduction

• The approach avoids deterministic PDE solvers by approximating SPDE field variables as a function of all relevant inputs, including stochastic and spatial or temporal parameters.The authors seek a flexible parameterization capable of approximating complex nonlinear maps.Source: §1 · p. 6
• Instead of squared-residual minimization used in related works [55] [56], the method minimizes a DNN-parameterized energy functional formed by taking an expectation over stochastic parameters.The functional is derived from the variational principle for the analogous deterministic PDE.Source: §1 · pp. 6–7
• The authors derive the variational principle for elliptic SPDEs, prove solution uniqueness in an appropriate Hilbert space, and show the DNN-space objective converges to a local minimum.The convergence claim is limited to the subset of the Hilbert space spanned by DNNs.Source: §1 · p. 7
• The paper numerically demonstrates benefits of the energy functional compared with the integrated squared residual.The introduction identifies this comparison as a contribution and points to a one-dimensional SBVP example.Source: §1 · pp. 7–8
• Across high-dimensional random input fields, including non-trivial mixtures, the authors report good performance on out-of-sample validation examples.The introduction identifies training across these input fields and out-of-sample validation as part of the numerical demonstrations.Source: §1 · p. 7
• The authors assess the resulting response surface for high-dimensional uncertainty propagation and use trained DNN approximators for uncertainty propagation and inverse problems.The introduction names both applications as part of the paper's scope.Source: §1 · pp. 7–8

2.1. Variational formulation of stochastic elliptic partial differential equations

• The target is U(x, ω) solving −∇·(A(x, ω)∇U(x, ω)) + C(x, ω)U(x, ω) = F(x, ω) on X.Source: §2.1 · p. 10
• The formulation imposes Dirichlet and Neumann boundary conditions almost surely; n(x) denotes the normal on ΓN.Source: §2.1 · p. 10
• Under the stated regularity, uniform ellipticity, and C ≥ η > 0 assumptions, the SBVP has a unique solution that minimizes J[U].Source: §2.1 · pp. 10–11
• The minimizer satisfies the SBVP's weak form almost surely, while any weak solution is a stationary point of J[U].Source: §2.1 · pp. 11, 13

2.2. Building a DNN approximator

• The input field is represented by finite-dimensional Ξ, a flattened pixel or discretization vector, with its probability density assumed or learned from data.Source: §2.2 · p. 14

2.2.1. Extension to boundary conditions:

• Dirichlet conditions are enforced through U=B+K N, where B has no adjustable parameters and K vanishes on the Dirichlet boundary; Neumann data remain in J.Source: §2.2.1 · pp. 14–15

2.2.2. The space of deep residual neural networks

• The ResNet maps spatial position and input Ξ through an initial dense layer, K residual blocks with L dense layers each, and a final linear layer.Source: §2.2.2 · p. 15
• Each residual block adds its input to its activation before passing the result to the next computation stage.Source: §2.2.2 · p. 16
• The network uses the Swish activation, with β set to 1 in this work.Source: §2.2.2 · p. 16

2.2.3. Training the ResNet

• The parameterized variational objective J(θ) is nonconvex even though the original functional J[U] is convex.Source: §2.2.3 · p. 17
• Training minimizes a Monte Carlo estimate of the expected energy functional; the estimate is unbiased, and the task is recast as stochastic minimization.Source: §2.2.3 · p. 18

3.1. Evaluation metrics

• The 1D evaluation compares DNN predictions with finite-volume solutions using relative RMSE E across test samples, relative L2 error, and the coefficient of determination R2.Relative L2 error compares solution vectors, while R2 is defined over FVM cell-center values.Source: §3.1 · pp. 20–21

3.2. Comparison of Energy Functional (EF) loss function vs integrated Squared Residual (SR) loss function with a 1D SBVP

• EF-trained ResNets had significantly smaller median and interquartile relative RMSE E than integrated-SR-trained networks at every tested length-scale; SR training failed as length-scale decreased.The comparison used six runs per length-scale. Figure 2 presents the box plots.Source: §3.2 · p. 23, Figure 2 · p. 24
• For EF-trained predictions at length-scale 0.03, relative L2 error was less than 0.05 and the R2 score was close to 0.99.The reported test predictions closely matched the finite-volume solutions.Source: §3.2 · p. 23, Figure 3 · p. 25

3.3. Stochastic boundary value problem in 2D

• The 2D unit-square SBVP models steady-state diffusion with a spatially varying coefficient and is tested with GRF, warped GRF, channelized, and multiple-length-scale GRF inputs.The input fields are uniformly bounded below by 0.005.Source: §3.3 · pp. 24, 26
• For the 2D SBVP, four input-field datasets were shown: exponential GRF, exponential warped GRF, channelized fields, and a length-scale-pair dataset comprising 60 distinct pairs.Source: §3.3 · pp. 26–27, Figure 6 · p. 29
• Separate DNNs for the four input fields each achieved E below 5.31% and captured fine-scale features in the solution responses.Training took around 5 to 7 hours, depending on network size.Source: Table 3 · p. 30, §3.3 · p. 30
• The single DNN achieved E = 4.56% across four test datasets, predicting well except for channelized fields, likely because that dataset had fewer training samples.The authors suggest that equal proportions of training samples from each field type could improve performance; with sufficient data across field types, one DNN may learn all solution responses.Source: §3.3 · pp. 30, 36

3.4. Testing DNNs generalizability to out-of-distribution inputs

• The generalizability study tests previously trained DNNs on inputs from distributions not seen during training, including GRFs with larger length-scales [0.3, 0.4].The study describes three cases; the supplied passages introduce Case A, which starts from a DNN trained on GRFs with length-scales [0.05, 0.08].Source: §3.4 · p. 36
• The DNN generalizes well to four out-of-distribution datasets except channelized fields, which are far from its training distribution.The tested datasets include warped and channelized fields and Gaussian random fields with different length scales and variance.Source: §3.4 · p. 38, Figure 15 · p. 39
• The ResNet generalizes to SPDE solutions at length-scales not used in training, but the DNN’s accuracy decreases for very fine length-scale values.Source: §3.4 · pp. 38–39
• For out-of-distribution stratified input data, the DNN trained on multiple length-scales generalizes very well and captures jumps at the stratification; E between DNN and FVM solutions is 3.8%.Source: §3.4 · pp. 39, 41

3.5. Uncertainty Propogation (UP) and Inverse problems

• Combining DNN solution approximators with uncertainty propagation or inverse methods significantly reduces the computational burden of repeated SPDE forward evaluations.The section demonstrates this combination through uncertainty-propagation and inverse-problem examples.Source: §3.5 · p. 41

3.5.1. Uncertainty Propogation (UP)

• Across four uncertainty-propagation cases, the study compares DNN estimates of solution mean, variance, and PDF with FVM estimates and reports mean and variance errors and R2 scores.Each case uses bounded Monte Carlo input samples; the FVM reference statistics are estimated from Monte Carlo samples.Source: §3.5.1 · p. 45, Table 4 · p. 45
• For uncertainty propagation Case 4, Figures 29–31 compare DNN and FVM SBVP means and variances, plus PDFs at two locations, using 105 MC samples.Source: Figure 29 · p. 49, Figure 30 · p. 49, Figure 31 · p. 49

3.5.2. Inverse Problem

• The inverse-problem input eA is the exponential of a zero-mean GRF with an exponential kernel, length-scale 0.03, variance 1, and bounds from 0.005 to 33.Measurements may be noisy and are assumed independent and identically distributed.Source: §3.5.2 · p. 50
• Because the posterior normalizing constant is difficult to evaluate but likelihood and prior sampling and unnormalized log-posterior gradients are available, the method uses MALA.MALA leverages the gradient to simulate a random walk targeting the posterior.Source: §3.5.2 · p. 51
• The MALA sampler uses step size 0.3, draws 15,000 posterior samples after discarding 2,000 burn-in samples and thinning by 10, and achieves 58.55% acceptance.The authors report convergence before drawing samples.Source: §3.5.2 · p. 51
• Posterior input-field samples capture a distribution over the true solution conditional on noisy observations; the inverse problem takes around 10–15 minutes.Figure 32 depicts posterior input-field samples and corresponding solution responses, with ground truth and experimental observations indicated.Source: §3.5.2 · p. 51, Figure 32 · p. 52

4. Conclusion

• The solver-free method recasts the SBVP variationally and minimizes its energy functional within DNNs that automatically satisfy Dirichlet conditions.Source: §4 · p. 53
• The energy minimization is cast as a stochastic optimization problem using an unbiased estimator and a standard variant of stochastic gradient descent.Source: §4 · p. 53
• Relative root mean square error between predicted and ground-truth FVM solutions over the entirety of their particular test datasets was reported to lie in a range beginning at 3.86.Source: §4 · p. 53
• Trained DNNs generalized to out-of-distribution inputs, including stratified fields and GRFs with length-scales outside the training range.Source: §4 · p. 53
• Uncertainty-propagation estimates had less than 1.35% relative L2 error for the mean and less than 10% for the variance.Source: §4 · p. 53
• Further work is needed on reducing training time, automating Dirichlet-condition enforcement, improving estimator variance, and analytically studying the energy loss over neural networks.Source: §4 · p. 53