PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep learning observables in computational fluid dynamics

Kjetil O. Lye, Siddhartha Mishra, Deep Ray

arXiv:1903.03040v2physics.comp-phcs.LGmath.NAphysics.flu-dynstat.ML

TL;DR

Large-scale computational fluid dynamics applications require many expensive forward PDE solves, motivating cheaper approximations of parameter-to-observable maps. The paper uses deep neural networks trained on few samples and combines them with Monte Carlo methods, achieving low prediction errors and one- to two-order-of-magnitude speedups under a generalization assumption.

  • Problem

    Computational fluid dynamics applications require many evaluations of expensive parameter-to-observable maps, making uncertainty quantification and related tasks computationally costly.

  • Method

    Deep fully connected neural networks learn observables from few samples, using low-discrepancy training points and ensemble training, then support deep learning MC and QMC algorithms.

  • Results

    The trained networks achieve low prediction errors with few samples, while deep learning QMC and MC methods provide about one and two orders of magnitude speedup over baseline QMC and MC, respectively.

  • Takeaways & Limitations

    Under good neural-network generalization, the proposed deep learning MC and QMC algorithms are guaranteed to outperform their baseline methods for uncertainty propagation.

  • Takeaways & Limitations

    The speedup guarantee depends on the underlying neural networks generalizing well, while theoretical error bounds can substantially overestimate practical generalization error.

Abstract

from arXiv · show

Many large scale problems in computational fluid dynamics such as uncertainty quantification, Bayesian inversion, data assimilation and PDE constrained optimization are considered very challenging computationally as they require a large number of expensive (forward) numerical solutions of the corresponding PDEs. We propose a machine learning algorithm, based on deep artificial neural networks, that predicts the underlying \emph{input parameters to observable} map from a few training samples (computed realizations of this map). By a judicious combination of theoretical arguments and empirical observations, we find suitable network architectures and training hyperparameters that result in robust and efficient neural network approximations of the parameters to observable map. Numerical experiments are presented to demonstrate low prediction errors for the trained network networks, even when the network has been trained with a few samples, at a computational cost which is several orders of magnitude lower than the underlying PDE solver. Moreover, we combine the proposed deep learning algorithm with Monte Carlo (MC) and Quasi-Monte Carlo (QMC) methods to efficiently compute uncertainty propagation for nonlinear PDEs. Under the assumption that the underlying neural networks generalize well, we prove that the deep learning MC and QMC algorithms are guaranteed to be faster than the baseline (quasi-) Monte Carlo methods. Numerical experiments demonstrating one to two orders of magnitude speed up over baseline QMC and MC algorithms, for the intricate problem of computing probability distributions of the observable, are also presented.

1 Introduction

Large-scale CFD tasks often require many expensive PDE solves to estimate observables and their statistics. The paper proposes deep neural networks to learn the input-to-observable map from few samples, then combines the surrogate with Monte Carlo methods for faster uncertainty propagation.

  • Motivation: Uncertainty quantification, Bayesian inversion, optimal control, design, and PDE-constrained optimization often require repeated evaluations of expensive CFD models.These applications may target observables rather than the full solution field.
  • Motivation: Observable statistics such as means, variances, higher moments, and probability density functions quantify uncertainty propagated from uncertain model inputs and design parameters.
  • Approach: Deep neural networks are proposed to learn the parameters-to-observable map from approximately O(100) computed samples using fully connected architectures.
  • Challenges: Designing accurate networks is difficult because CFD parameter-to-observable maps may have low regularity, while generalization-error bounds degrade with few expensive training samples.
  • Contribution: The deep learning Monte Carlo and Quasi-Monte Carlo methods are guaranteed to outperform baseline methods when the trained networks generalize well.

2.1 The problem

The paper studies parameterized convection-diffusion PDEs whose uncertain inputs determine solution fields and observables. It formalizes the parameter-to-observable map and assumes accurate high-resolution numerical approximations are available for training and evaluation.

  • Parameterized PDE: The parameterized problem is a convection-diffusion PDE with parameter-dependent fluxes, domains, initial data, and boundary conditions.
  • Parameterized PDE: The parameter space is taken as Y = [0, 1]^d, while the solution U depends on time, spatial location, and parameter vector y.
  • Interpretation: Uncertainty in domains, coefficients, and initial or boundary data is represented through parameters in Y, while Y can also encode design or control parameters.
  • Observables: Observables are functionals of the PDE solution, with lift and drag identified as examples that fit the general observable form.
  • Numerical approximation: The training framework assumes high-resolution numerical solutions U_Δ(y) and accurate quadrature-based approximations L_Δ(y) are available for sampled parameters.

2.2 Deep learning the parameters to observable map

The proposed surrogate learns the parameter-to-observable map from a small set of expensive CFD evaluations using fully connected neural networks. Training minimizes a regularized loss with stochastic optimization, producing a network surrogate for subsequent evaluations.

  • Training set: A small training set S is selected, and high-resolution CFD simulations compute L_Δ(y) for each sampled parameter, typically with N at most O(100).
  • Training set: Training points may be random or generated by low-discrepancy sequences such as Sobol or Halton points, which are better spread across parameter space.
  • Network architecture: The surrogate uses a feedforward multilayer perceptron composed of affine transformations and scalar nonlinear activations across hidden layers.
  • Network architecture: The network is restricted to fully connected architectures without an assumed sparsity structure.
  • Optimization: Weights are trained by minimizing a possibly regularized loss with full-batch or mini-batch stochastic gradient methods, including ADAM.
  • Output: The resulting locally optimized network L_θ* serves as the neural surrogate for the parameter-to-observable map.

2.3 Theory

Universal approximation results motivate neural-network approximations of the parameters-to-observable map, but low regularity, limited data, and high dimensionality create severe theoretical and practical challenges. Numerical experiments illustrate both pessimistic size estimates and large prediction errors in data-poor settings.

  • Approximation theory: Constructive ReLU approximation results bound network size for functions in Sobolev spaces, motivating regularity analysis of the parameters-to-observable map.The approximation rate depends on smoothness, dimension, and target accuracy.
  • Map regularity: Regularity of the parameters-to-observable map depends on space-time and parametric regularity of the underlying PDE solution field.For generic convection-diffusion equations, such regularity is generally unavailable; scalar conservation laws may provide only BV regularity.
  • Map regularity: L1 stability for scalar conservation laws yields bounds for the parameters-to-observable map and its monotone numerical approximation under stated assumptions.The proof uses contractivity of the solution operator together with Lipschitz regularity of the observable function.
  • Map regularity: Higher parametric derivative bounds cannot generally be expected because the underlying solution field lacks differentiability.The stated bounds extend to conservation laws and incompressible Navier–Stokes equations when the field is stable with respect to input parameters.
  • Theoretical difficulty: O(10^12) tunable parameters result from a theoretical one-percent approximation requirement for the six-dimensional RAE2822 lift-and-drag map.This estimate illustrates how approximation-theoretic bounds can become impractical for the target observable map.
  • Generalization: Generalization bounds based on standard complexity measures can be highly pessimistic, while practical data-poor regimes may still yield 30–100% prediction errors.The text notes that conventional bounds may overestimate generalization error by tens of orders of magnitude, yet stringent theoretical assumptions remain insufficient in practice.
  • Illustrative numerical experiment: 128 training samples produce unacceptably high errors for the unscaled sum of sines, while scaling reduces errors by almost an order of magnitude.The experiment uses 8192 Sobol test points in [0,1]^6; the reduction is attributed to smaller derivatives of the scaled map.

2.4 Hyperparameters and Ensemble training.

The paper uses ensemble training to select neural-network architectures and training hyperparameters in a data-poor regime. Each sampled hyperparameter configuration is trained separately, and the resulting loss determines performance comparisons.

  • Hyperparameter selection: An ensemble training algorithm samples candidate hyperparameters, including network depth and layer widths, to address low-prediction-error requirements.The procedure is designed for the theoretically difficult data-poor setting.
  • Ensemble training: Each hyperparameter choice is evaluated by training one network with ADAM and minimizing its resulting loss function.The procedure has similarities to an active-learning procedure described in prior work.

3 Uncertainty quantification in CFD with deep learning.

The section formulates uncertainty quantification as approximation of the observable’s push-forward probability measure and combines deep neural networks with Monte Carlo or Quasi-Monte Carlo sampling. Under a generalization-error assumption, the resulting deep-learning estimators can reduce the cost of distributional computation relative to baseline methods.

  • Problem formulation: The target is the push-forward measure of uncertain input parameters through the high-resolution observable map, from which moments and probability distributions can be computed.The section emphasizes that computing the full probability distribution is harder than estimating only mean and variance.
  • Baseline methods: QMC uses deterministic low-discrepancy point sets, and for moderate dimensions its integration error improves with a rate J_q^-α for 1/2 < α ≤ 1.The section states that this rate allows QMC to outperform baseline Monte Carlo under the stated dimensionality assumption.
  • Deep-learning acceleration: DLQMC replaces repeated high-resolution evaluations in QMC with a neural approximation of the parameters-to-observable map trained on N << J_q CFD samples.The algorithm selects QMC points, evaluates a smaller training subset with the CFD solver, and trains an optimal neural network before evaluating the remaining points cheaply.
  • Deep-learning acceleration: Theorem 3.3 guarantees a DLQMC speedup under O(ε) neural-network generalization error and the baseline QMC error estimate.The speedup compares the costs of baseline QMC and DLQMC for approximating the push-forward measure to O(ε) accuracy in the 1-Wasserstein metric.
  • Complexity and assumptions: The network evaluation cost is expected to be much lower than a CFD solve, so substantial speedups follow when N is much smaller than J_q and the learned variation remains comparable.The argument assumes V* ≈ V_Δ and contrasts network cost O(M) with the high-resolution CFD cost.

4 Implementation of ensemble training

The implementation uses fully connected ReLU networks with compact architectures and an ensemble over training choices. It varies loss, regularization, initialization, sampling, and optimization settings to select networks using training-based surrogates for unavailable generalization error.

  • Network and loss design: The experiments use fully connected ReLU networks whose total size is O(d + N), consistent with the prescribed architecture scaling.The network width and depth are selected through a reference architecture for each experiment.
  • Network and loss design: The ensemble varies MAE versus MSE losses, regularization exponents, and λ values spanning 7.8 × 10^-5, 7.8 × 10^-6, 7.8 × 10^-7, and 0.These choices provide a tractable search over loss and regularization configurations.
  • Initialization and selection: Because the loss is non-convex, different SGD or ADAM initializations can reach local minima with different loss and generalization profiles.The starting value is therefore treated as a key hyperparameter and multiple networks are trained in parallel.
  • Initialization and selection: The selection rules include lowest training error, lowest validation loss, lowest training mean error, and lowest training Wasserstein error.Validation-based selection sacrifices some training data, while true generalization error is unavailable.
  • Sampling and optimization: Training sets use either randomly distributed Monte Carlo points or more uniformly distributed QMC points, while learning rate and epoch count are additional hyperparameters.The experiments compare both sampling choices to determine which is preferable in practice.

5 Numerical results

The numerical experiments use the two-dimensional compressible Euler equations for flow past the RAE 2822 airfoil. The setup defines the principal gas-dynamic variables and uses γ = 1.4 in the simulations.

  • Governing equations: The experiments solve the compressible Euler equations of gas dynamics as the underlying PDE.The setting is two-dimensional.
  • Governing equations: The state variables include density ρ, velocity u = (u, v), pressure p, and total energy per unit volume E.These variables define the flow state used in the numerical experiments.
  • Flow quantities: The ratio of specific heats is γ = 1.4, and the setup also defines sound speed, Mach number, and temperature through the ideal gas law.The associated relations include a = √(γp/ρ), M = |u|/a, and p = ρRT.

5.1 Flow past an airfoil

The airfoil experiment learns lift and drag from six uncertain flow and geometry parameters using an ensemble of fully connected neural networks. With 128 training samples, selected networks achieve low prediction errors, while sensitivity analyses show that drag is more affected by hyperparameter choices than lift.

  • Problem description: The experiment varies six parameters describing free-stream conditions and perturbed upper and lower airfoil surfaces, with lift and drag as observables.The parameter domain is Y = [0, 1]6, and each observable is computed on the perturbed airfoil surface.
  • Data and training setup: The study uses Sobol points to construct training, validation, and test sets, with 128 points assigned to training and 128 to validation.The experiment evaluates trained networks on a larger Sobol test set.
  • Ensemble training: 570 networks are trained from 114 hyperparameter configurations and five random retrainings per configuration.The ensemble varies network and optimization hyperparameters before selecting the best retraining for each configuration.
  • Training results: Three orders of loss reduction occur for lift and two orders for drag during ADAM training.The reductions are observed for the best-performing networks as training proceeds across epochs.
  • Prediction accuracy: Less than 1% relative error is obtained for lift and less than 2% for drag using 128 training samples.These errors are reported for the best-performing networks on the test set.
  • Comparison and sensitivity: Neural networks outperform linear least squares by approximately 5 times for lift and 20 times for drag, indicating greater nonlinearity in the drag map.Linear least-squares errors are approximately 4% for lift and 36% for drag.
  • Comparison and sensitivity: Hyperparameter sensitivity is greater for drag: errors vary substantially, ADAM generally outperforms standard SGD, and loss-function differences are minor except for a drag outlier.Retraining and network-size sensitivity are also reported as greater for drag than for lift.

5.1.5 Network performance with respect to number of training samples.

Prediction error generally decreases as training samples increase, while the resulting DLQMC method reproduces reference observable distributions and substantially reduces computational cost.

  • Training-sample study: For N = 32, 64, 128, and 256, ensemble training evaluates minimum, maximum, and mean relative L2 prediction errors across hyperparameter configurations.The reported baseline experiments used N = 128 training samples.
  • Training-sample study: The error decays faster with sample count than theory predicts for lift and drag, with lower-than-parameter-count constants indicating high network compressibility.The reported exponents are α1 = 0.81 for lift and α2 = 0.92 for drag.
  • Uncertainty quantification: The DLQMC histograms for lift and drag show very good qualitative agreement with reference measures computed from the 1001-sample test set.The comparison uses the best-performing network for each observable.
  • Uncertainty quantification: With N = 128 training samples, DLQMC achieves half-an-order to one-order-of-magnitude speedups over baseline QMC for lift and drag.Drag speedups are slightly better than lift speedups.
  • Uncertainty quantification: DLQMC speedups peak at 64 training samples for lift and 128 for drag, reflecting interaction between decreasing network error and baseline-QMC error convergence.The speedup compares costs at errors of the same Wasserstein magnitude.
  • Monte Carlo comparison: DLQMC achieves at least two orders of magnitude speedup over baseline MC because QMC already outperforms MC and DLQMC further accelerates QMC.The real MC speedup is obtained by squaring the raw speedup to account for MC’s square-root convergence.

5.2 A stochastic shock tube problem

The stochastic shock tube tests the method on a one-dimensional Euler problem whose high-variance initial data make neural-network prediction harder than the airfoil case. Despite this difficulty, the trained networks remain accurate and DLQMC substantially outperforms baseline QMC.

  • Problem setup: The stochastic shock tube uses one-dimensional compressible Euler equations with density observables defined as average integrals over three fixed intervals.The regions are I1 = [−1.5, −0.5], I2 = [0.8, 1.8], and I3 = [2, 3].
  • Problem setup: Initial-data standard-deviation-to-mean ratios reach 100%, making this problem substantially more stochastic than the flow-past-airfoils problem.The paper describes this variance as almost an order of magnitude larger than in the airfoil problem.
  • Problem setup: High variance can produce large generalization errors, so the shock tube is expected to be harder for neural networks to predict than the airfoil problem.The paper explicitly connects this expectation to prior analysis of variance and generalization error.
  • Network results: Most hyperparameter configurations produce errors comparable to the best networks, and a large majority reduce prediction error by a factor of 3–4 versus linear least-squares regression.Some outlier configurations remain, but they do not represent the majority.
  • Network results: Mean prediction error decays with exponent 0.5 for observables L2 and L3, while L1 decays somewhat more slowly; low constants still indicate substantial compression.The authors interpret this compression as enabling accurate approximation of the intricate problem.
  • Uncertainty quantification: DLQMC reaches maximum real speedups of about 16 for L1 and L2 and 5 for L3 over baseline QMC.The speedups are measured across hyperparameter configurations as training-sample count varies.

6 Discussion

The paper addresses expensive many-query CFD by learning parameter-to-observable maps with fully connected neural networks trained on few samples. Experiments support low errors, robustness, and large gains when the surrogate is combined with QMC, while extension from observables to full fields remains harder.

  • Motivation: Many-query CFD tasks require numerous expensive solver evaluations, making their cumulative computational cost potentially prohibitive.The motivating applications include uncertainty quantification, Bayesian inversion, optimal design, and control.
  • Approach: The method learns observables rather than full PDE solution fields, targeting maps that are more regular and may have lower variance.The paper presents this choice as one of three strategies for overcoming data scarcity and low regularity.
  • Approach: Low-discrepancy training points are combined with ensemble training to reduce generalization error and systematically search hyperparameters.The ensemble also tests sensitivity to hyperparameter choices.
  • Findings: Across two compressible-Euler experiments, networks trained on few samples achieve low prediction errors, robust performance across many hyperparameters, and evaluation costs several orders of magnitude below full CFD solves.The authors attribute unexpectedly low errors to generalization and high compression.
  • Findings: Theorem 3.3 guarantees DLQMC will outperform baseline QMC when the learned neural network generalizes well, and experiments support this prediction.The approach uses the trained surrogate to approximate observable probability distributions.
  • Scope: The approach is data-driven and can extend to other PDE types, but learning full solution fields may be harder because fields are less regular than observables.The paper identifies Navier–Stokes and other elliptic or parabolic PDEs as possible extensions.
Loading 1903.03040v2…