DPM: A deep learning PDE augmentation method (with application to large-eddy simulation)

Jonathan B. Freund; Jonathan F. MacArt; Justin Sirignano

Source-linked AI summary

DPM: A deep learning PDE augmentation method (with application to large-eddy simulation)

Jonathan B. Freund, Jonathan F. MacArt, Justin Sirignano

arXiv:1911.09145v1Nov 20, 2019cs.LGcs.CEstat.ML

TL;DR

Limited data and unresolved physics challenge scientific machine learning, especially when PDE models contain unclosed terms and numerical errors. The paper embeds a neural network in a known-physics PDE and trains it through adjoint-based full-PDE optimization, with stochastic acceleration. In LES turbulence predictions, the resulting DPM outperforms established closure models and a priori-trained models, including on coarse meshes where established models become qualitatively incorrect.

Problem
Scientific machine learning must learn unknown or unrepresented physics from limited data, while LES additionally contains unclosed sub-filter-scale terms and discretization errors.
Method
The DPM embeds a neural network in a known-physics PDE and estimates it by optimizing the full PDE with adjoint PDE gradients and stochastic time-interval sampling.
Results
The DPM recovers resolved turbulence accurately, outperforms established LES models, and corrects numerical discretization errors.
Takeaways & Limitations
DPMs can leverage known and resolvable physics while learning unresolved effects, with applications broader than Navier–Stokes turbulence.
Takeaways & Limitations
Successful extrapolation requires training sets with sufficiently rich variability, and some validation is needed when represented physics is absent.

Abstract

from arXiv · show

Machine learning for scientific applications faces the challenge of limited data. We propose a framework that leverages a priori known physics to reduce overfitting when training on relatively small datasets. A deep neural network is embedded in a partial differential equation (PDE) that expresses the known physics and learns to describe the corresponding unknown or unrepresented physics from the data. Crafted as such, the neural network can also provide corrections for erroneously represented physics, such as discretization errors associated with the PDE's numerical solution. Once trained, the deep learning PDE model (DPM) can make out-of-sample predictions for new physical parameters, geometries, and boundary conditions. Our approach optimizes over the functional form of the PDE. Estimating the embedded neural network requires optimizing over the entire PDE, which itself is a function of the neural network. Adjoint partial differential equations are used to efficiently calculate the high-dimensional gradient of the objective function with respect to the neural network parameters. A stochastic adjoint method (SAM), similar in spirit to stochastic gradient descent, further accelerates training. The approach is demonstrated and evaluated for turbulence predictions using large-eddy simulation (LES), a filtered version of the Navier--Stokes equation containing unclosed sub-filter-scale terms. The DPM outperforms the widely-used constant-coefficient and dynamic Smagorinsky models, even for filter sizes so large that these established models become qualitatively incorrect. It also significantly outperforms a priori trained models, which do not account for the full PDE. A relaxation of the discrete enforcement of the divergence-free constraint is also considered, instead allowing the DPM to approximately enforce incompressibility physics.

1 Introduction

The paper embeds a neural network within a PDE to represent unknown or unrepresented physics while retaining known physics, addressing limited data and enabling scenario generalization. Applied to LES, the DPM jointly accounts for modeling and discretization errors and outperforms established and a priori-trained alternatives.

Motivation and framework: The framework embeds a neural network in a PDE that expresses known physics and learns unknown or unrepresented physics from limited data.The PDE may be incomplete without being incorrect, making it a building block for machine-learning augmentation.
Motivation and framework: The DPM estimates its neural network by optimizing the full PDE solution against trusted data rather than training the network independently.This optimization is challenging because the objective depends on a PDE whose solution is a nonlinear function of the network parameters.
Generalization: The trained DPM supports out-of-sample predictions for new physical parameters, initial conditions, boundary conditions, and geometries.The paper evaluates this capability for turbulence predictions using large-eddy simulation.
LES application: In LES, the DPM models unclosed sub-grid-scale physics and corrects discretization errors that become important on coarse meshes.LES uses coarse grids to reduce cost, but filtering introduces unclosed terms and unresolved scales.
LES application: The DPM recovers resolved turbulence spectra on significantly coarser LES meshes than established models and outperforms a priori-trained models for LES.The paper attributes the a priori approach’s poor LES performance to its decoupling of optimization from the full PDE.

2 Stochastic Adjoint Method

The stochastic adjoint method makes full-PDE training computationally feasible by obtaining neural-network gradients through adjoint PDEs rather than parameter-wise differentiation. It further accelerates optimization through stochastic time-interval sampling and remains compatible with discretized incompressible-flow equations.

Adjoint-based gradients: Directly differentiating the PDE with respect to every neural-network parameter is computationally intractable for networks often exceeding 10^5 parameters.Finite-difference gradients would likewise require an intractable number of evaluations.
Adjoint-based gradients: The adjoint PDE computes the objective gradient using only p PDEs per update, matching the dimension of the PDE state u.The adjoint equation can be viewed as a continuous-time analogue of neural-network backpropagation.
Computational trade-offs: For multiple scenarios, the method requires separate adjoint PDEs for each scenario and still incurs significant computation when the scenario and time counts are large.This is the principal computational trade-off identified for the adjoint approach.
Stochastic adjoint method: The stochastic adjoint method accelerates training by randomly sampling scenarios and time intervals, enabling more training iterations per unit computational time.Each sampled interval is solved forward, evaluated, and differentiated with a time-reversed adjoint PDE.
Incompressible-flow formulation: For incompressible Navier–Stokes equations, pressure enforces the divergence-free condition, and the adjoint equations are derived consistently with the time-discretized PDE.Operator splitting and a standard projection method are used for computation.

3 Sub-grid-scale closure for incompressible turbulence

LES reduces computational cost by resolving only the largest turbulence scales, but filtering creates unclosed sub-grid-scale terms whose accurate modeling is essential. The DPM produces qualitatively correct coarse-grid turbulence where dynamic Smagorinsky becomes qualitatively incorrect.

LES formulation: Filtering the Navier–Stokes equations introduces unclosed terms associated with unresolved scales, requiring sub-grid-scale closure models.The DPM is designed to account for these terms together with numerical discretization errors.
LES formulation: DNS resolves all turbulence scales at very fine spatial and temporal resolution, making it prohibitively expensive for flows with small Kolmogorov scales.LES reduces expense by resolving only the largest scales on coarser grids.
Evaluation: The study evaluates DPM, Smagorinsky, and dynamic Smagorinsky closures quantitatively using decaying isotropic turbulence and limited high-fidelity DNS data.The objective is to improve accuracy while resolving fewer scales and reducing computational cost.
DPM comparison: For a coarser filter on an N = 64^3 grid, dynamic Smagorinsky produces a qualitatively incorrect solution, whereas the adjoint-trained DPM remains qualitatively correct.Figure 2 compares filtered N = 1024^3 DNS, dynamic Smagorinsky LES, and adjoint-trained DPM LES.

4 Discretization errors and learning corrections

LES discretization errors can be comparable to or larger than the filtered velocity-gradient magnitude, so the DPM learns corrections for both closure mismatch and numerical errors. The approach can also relax coarse-grid divergence enforcement, but its trained model depends on mesh, numerical schemes, and sufficiently varied data.

Scope and limitations: The learned correction is linked to mesh density and numerical schemes, and sufficiently large, variable training data are needed for generalization.Efficient LES applications may not vary resolutions widely, but richer variability is needed for extrapolation.
Learning corrections: Training on coarse LES data allows the DPM to learn both closure mismatch and discretization errors.The discrete operators and their approximations enter the learned correction through the full PDE.
Quantifying discretization errors: LES discretization errors can be comparable to the average DNS velocity-gradient magnitude and increase with mesh size.At Δ/Δx_dns = 32, the error becomes greater than the average velocity-gradient magnitude.
Quantifying discretization errors: Even exact modeling of the unclosed sub-filter-scale term would leave the LES calculation inaccurate when discretization errors are large.
Relaxing incompressibility enforcement: The filtered DNS velocity fails the coarse-grid discrete divergence-free condition, with divergence comparable to the average velocity-gradient magnitude.On the DNS grid, the filtered velocity satisfies the condition within a small factor of the Poisson-solver tolerance, 10^-9.
Relaxing incompressibility enforcement: The DPM can exploit nonzero coarse-grid divergence to learn incompressibility physics rather than enforcing the discrete divergence-free constraint exactly.This extension is shown to reproduce key features of turbulence physics.

5 Numerical results

The numerical experiments evaluate DPM training and model variants on decaying isotropic-turbulence DNS, including out-of-sample prediction, a priori-training comparisons, relaxed incompressibility, and computational cost. Across these tests, the adjoint-trained DPM better reproduces turbulence evolution and spectra than established and a priori-trained alternatives, while relaxed divergence enforcement can reduce cost.

5.1 DNS data: The DPM learns Reynolds-number scaling from instantaneous fields while closing the LES equations, including numerical error terms, better than established models.The training varies dimensional time scales so model inputs differ by factors of 10 for nominally similar instantaneous data.
5.1 DNS data: Three DNS datasets train the models and three provide out-of-sample tests across varied turbulence time scales and initial dissipation rates.The six datasets span µ/µ0 values from 0.5 to 2.0, with two random phasings each; initial TKE dissipation varies by a factor of 64.
5.4 Out-of-sample model comparison: The DPM outperforms both dynamic and fixed-coefficient Smagorinsky models on all three out-of-sample decay tests at filter size ∆/∆x = 16.The established models follow the same decay-rate profile because the test cases share Reynolds number, whereas the DPM learns the correct rescaling from different initial dissipation rates.
5.4 Out-of-sample model comparison: The adjoint-trained DPM reproduces the energy spectrum qualitatively better than eddy-viscosity models at the large filter size ∆/∆x_dns = 16.The constant-coefficient and dynamic Smagorinsky spectra are nearly identical, with high-wavenumber energy consistent with their underpredicted decay rates.
5.5 Comparison with a priori training: The adjoint-trained DPM performs substantially better than the a priori-trained closure because full-PDE optimization accounts for nonlinear coupling and coarse-grid numerical errors.A priori training also requires a full description of the modeled divergence term, which is sparse and most directly available when the unclosed term can be computed explicitly.
5.6 Relaxation of the divergence-free constraint: Relaxing strict divergence-free enforcement preserves comparable or improved spectral agreement, while learned incompressibility reduces solution cost to approximately one-twentieth of the legacy-solution cost.The relaxed approach eliminates the elliptic pressure-projection solve; larger networks improve accuracy, and deeper networks reduce numerical instability.
5.6 Relaxation of the divergence-free constraint: The framework’s extrapolation capacity may diminish when physics is less explicitly represented, so validation remains necessary for model-reduction applications.This is an author-identified scope boundary for extending the DPM beyond the demonstrated setting.

6 Conclusion

The DPM learns unclosed turbulence physics within the filtered Navier–Stokes equations from filtered DNS data, while adjoint PDE training can also correct discretization errors. It accurately represents resolved turbulence, outperforms established eddy-viscosity models, and supports broader physics-learning formulations.

Scope and evidence: The results demonstrate the promise of DPMs for problems with unresolved physics using limited high-fidelity numerical data for training.The conclusion identifies this as an initial demonstration of learning unrepresented or unknown physics.
DPM formulation: The DPM learns the unclosed LES sub-grid-scale stress tensor from filtered DNS data and evaluates its out-of-sample accuracy.The model is embedded in the filtered Navier–Stokes equations rather than trained independently of the PDE.
Turbulence results: The DPM recovered an accurate representation of resolved turbulence and outperformed established eddy-viscosity models, including dynamic Smagorinsky.The comparison uses resolved kinetic energy decay rate and resolved kinetic energy spectrum against exact filtered DNS observations.
Discretization-error correction: Adjoint PDE-based training was able to correct numerical discretization errors that affect LES performance on coarse meshes.The errors were interpreted as additional unclosed terms in the discrete governing equations on the coarse LES grid.
Learned incompressibility: A relaxed formulation can learn incompressibility physics from data without exactly enforcing the discrete divergence-free constraint, reducing computational cost.This formulation learns the coarse-grid pressure physics instead of requiring exact discrete enforcement.
Limitations and outlook: Further progress requires training sets with sufficiently rich variability to enable extrapolation to more complex Navier–Stokes turbulence flows.Extending the approach to more complex flows is identified as an important challenge.
Broader applicability: The method may extend beyond turbulence to intricate sub-grid-scale problems involving mixing, combustion, or additional mechanisms coupled with turbulence.Its formulation can incorporate experimental data even when the physical description is unclear, while retaining known and resolvable physics.

A.1 Divergence-free projection of DNS fields

The appendix projects filtered DNS velocity fields onto the nearest field satisfying the discretized divergence-free condition on the coarse LES grid. Lagrange multipliers reduce this constrained projection to a pressure-like discrete Poisson problem.

Projection setup: Because filtered DNS fields do not satisfy the coarse-grid discrete divergence-free condition, each field is first projected onto that constraint.The projection is performed at each time t before constructing the target field.
Constrained optimization: The projected field is chosen as the nearest field to the filtered DNS field while satisfying the discretized divergence-free condition.This is formulated as a constrained ℓ2 optimization problem on the coarse LES grid.
Solution method: A pressure-like Lagrange multiplier solves a discrete Poisson equation to obtain the constrained minimizer.The gradients use the same second-order, staggered-mesh central-difference scheme as the flow solver.