PaperlayerView exact revision on arXiv

Source-linked AI summary

Modeling the Dynamics of PDE Systems with Physics-Constrained Deep Auto-Regressive Networks

Nicholas Geneva, Nicholas Zabaras

arXiv:1906.05747v3physics.comp-phcs.LGstat.ML

TL;DR

Expensive PDE simulations and data-hungry traditional deep-learning surrogates motivate a physics-constrained alternative. The paper develops an auto-regressive convolutional surrogate with Bayesian uncertainty quantification, tests it across nonlinear transient PDEs, and reports accurate predictions without output training data. Its main remaining challenges include training cost and limitations of the single-mode Gaussian posterior approximation.

  • Problem

    PDE surrogate modeling can require substantial training data even though scientific and engineering simulations are expensive and often repeated.

  • Method

    The paper combines a physics-constrained auto-regressive dense encoder-decoder convolutional network with a Bayesian framework for epistemic and aleatoric uncertainty quantification.

  • Results

    The model accurately reproduces tested nonlinear transient PDE dynamics without output training data, including turbulent statistics and complex shock interactions.

  • Takeaways & Limitations

    Physics-constrained auto-regressive surrogates can model diverse transient PDE systems while providing predictive uncertainty and comparisons with traditional numerical methods.

  • Takeaways & Limitations

    Training cost remains a significant challenge for time-series problems, and the single-mode Gaussian posterior approximation cannot robustly handle multimodal data.

Abstract

from arXiv · show

In recent years, deep learning has proven to be a viable methodology for surrogate modeling and uncertainty quantification for a vast number of physical systems. However, in their traditional form, such models can require a large amount of training data. This is of particular importance for various engineering and scientific applications where data may be extremely expensive to obtain. To overcome this shortcoming, physics-constrained deep learning provides a promising methodology as it only utilizes the governing equations. In this work, we propose a novel auto-regressive dense encoder-decoder convolutional neural network to solve and model non-linear dynamical systems without training data at a computational cost that is potentially magnitudes lower than standard numerical solvers. This model includes a Bayesian framework that allows for uncertainty quantification of the predicted quantities of interest at each time-step. We rigorously test this model on several non-linear transient partial differential equation systems including the turbulence of the Kuramoto-Sivashinsky equation, multi-shock formation and interaction with 1D Burgers' equation and 2D wave dynamics with coupled Burgers' equations. For each system, the predictive results and uncertainty are presented and discussed together with comparisons to the results obtained from traditional numerical analysis methods.

1. Introduction

PDE simulation is broadly important but often computationally expensive, motivating efficient surrogate models. This paper extends physics-constrained deep learning to dynamical PDEs with auto-regressive prediction, Bayesian uncertainty quantification, and comparisons against numerical methods.

  • Motivation: PDE simulations underpin many scientific and engineering applications, but achieving desired resolution or accuracy remains computationally expensive.Repeated simulations in optimization and inverse problems heighten the need for computational efficiency.
  • Motivation: Surrogate models reduce this burden by providing fast approximate imitations of standard numerical solvers.
  • Related work: Physics-constrained learning uses governing-equation-based loss functions to solve PDE problems without relying on conventional output training data.Earlier work applied this idea to parameterized initial/boundary value problems and elliptic-PDE surrogate modeling.
  • Contributions: The paper generalizes physics-constrained deep-learning surrogate models from elliptic systems to dynamical PDEs using a deep auto-regressive dense encoder-decoder.
  • Contributions: A Bayesian extension quantifies both epistemic and aleatoric uncertainty in predictions.
  • Contributions: The model is tested on chaotic, multi-shock, and coupled nonlinear PDE systems, with prediction accuracy, uncertainty, and computational efficiency compared against state-of-the-art numerical methods.

2. Problem Definition

The paper formulates transient PDE surrogate modeling as discrete time integration learned by a convolutional encoder-decoder. Auto-regressive execution uses prior predicted states, while physics-constrained losses train the model from initial states without output data.

  • Problem Definition: The target is a surrogate for nonlinear dynamical systems evolving in space and time, trained from governing equations rather than output training data.
  • Problem Definition: Transient PDEs specify temporal evolution, spatial and nonlinear operators, boundary conditions, and an initial-state distribution.
  • Time Integration: The convolutional encoder-decoder performs discrete time integration by mapping system states at prior time steps to the next state.The model is parameterized by a learned function with convolutional-network weights.
  • Auto-regressive Modeling: The AR-DenseED predicts N time steps auto-regressively from a discretized initial state, reusing its own predictions as future inputs.
  • Physics-Constrained Training: Physics-constrained training optimizes the neural representation against numerical time integration and discretized PDE terms.The specific numerical state inputs depend on the selected integrator, such as forward Euler or Crank-Nicolson.
  • Surrogate Formulations: The surrogate can represent deterministic responses or predictive response densities for initial conditions sampled from an initial-state distribution.When the true distribution is unknown, it may be approximated to assemble training initial states; finite training sets cover only part of possible responses.

3. Auto-regressive Dense Encoder-Decoder

AR-DenseED models dynamical PDE evolution through sequential convolutional encoder-decoder predictions using prior states as inputs. Physics-constrained training compares predictions with numerical time integration while supporting multi-step back-propagation and periodic boundaries.

  • Auto-regressive architecture: AR-DenseED predicts dynamics autoregressively by using k + 1 previous states as input for each forward time-step.The same model is reused sequentially, with predictions becoming inputs for later steps.
  • Auto-regressive architecture: Multiple previous time-steps improve training stability, whereas using too many slows training without considerable predictive improvements.The number of previous states is a tunable hyper-parameter, and the tested systems required k ≥1 for stable training.
  • Auto-regressive architecture: Encoding convolutions reduce feature-map dimensionality, decoding convolutions restore it, and dense blocks preserve dimensionality across densely connected layers.The architecture is organized as successive encoding and decoding convolutions interleaved with dense blocks.
  • Physics-constrained training: The physics-constrained loss minimizes the difference between neural-network predictions and targets generated by a consistent numerical time integrator.This L2 minimization is analogous to minimizing the discretized PDE residual across the domain.
  • Physics-constrained training: For large time-steps with CFL > 1, the model uses predicted next states when evaluating spatial gradients, enabling implicit-like integration without matrix inversions during optimization.The formulation can also use other time-integration algorithms, including multiple smaller explicit steps, though that alternative was not investigated.
  • Physics-constrained training: Periodic boundaries are enforced with circular padding during convolutions and loss evaluation, equivalent to using ghost nodes in numerical methods.The approach can also incorporate an additional loss term for boundary conditions in systems requiring it.

4. Bayesian AR-DenseED

The Bayesian AR-DenseED framework represents uncertainty in data-free physics-constrained predictions by combining a probabilistic likelihood with priors over model and discretization-error parameters. It uses numerical time integration to generate target states on the fly while accounting for temporal and spatial discretization errors.

  • Uncertainty formulation: The model separates aleatoric uncertainty in observations from epistemic uncertainty in model predictions.Epistemic uncertainty reflects factors including limited training scenarios and limited model expressibility.
  • Discretization error: Temporal and spatial discretization errors are modeled separately, with their relative importance depending on the integration scheme and resolution.For large time-steps with implicit methods, temporal integration can become the primary error source through numerical diffusion.
  • Data-free likelihood: The likelihood remains data-less because the numerical time-integrator generates target predictions on the fly from the model’s deterministic evolution.Under a Markov assumption, the likelihood of an entire sequence is formed as a product of individual time-step likelihoods.
  • Optimization objective: The probabilistic formulation recovers the standard L2 loss when minimizing the negative log likelihood, equivalent to minimizing the strong residual of the discretized PDE.This equivalence applies to both implicit and explicit time-integration schemes.
  • Parameter priors: The Bayesian network places a factorizable Gaussian prior on weights with Gamma-distributed precision, using a wider-support Student’s T prior to relax sparsity.The paper notes that a narrow prior was found to damage predictive capability, motivating weaker sparsity promotion.
  • Posterior approximation: SWAG approximates the posterior, but its single-mode Gaussian assumption cannot robustly represent multimodal distributions.The sampling learning rate must explore the converged minima without jumping to other local minima.

5. Kuramoto-Sivashinsky Equation

The paper evaluates AR-DenseED on the chaotic Kuramoto-Sivashinsky equation, comparing deterministic and Bayesian predictions with spectral ETDRK4 simulations. The model remains physically reasonable beyond the interval where chaotic divergence prevents pointwise agreement.

  • System: The K-S equation is a fourth-order nonlinear PDE known for spatio-temporal chaos on sufficiently large periodic domains.The paper uses it as a prototypical weakly turbulent system for testing complex-dynamics prediction.
  • Model and training: AR-DenseED uses two previous states as inputs and advances the discretized system autoregressively with a 0.1 time step.The K-S model contains just over 4800 learnable weights and uses 96 spatial points.
  • Deterministic predictions: After 1000 neural-network iterations, predictions remain consistent with the numerical solver for t = [0, 30] before chaotic divergence, while remaining qualitatively stable at longer times.The tests use three initial conditions and compare AR-DenseED predictions against spectral ETDRK4 results.
  • Deterministic predictions: Mean MSE begins increasing around t = 20, and AR-DenseED has fully diverged from the numerical simulator by t > 40.Despite this pointwise divergence, the predicted fields remain physical rather than becoming unstable.
  • Turbulent statistics: Time-averaged spectral energy density remains accurate at larger wavelengths, where most energy is concentrated, while discrepancies grow at higher frequencies with decaying energy.The statistic averages predictions over t = [0, 500], corresponding to at least 5000 stable prediction steps.
  • Computational comparison: For 5000 K-S time steps, AR-DenseED is less computationally effective than the highly efficient spectral ETDRK4 method, despite modeling the chaotic dynamics successfully.The wall-clock comparison averages 10 independent simulation runs.
  • Bayesian predictions: BAR-DenseED posterior samples initially agree but become distinct after t > 40, reflecting uncertainty growth from the chaotic K-S dynamics.The posterior uses SWAG, with predictive error estimated from 30 model samples over 200 test cases.

6. 1D Viscous Burgers’ Equation

The 1D viscous Burgers’ benchmark tests physics-constrained auto-regressive prediction on variable initial conditions containing multiple waves and shocks. AR-DenseED reproduces complex shock dynamics and offers Bayesian uncertainty estimates, while errors grow during extrapolation and some outlier cases perform poorly.

  • Problem setup: The benchmark uses periodic 1D viscous Burgers’ dynamics with velocity u, viscosity ν, and variable initial conditions containing multiple waves.The random initial condition is generated with a truncated Fourier series, producing multiple shock formations and interactions.
  • Model and training: The model uses five previous time-steps as inputs, a 512-point spatial discretization, and ∆t = 0.005 for auto-regressive prediction.The training uses 2560 randomly sampled scenarios and permits unrolling for up to 200 time-steps.
  • Deterministic predictions: AR-DenseED accurately predicts shock formations and intersections across four test cases, including predictions beyond the training time range.Testing predicts 400 time-steps, with t > 1.0 constituting extrapolation; the FEM target uses ∆t = 0.001.
  • Error evaluation: Prediction error initially spikes during shock formation and intersection, then grows in the extrapolation region; mean errors exceed medians because of poorly performing outlier cases.The authors identify compounding error from poor initial predictions as a drawback of the auto-regressive approach.
  • Computational comparison: The study compares AR-DenseED computational cost with FEM and FDM solvers for 400 time-steps of the 1D Burgers’ system.The supplied table caption specifies that wall-clock estimates average 10 independent simulation runs.
  • Probabilistic predictions: BAR-DenseED places most predictive variance near shocks, while uncertainty bands widen during extrapolation and still capture the true solution structure.Posterior samples show small early differences can change later wave locations; 90 parameter samples and 30 predictive samples are used.

7. 2D Coupled Burgers’ Equation

The 2D coupled Burgers’ benchmark extends the model to interacting x- and y-velocity fields with complex wave structures and discontinuities. AR-DenseED predicts these dynamics into extrapolation, while Bayesian uncertainty increases over time and can improve stability beyond the training range.

  • Problem setup: The 2D coupled Burgers’ system combines nonlinear advection and diffusion, with x- and y-velocity components on a periodic domain.The benchmark uses ν = 0.005 and domain size {x, y} ∈[0, 1].
  • Problem setup: Random Fourier-series initial conditions evolve into distinct structures as waves form, mix, interact, and dissipate.These dynamics make the 2D system substantially more complex than standard advection or diffusion examples.
  • Model and training: The largest AR-DenseED example predicts both velocity components through two output channels and contains about 72000 learnable parameters.Spatial gradients are approximated using Sobel-filter 2D convolutions in the physics-constrained formulation.
  • Deterministic predictions: AR-DenseED predicts 200 time-steps, including t > 0.5 extrapolation, and accurately reproduces complex structures and discontinuities in two test cases.The target is a high-fidelity FEM simulation on a 128 × 128 discretization interpolated to a 64 × 64 grid; errors concentrate on shock interfaces and wave fronts.
  • Figures and profiles: The 2D figures compare FEM targets, AR-DenseED predictions, and L1 errors separately for x- and y-velocity components.Profile plots additionally show predictive expectation and standard deviation at selected horizontal and vertical locations and times.
  • Error and cost evaluation: Prediction error remains stable within the training region but grows faster for the mean than the median during extrapolation, indicating poor outlier cases.The study evaluates generalized MSE and ESE over 200 test scenarios and compares computational cost with FEM.
  • Probabilistic predictions: BAR-DenseED posterior variance increases as time progresses, while predictive standard deviation captures the true solution for almost all times.The Bayesian model also achieves smaller and more stable prediction error during extrapolation beyond the training range.

8. Conclusion

The paper presents a physics-constrained deep auto-regressive convolutional network for transient PDE dynamics, with Bayesian uncertainty quantification and zero output training data. Across three PDE systems, it reports accurate predictions, extrapolation beyond the training time range, and substantially faster surrogate modeling than numerical solvers.

  • The model learns and surrogate-models transient PDE dynamics using governing equations to train without output data.
  • Bayesian modeling provides uncertainty quantification for both epistemic and aleatoric uncertainty.
  • The model accurately reproduces turbulent statistics, multi-shock formation and intersections, and complex coupled Burgers’ wave dynamics across three PDE systems.
  • The model successfully extrapolates beyond the time range used during training.
  • Convolutional physics-constrained surrogate models can be orders of magnitude faster than state-of-the-art numerical solvers.
  • Training cost remains a major challenge as the model is extended to more complex and larger systems.

Appendix A. Kuramoto-Sivashinsky

The Kuramoto-Sivashinsky implementation uses a compact auto-regressive dense encoder-decoder with convolutional encoding, dense feature growth, and decoding by up-sampling. Two previous time-steps provide the model inputs.

  • The Kuramoto-Sivashinsky model encodes two previous time-steps through a convolution, dense block, and decoding block.The model has 4821 learnable parameters, with dense-block growth rate 4 and length 4.
  • The encoder maps the 1D input {u_n, u_n−1} ∈ R^d to latent variables with dimensionality d/2.
  • Training used a relatively small model because smaller models were more stable and faster, with circular padding supporting periodic boundaries.
  • Dense blocks use residual connections that stack convolution outputs and substantially increase feature-channel count.
  • The decoding block uses sequential normalization, ReLU, convolutions, and nearest-neighbor up-sampling to restore the desired data dimensions.

Appendix A.1. Training Initial States

The training initial states are generated without simulator output data using truncated Fourier series with random coefficients. The construction is designed to provide physically realizable initial conditions for exploring the Kuramoto-Sivashinsky system.

  • Initial states are generated from a truncated Fourier series with random coefficients instead of simulator data.
  • The initial condition rescales the sampled function w(x) using a random amplitude parameter a.
  • The coefficient construction uses random variables for Fourier coefficients, phase, and amplitude around an a priori mean amplitude estimate.The a priori mean amplitude a0 is set to 2.5.
  • The number of unstable modes can be estimated from the domain length using k0 = L/(2π).
  • The resulting function is intended to provide physically realizable initial conditions for AR-DenseED to explore the K-S system.

Appendix B. 1D Viscous Burgers’ System

The 1D viscous Burgers’ implementation uses a larger auto-regressive dense encoder-decoder than the Kuramoto-Sivashinsky model and takes five previous time-steps as inputs. Sharp gradients made this system the most difficult of the three to train.

  • The 1D Burgers’ model uses an encoding convolutional block, dense block, and decoding block with five previous time-steps as inputs.Its dense block has growth rate 4 and length 1.
  • Sharp gradients in the low-viscosity Burgers’ system likely make training difficult by contributing to exploding optimization gradients.
  • The Burgers’ model has 13442 learnable parameters, approximately three times the size of the Kuramoto-Sivashinsky model.
  • Training parameters for the AR-DenseED and BAR-DenseED Burgers’ models are listed in Table B.5.

Appendix C. 2D Coupled Burgers’ System

The 2D coupled Burgers’ model is an auto-regressive dense encoder-decoder convolutional network that predicts both velocity components from prior time-steps. The appendix describes its architecture and training-parameter specification.

  • Model inputs and outputs: The model predicts the x and y velocity components and uses both components as inputs.With three previous time-steps, the input χn+1 has six channels.
  • Network architecture: The AR-DenseED architecture has 71,953 learnable parameters, with encoding, dense, and decoding convolutional blocks.Its dense block has growth rate 4 and length 4.
  • Model inputs and outputs: Five previous time-steps are used as inputs in the illustrated AR-DenseED model.
  • Training setup: A table specifies the AR-DenseED and BAR-DenseED training parameters for the 2D coupled Burgers’ system.
Loading 1906.05747v3…