Learning in Modal Space: Solving Time-Dependent Stochastic PDEs Using Physics-Informed Neural Networks

Dongkun Zhang; Ling Guo; George Em Karniadakis

Source-linked AI summary

Learning in Modal Space: Solving Time-Dependent Stochastic PDEs Using Physics-Informed Neural Networks

Dongkun Zhang, Ling Guo, George Em Karniadakis

arXiv:1905.01205v2May 3, 2019cs.LGmath.NAphysics.comp-phstat.ML

TL;DR

Long-time integration of nonlinear time-dependent SPDEs remains challenging, especially when standard DO/BO representations require restrictive covariance or eigenvalue assumptions. The paper introduces NN-DO/BO PINNs that impose these constraints implicitly in the loss, and demonstrates their use across forward, inverse, and noisy-data problems. The reported examples show applicability to uncertainty propagation while retaining explicit limitations in accuracy and training cost.

Problem
Long-time integration of nonlinear time-dependent SPDEs is an open scientific-computing problem, while standard DO/BO methods require invertible covariance matrices or no eigenvalue crossing.
Method
NN-DO/BO methods use PINN surrogates based on generalized KL expansions and impose DO/BO constraints implicitly through the loss function.
Results
NN-DO/BO methods handle deterministic-initial-condition cases, eigenvalue crossings, and time-dependent stochastic inverse problems across the reported numerical examples.
Takeaways & Limitations
The simulations demonstrate that NN-DO/BO methods can quantify uncertainty propagation across a range of physical problems, including forward and inverse settings.
Takeaways & Limitations
Absolute errors cannot reach below about 10^-5, and training is especially costly for long-time integration.

Abstract

from arXiv · show

One of the open problems in scientific computing is the long-time integration of nonlinear stochastic partial differential equations (SPDEs). We address this problem by taking advantage of recent advances in scientific machine learning and the dynamically orthogonal (DO) and bi-orthogonal (BO) methods for representing stochastic processes. Specifically, we propose two new Physics-Informed Neural Networks (PINNs) for solving time-dependent SPDEs, namely the NN-DO/BO methods, which incorporate the DO/BO constraints into the loss function with an implicit form instead of generating explicit expressions for the temporal derivatives of the DO/BO modes. Hence, the proposed methods overcome some of the drawbacks of the original DO/BO methods: we do not need the assumption that the covariance matrix of the random coefficients is invertible as in the original DO method, and we can remove the assumption of no eigenvalue crossing as in the original BO method. Moreover, the NN-DO/BO methods can be used to solve time-dependent stochastic inverse problems with the same formulation and computational complexity as for forward problems. We demonstrate the capability of the proposed methods via several numerical examples: (1) A linear stochastic advection equation with deterministic initial condition where the original DO/BO method would fail; (2) Long-time integration of the stochastic Burgers' equation with many eigenvalue crossings during the whole time evolution where the original BO method fails. (3) Nonlinear reaction diffusion equation: we consider both the forward and the inverse problem, including noisy initial data, to investigate the flexibility of the NN-DO/BO methods in handling inverse and mixed type problems. Taken together, these simulation results demonstrate that the NN-DO/BO methods can be employed to effectively quantify uncertainty propagation in a wide range of physical problems.

1. Introduction

The paper targets time-dependent nonlinear SPDEs by combining PINNs with DO/BO stochastic representations, embedding their constraints implicitly in neural-network loss functions. This formulation addresses key restrictions of standard DO/BO methods and extends to stochastic inverse problems.

PINNs augment data-mismatch loss with PDE residuals evaluated through automatic differentiation, supporting physics-informed approximation of PDE solutions.The approach can require less training data, respect conservation laws, and predict system states beyond interpolation within the training domain.
Time-dependent stochastic KL representations evolve spatial modes and stochastic coefficients simultaneously, requiring DO or BO constraints to remove redundancy.Standard DO/BO formulations require explicit evolution equations and can become unstable with singular or ill-conditioned covariance matrices; BO can also fail at eigenvalue crossings.
NN-DO/BO methods combine PINN surrogates with generalized KL expansions and impose DO/BO constraints directly in the loss function.This avoids deriving explicit temporal-derivative expressions for the modes.
NN-DO does not require an invertible covariance matrix, including for SPDEs with deterministic initial conditions.This removes a standard DO assumption that fails when all random coefficients initially vanish.
NN-BO can handle eigenvalue crossings, while the same formulation and code apply to time-dependent stochastic inverse problems or sparse noisy data.The inverse-problem formulation has the same computational complexity as the forward formulation.

2. Problem Setup

The paper formulates a time-dependent SPDE on a bounded spatial domain and finite time interval over a probability space. Its goal is to solve the equation and evaluate the solution’s mean and standard deviation.

The SPDE is accompanied by initial and boundary conditions denoted by u0(x; ω) and h(t, x; ω).These conditions supplement the differential equation and boundary operator in the problem setup.
The quantity of interest u(x, t; ω) is assumed to be a second-order random field.The stated targets are the mean and standard deviation of the solution.

3. An Overview of the DO and BO Decomposition Methods

The generalized KL representation separates a time-dependent random field into a mean and stochastic fluctuation modes, with DO and BO constraints removing redundancy. The methods differ in how orthogonality is enforced and carry covariance scaling in different components.

Generalized KL representation: The generalized KL expansion decomposes the random field into a deterministic mean and time-dependent spatial modes with zero-mean stochastic coefficients.Both modes and coefficients evolve with the system dynamics, unlike standard polynomial chaos coefficients.
Dynamically Orthogonal representation: The DO representation imposes a dynamical orthogonality condition so evolving spatial modes remain orthogonal to their span.The DO condition preserves orthonormality and unit length of the spatial bases.
Bi-Orthogonal representation: The BO representation imposes static orthogonality on both spatial bases and stochastic coefficients.In BO, spatial bases carry covariance eigenvalues, whereas stochastic coefficients carry them in DO.
Comparison: DO and BO are theoretically equivalent through a differential transformation, although their constraints govern the evolution of bases differently.DO restricts spatial-mode evolution to directions normal to their span; BO adds orthogonality restrictions on random coefficients instead.
Limitations and proposed direction: The original DO method requires an invertible coefficient covariance matrix, while BO requires no eigenvalue crossing to obtain explicit evolution matrices.These assumptions can fail for deterministic initial conditions and eigenvalue crossings, respectively.

4.1. Physics-informed neural network

A PINN approximates a differential-equation solution with a neural network while enforcing both observations and the governing physics through a residual loss. Automatic differentiation constructs the residual from the surrogate network during training.

PINN formulation: A PINN uses a neural-network surrogate for the solution and augments data mismatch with the residual of the governing differential equation.The residual is evaluated through automatic differentiation and training minimizes a loss reflecting both constraints.
PINN formulation: The surrogate network takes spatial coordinates as input and outputs a vector with the same dimension as the solution.Its parameters include the network weights and biases, which are tuned during training.
Physics residual: Automatic differentiation computes derivatives of the surrogate with respect to its inputs to form the physics residual network.The residual network shares parameters with the solution surrogate and should output zero across the domain.
Training: Training minimizes a loss combining observed solution values with the requirement that the physics residual vanish.This formulation also supports inference of additional parameters when they are included in the residual network.

4.2. A weak formulation interpretation of the DO and BO methods

The DO and BO evolution equations can be interpreted as a weak formulation obtained by applying expectation and inner-product operators to the SPDE’s finite stochastic expansion. The resulting basis functions and coefficients provide valid physical- and probability-space bases.

Weak formulation derivation: The derivation first applies expectation to the SPDE after replacing the solution with its finite stochastic expansion.The zero-mean coefficient condition simplifies the resulting mean equation.
Weak formulation derivation: Projection onto each spatial mode and multiplication by each stochastic coefficient generate the remaining weak equations.These operations produce equations for the spatial and stochastic components.
Constraint enforcement: Substituting the truncated expansion and imposing DO or BO constraints simplifies the projected equations into the weak formulation.The constraints remove redundancy in the representation.
Basis interpretation: Orthogonal spatial modes and stochastic coefficients form valid bases in the physical domain and probability space, respectively.The DO representation is equivalent to BO, so its coefficients do not degenerate in this construction.

4.3. NN-DO/BO Methods

NN-DO/BO methods represent the SPDE solution with several neural networks and enforce the weak SPDE, initial and boundary conditions, and DO/BO constraints through a weighted loss. The implicit constraint formulation avoids explicit temporal-mode derivatives and supports both NN-DO and NN-BO variants.

Network construction: Four neural networks represent the mean, scaling factors, spatial modes, and stochastic coefficients in the generalized expansion.Together they construct a surrogate for the truncated SPDE solution.
Loss formulation: The loss combines weak-SPDE residuals, initial and boundary conditions, DO/BO constraints, and regularization terms.Each component is formed from mean squared errors evaluated at sampled physical, stochastic, and temporal points.
Initial and boundary conditions: Initial conditions for deterministic data set the scaling factors to zero while choosing orthonormal spatial bases and unit-variance stochastic bases.The initial condition is imposed through an additional penalty term.
Initial and boundary conditions: Periodic boundary conditions can be imposed directly by replacing x with sin(2πx/L) and cos(2πx/L) in the mean and mode networks.This removes the separate periodic-boundary loss term.
Loss weighting: The training objective uses relatively large weights for initial, boundary, and DO/BO losses, while regularization receives a small weight.The authors state that the distribution of weights remains an open question for future research.
Algorithm: The algorithm supports either DO or BO constraints, producing the NN-DO and NN-BO methods.Training samples points in physical, stochastic, and time domains, minimizes the loss, and reconstructs the SPDE solution.
Advantages: Unlike the original methods, NN-DO/BO does not require explicit temporal derivatives of the modes, an invertible covariance matrix, or no eigenvalue crossing.The constraints are incorporated implicitly in the neural-network loss.

5. Simulation Results

The paper evaluates NN-DO/BO methods on benchmark and nonlinear stochastic problems, including forward and inverse settings with rough initial data.

Two benchmark cases are designed to have exact DO and BO representations.
A nonlinear diffusion-reaction equation uses a 19-dimensional random input and rough initial conditions given as discrete point values.
An inverse problem is included to demonstrate the capacity of the proposed NN-DO/BO methods.

5.1. Application to a Linear Stochastic Problem

The linear stochastic advection example tests NN-DO and NN-BO against exact solutions. NN-DO handles deterministic initial conditions, while both formulations accurately recover solution components and statistics.

Problem setup: The problem uses a random advection velocity ξ ∼ N(0, σ2) with σ = 0.8 on [−π, π] until T = π.Periodic boundary conditions are imposed.
Reference solutions: Exact mean, variance, DO components, and BO components are available for comparison.
Case 1: NN-DO Method: The standard DO method cannot be directly applied with deterministic initial conditions, but NN-DO obtains good results.
Case 1: NN-DO Method: NN-DO bases agree well with exact bases, and its random coefficients evolve over time while preserving orthogonality.
Case 2: NN-BO Method: NN-BO solutions show good agreement with exact reference solutions for scaling factors, bases, random coefficients, mean, and variance.Its performance is reported as very good and similar to NN-DO.
Case 2: NN-BO Method: During training, MSEw decreases from more than 0.1 to less than 0.001, while MSEIC and MSEBO remain around 10^-5 to 10^-4.The original-equation loss MSE0 decays but contributes little because of its small weight.

5.2. Application to Nonlinear Stochastic Problem

The NN-DO and NN-BO methods are tested on a nonlinear stochastic Burgers’ equation over long time intervals, including cases challenging the standard DO/BO formulations. Both methods closely reproduce exact solution components and maintain low errors through t = 10π.

Problem setup: The Burgers’ equation experiment uses a manufactured solution with two uniformly distributed random variables, allowing exact DO and BO components for comparison.The physical domain is [−π, π], the viscosity is ν = 0.1, and the stochastic forcing is parameterized by ξ1 and ξ2.
Training strategy: The time interval [0, 10π] is divided into ten subdomains, with independent neural networks trained sequentially and linked through interval-end initial conditions.Each subdomain has length π, supporting long-time integration while propagating the previous interval’s terminal result forward.
NN-DO results: The NN-DO solution accurately matches exact bases, means, and variances, including the increasingly complex variance structure between t = 5π and t = 10π.The two scaling factors retain comparable amplitude rather than decaying, indicating fully developed stochasticity in the manufactured system.
NN-DO results: All NN-DO relative L2 errors at T = 10π are around or below 1%, while mean and variance errors remain below 2% for most of the time.The relative variance error is slightly higher than the relative mean error.
NN-BO results: The NN-BO method remains accurate despite substantial eigenvalue crossings throughout the time evolution and within individual time subdomains.The standard BO method would fail in this setting because explicit formulas for matrices M and S are unavailable, whereas all NN-BO relative L2 errors are less than 1%.

5.3. Application to Nonlinear Di

The NN-BO method is applied to nonlinear stochastic diffusion-reaction equations in forward and inverse settings, including noisy initial data. It produces accurate solution statistics and modal components while identifying unknown coefficients from sparse mean measurements.

Problem setup: The diffusion-reaction problem has nonlinear source term bu^2, Gaussian-process forcing, Dirichlet boundaries, and deterministic initial condition −sin(πx).The forward task solves for u with known a and b, whereas the inverse task also identifies unknown coefficients from additional solution information.
Forward problem: Using six BO modes, NN-BO accurately predicts means, modal functions, and random coefficients at both t = 0.1 and t = 1.0.In the 19-dimensional stochastic problem, the first four modes gain energy from the nonlinear source while the fifth and sixth remain relatively stable.
Noisy initial data: With noisy sensor measurements as the initial condition, NN-BO slightly underestimates final-time variance because of the truncated expansion, while noisy initialization has limited effect on the final prediction.The comparison uses 30 uniformly placed sensors and Monte Carlo reference statistics.
Inverse problem: For the inverse problem, six mean measurements at three spatial locations and two times are used to infer unknown diffusion and reaction coefficients.The hidden values are a = 0.5 and b = 0.3, and the coefficients are trained as variables through an additional measurement-data loss term.
Inverse problem: NN-BO remains accurate for the inverse problem, reproducing solution statistics, BO modes, scaling factors, and convergent estimates of a and b against reference solutions.The inverse representation uses four BO modes, with coefficient references generated from the forward problem.

6. Summary

The paper introduces NN-DO/BO PINNs for time-dependent SPDEs by enforcing DO/BO constraints implicitly in the loss function. Numerical examples show accurate uncertainty propagation across long-time, nonlinear, noisy-data, and inverse settings, while training cost and optimization accuracy remain limitations.

Summary: NN-DO/BO combines PINNs with DO or BO constraints to remove redundancy in generalized Karhunen-Loeve representations of time-dependent SPDEs.The constraints are imposed implicitly through the neural-network loss rather than explicit temporal derivative formulas.
Summary: The proposed methods avoid the original DO covariance-invertibility assumption and the original BO no-eigenvalue-crossing assumption.This broadens the stated applicability to problems such as deterministic initial conditions and eigenvalue crossings.
Summary: Numerical studies report accurate results for benchmark DO/BO problems and nonlinear diffusion-reaction forward and inverse problems.The examples include noisy initial data and demonstrate the methods’ use for uncertainty quantification in the tested settings.
Limitations: Absolute errors cannot reach below about 10^-5 because PINN training solves a non-convex optimization problem without theoretical guarantees of a global minimum.The authors also identify excessive training cost, especially for long-time integration, and suggest time-parallel methods such as parareal as a promising direction.