Numerical Gaussian Processes for Time-dependent and Non-linear Partial Differential Equations

Maziar Raissi; Paris Perdikaris; George Em Karniadakis

Source-linked AI summary

Numerical Gaussian Processes for Time-dependent and Non-linear Partial Differential Equations

Maziar Raissi, Paris Perdikaris, George Em Karniadakis

arXiv:1703.10230v1Mar 29, 2017stat.MLmath.APmath.DSmath.NA

TL;DR

The paper addresses solving time-dependent PDEs from noisy black-box initial data while quantifying the resulting uncertainty. It introduces numerical Gaussian processes whose priors are placed to encode temporal discretizations without spatial operator discretization. Across linear and nonlinear benchmark problems, the method recovers latent solutions and propagates uncertainty, including over long integrations.

Problem
The paper asks how to learn solutions of time-dependent PDEs from noisy black-box initial conditions while quantifying uncertainty in those solutions.
Method
The method constructs numerical Gaussian processes by placing Gaussian-process priors so temporal numerical schemes encode PDE operators without spatial discretization.
Results
Across linear and nonlinear benchmark problems, the method recovers accurate latent-solution approximations and consistently propagates uncertainty, including during very long time integration.
Takeaways & Limitations
The approach provides a structured, physics-informed and data-efficient learning framework for uncertainty-aware PDE solution inference from noisy data.
Takeaways & Limitations
The present method has cubic computational scaling with the total number of training data points.

Abstract

from arXiv · show

We introduce the concept of numerical Gaussian processes, which we define as Gaussian processes with covariance functions resulting from temporal discretization of time-dependent partial differential equations. Numerical Gaussian processes, by construction, are designed to deal with cases where: (1) all we observe are noisy data on black-box initial conditions, and (2) we are interested in quantifying the uncertainty associated with such noisy data in our solutions to time-dependent partial differential equations. Our method circumvents the need for spatial discretization of the differential operators by proper placement of Gaussian process priors. This is an attempt to construct structured and data-efficient learning machines, which are explicitly informed by the underlying physics that possibly generated the observed data. The effectiveness of the proposed approach is demonstrated through several benchmark problems involving linear and nonlinear time-dependent operators. In all examples, we are able to recover accurate approximations of the latent solutions, and consistently propagate uncertainty, even in cases involving very long time integration.

1. Introduction

The paper develops numerical Gaussian processes to incorporate physical structure into learning from noisy black-box initial data for time-dependent PDEs. The approach uses Gaussian-process priors with numerical schemes to represent operators, propagate uncertainty, and avoid spatial discretization.

The motivating gap is that many machine-learning methods do not explicitly leverage physical laws underlying observed data.
The problem setting involves noisy observations of a black-box initial function and prediction of the latent solution with propagated uncertainty.
Gaussian-process regression supplies a probabilistic prior and posterior variance estimates for quantifying uncertainty in function-valued solutions.
The resulting covariance functions define numerical Gaussian processes, whose structure follows directly from the chosen numerical scheme and prior.
The framework is generalized to arbitrary linear multi-step methods and Runge-Kutta methods.
Properly placed Gaussian-process priors combine with numerical schemes to encode differential operators without inverting them or spatially discretizing them.

2. Linear Multi-step Methods

The framework extends numerical Gaussian processes from Euler stepping to general linear multi-step schemes. Algebraic reformulation and prior placement encode each scheme in the resulting joint process while avoiding spatial operator inversion.

General linear multi-step methods are parameterized by coefficients α_i and β_i, with different choices producing specific schemes.
The methods are equivalently rewritten using operators that combine the identity with Δt times the spatial differential operator.
The construction shifts intermediate terms by a scheme-dependent τ to express prior-linked states at compatible time locations.
Trapezoidal rule: For the trapezoidal rule, the relations P_xu_n+1/2 = u_n = Q_xu_n−1/2 and P_xu_n−1/2 = u_n−1 = Q_xu_n−3/2 connect neighboring states.
Trapezoidal rule: Placing a Gaussian-process prior on u_n−1/2 captures the trapezoidal rule while avoiding spatial discretization because no operator inversion is required.

2.1. Prior

The prior construction assigns independent Gaussian processes to appropriately shifted states, then uses numerical operators to produce a correlated multi-output process. Boundary observations can be incorporated through noisy linear transformations.

For each shifted state, the method assigns a Gaussian-process prior with its own covariance kernel k_j,j(x, x′; θ_j).
Applying the numerical operators to these processes yields the numerical Gaussian process and its covariance functions.
The covariance kernels capture the entire structure of the linear multi-step method.
Although the component priors are independent, the resulting numerical Gaussian process has a fully correlated structure.
Boundary conditions: Boundary information can be represented by noisy observations of a linear transformation B_x of the solution.
Boundary conditions: The numerical examples cover Dirichlet, Neumann, mixed, and periodic boundary conditions.

2.2. Work flow and computational cost

The workflow alternates Gaussian-process training and posterior prediction to generate artificial data at successive time steps. Its principal computational burden is cubic scaling from covariance-matrix factorization, while uncertainty is marginalized during propagation.

The workflow begins by training kernel hyperparameters using initial and boundary data.
Posterior prediction generates artificial data for the next time step, with spatial locations sampled uniformly and values sampled from a normal distribution.
Computational cost: Training scales cubically with the total number of training points because it requires Cholesky factorization of full covariance matrices.
Subsequent steps retrain hyperparameters and repeat prediction until the final integration time is reached.
Uncertainty propagation: Because the intermediate artificial state is random, it is marginalized to obtain consistent uncertainty estimates for the next state.
The workflow consists of Gaussian-process regressions at every time step, using previous hyperparameters as initial guesses to accelerate training.

2.3. Training

Training uses marginal likelihood to fit Gaussian-process hyper-parameters, balancing data fit against model complexity.

The hyper-parameters θ_i are trained by minimizing the Negative Log Marginal Likelihood.
The training data include noisy boundary observations and artificially generated data from the previous time step.
Marginal likelihood provides a regularization mechanism that balances data fit and model complexity.

2.4. Posterior

The posterior is obtained by conditioning the numerical Gaussian process at new test points, producing predictions for the solution.

The method uses a conditional distribution to predict u_n at a new test point x_n*.

2.5. Propagating Uncertainty

Uncertainty propagation marginalizes artificially generated intermediate data before generating the next time-step data.

Artificial data from the previous time step are marginalized to propagate uncertainty through time.
The resulting posterior distribution generates artificial data for the next time step.

2.6. Example: Burgers’ equation (Backward Euler)

The Burgers’ equation example applies a backward Euler numerical Gaussian process to noisy black-box initial data and nonlinear dynamics. It propagates posterior uncertainty while recovering accurate behavior and first-order temporal convergence.

2.6. Example: Burgers’ equation (Backward Euler): Burgers’ equation tests the method on a nonlinear PDE with noisy measurements of a black-box initial function and Dirichlet boundary conditions.
2.6. Example: Burgers’ equation (Backward Euler): The nonlinear product of Gaussian processes is avoided by replacing the nonlinear term with the previous posterior mean.
2.6. Example: Burgers’ equation (Backward Euler): Figure 1 uses backward Euler with ∆t = 0.01 and 31 artificial data points per time step to show posterior distributions across time snapshots.
2.6. Example: Burgers’ equation (Backward Euler): A neural-network covariance prior is chosen because small viscosity can produce discontinuities in Burgers’ equation solutions.
2.6. Example: Burgers’ equation (Backward Euler): The method propagates an infinite collection of correlated Gaussian random variables through the nonlinear Burgers’ dynamics.
2.6. Example: Burgers’ equation (Backward Euler): The relative spatial L2 error grows as O(∆t), confirming first-order temporal accuracy until saturation below approximately 10^-3.

2.7. Example: Wave Equation (Trapezoidal Rule)

The wave-equation example places Gaussian process priors within a trapezoidal discretization to infer both solution components from noisy initial data while propagating uncertainty. The resulting numerical Gaussian process is reported as second-order accurate in time, with convergence assessed over integration time, step size, and training-point count.

Problem formulation: The wave equation is rewritten as a first-order system by introducing v := u_t, enabling joint inference of u and v.The example uses noisy measurements of both black-box initial functions u_0 and v_0 and seeks uncertainty-aware solutions for t > 0.
Uncertainty quantification: The posterior compares the true solution, posterior mean, and two-standard-deviation uncertainty bands across time snapshots using artificial observations for both u and v.The figures use 51 artificial data points for u and 49 for v, randomly located in [0, 1].
Method: The trapezoidal-rule formulation guides Gaussian process prior placement so the numerical process encodes the wave-equation structure without spatial operator discretization.Rearranging and shifting the discretized equations identifies the locations for priors on intermediate quantities and the two solution components.
Method: Independent squared-exponential Gaussian processes are assigned to the half-step variables for u and v, producing a structured multi-output numerical Gaussian process.The squared-exponential kernels imply smooth approximations, while the resulting covariance functions capture the trapezoidal rule applied to the wave equation.
Numerical study: Second-order temporal convergence is observed through relative spatial L2-error evolution to T = 1.5 and step-size refinement, with additional experiments varying training-point count.The study notes that the u error is not always lower than the v error; that ordering occurs at T = 0.2 in the reported results.

3. Runge-Kutta Methods

The paper extends numerical Gaussian processes to general Runge-Kutta schemes, encoding their stage structure in covariance functions rather than discretizing spatial differential operators. A two-stage Gauss-Legendre example demonstrates fourth-order temporal accuracy and supports uncertainty-aware inference for long-time advection problems.

Runge-Kutta formulation: A q-stage Runge-Kutta discretization defines both the next solution and intermediate stage functions through weights a_ij and b_i.The stage functions satisfy u_{n+τ_i}(x) = u(t_n + τ_i∆t, x), and parameter choices determine whether the scheme is implicit or explicit.
Computational implication: For a fixed number of stages q, implicit and explicit time marching have identical cost because both reduce to sequences of regression problems.This representation avoids repeatedly solving linear or nonlinear systems while retaining the stability properties of fully implicit schemes.
Prior construction: The numerical Gaussian process is constructed from q + 1 mutually independent Gaussian processes whose covariance functions encode the complete Runge-Kutta structure.The method can inspect the same artificial data through different kernels associated with the next-step and stage processes.
Advection example: The two-stage Gauss-Legendre method is fourth-order accurate and is applied to advection with periodic boundary conditions and noisy initial data.The example uses the numerical Gaussian process to infer u(t, x) with quantified uncertainty for t > 0.
Numerical study: The advection experiments assess posterior inference with artificial data and verify fourth-order temporal accuracy through time-evolution, step-size, and training-point convergence studies.The qualitative convergence conclusions are reported as consistent across the temporal and spatial experiments.

4. Concluding Remarks

The framework encodes PDE physics into Gaussian-process priors for inference with noisy initial or boundary data, while propagating uncertainty over time. The authors identify computational scaling and unresolved probabilistic numerical-analysis questions as important limitations and directions for future work.

The framework encodes physical laws from partial differential equations into Gaussian-process priors for nonparametric Bayesian regression.
The proposed algorithms infer solutions to time-dependent and nonlinear partial differential equations while quantifying and propagating uncertainty from noisy initial or boundary data.
The method propagates entire probability distributions in time, unlike classical deterministic numerical methods for partial differential equations.
The present implementation has cubic computational scaling with respect to the total number of training data points.
Future work includes more efficient inference, applications to complex physical systems, parameter and inverse problems, and probabilistic time integration for time-stepping uncertainty.
Theoretical questions involving prior consistency, posterior robustness, and posterior contraction rates remain largely unexplored for numerical Gaussian processes.

5. Appendix

The appendix specifies covariance functions, training and prediction ingredients, boundary-data handling, and uncertainty propagation for the Burgers, wave, advection, and heat equation examples.

The appendix gives covariance-function constructions for the Burgers’ equation example.
Kernel computations require derivatives of kernels, which can be performed with symbolic-computation software.
The wave equation example is accompanied by covariance functions and straightforward kernel computations.
The hyper-parameters θu and θv are trained by minimizing the resulting Negative Log Marginal Likelihood.
Boundary data in the heat-equation example correspond to Dirichlet boundary conditions.
The appendix describes conditional-distribution prediction, artificial-data generation, and marginalization to propagate uncertainty for the advection and heat equation examples.