Source-linked AI summary
Three Ways to Solve Partial Differential Equations with Neural Networks -- A Review
Jan Blechschmidt, Oliver G. Ernst
TL;DR
Neural-network PDE solvers address the challenge of applying numerical methods to complex or high-dimensional problems, though they remain limited against classical methods in low dimensions and lack mature error analysis. This review presents PINNs, Feynman-Kac methods, and BSDE-based methods, alongside executable Jupyter notebooks, and concludes that their strengths differ by problem regime.
Problem
Neural-network PDE solvers are investigated for scientific machine learning, but they typically cannot compete with classical methods in low to moderate dimensions and lack mature error analysis.
Method
The review contrasts PINNs, Feynman-Kac methods, and BSDE-based methods, with accompanying Jupyter notebooks that explain implementations and numerical experiments.
Results
The reviewed methods illustrate different strengths: PINNs suit low-dimensional complex nonlinear PDEs, while Feynman-Kac and BSDE methods target high-dimensional linear and semilinear parabolic problems.
Takeaways & Limitations
The review provides an accessible overview and practical experimentation resources for assessing neural-network PDE methods across different problem classes.
Takeaways & Limitations
Selecting neural-network architectures and parameters remains an open problem, and neural-network PDE solvers lack the mature error analysis of traditional methods.
Abstract
from arXiv · showhide
Neural networks are increasingly used to construct numerical solution methods for partial differential equations. In this expository review, we introduce and contrast three important recent approaches attractive in their simplicity and their suitability for high-dimensional problems: physics-informed neural networks, methods based on the Feynman-Kac formula and methods based on the solution of backward stochastic differential equations. The article is accompanied by a suite of expository software in the form of Jupyter notebooks in which each basic methodology is explained step by step, allowing for a quick assimilation and experimentation. An extensive bibliography summarizes the state of the art.
1 INTRODUCTION
The review introduces neural-network methods for numerical solution of linear and nonlinear PDEs, motivated by persistent challenges such as the curse of dimensionality. It focuses on three approaches and provides accompanying notebooks for accessible experimentation.
- Motivation: Classical PDE discretization schemes face the curse of dimensionality, with complexity increasing at least by 2^d when degrees of freedom double in each of d directions.This challenge is especially relevant to high-dimensional problems.
- Motivation: Neural networks are attractive because their compositional structure and optimization-based parametrization suit highly nonlinear and high-dimensional PDE approximations.The review contrasts this with additive trial functions used in conventional Galerkin, collocation, and finite-volume methods.
- Scope: Neural-network PDE solvers generally do not compete with classical methods in low to moderate dimensions and lack equally mature error analysis.Their training requires highly nonlinear large-scale optimization, while classical methods often solve algebraic systems.
- Scope: The review examines physics-informed neural networks, Feynman-Kac methods, and backward stochastic differential equation methods in detail.Additional scientific machine learning approaches are collected separately.
- Resources: A collection of commented Jupyter notebooks supplies Python implementations and numerical experiments for the methods reviewed in Sections 2–4.The notebooks can run in Google Colaboratory without local installation.
2 PHYSICS-INFORMED NEURAL NETWORKS
Physics-informed neural networks approximate PDE solutions by embedding the governing equation and initial or boundary conditions into a neural-network training objective. Their main appeal is flexibility across challenging nonlinear PDEs, while their practical limitations include optimization difficulty and less mature error analysis.
- Continuous Time Approach: PINNs target PDE problems in the small-data setting, where only the PDE problem data is available rather than many variable-value pairs.The method constructs a neural-network approximation u_θ(t, x) to the PDE solution.
- Continuous Time Approach: Automatic differentiation evaluates the time derivative, spatial differential operator, and PDE residual of the neural-network approximation at collocation points.The residual is incorporated into the loss together with initial and boundary-condition misfit terms.
- Continuous Time Approach: PINNs are flexible across challenging nonlinear PDEs, whereas classical numerical approximations typically require tailoring to each PDE.This flexibility is presented as the principal strength of PINNs, rather than mitigation of the curse of dimensionality.
- Continuous Time Approach: Multilayer feed-forward networks approximate solutions through compositions of affine maps and nonlinear activation functions.Their compositional structure is associated with strong approximation properties in many applications.
- Continuous Time Approach: The loss functional combines mean squared PDE residual with mean squared misfit to initial and boundary conditions.Training data consists entirely of time-space coordinates, and individual loss-term weighting may improve convergence.
3 LINEAR PDES IN HIGH DIMENSIONS: THE FEYNMAN-KAC FORMULA
The review presents the Feynman-Kac approach for approximating backward Kolmogorov PDE solutions in high dimensions by connecting PDEs to stochastic processes and neural-network regression. It combines theoretical justification, data-generation procedures, numerical experiments, and implementation guidance.
- 3 LINEAR PDES IN HIGH DIMENSIONS: THE FEYNMAN-KAC FORMULA: The approach targets high-dimensional PDEs arising as backward Kolmogorov equations of Itô diffusions, including applications such as financial valuation and stochastic control.
- 3.1 The Feynman-Kac Formula: For a Kolmogorov backward equation, the Feynman-Kac theorem expresses u(t, x) as the conditional expectation of g(X_T) given X_t = x.
- 3.1 The Feynman-Kac Formula: Under globally Lipschitz drift and diffusion coefficients, the associated SDE has a pathwise unique strong solution, enabling the stochastic-process representation.
- 3.1 The Feynman-Kac Formula: Applying Itô’s formula and conditional-expectation arguments confirms that the Feynman-Kac representation solves the corresponding Kolmogorov PDE.
- 3.2.1 Generation of Training Data: The method learns u_θ(0, x) from stochastic-process data, avoiding artificial truncation boundary conditions because sample paths can leave the bounded domain.
- 3.2.2 Neural Network Approximation: The neural-network approximation is theoretically linked to a continuous function u* satisfying u*(x) = u(0, x) under suitable assumptions.
- 3.2.3 Example: Heat equation: In the 100-dimensional heat-equation experiment, final approximation quality depended heavily on learning-rate selection, with conservative smaller steps appearing preferable to initially larger exponentially decaying steps.
- 3.4 Summary and Extensions: The reviewed approach can be implemented as a regression problem using directly sampled or time-stepped SDE data, while general network-architecture and parameter-selection guidance remains future work.
4 SEMILINEAR PDES IN HIGH DIMENSIONS
The section extends the PDE-SDE methodology to semilinear PDEs through backward stochastic differential equations and a neural-network deep BSDE solver. The method approximates solution values by simulating forward and backward stochastic processes and learning the unknown initial value and gradient processes.
- Problem: Semilinear PDEs allow lower-order terms to depend nonlinearly on the solution and its transformed gradient.This dependence produces the semilinear problem studied in the section.
- PDE-BSDE connection: The method uses the correspondence between nonlinear PDEs and backward stochastic differential equations.It extends the Feynman-Kac-based approach used for linear PDEs.
- PDE-BSDE connection: Under suitable regularity assumptions, solving the associated forward-backward stochastic differential equation yields the PDE solution through Y_0 = u(0, x).The forward SDE can be solved independently because it does not depend on Y_t or Z_t.
- Deep BSDE solver: The deep BSDE solver discretizes the forward and backward SDEs with Euler-Maruyama time stepping and trains neural networks to approximate the unknown gradient processes.The initial approximations of u(0, x) and its transformed gradient are learned parameters, while later gradient values are represented by neural networks.
- Examples: For the linear-quadratic Gaussian control problem, all tested models display similar performance.The authors report that the Simple model required less than 4 seconds while providing essentially the same approximation quality as more complex models.
- Examples: For the Allen-Cahn example, the Simple model produced a rough solution approximation within 7 seconds, while deeper and wider networks yielded only small relative-error decreases.The section also notes that the accompanying notebook includes a Burgers-type PDE example.
- Extensions: The deep BSDE solver can solve semilinear PDEs and can also be applied directly to BSDEs without explicitly treating a PDE.The described solver was presented and developed in earlier work.
5 EXTENSIONS AND RELATED WORK
The section surveys extensions and related scientific machine-learning approaches for PDEs beyond the three methods treated in detail. These include variational, residual-based, Hamilton-Jacobi, multilevel, multiscale, neural-operator, and software-based approaches.
- Scope: The review’s related-work discussion is necessarily incomplete because scientific machine learning for PDEs is developing rapidly.The section presents selected recent developments beyond the three main approaches.
- Residual methods: The deep Galerkin method approximates nonlinear parabolic PDE solutions with neural networks trained by minimizing a strong-solution residual.For high-dimensional problems, Monte Carlo methods are used rather than automatic differentiation to compute second derivatives.
- Additional approaches: Other surveyed directions include high-dimensional Hamilton-Jacobi methods, multilevel Picard approximations, branching diffusion processes, and multiscale neural networks.The Hamilton-Jacobi approach based on the Hopf formula has computational expense that behaves polynomially in spatial dimension.
- Neural operators: Neural operators infer mappings between function spaces using mesh-free, infinite-dimensional operators and observational training data.These methods differ from the reviewed finite-dimensional neural-network mappings and require no prior knowledge of the underlying PDE.
- Hybrid methods: Machine learning can also be used to accelerate existing numerical methods for solving partial and ordinary differential equations.This direction is represented by a general data-driven procedure cited in the review.
- Variational methods: The deep Ritz method reformulates variational problems as energy minimization problems.Boundary conditions can be enforced weakly through a penalty term, with Monte Carlo integration and mini-batch stochastic optimization used in training.
- Software: NeuralPDE.jl provides implementations for PINNs, forward-backward SDEs, optimal-stopping problems, and Kolmogorov backward equations.The package is written in Julia and is available as an open-source software package.
6 CONCLUSION
The review presents PINNs, Feynman-Kac methods, and BSDE methods as versatile neural-network approaches for PDE solution. It characterizes PINNs as better suited to low-dimensional complex nonlinear PDEs, while the stochastic approaches target high-dimensional linear and semilinear parabolic problems.
- Conclusion: The reviewed methods illustrate the versatility of machine-learning algorithms for solving PDEs and are described as promising approaches.This is the section’s overall conclusion.
- Conclusion: PINNs are described as best suited for low-dimensional but complex nonlinear PDEs.Their emphasis is on challenging physics features rather than high-dimensional problems.
- Conclusion: Feynman-Kac and BSDE methods promise to extend simulation capabilities for high-dimensional linear and semilinear parabolic problems in non-variational form.The conclusion links their relevance to settings where classical approaches are infeasible because of the curse of dimensionality.