Source-linked AI summary
An overview on deep learning-based approximation methods for partial differential equations
Christian Beck, Martin Hutzenthaler, Arnulf Jentzen, Benno Kuckuck
TL;DR
High-dimensional nonlinear PDEs are difficult to approximate without computational costs suffering from the curse of dimensionality. The article introduces deep learning-based and related Monte Carlo approaches by reviewing selected results, proof ideas, implementations, and literature. It reports theoretical expressive-power results and highlights numerical feasibility at dimensions of 1000 or 10 000, while noting important scope and performance limitations.
Problem
High-dimensional nonlinear PDEs are rarely solvable analytically, while standard approximation methods can scale exponentially with dimension or reciprocal accuracy.
Method
The article synthesizes selected deep learning-based and Monte Carlo PDE methods, revisits mathematical results and proof ideas, and supplies illustrative implementations.
Results
The reviewed results include neural networks with approximation error at most ε and parameter counts polynomial in PDE dimension and 1/ε, alongside feasible approximate solutions for semilinear PDEs in 1000 or 10 000 dimensions.
Takeaways & Limitations
Deep learning-based methods have made high-dimensional PDE approximation a rapidly developing research area with applications and examples that were recently considered out of reach.
Takeaways & Limitations
The article is not comprehensive, its implementations are intentionally naïve rather than state-of-the-art, and rigorous proof that deep learning overcomes the curse of dimensionality remains incomplete.
Abstract
from arXiv · showhide
It is one of the most challenging problems in applied mathematics to approximatively solve high-dimensional partial differential equations (PDEs). Recently, several deep learning-based approximation algorithms for attacking this problem have been proposed and tested numerically on a number of examples of high-dimensional PDEs. This has given rise to a lively field of research in which deep learning-based methods and related Monte Carlo methods are applied to the approximation of high-dimensional PDEs. In this article we offer an introduction to this field of research by revisiting selected mathematical results related to deep learning approximation methods for PDEs and reviewing the main ideas of their proofs. We also provide a short overview of the recent literature in this area of research.
1 Introduction
High-dimensional nonlinear PDEs are difficult to solve because standard numerical methods can suffer from the curse of dimensionality. The article introduces deep learning-based PDE approximation by reviewing selected theory, proof ideas, implementations, and related literature.
- The challenge: Standard approximation methods can require computational work that grows exponentially with PDE dimension or reciprocal accuracy.The article frames polynomial dependence on both quantities as the desired way to overcome the curse of dimensionality.
- Recent progress: Deep learning-based approximation methods have recently shown particular promise for high-dimensional PDEs after earlier applications to low-dimensional problems.The high-dimensional PDE literature expanded substantially from 2017 onward.
- Related setting: Parametric PDE problems can become high-dimensional when many initial-condition or additional parameters determine a family of solutions.This creates a high-dimensional approximation problem even when the underlying PDE has low spatial dimension.
- Article scope: The article reviews selected mathematical results and proof ideas, provides PyTorch implementations, illustrates applications, and surveys related deep learning-based and Monte Carlo methods.Its detailed treatment includes the deep Galerkin and deep splitting methods.
2 Deep learning-based approximation methods for linear PDEs
This section develops deep learning-based approximation methods for linear PDEs by reformulating PDE solutions as stochastic optimization problems and approximating those problems with neural networks. The approach is motivated by evidence that deep learning may be more efficient than standard Monte Carlo when approximating solutions across regions rather than at single points.
- Motivation: Linear Kolmogorov PDEs can be approximated without the curse of dimensionality using standard Monte Carlo methods, while deep learning may be more efficient over entire high-dimensional regions.The reported numerical comparison concerns approximation on high-dimensional cubes rather than only at a fixed space-time point.
- Method: The approximation strategy reformulates a linear Kolmogorov PDE as an infinite-dimensional stochastic optimization problem whose unique solution is the PDE solution.It then replaces this problem with finite-dimensional stochastic optimization problems involving deep neural networks and approximately computes their minimizers.
- Method: For linear heat PDEs, the reformulation follows from the Feynman–Kac formula and the fact that expectation is the best L2-approximation of a random variable.Theorem 2.1 provides the corresponding mathematical justification.
- Results: Over an entire timespan, the PDE solution can likewise be characterized through a stochastic optimization problem using an independent uniformly distributed time variable.Theorem 2.2 establishes uniqueness and identifies every minimizer with the PDE solution under the stated continuity and independence assumptions.
- Results: At terminal time, the unique global minimizer of the stochastic objective equals the heat-equation solution throughout the spatial cube.The result assumes a standard normal random variable and an independent uniformly distributed spatial random variable.
- Extensions: For general linear Kolmogorov PDEs, nonlinear drift and diffusion coefficients require approximating the associated stochastic process with schemes such as Euler–Maruyama or Milstein.Unlike the heat-equation case, the relevant process cannot be simulated exactly.
3 Deep learning-based approximation methods for nonlinear PDEs
The article next turns to deep learning-based approximation methods for possibly nonlinear PDEs. It focuses on two methods, including the deep Galerkin method and deep splitting method, before surveying the wider literature.
- Scope: The nonlinear-PDE section focuses on two particular deep learning-based approximation methods and then briefly reviews the wider literature.The section follows the preceding treatment of linear heat PDEs.
- Deep Galerkin method: One focal method is the deep Galerkin method proposed by Sirignano and Spiliopoulos.The passage also points to earlier related work by Dissanayake and Phan-Thien.
- Deep splitting method: The second focal method is the deep splitting method, which is based on the method presented for linear PDEs.The article treats both methods before providing a broader literature overview.
3.1 The deep Galerkin method
The deep Galerkin method casts certain nonlinear PDE solutions as stochastic optimization problems, then approximates the optimization over neural-network functions using stochastic gradient descent.
- Stochastic optimization formulation: The method represents the solution of a semilinear PDE as a global minimizer of a nonnegative stochastic objective.The objective measures initial-condition and PDE-residual errors under sampled random variables.
- Scope and implementation: The method is presented concretely for semilinear heat PDEs but can be adapted to a wider range of PDEs.The article reports simulations applying the method to semilinear parabolic PDEs in up to 200 dimensions.
- Neural-network approximation: A finite-dimensional optimization problem is obtained by restricting the function space to DNNs with fixed architecture and activation functions.This parameterization replaces optimization over C1,2 functions with optimization over neural-network realizations.
- Training procedure: Stochastic gradient descent approximately minimizes the neural-network objective, with automatic differentiation computing the required derivatives.The resulting minimizer is treated as an approximation to the PDE solution.
- Training procedure: The implementation samples time and space points to estimate the initial-condition and dynamic PDE-residual losses.The supplied implementation uses independent realizations of the relevant random variables and computes spatial derivatives by automatic differentiation.
3.2 The deep splitting approximation method for semilinear PDEs
The deep splitting method approximates semilinear PDEs by partitioning the time interval, freezing the nonlinearity on each subinterval, and consecutively applying a deep-learning method for linear PDEs. Its approximation error converges to zero as the number of subintervals increases.
- Method construction: The method splits the time interval [0,T] into N subintervals and constructs a linear PDE on each subinterval by freezing the semilinear nonlinearity.The resulting linear problems are solved successively using deep-learning-based approximation methods.
- Method construction: Deep splitting combines splitting approximations with deep-learning approximations for linear PDEs to approximate semilinear PDE solutions.This combination is formalized through an approximation theorem for a series of linear heat PDEs.
- Assumptions: The theorem assumes polynomial growth of the solutions and their derivatives, together with Lipschitz continuity and polynomial-growth conditions for the nonlinearity.These assumptions support the stated existence and approximation results.
- Approximation result: For every dimension d and spatial point x, the terminal-time approximation error converges to 0 as N tends to infinity.The theorem establishes this convergence for the constructed linear-PDE approximations and the target semilinear-PDE solution.
- Implementation and simulations: The article reports numerical simulations of deep splitting for semilinear parabolic PDEs in up to 1000 dimensions.A simple PyTorch implementation and the corresponding simulation source code are provided.
3.3 Other deep learning-based approximation methods for PDEs
Beyond the detailed methods, the article surveys deep learning-based PDE approximation approaches based on BSDEs, residual minimization, variational formulations, and related extensions. It emphasizes that this literature is broad and cannot be covered comprehensively.
- Scope of the survey: The article explicitly provides only an incomplete view of a rapidly developing research field and does not claim comprehensiveness.The authors direct readers to broader surveys for more extensive coverage.
- BSDE-based methods: BSDE-based methods exploit equivalences between PDEs and backward stochastic differential equations to approximate solution gradients or construct PDE schemes.The surveyed literature covers semilinear, quasilinear, elliptic, and fully nonlinear PDE settings.
- Residual-minimization methods: PINN/DGM approaches minimize PDE residuals, with extensions covering non-local PDEs, integral equations, adaptive collocation, and training improvements.Some non-local approaches remain susceptible to the curse of dimensionality, while high-dimensional variants have also been investigated.
- Other formulations: Other surveyed approaches include deep Ritz methods based on variational formulations, Petrov–Galerkin methods, Feynman–Kac representations, adaptive sampling, and Gaussian-process regression.The section presents these as distinct strands of the broader approximation literature.
4 Approximation methods for high-dimensional PDEs not based on machine learning
The article also discusses non-machine-learning approaches for high-dimensional nonlinear PDEs, especially full history recursive multilevel Picard methods. It notes that rigorous evidence for overcoming the curse of dimensionality remains limited for deep-learning methods.
- Alternative approaches: The article surveys non-machine-learning approaches including spatial-grid BSDE methods and methods for high-dimensional Hamilton–Jacobi–Bellman PDEs.These approaches are presented alongside other attempts to approximate high-dimensional nonlinear PDEs.
- Rigorous guarantees: As of the article’s writing, deep-learning methods had not been rigorously proved to overcome the curse of dimensionality for high-dimensional PDE approximation.The statement concerns rigorous proof, despite promising numerical simulations.
- Multilevel Picard methods: The article identifies full history recursive multilevel Picard methods as the only class established in the cited literature to overcome the curse of dimensionality for semilinear PDEs with general time horizons.These methods reformulate the PDE as a stochastic fixed-point equation, apply Picard iterations, and approximate expectations using multilevel Monte Carlo.
- Multilevel Picard methods: MLP approximations are full history recursive because each iterate uses realizations from all preceding iterates.The cited literature extends and studies these methods for several semilinear PDE classes.
5 Simulations
The simulations illustrate deep Galerkin and deep splitting approximations for semilinear PDEs across dimensions up to 1000, using three initial values. Deep splitting generally achieves higher accuracy because it exploits the PDE structure, whereas deep Galerkin is more broadly applicable but harder to train reliably.
- Simulation setup: The simulations approximate a sine-Gordon-type PDE with three initial values in dimensions up to d = 1000.The deep Galerkin simulations cover dimensions through 200, while deep splitting simulations extend through 1000.
- Scope: The implementations are intentionally naïve and designed for understandable illustrations rather than state-of-the-art performance.The article directs readers to related literature for extensive numerical studies of these methods.
- Comparison: Deep Galerkin generally has lower accuracy than deep splitting because deep splitting is tailored to the semilinear parabolic PDEs, while deep Galerkin uses none of their structure.The broader applicability of deep Galerkin also makes it relatively easy to formulate, although reliable convergence can be challenging.
- Deep Galerkin method: Deep Galerkin simulations evaluate the solution at t = 1/2 and x = 0 for dimensions up to 200, using three different initial values.For each dimension and initial value, the method was run independently 20 times.
- Evaluation: The result tables report means, standard deviations, reference values, and estimated absolute and relative L1-errors across independent runs.The deep Galerkin tables compare approximations with reference values computed using the MLP method.
- Deep splitting method: Deep splitting simulations evaluate the same target at t = 1/2 and x = 0 for dimensions up to 1000 by splitting the time interval into 30 subintervals and training 30 neural networks.The results use the same three initial values and repeated-run evaluation format as the deep Galerkin simulations.
6 Theoretical results for DNN approximations for PDEs
Theoretical work shows that DNNs can have sufficient expressive power for curse-of-dimensionality-free approximation of several high-dimensional PDE classes, but optimization guarantees remain incomplete. The section reviews a representative theorem for nonlinear heat PDEs with Lipschitz nonlinearities and sketches its proof strategy.
- Literature overview: No rigorous result yet fully explains the practical success of deep learning-based PDE approximation or proves curse-of-dimensionality-free computational complexity.Existing results establish expressive power, not complete end-to-end algorithmic guarantees.
- Literature overview: Recent analyses show that DNN approximations can use a number of parameters growing at most polynomially in PDE dimension d and reciprocal accuracy 1/ε.This expressive-power result covers several PDE classes, including nonlinear heat PDEs with Lipschitz continuous nonlinearities.
- Open limitations: No result is known showing that SGD optimization error with one random initialization avoids the curse of dimensionality or necessarily converges to zero, apart from a special output-layer-only setting.The exception concerns shallow networks learning Black–Scholes-type PDEs under suitable assumptions.
- Proof strategy: The theorem assumes neural networks approximate the initial conditions and uses perturbed PDEs, DNN approximations of the nonlinearity, and MLP methods to establish the result.The proof combines perturbation of the equation with a curse-of-dimensionality-free MLP approximation result for nonlinear heat PDEs.
- Approximation error bounds: Theorem 6.1 gives neural-network approximations whose L2 error at terminal time is at most ε while parameter counts grow polynomially in d and 1/ε.The error is measured on the region [0, 1]^d between the exact terminal solution and the network realization.
- Scope and limitations: The theorem’s constants may depend on the time horizon T, with the authors expecting K to grow exponentially in T without additional assumptions such as dissipativity.They expect the exponent c can be chosen independently of T.
7 Conclusion
The article surveys deep learning and selected Monte Carlo methods for high-dimensional PDE approximation. It concludes that these tools have made approximation in dimensions such as 1000 or 10,000 feasible while leaving substantial theoretical work ahead.
- Conclusion: The article surveys deep learning-based approximation methods for high-dimensional PDEs together with selected related Monte Carlo methods.It focuses on methods drawn from the recent literature.
- Conclusion: Approximate solutions to semilinear PDEs in 1000 or 10 000 space dimensions have become feasible, according to the authors’ assessment.The conclusion describes such problems as previously out of reach.
- Future directions: The authors expect these developments to motivate new approximation methods and rigorous analyses of their approximation capabilities.This is presented as a direction for future research.
8 Source code for simulations
The appendix supplies the Python source code used for the numerical simulations, including utilities for repeated jobs and reporting, plus core implementations of the deep Galerkin and deep splitting methods.
- Source code: The appendix provides downloadable Python source files used to obtain the simulation results.The files are included with the arXiv source archive.
- Deep Galerkin method: The deep Galerkin implementation builds fully connected feed-forward networks, samples collocation points, computes PDE losses and gradients, and updates parameters during training.The implementation includes Xavier initialization, Adam optimization, and exponential learning-rate decay.
- Deep splitting method: The deep splitting implementation trains a sequence of neural networks across time-splitting stages and evaluates the final model at the origin.Its code generates Gaussian inputs for subintervals and computes stagewise losses.