Source-linked AI summary
DGM: A deep learning algorithm for solving partial differential equations
Justin Sirignano, Konstantinos Spiliopoulos
TL;DR
High-dimensional PDEs are difficult to solve numerically because meshes become infeasible, motivating a meshfree alternative. The paper trains deep neural networks on randomly sampled points to satisfy PDE, initial, and boundary conditions, and reports accurate results up to 200 dimensions alongside approximation theory for a PDE class.
Problem
High-dimensional PDEs pose a longstanding numerical challenge, while mesh-based methods become infeasible as dimensionality grows.
Method
DGM uses a deep neural network trained on randomly sampled time and space points to satisfy the differential operator, initial condition, and boundary conditions.
Results
The method accurately solves high-dimensional free-boundary PDEs up to 200 dimensions and also produces accurate results for a high-dimensional Hamilton-Jacobi-Bellman PDE.
Takeaways & Limitations
The paper provides a meshfree approach for high-dimensional PDE computation and proves neural-network approximation power for a class of quasilinear parabolic PDEs.
Takeaways & Limitations
The analysis uses homogeneous boundary conditions in its concrete setup and stochastic-gradient training may converge only to a local minimum for non-convex networks.
Abstract
from arXiv · showhide
High-dimensional PDEs have been a longstanding computational challenge. We propose to solve high-dimensional PDEs by approximating the solution with a deep neural network which is trained to satisfy the differential operator, initial condition, and boundary conditions. Our algorithm is meshfree, which is key since meshes become infeasible in higher dimensions. Instead of forming a mesh, the neural network is trained on batches of randomly sampled time and space points. The algorithm is tested on a class of high-dimensional free boundary PDEs, which we are able to accurately solve in up to $200$ dimensions. The algorithm is also tested on a high-dimensional Hamilton-Jacobi-Bellman PDE and Burgers' equation. The deep learning algorithm approximates the general solution to the Burgers' equation for a continuum of different boundary conditions and physical conditions (which can be viewed as a high-dimensional space). We call the algorithm a "Deep Galerkin Method (DGM)" since it is similar in spirit to Galerkin methods, with the solution approximated by a neural network instead of a linear combination of basis functions. In addition, we prove a theorem regarding the approximation power of neural networks for a class of quasilinear parabolic PDEs.
1 Deep learning and high-dimensional PDEs
High-dimensional PDEs are difficult for mesh-based numerical methods, so the paper proposes DGM, a meshfree deep-learning approach that trains neural networks on randomly sampled points. It combines Galerkin-style residual minimization with neural-network approximation and establishes approximation results for a class of quasilinear parabolic PDEs.
- Motivation: Finite-difference meshes scale as O(d+1) with d spatial and one time dimension, becoming computationally intractable in moderately high dimensions.The difficulty comes from the explosion in grid points and the demand for smaller time steps.
- Method: DGM replaces Galerkin basis expansions with a deep neural network trained to satisfy the differential operator, initial condition, and boundary conditions.The network is optimized using stochastic gradient descent at randomly sampled spatial points.
- Method: Randomly sampled spatial points eliminate mesh formation and convert the PDE problem into a machine-learning problem.This avoids the infeasible high-dimensional mesh while retaining sequential processing of sampled points.
- Applications: The algorithm is also tested on a high-dimensional Hamilton-Jacobi-Bellman PDE and on Burgers’ equation across different boundary, initial, and physical conditions.The Burgers’ equation experiment approximates a general solution over a high-dimensional problem-setup space.
- Applications: The paper targets high-dimensional free-boundary PDEs relevant to American-option pricing, where portfolio dimensions can range from dozens to hundreds of stocks.The number of PDE space dimensions equals the number of stocks in the portfolio.
- Approximation theory: For a class of quasilinear parabolic PDEs, neural networks can achieve vanishing objective error and converge to the PDE solution in Lρ for ρ < 2 as hidden units increase.The result combines neural-network approximation properties with PDE continuity and compactness arguments.
2 Algorithm
DGM represents the PDE solution with a neural network and minimizes an objective measuring residuals from the differential operator, boundary conditions, and initial condition. Stochastic gradient descent estimates this objective from randomly sampled time and space points, avoiding mesh construction but potentially converging only to a local minimum.
- Network objective: DGM approximates u(t, x) with a neural network f(t, x; θ), whose differential operators can be calculated analytically.The parameters θ determine the reduced-form approximation.
- Network objective: The objective J(f) measures squared violations of the PDE operator, boundary condition, and initial condition; J(f) = 0 implies a PDE solution.Minimizing J produces a function that closely satisfies all three requirements.
- Stochastic optimization: Large spatial dimension makes direct minimization of J infeasible because its integral over the domain is computationally intractable.The algorithm replaces direct integration with stochastic estimates from random samples.
- Stochastic optimization: Each iteration samples interior, boundary, and initial-condition points, evaluates the squared error, takes a descent step, and repeats until convergence.The sampled set includes time-space points, boundary points, and a spatial point for the initial condition.
- Stochastic optimization: The sampled gradient steps are unbiased estimates of the objective gradient, so stochastic gradient descent proceeds on average in a descent direction.Under relatively mild technical conditions, the parameter sequence converges to a critical point of J.
- Limitation: Because neural-network optimization is non-convex, stochastic gradient descent may converge to a local rather than global minimum.This limitation is shared with non-convex optimization generally and is not specific to DGM.
3 A Monte Carlo Method for Fast Computation of Second Derivatives
The paper introduces a Monte Carlo scheme to reduce the cost of computing the second derivatives required by DGM in high-dimensional PDEs. The modified gradient estimate lowers computational expense but introduces bias and variance relative to the original method.
- Computational challenge: Second-derivative calculations cost O(d^2 × N), compared with O(d × N) for first derivatives, where d is dimension and N is batch size.In d = 200 dimensions, 20,000 second derivatives may be required.
- Monte Carlo approximation: The DGM gradient requires second-derivative terms from the differential operator, motivating their Monte Carlo approximation.The method targets the second derivatives entering the stochastic-gradient update.
- Monte Carlo approximation: The approximation uses Gaussian perturbations with covariance determined by the diffusion coefficients and has O(Δ) bias.A Taylor expansion establishes the approximation error rate.
- Variance reduction: Antithetic variates use paired positive and negative perturbations to improve the second-derivative approximation without computational cost depending on Δ.The construction permits arbitrarily small Δ while retaining the stated computational property.
- Modified algorithm: The modified algorithm samples interior, boundary, and initial-condition points, combines the approximate interior gradient with the other gradient terms, and iterates.The resulting estimate approximates the full objective gradient at each sampled batch.
- Trade-off: The modified method is computationally less expensive but adds bias and variance, whereas the original method is unbiased with lower variance but more expensive.The added variance is described as i.i.d. noise that averages out over many samples.
4 Numerical Analysis for a High-dimensional Free Boundary PDE
The paper applies DGM to high-dimensional American option PDEs, using an iterative treatment of the free boundary and random batches to train the neural network. It evaluates approximate solutions against semi-analytic or error-bound references across time, space, and dimensions.
- Problem: American option PDEs are challenging because they combine high dimensionality, a free boundary, and a nondifferentiable payoff function.The option price is u(0, X0), and the solution satisfies u(t, x) ≥ g(x).
- Algorithm: The algorithm simulates points above and below the free boundary, selecting them using the current neural-network estimate.The approximate free boundary is updated iteratively through the current parameter estimate θn.
- Algorithm: The free-boundary training procedure generates random interior, boundary, and payoff points, then takes descent steps until convergence.The sampled sets are combined to approximate the objective function and update the network parameters.
- Implementation: The neural network uses gated recurrent-style layers with element-wise updates and is designed to represent sharp turns caused by the terminal payoff.The architecture uses Z, G, R, and H gates across L layers, with x⃗ = (t, x) as input.
- Results: The method provides approximate solutions over all time and space, while Figure 2 compares absolute and percent errors for a 20-dimensional American option PDE.Large absolute errors can occur where the solution is large, while percent error remains small there.
5 High-dimensional Hamilton-Jacobi-Bellman PDE
The paper tests DGM on a 21-dimensional HJB PDE derived from optimal control of a stochastic heat equation. The reported average percent error over the entire space is 0.1%.
- Problem: The HJB equation represents optimal control of a stochastic heat equation whose control guides temperature toward a target profile while minimizing energy.The stochastic heat equation includes Brownian-sheet fluctuations and fixed endpoint temperatures.
- Problem: Spatial discretization converts the stochastic heat equation into a finite-dimensional SDE system and a corresponding high-dimensional PDE.The PDE value function has d spatial dimensions, with boundary values represented by fixed endpoint components.
- Method: DGM solves the resulting PDE in d = 21 dimensions using the deep learning algorithm.The experiment uses L = 10^-1, α = 10^-4, σ = 10^-1, λ = 1, γ = 1, and target profile v̄(x) = 0.
- Evaluation: The contour plot estimates percent error by sampling spatial points, calculating pointwise error against the semi-analytic solution, and aggregating over a two-dimensional subspace.The horizontal axis is the 11-th dimension and the vertical axis is the average of all dimensions.
- Results: 0.1% is the average percent error over the entire space.The PDE has a semi-analytic solution obtained through an iterative Riccati-equation method.
6 Burgers’ equation
The paper extends DGM to approximate a parameterized family of Burgers’ equations without repeatedly solving each setup separately. A single neural network is trained on random time, space, and setup points and performs closely against exact solutions across several conditions.
- Problem: Solving PDEs across many physical, boundary, and initial conditions is expensive because discretizing the high-dimensional setup space requires exponentially many grid points.The setup is represented by p ∈ P, and traditional methods re-solve the PDE at many points in P.
- Method: DGM approximates the general solution u(t, x; p) across boundary conditions, initial conditions, and physical conditions with one neural network.Training uses stochastic gradient descent on randomly sampled time, space, and problem-setup points.
- Method: The Burgers’ setup space is P = (ν, α, a, b) ∈ R4, and the network covers the specified ranges of time, space, viscosity, coefficients, and boundary values.The network uses 6 layers with 200 units per layer for this experiment.
- Results: The deep learning solution is very close to the exact finite-difference solution across several Burgers’ problem setups at t = 1.The tested setups vary ν, α, a, and b, while Figure 6 also examines changing steepness and shock-layer formation.
7 Neural Network Approximation Theorem for PDEs
The paper proves neural networks can make the PDE residual, boundary-condition error, and initial-condition error arbitrarily small for a class of quasilinear parabolic PDEs, with convergence to the PDE solution under stronger assumptions.
- 7 Neural Network Approximation Theorem for PDEs: The theorem applies to quasilinear parabolic PDEs with divergence-form principal terms and specified growth, smoothness, and Lipschitz assumptions on nonlinearities.The assumptions include a compact domain, a unique classical solution, and locally Lipschitz nonlinear terms with at most polynomial growth.
- 7.2 Convergence of the neural network to the PDE solution: The proof combines neural-network approximation of functions and derivatives with PDE continuity and compactness arguments.Vanishing L2 source terms are used to establish convergence after showing the networks satisfy the operator and conditions increasingly well.
- 7.1 Convergence of the L2 error J(f): Neural networks with sufficiently many hidden units can make the objective J(f), measuring PDE, boundary, and initial-condition errors, arbitrarily small.The result concerns multilayer feed-forward networks and uses assumptions ensuring a classical PDE solution exists.
- 7.2 Convergence of the neural network to the PDE solution: The convergence theorem is established for homogeneous boundary data, while extensions to smooth inhomogeneous data are expected but not proved here.The restriction to zero boundary data is introduced to avoid technical difficulties.
8 Conclusion
The conclusion presents DGM as a meshfree neural-network approach for high-dimensional PDEs, reports accurate results in several settings, and identifies broader PDE classes and difficult solution behaviors as requiring further study.
- 8 Conclusion: DGM trains a deep neural network to satisfy the differential operator, initial condition, and boundary conditions using randomly sampled time-space points instead of a mesh.This meshfree design addresses the infeasibility of meshes in higher dimensions.
- 8 Conclusion: Accurate numerical results are reported for high-dimensional free-boundary PDEs in up to 200 dimensions and for a high-dimensional Hamilton-Jacobi-Bellman PDE.The paper also applies the method to Burgers’ equation across varying conditions in the broader context.
- 8 Conclusion: DGM can be modified for hyperbolic, elliptic, and partial-integral PDEs, but its numerical performance for those classes remains uninvestigated.The conclusion presents these evaluations as future work rather than established results.
- 8 Conclusion: Highly non-monotonic or oscillatory PDE solutions may be more challenging, and further architectural development and numerical testing are required.The authors conclude that the reported results justify further exploration of neural-network approaches for PDEs.
- 8 Conclusion: The paper leaves broader convergence theorems, stability analysis, and more direct variational formulations of PDE-learning algorithms for future work.These open questions extend beyond the quasilinear parabolic class treated theoretically.
A Proofs of convergence results
The convergence proof combines neural-network approximation, uniform bounds, compactness, and nonlinear-term convergence to show that approximate solutions converge to the unique PDE solution. Under equicontinuity, the convergence strengthens from Lρ convergence to uniform convergence.
- Approximation foundation: Neural-network density supplies approximations to sufficiently smooth PDE solutions, while the PDE objective vanishes for an exact solution.The proof invokes uniform 2^-density on compact subsets of C2(R^(1+d)) and uses the PDE residual objective.
- Compactness and limit identification: Uniform energy bounds yield subsequences converging strongly in L2(ΩT) to a limit function u.Compact embeddings and a compactness result establish relative compactness before identifying the limit.
- Convergence of the learned solution: The neural-network sequence f^n converges to u in Lρ(ΩT) for every ρ < 2, because its difference from the zero-boundary auxiliary solution vanishes strongly.The argument combines convergence of f^n − f̂^n with strong convergence of f̂^n to u.
- Uniform convergence: Uniform convergence follows when the neural-network sequence is uniformly bounded and equicontinuous.Equicontinuity first gives uniform convergence of the boundary values, and Arzelà–Ascoli then upgrades the Lρ result.