Source-linked AI summary
DeepXDE: A deep learning library for solving differential equations
Lu Lu, Xuhui Meng, Zhiping Mao, George E. Karniadakis
TL;DR
PDE solving with deep learning is an emerging Scientific Machine Learning problem. The paper presents PINNs and the DeepXDE library, adds residual-based adaptive refinement, and demonstrates the approach across forward and inverse problems. DeepXDE provides compact, customizable software for education and computational science, while PINNs remain slower than finite elements for forward problems.
Problem
Deep learning has been highly successful in other applications, but its use for solving PDEs has only recently emerged.
Method
The paper presents PINNs that embed PDE residuals into neural-network losses through automatic differentiation, together with the DeepXDE Python library and residual-based adaptive refinement.
Results
Numerical examples verify PINN effectiveness and DeepXDE capability for forward and inverse problems, including improved discontinuity capture with RAR for Burgers equation.
Takeaways & Limitations
DeepXDE provides compact, customizable PINN software intended for classroom use and computational-science research.
Takeaways & Limitations
PINNs are currently slower than finite elements for forward problems, and effective neural-network architectures are still selected empirically.
Abstract
from arXiv · showhide
Deep learning has achieved remarkable success in diverse applications; however, its use in solving partial differential equations (PDEs) has emerged only recently. Here, we present an overview of physics-informed neural networks (PINNs), which embed a PDE into the loss of the neural network using automatic differentiation. The PINN algorithm is simple, and it can be applied to different types of PDEs, including integro-differential equations, fractional PDEs, and stochastic PDEs. Moreover, from the implementation point of view, PINNs solve inverse problems as easily as forward problems. We propose a new residual-based adaptive refinement (RAR) method to improve the training efficiency of PINNs. For pedagogical reasons, we compare the PINN algorithm to a standard finite element method. We also present a Python library for PINNs, DeepXDE, which is designed to serve both as an education tool to be used in the classroom as well as a research tool for solving problems in computational science and engineering. Specifically, DeepXDE can solve forward problems given initial and boundary conditions, as well as inverse problems given some extra measurements. DeepXDE supports complex-geometry domains based on the technique of constructive solid geometry, and enables the user code to be compact, resembling closely the mathematical formulation. We introduce the usage of DeepXDE and its customizability, and we also demonstrate the capability of PINNs and the user-friendliness of DeepXDE for five different examples. More broadly, DeepXDE contributes to the more rapid development of the emerging Scientific Machine Learning field.
1. Introduction.
Scientific Machine Learning applies deep learning to PDEs, replacing traditional discretization with neural-network solution approximations. The paper presents PINNs and DeepXDE as methods and software for broad scientific-computing use.
- Deep learning for PDEs has recently emerged as a Scientific Machine Learning sub-field.
- PINNs use automatic differentiation to constrain neural networks by minimizing PDE residuals, supporting inverse, integro-differential, fractional, and stochastic problems.
- DeepXDE implements PINN algorithms as both classroom education software and a research tool for computational science and engineering.
- DeepXDE supports multi-physics problems, constructive-solid-geometry domains, time-dependent PDEs, callbacks, and compact code resembling mathematical formulations.
2. Algorithm and theory of physics-informed neural networks.
The paper develops PINNs around neural-network function approximators and presents DeepXDE as their implementation framework. It covers network architectures, implementation, and software availability.
- PINNs use feedforward neural networks and residual networks as compositional approximators for PDE solutions.
- A feedforward network is recursively formed from linear transformations, biases, and element-wise nonlinear activation functions.
- DeepXDE provides the paper’s Python implementation of PINN algorithms for education and computational-science research.
2.2. Automatic differentiation.
Automatic differentiation computes neural-network derivatives through repeated chain-rule evaluations. Compared with finite differences, it reduces the number of forward passes, especially for high-dimensional inputs.
- PINNs can compute required network derivatives using analytical, numerical, symbolic, or automatic differentiation methods.
- Automatic differentiation applies the chain rule through a forward pass for values and a backward pass for derivatives.
- AD computes all partial derivatives with one forward and one backward pass, whereas finite differences require d_in + 1 forward passes.
- Automatic differentiation can be applied recursively to compute higher-order derivatives.
2.3. Physics-informed neural networks (PINNs) for solving PDEs.
PINNs approximate PDE solutions with neural networks, enforce equation and boundary conditions through residual losses, and train by minimizing those losses. The section also discusses adaptive points, frequency learning, and constraint choices.
- 2.3. Physics-informed neural networks (PINNs) for solving PDEs.: A PINN constructs a neural-network surrogate and obtains its input derivatives through automatic differentiation.
- 2.3. Physics-informed neural networks (PINNs) for solving PDEs.: Training uses residual points in the domain and on boundaries or initial-condition surfaces.
- 2.3. Physics-informed neural networks (PINNs) for solving PDEs.: The loss is a weighted sum of L2 residual norms for the PDE and boundary conditions, with derivatives handled by automatic differentiation.
- 2.3. Physics-informed neural networks (PINNs) for solving PDEs.: PINNs train network parameters by minimizing a highly nonlinear, non-convex loss using gradient-based optimizers.
- 2.3. Physics-informed neural networks (PINNs) for solving PDEs.: Boundary and initial conditions can be enforced softly through the loss or, for simple cases, hard through the surrogate architecture.
- 2.3. Physics-informed neural networks (PINNs) for solving PDEs.: Residual points may be fixed, resampled, or adaptively refined during training, with mini-batches available for large point sets.
- 2.3. Physics-informed neural networks (PINNs) for solving PDEs.: PINNs learn all frequencies almost simultaneously when solving the stated Poisson example, unlike ordinary neural-network function approximation.
2.4. Approximation theory and error analysis for PINNs.
PINNs can approximate PDE solutions and their derivatives with neural networks, but total error reflects approximation, optimization, and generalization components.
- PINNs require a neural network to satisfy both the PDE and boundary conditions while approximating the solution and its partial derivatives.
- The chosen network architecture defines a function family F, whose best approximation uF may differ from the PDE solution u.
- PINN total error combines approximation, optimization, and generalization errors associated with representation, training, and residual-point sampling.
- Current PINN error estimation remains unresolved, and larger networks trade smaller approximation errors against potentially higher generalization errors.
2.5. Comparison between PINNs and FEM.
PINNs and FEM both approximate PDE solutions, but PINNs use neural-network surrogates and loss minimization instead of mesh-based algebraic systems.
- FEM approximates solutions with piecewise polynomials, whereas PINNs use neural networks parameterized by weights and biases.
- FEM typically requires mesh generation, while PINNs are mesh-free and can use grids or random points.
- FEM converts PDEs into algebraic systems using stiffness and mass matrices, whereas PINNs embed PDEs and boundary conditions into a loss function.
2.6. PINNs for solving integro-differential equations.
For integro-differential equations, PINNs retain automatic differentiation for integer-order derivatives and numerically approximate integral operators.
- PINNs analytically derive integer-order derivatives using automatic differentiation when solving integro-differential equations.
- Integral operators are approximated numerically with classical methods such as Gaussian quadrature.
- The integral approximation introduces a fourth error component, the discretization error Edis.
- The modified procedure first approximates the integral with Gaussian quadrature and then solves the resulting PDE using a PINN.
- PINNs can also be extended to fractional and stochastic differential equations, although those cases are not discussed here because of page limits.
2.7. PINNs for solving inverse problems.
PINNs handle inverse problems by incorporating measurements into the loss and jointly optimizing network parameters with unknown physical parameters.
- Inverse problems contain unknown parameters λ and additional measurements at points Ti alongside the differential equation and boundary conditions.
- Compared with forward problems, inverse PINNs add an extra measurement-based loss term.
- The network parameters θ and unknown parameters λ are optimized jointly by minimizing the combined loss.
2.8. Residual-based adaptive refinement (RAR).
Residual-based adaptive refinement (RAR) improves residual-point placement during PINN training by adding points where PDE residuals are largest, targeting steep solution gradients.
- RAR addresses inefficient random residual-point distributions for PDEs whose solutions contain steep gradients or sharp fronts.The method is motivated by the difficulty of designing an effective point distribution when the solution is unknown.
- RAR continues adding points until the mean residual is smaller than the threshold E0.
- The refinement procedure begins with initial residual points, trains for limited iterations, and then evaluates the residual during training.The cycle continues while the mean residual exceeds threshold E0.
- RAR estimates the mean PDE residual using Monte Carlo integration over randomly sampled locations.The estimate uses the average residual values at sampled locations S.
- RAR adds m new residual points at locations with the largest residuals in the sampled set S.After adding points, the procedure returns to training and repeats the refinement cycle.
3. DeepXDE usage and customization.
DeepXDE provides a compact, configurable workflow for defining and solving differential equations, while supporting custom geometries, networks, callbacks, and additional learning tasks.
- Usage: DeepXDE solves differential equations by specifying geometry, PDEs, boundary or initial conditions, training data, network architecture, and hyperparameters.Its code is designed to resemble the mathematical formulation and is shorter than code for traditional numerical methods.
- Usage: The standard workflow defines the domain and PDE, combines them with conditions and training data, constructs a network, trains a model, and predicts solutions.Time-independent and time-dependent problems use data.PDE and data.TimePDE, respectively.
- Geometry: Constructive solid geometry builds complex two- and three-dimensional domains from primitive geometries using union, difference, and intersection operations.The geometry module includes primitives such as intervals, triangles, rectangles, polygons, disks, cuboids, and spheres.
- Usage: DeepXDE supports Dirichlet, Neumann, Robin, periodic, operator, and initial conditions, along with feed-forward and residual neural networks.Users can also select loss types, metrics, optimizers, learning-rate schedules, initializations, and regularizations.
- Additional capabilities: DeepXDE also approximates functions from multi-fidelity data and learns nonlinear operators in addition to solving differential equations.
- Customization: DeepXDE is loosely coupled and highly configurable, allowing users to add new geometries, neural networks, callbacks, and training behaviors.Customization examples include implementing geometry methods, network interfaces, and callback functions at training stages.
4. Demonstration examples.
The examples demonstrate PINNs and DeepXDE across forward, adaptive-refinement, inverse, and integro-differential problems, including challenging geometries and steep solution gradients. The reported results show accurate solutions, improved capture near discontinuities with RAR, and recovery of system parameters from observations.
- Setup: DeepXDE uses tanh activation and documented hyperparameters across five demonstration examples.The examples use the settings summarized in Table 3.
- Forward PDEs: 1200 equation points and 120 boundary points produce a PINN solution compared with a spectral element solution on an L-shaped Poisson domain.The absolute error is also reported for the comparison.
- RAR for Burgers equations: RAR places 40 additional residual points near the sharp interface and captures the 1D Burgers discontinuity better than PINN training without RAR.The comparison uses ν = 0.01/π and 2500 initial residual points, followed by adaptive refinement.
- RAR for Burgers equations: At Re = 5000, RAR produces an effective 2D Burgers solution using 210 residual points, compared with a solution without RAR using the same total.Only 10 extra residual points are added adaptively after 200 initial samples.
- Inverse problems: The inverse examples recover Lorenz parameters near their true values and estimate diffusion-reaction coefficients from 40000 concentration observations.The Lorenz estimates are (10.002, 14.999, 2.668), while the identified D and kf are 1.98 × 10−3 and 0.0971.
- Integro-differential equations: For a Volterra integro-differential equation, Gaussian-Legendre quadrature of degree 20 yields an L2 relative error of 2 × 10−3.The numerical solution is shown against the exact solution using 12 equispaced residual points.
5. Concluding Remarks.
PINNs use automatic differentiation to solve diverse PDEs, while DeepXDE provides a compact, customizable implementation for education and computational science. The paper also proposes residual-based adaptive refinement to improve training efficiency, while noting important limitations in speed, architecture selection, and formulation choices.
- PINNs use automatic differentiation to handle differential operators, providing a mesh-free approach for solving different types of PDEs.
- PINNs extend to integro-differential equations and inverse problems, while residual-based adaptive refinement improves residual-point distribution and training efficiency.
- DeepXDE implements PINNs with compact user code that follows the mathematical formulation and supports customization for new problem requirements.
- PINNs are currently slower than finite elements for forward problems, although offline training can alleviate this limitation.
- Effective neural-network architecture selection remains empirical, and strong-form enforcement is not the only viable formulation because weak or variational forms may also be effective.