Source-linked AI summary
Efficient training of physics-informed neural networks via importance sampling
Mohammad Amin Nabian, Rini Jasmine Gladstone, Hadi Meidani
TL;DR
PINN training can be computationally inefficient because uniform collocation batches may provide little gradient information. This paper proposes loss-proportional importance sampling, including a piecewise-constant approximation, and reports improved convergence and computational performance across elasticity, diffusion, and plane stress examples. The approach is straightforward to add to existing PINN codes without new hyperparameters, though broader challenging PDE settings remain to be investigated.
Problem
Uniform collocation sampling can waste computation on points with already-small residuals, limiting PINN training efficiency.
Method
The paper trains PINNs by sampling collocation points from a distribution proportional to the loss, using a piecewise-constant loss approximation for faster importance sampling.
Results
The importance-sampling approach improves convergence behavior and computational performance versus uniform sampling across elasticity, diffusion, and plane stress examples.
Takeaways & Limitations
The approach is straightforward to implement in existing PINN codes and introduces no new hyperparameters.
Takeaways & Limitations
Further investigation is needed for stochastic PDEs and time-dependent PDEs with highly oscillatory or non-monotonic solutions.
Abstract
from arXiv · showhide
Physics-Informed Neural Networks (PINNs) are a class of deep neural networks that are trained, using automatic differentiation, to compute the response of systems governed by partial differential equations (PDEs). The training of PINNs is simulation-free, and does not require any training dataset to be obtained from numerical PDE solvers. Instead, it only requires the physical problem description, including the governing laws of physics, domain geometry, initial/boundary conditions, and the material properties. This training usually involves solving a non-convex optimization problem using variants of the stochastic gradient descent method, with the gradient of the loss function approximated on a batch of collocation points, selected randomly in each iteration according to a uniform distribution. Despite the success of PINNs in accurately solving a wide variety of PDEs, the method still requires improvements in terms of computational efficiency. To this end, in this paper, we study the performance of an importance sampling approach for efficient training of PINNs. Using numerical examples together with theoretical evidences, we show that in each training iteration, sampling the collocation points according to a distribution proportional to the loss function will improve the convergence behavior of the PINNs training. Additionally, we show that providing a piecewise constant approximation to the loss function for faster importance sampling can further improve the training efficiency. This importance sampling approach is straightforward and easy to implement in the existing PINN codes, and also does not introduce any new hyperparameter to calibrate. The numerical examples include elasticity, diffusion and plane stress problems, through which we numerically verify the accuracy and efficiency of the importance sampling approach compared to the predominant uniform sampling approach.
1. Introduction
PINNs solve PDE-governed problems by optimizing neural-network solutions against physics and condition residuals. The paper targets inefficient uniform collocation sampling with importance sampling to improve training convergence and efficiency.
- PINNs represent PDE solutions with differentiable neural networks whose parameters minimize squared residuals at collocation points using automatic differentiation and stochastic gradient descent.
- Uniformly sampled batches can evaluate points whose differential-operator residuals are already small, yielding little gradient information and delaying convergence.
- Importance sampling has improved convergence in other neural-network tasks by drawing samples according to a proposal distribution proportional to the loss-gradient norm.
- The paper proposes importance-sampling-based PINN training and a piecewise-constant loss approximation, implemented with few code changes and no new hyperparameters.
- The paper evaluates the approach through numerical examples covering elasticity, diffusion, and plane stress problems.
2. Deep Learning of Differential Equations
PINNs formulate differential-equation solution as constrained optimization over neural-network parameters. Training uses automatic differentiation and mini-batch SGD on collocation points, with boundary and initial conditions incorporated into the loss.
- 2.1. Feed-Forward Fully-Connected Deep Neural Networks: A feed-forward fully-connected network maps a d-dimensional input to a k-dimensional output through weighted layers, biases, and element-wise nonlinear activation functions.
- 2.1. Feed-Forward Fully-Connected Deep Neural Networks: Model parameters are optimized iteratively with SGD variants, updating weights and biases using a step size and gradients computed by backpropagation.
- 2.2. Physics-Informed Neural Networks: PINNs approximate differential-equation solutions with neural networks whose parameters enter a general differential operator over a bounded spatial domain and its boundary.
- 2.2. Physics-Informed Neural Networks: The solution is reduced to constrained optimization, commonly reformulated with soft constraints as additive penalty terms in a modified loss function.
- 2.2. Physics-Informed Neural Networks: Mini-batch SGD estimates each loss gradient from m selected collocation points and updates the model parameters through iterative descent.
- 2.2. Physics-Informed Neural Networks: The baseline training algorithm uniformly samples a batch from generated collocation points before each descent step, while low-discrepancy sequences provide an alternative point generator.
3. Importance Sampling for Training of PINNs
The paper designs importance sampling for PINN training to accelerate convergence, replacing uniform sampling with distributions based on loss-related quantities. A piecewise constant loss approximation further reduces the cost of constructing the sampling distribution.
- The objective is to design a sampling distribution q that accelerates convergence of the PINN stochastic-gradient update.
- Because computing loss gradients at every collocation point is expensive, the method uses the loss value itself as the importance metric.
- The method approximates loss values by evaluating them at S<N seed points and assigning each collocation point the value of its nearest seed.
- Nearest-seed assignment forms Voronoi cells with constant loss values, enabling faster importance sampling while preserving the proposed sampling workflow.
- Algorithm 2 samples collocation points from the resulting multinomial distribution after generating collocation points, assigning nearest seeds, and evaluating seed losses.
4. Numerical Examples
The numerical examples evaluate piecewise-constant importance sampling for PINN solutions to elasticity and plane-stress problems. Across the reported comparisons, the approach preserves solution accuracy while improving training efficiency relative to uniform and exact-loss importance sampling.
- Numerical examples: The study applies piecewise-constant importance sampling to elasticity, plane-stress, and diffusion PINN examples.The supplied passages specifically describe elasticity and plane-stress examples, while the numerical-examples overview also identifies diffusion.
- 4.1. A two-dimensional isotropic elasticity problem: The elasticity example defines displacement fields on an irregular plate and compares the PINN solution with an exact solution.Dx and Dy denote displacement in the x and y directions, respectively.
- 4.1. A two-dimensional isotropic elasticity problem: The proposed elasticity PINN solution closely agrees with the exact solution, verifying the approach’s numerical accuracy.The comparison uses importance sampling with a piecewise-constant loss approximation.
- 4.1. A two-dimensional isotropic elasticity problem: Importance sampling with piecewise-constant loss accelerates convergence compared with uniform sampling while maintaining a good loss approximation.The comparison considers loss across iterations and elapsed time; the piecewise-constant method also uses fewer forward model evaluations than exact-loss importance sampling.
- 4.1. A two-dimensional isotropic elasticity problem: The piecewise-constant approximation error is evaluated against seed count, and seed-size choices are examined for their effects on training performance.The reported analyses use relative L2 approximation error and compare loss against iterations and elapsed time.
- 4.2. A plane stress problem: In the plane-stress example, importance sampling computes plate displacements and provides better computational performance than uniform sampling in iterations and elapsed time.The example concerns a two-dimensional plate with multiple holes and plane-stress equations.
5. Conclusion
The paper presents importance sampling as a step toward improving PINN computational efficiency, while acknowledging that broader validation remains necessary for more challenging problems.
- PINNs offer advantages over classical PDE methods but are not expected to dominate the best established methods in efficiency and robustness.
- The proposed approach selects PINN training points using a proposal distribution proportional to a piece-wise constant approximation of the loss function.
- The approach is straightforward to implement in existing PINN codes and introduces no new hyperparameters.
- Theoretical evidence and three numerical examples demonstrated improved PINN training efficiency for elasticity and transient diffusion problems.
- Further investigation is needed for stochastic PDEs and time-dependent PDEs with highly oscillatory or non-monotonic solutions.