Source-linked AI summary
A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks
Chenxi Wu, Min Zhu, Qinyang Tan, Yadhu Kartha, Lu Lu
TL;DR
PINNs depend on residual-point distributions, yet prior work mainly used simple fixed sampling schemes. This paper compares ten uniform, resampling, and adaptive methods across six PDE problems, finding that RAD and RAR-D substantially improve accuracy, especially with few residual points, while RAR-D offers a more computationally efficient option.
Problem
Residual-point sampling is important for PINN performance, but previous studies mainly used equispaced grids and uniformly random sampling.
Method
The study compares ten sampling methods across four forward and two inverse PDE problems, including six uniform methods, Random-R, RAR, RAD, and RAR-D.
Results
RAD performs best across all forward and inverse problems, while RAD and RAR-D improve PINN accuracy by orders of magnitude, especially with few residual points.
Takeaways & Limitations
RAR-D achieves comparable accuracy to RAD while gradually increasing residual points, making it preferable when computational resources are limited.
Takeaways & Limitations
The brute-force sampling approach may require alternative methods for high-dimensional problems, and the chosen residual-based probability density may not be optimal.
Abstract
from arXiv · showhide
Physics-informed neural networks (PINNs) have shown to be an effective tool for solving forward and inverse problems of partial differential equations (PDEs). PINNs embed the PDEs into the loss of the neural network, and this PDE loss is evaluated at a set of scattered residual points. The distribution of these points are highly important to the performance of PINNs. However, in the existing studies on PINNs, only a few simple residual point sampling methods have mainly been used. Here, we present a comprehensive study of two categories of sampling: non-adaptive uniform sampling and adaptive nonuniform sampling. We consider six uniform sampling, including (1) equispaced uniform grid, (2) uniformly random sampling, (3) Latin hypercube sampling, (4) Halton sequence, (5) Hammersley sequence, and (6) Sobol sequence. We also consider a resampling strategy for uniform sampling. To improve the sampling efficiency and the accuracy of PINNs, we propose two new residual-based adaptive sampling methods: residual-based adaptive distribution (RAD) and residual-based adaptive refinement with distribution (RAR-D), which dynamically improve the distribution of residual points based on the PDE residuals during training. Hence, we have considered a total of 10 different sampling methods, including six non-adaptive uniform sampling, uniform sampling with resampling, two proposed adaptive sampling, and an existing adaptive sampling. We extensively tested the performance of these sampling methods for four forward problems and two inverse problems in many setups. Our numerical results presented in this study are summarized from more than 6000 simulations of PINNs. We show that the proposed adaptive sampling methods of RAD and RAR-D significantly improve the accuracy of PINNs with fewer residual points. The results obtained in this study can also be used as a practical guideline in choosing sampling methods.
1 Introduction
The study addresses the overlooked importance of residual-point distribution in PINNs through a comprehensive comparison of uniform, resampling, and adaptive sampling strategies. It evaluates these methods across forward and inverse PDE problems to identify accuracy and efficiency differences.
- Motivation: PINNs evaluate PDE loss at scattered residual points, whose location and distribution can strongly affect performance.Earlier studies mainly used equispaced grids and uniformly random sampling, leaving residual-point sampling comparatively overlooked.
- Uniform sampling: The study compares six uniform sampling methods, including grids, random sampling, Latin hypercube sampling, and three low-discrepancy sequences.The low-discrepancy sequences are Sobol, Halton, and Hammersley.
- Resampling: The authors systematically test uniform resampling because selecting new residual points during training had not yet been comprehensively evaluated.They investigate both the number of residual points and the resampling period.
- Adaptive sampling: RAD samples from a residual-dependent probability distribution, while RAR-D combines residual-based refinement with distribution-based point addition.Both methods are designed as automatic adaptive nonuniform sampling approaches.
- Evaluation: The evaluation covers four forward and two inverse PDE problems, with more than 6000 PINN simulations performed.The study compares the sampling methods across many experimental setups.
2 Methods
The methods section reviews PINNs, introduces non-adaptive sampling strategies, proposes two adaptive residual-based methods, and compares them with related approaches.
- Methods: The section covers PINNs, uniform sampling strategies, two adaptive nonuniform residual-based methods, and comparisons with related methods.The adaptive methods target improved accuracy and training efficiency.
2.1 PINNs in solving forward and inverse PDEs
PINNs use neural networks to approximate PDE solutions by minimizing losses that encode physics, boundary conditions, and, for inverse problems, observed data. Forward problems estimate solutions with known parameters, whereas inverse problems simultaneously infer unknown parameters from observations.
- Forward and inverse problems: The initial condition is treated as a Dirichlet boundary condition in the PINN formulation.The solution notation u(x) is used for the field represented by the network.
- Forward and inverse problems: In a forward problem, a neural network approximates the solution u while known PDE parameters λ remain fixed.The network parameters θ are trained by minimizing the PINN loss.
- Forward and inverse problems: PINN losses combine PDE and boundary-condition terms, with residual points sampled inside the domain and on its boundaries.The weights wf and wb balance the corresponding loss terms.
- Forward and inverse problems: Inverse problems add an observed-data misfit term to the loss and train the network parameters θ simultaneously with unknown parameters λ.The additional data term is weighted by wi.
2.2 Uniformly-distributed non-adaptive sampling
The paper describes fixed residual-point sampling using grids, random, stratified, and low-discrepancy methods, then introduces periodic random resampling during training. These choices determine how residual points populate the computational domain.
- Fixed residual points: Residual-point sampling is central to PINN accuracy and computational efficiency because PDE loss is evaluated at those points.The reviewed fixed-point approaches include grids, random sampling, LHS, and Sobol sequences.
- Fixed residual points: Six uniform methods are considered: equispaced grids, uniformly random sampling, Latin hypercube sampling, Halton, Hammersley, and Sobol sequences.The methods generate different point distributions in the computational domain.
- Fixed residual points: Latin hypercube sampling stratifies random samples across equal-probability intervals.It is presented as a stratified Monte Carlo sampling method.
- Fixed residual points: The low-discrepancy methods generate highly uniform point sequences, with Hammersley and Sobol having distinct construction rules.Hammersley uses equidistant points in its first dimension, while Sobol is a base-2 digital sequence.
- Uniform points with resampling: Random-R periodically resamples uniformly random residual points instead of keeping one fixed set throughout training.Its resampling period N is an important accuracy-related hyperparameter.
2.3 Nonuniform adaptive sampling
Adaptive nonuniform sampling improves residual-point distributions by focusing training points according to PDE residuals. RAD resamples all points from a residual-based PDF, while RAR-D combines this distribution with iterative point addition.
- RAR: RAR adds residual points at locations with large PDE residuals during training, making it a greedy adaptive method.
- RAD: RAD samples all residual points according to a PDF based on the PDE residual, rather than concentrating only on the largest residual.
- Motivation: RAD and RAR-D were proposed as improved adaptive methods because an earlier residual-proportional approach performs poorly for some PDE cases.
- RAD: RAD approximates its sampling distribution by evaluating a dense candidate set, normalizing probabilities, and sampling a subset accordingly.
- RAD: RAD's hyperparameters k and c control the balance between sampling large-residual regions and maintaining a more uniform distribution.k = 0 gives uniform sampling; increasing k emphasizes large residuals, while increasing c promotes uniformity.
- RAR-D: RAR-D combines RAR's repeated point addition with RAD's residual-based PDF, and approaches RAR-G as k becomes very large.The study reports k = 2 and c = 0 as a usually good default combination, while noting that optimal values are problem-dependent.
2.4 Comparison with related work
The paper frames several related adaptive sampling methods as special cases or close variants of RAD and RAR-D. Its added contribution is the tunable pair k and c, whose values strongly influence residual-point distributions.
- Special cases: RAR-G is a special case of RAR-D obtained with a large value of k.
- Special cases: The method of Nabian et al. is a special case of RAD with k = 1 and c = 0.
- Special cases: Gao et al.'s method is a special case of RAD with c = 0.
- Special cases: Tang et al. proposed one method equivalent to RAD and another equivalent to RAR-D, both with k = 2 and c = 0.
- Related variants: Other related methods modify RAR-G by using subdomains, node generation, point removal, or uniformly random replacement.The cited variants differ in how they add or remove points and in their dimensionality or computational-cost properties.
- Novelty: The paper states that k and c significantly influence residual-point distributions and that suitable values can make RAD and RAR-D outperform other methods.
- Connection to weighting: Point-wise weighting can be interpreted as a special case of adaptive sampling through an equivalent residual-weighted loss formulation.
3 Results
Across six forward and inverse PDE problems, residual-point sampling strongly affected PINN accuracy. RAD was consistently strongest overall, while the best uniform strategy depended on solution complexity and sampling regime.
- RAD performed best among the 10 sampling methods across all forward and inverse problems.
- For complicated solutions such as Burgers’ and multi-scale wave equations, RAD and RAR-D achieved errors magnitudes lower than competing methods.
- For smooth solutions such as diffusion and diffusion-reaction equations, Hammersley and Random-R also produced sufficiently low errors.
- Among fixed-point uniform methods, Halton, Hammersley, and Sobol generally outperformed Random and LHS, which both outperformed Grid.
- At 50 diffusion residual points, Grid and Random had errors about one order of magnitude larger than Hammersley and Sobol.
- With 2000 Burgers’ residual points, RAD and RAR-D reached 0.02% L2 relative error versus 1.69 ± 1.67% for Random-R.
4 Conclusions
The paper compares uniform and adaptive residual-point sampling for PINNs across forward and inverse PDE problems. RAD and RAR-D improve accuracy, while the study provides practical choices based on computational constraints and sampling restrictions.
- The study evaluates ten residual-point sampling methods across four forward and two inverse PDE problems under varied setups.The methods include six uniform schemes, uniform resampling, RAD, RAR-D, and an existing adaptive method.
- RAD and RAR-D significantly improve PINN accuracy, especially when few residual points are available or solutions are complicated.The reported difficult cases include Burgers’ equation with steep gradients and the wave equation with multi-scale behavior.
- RAD with k = 1 and c = 1 is recommended as a default, with k and c tunable to balance locations having large and small PDE residuals.The recommendation is presented as a practical guideline for solving a new PDE.
- RAR-D can provide comparable accuracy to RAD at lower computational cost by gradually increasing the number of residual points.RAR-D with k = 2 and c = 0 is preferred when computational resources are limited.
- Random-R is suitable when adaptive sampling is unavailable, but its resampling period should avoid both extremes.This applies when sampling points according to a probability density function is difficult.
- Low-discrepancy sequences such as Hammersley are preferred over Grid, Random, or LHS when residual points must remain fixed.The guideline specifically mentions PINNs using the augmented Lagrangian method.
- The brute-force point-sampling approach used by RAD and RAR-D is sufficient for many PDEs but requires other methods for high-dimensional problems.Generative adversarial networks are given as an example of an alternative for high-dimensional settings.
- The adaptive sampling probability p(x) ∝ ε^k(x) / (E[ε^k(x)] + c) works well in this study, but another probability density may perform better.The paper identifies meta-learning as a possible way to learn a new density.