PaperlayerView exact revision on arXiv

Source-linked AI summary

Multilevel domain decomposition-based architectures for physics-informed neural networks

Victorita Dolean, Alexander Heinlein, Siddhartha Mishra, Ben Moseley

arXiv:2306.05486v3math.NAcs.NA

TL;DR

PINNs often struggle with high-frequency and multi-scale solutions, motivating approaches that better handle such problems. This paper adds multiple overlapping domain-decomposition levels to FBPINNs and evaluates their scaling. Multilevel FBPINNs consistently outperform PINNs across the studied problems and improve accuracy over one-level FBPINNs at large subdomain counts.

  • Problem

    PINNs often struggle with high-frequency and multi-scale solutions, where accuracy can fall as solution complexity and computational cost increase.

    Source: §1 · pp. 3–4

  • Method

    The paper extends FBPINNs with multiple levels of overlapping domain decompositions and assesses accuracy through strong and weak scaling tests.

    Source: §1 · p. 4, §2.2 · p. 12

  • Results

    Across all studied problems, multilevel FBPINNs consistently outperform tested one-level FBPINNs and PINNs, and are more accurate and computationally efficient than tested PINNs.

    Source: §4 · p. 32

  • Takeaways & Limitations

    At large subdomain counts, multilevel FBPINNs improve accuracy over one-level FBPINNs, suggesting coarse levels support scalability by improving global communication.

    Source: §1 · p. 5, §4 · p. 32

    The study uses uniform rectangular decompositions and an exponential level structure; irregular decompositions for complex geometries or varying solution complexity may be beneficial but are practically challenging.

    Source: §4 · p. 33

Abstract

from arXiv · show

Physics-informed neural networks (PINNs) are a powerful approach for solving problems involving differential equations, yet they often struggle to solve problems with high frequency and/or multi-scale solutions. Finite basis physics-informed neural networks (FBPINNs) improve the performance of PINNs in this regime by combining them with an overlapping domain decomposition approach. In this work, FBPINNs are extended by adding multiple levels of domain decompositions to their solution ansatz, inspired by classical multilevel Schwarz domain decomposition methods (DDMs). Analogous to typical tests for classical DDMs, we assess how the accuracy of PINNs, FBPINNs and multilevel FBPINNs scale with respect to computational effort and solution complexity by carrying out strong and weak scaling tests. Our numerical results show that the proposed multilevel FBPINNs consistently and significantly outperform PINNs across a range of problems with high frequency and multi-scale solutions. Furthermore, as expected in classical DDMs, we show that multilevel FBPINNs improve the accuracy of FBPINNs when using large numbers of subdomains by aiding global communication between subdomains.

1. Introduction

PINNs often lose accuracy and become more costly on high-frequency and multi-scale solutions. The paper introduces multilevel FBPINNs and reports improved performance over PINNs, with better accuracy than one-level FBPINNs at large subdomain counts.

1. Introduction

  • PINNs often struggle with high-frequency and multi-scale solutions; the paper adds multiple levels of overlapping domain decompositions and defines strong and weak scaling tests.Neural networks tend to learn high frequencies more slowly, while more complex solutions typically require more collocation points and parameters.Source: §1 · pp. 3–4
  • Across tested problems, multilevel FBPINNs outperform PINNs in accuracy and computational cost, and improve FBPINN accuracy at large subdomain counts by aiding global communication.The evaluation covers high-frequency and multi-scale problems.Source: §1 · pp. 4–5

1.1. Related work

  • Prior PINN domain-decomposition methods include overlapping Schwarz approaches and methods using non-overlapping decompositions; learned partition-of-unity functions were also studied for function approximation rather than PDE-based PINNs.The cited methods include D3M, DeepDDM, cPINN, and XPINNs.Source: §1.1 · p. 5

1.2. Neural networks

  • Neural networks are treated as mathematical functions with learnable parameters, and this work uses feedforward fully connected networks.Their layer maps apply activation functions to affine transformations.Source: §1.2 · p. 6

1.3. Physics-informed neural networks

  • PINNs approximate differential-equation solutions with neural networks and train by minimizing residuals; their functional approximation is mesh-free and can also support inverse problems.The loss balances PDE residuals with boundary-condition residuals, using sampled collocation and boundary points.Source: §1 · p. 3, §1.3 · p. 7

1.3.1. Hard constrained PINNs

  • Hard-constrained PINNs embed boundary conditions in the solution ansatz, but choosing a suitable operator for complex geometries or boundary conditions may be difficult or impossible.This strategy is problem-dependent; soft enforcement may be used instead.Source: §1.3.1 · p. 8

2. Methods

Multilevel FBPINNs extend overlapping-domain neural networks with progressively finer decompositions, drawing on classical domain decomposition methods. The section defines the architecture, explains its computational-cost argument, and illustrates its use on Laplace problems.

2.1. Finite basis physics-informed neural networks

  • FBPINNs replace one global network with smaller networks on overlapping subdomains, whose locally normalized inputs can make high-frequency problems lower-frequency locally.The local problems are coupled, and their solutions are summed to approximate the global solution.Source: §2.1 · p. 9, Figure 1 · p. 9

2.1.1. Mathematical definition

  • Each FBPINN subdomain network is windowed to its subdomain, and the window functions form a partition of unity for the global solution decomposition.The solution is inserted into the PINN loss and trained using the PINN optimization scheme.Source: §2.1.1 · pp. 10–11

2.1.2. Computational efficiency of FBPINNs versus PINNs

  • FBPINN evaluation costs O(NC S̃), where C is the average subdomains per collocation point, rather than the naive O(NJ S̃).The text notes that subdomain networks can often remain fixed in size as complexity grows, whereas a global PINN network typically grows.Source: §2.1.2 · p. 11

2.1.3. FBPINNs versus other methods for combining PINNs with DD

  • XPINNs use non-overlapping subdomains and interface-coupling losses, whereas FBPINNs retain continuity without extra loss terms; adaptive expert approaches cost more to train.The compared methods include Gated-PINNs and APINNs, which evaluate all experts at each coordinate.Source: §2.1.3 · pp. 11–12

2.2. Multilevel FBPINNs

  • Multilevel FBPINNs add multiple overlapping domain decompositions to the solution ansatz, following the classical motivation for coarse levels when many subdomains are used.The paper hypothesizes that this multilevel structure will improve FBPINN performance.Source: §2.2 · p. 12

2.2.1. Example of a multilevel FBPINN

  • Three-level examples use 1, 2, and 4 subdomains in 1D and 1×1, 2×2, and 4×4 in 2D, with reported close agreement to exact Laplace solutions.The 1D and 2D examples use uniform decompositions and an overlap ratio of 1.9.Source: §2.2.1 · pp. 14–16

2.2.2. Multilevel FBPINNs versus classical multilevel DDMs

  • Unlike classical Schwarz methods, FBPINNs construct the global space from neural-network local spaces and minimize a non-convex global loss; coarse levels target global information transfer.Classical one-level Schwarz convergence can deteriorate as subdomain counts increase because information travels only through overlaps.Source: §2.2.2 · p. 17

3. Numerical results

The numerical section assesses model accuracy and computational cost across Laplace and Helmholtz problems. It defines strong and weak scaling in terms of changing model capacity and problem complexity.

3. Numerical results

  • The experiments assess multilevel FBPINN accuracy and computational cost against PINNs, Fourier-feature PINNs, SA-PINNs, and one-level FBPINNs.The section frames scaling tests as analogous to those used for classical domain decomposition.Source: §3 · p. 18

3.1.1. Homogeneous Laplacian problem in two dimensions

  • The homogeneous 2D Laplacian problem is used for ablations of level count, subdomain count, overlap ratio, and subdomain-network size.The problem has homogeneous Dirichlet boundary conditions on the unit square.Source: §3.1.1 · p. 18

3.1.2. Multi-scale Laplacian problem in two dimensions

  • The multi-scale 2D Laplacian varies the number and frequencies of solution components to test scaling as more components are added.The source term controls the components and their frequencies.Source: §3.1.2 · p. 19

3.1.3. Helmholtz problem in two dimensions

  • The 2D Helmholtz problem varies wave number to test performance on standing-wave patterns, comparing models with finite-difference solutions because the exact solution is unknown.The setup uses homogeneous Dirichlet conditions and a centered Gaussian point source.Source: §3.1.3 · p. 19

3.2. Definition of strong and weak scaling

  • Strong scaling fixes problem complexity while increasing model capacity; ideal scaling improves convergence rate or accuracy at the capacity-growth rate.Capacity increases through more levels, subdomains, or larger subdomain networks.Source: §3.2 · p. 20
  • Weak scaling increases problem complexity and model capacity at the same rate; ideal scaling keeps convergence rate or accuracy approximately constant.Tests were limited to model-capacity scaling because the models were trained on a single GPU.Source: §3.2 · p. 20

3.3. Common implementation details

The experiments use a common exponentially increasing level structure and shared implementation choices for decompositions, networks, optimization, and evaluation.

3.3. Common implementation details

  • Multilevel FBPINNs use J(l) = 2^d(l−1) subdomains per level, an exponential structure intended to span multiple solution-frequency scales with relatively few levels.This structure is a stated hypothesis, not an experimentally established optimal choice.Source: §3.3 · p. 20, Figure 4 · p. 21
  • FBPINNs use uniform rectangular decompositions, shared window functions, identical fully connected subdomain networks, and inputs normalized over each subdomain.PINN inputs are instead normalized over the global domain.Source: §3.3 · p. 21
  • Models use hard-constrained losses, shared collocation and testing points for comparisons, Adam optimization, and normalized L1 test loss averaged over ten random seeds.Fourier-feature PINNs use a lower learning rate because convergence was unstable at a larger rate.Source: §3.3 · pp. 21–22
  • All models are trained on a single NVIDIA RTX 3090 GPU.Consequently, the reported scaling tests concern model capacity rather than hardware parallelization.Source: §3.3 · p. 22

3.4. Results

Across the tested high-frequency and multi-scale problems, multilevel FBPINNs model solutions that standard PINNs often handle poorly, while their accuracy depends on problem complexity and model configuration. The results also show that multiple levels help maintain accuracy as the number of subdomains grows.

3.4.1. Homogeneous Laplacian problem in two dimensions

  • For the homogeneous Laplacian, increasing overlap and subdomain-network parameters improves multilevel FBPINN accuracy, while one-level FBPINN accuracy falls as subdomains are added; the PINN is less accurate and less stable.The authors suggest that additional levels enable transfer of global information between subdomains.Source: §3.4.1 · pp. 23–24, Figure 5 · p. 23

3.4.2. Multi-scale Laplacian problem in two dimensions

  • In multi-scale Laplacian weak scaling, the models capture all tested frequency components, but normalized L1 accuracy declines somewhat as problem complexity increases, yielding near—but not perfect—weak scaling.The test increases frequency components alongside model capacity and collocation points.Source: Figure 7 · p. 26, §3.4.2 · pp. 27–28

3.4.3. Helmholtz problem in two dimensions

  • In Helmholtz ablations, accuracy improves with greater overlap, more subdomain-network parameters, and more levels; one-level FBPINNs converge more slowly and finish less accurately than the multilevel model.Below a test loss of 10^-1, random initializations produce substantial variation, with final loss between 10^-2 and 10^-1.Source: Figure 8 · p. 28, §3.4.3 · p. 31
  • In Helmholtz weak scaling, multilevel FBPINNs accurately model every tested solution except the highest-wave-number case, while PINN and SA-PINN fail and Fourier-feature PINNs take an order of magnitude longer.At the highest wave number, the multilevel FBPINN captures the dominant frequency and overall concentricity but misses more complex motifs.Source: Figure 9 · p. 29, Figure 10 · p. 30, §3.4.3 · pp. 31–32

4. Discussion

Across the problems studied, multilevel FBPINNs outperform the tested one-level FBPINNs and PINNs, and coarse levels are associated with better accuracy when many subdomains are used. The discussion identifies limits in problem scope, theoretical understanding, and computational efficiency, alongside directions for further work.

4. Discussion

  • Across all problems studied, multilevel FBPINNs outperform the tested one-level FBPINNs and PINNs, and are more accurate than one-level FBPINNs with many subdomains.The authors suggest coarse levels improve global communication between subdomains.Source: §4 · p. 32
  • The study is limited to rectangular geometries, homogeneous solution complexity, uniform rectangular decompositions, and an exponential level structure.Irregular decompositions may better fit complex geometries and spatially varying complexity, but may require asynchronous training and difficult design choices.Source: §4 · pp. 32–33
  • Why optimization of high wave number Helmholtz problems is challenging remains unclear, and convergence of classical DDMs for these problems is also not fully understood.The authors identify theoretical convergence properties of multilevel FBPINNs as a direction for further study.Source: §4 · p. 33
  • Although multilevel FBPINNs are over an order of magnitude more efficient than the tested PINNs, their training may remain slower than traditional finite-difference or finite-element solvers.The authors attribute this to the non-convex optimization of (FB)PINNs compared with the linear solves traditional methods typically use.Source: §4 · p. 33
  • The authors propose training across varied inputs, such as boundary conditions and PDE coefficients, so multilevel FBPINNs can serve as surrogates without retraining for each new solution.They also identify multi-GPU training as a direction for investigation.Source: §4 · p. 33
  • Solution evaluation has cost O(NC S̃), where N is the number of collocation points, C the average number of subdomains containing a point, and S̃ the cost of one subdomain network evaluation.This cost is independent of the total number of subdomains; multi-GPU communication may complicate perfect linear scaling, and overlapping levels increase communication while improving numerical scalability.Source: §4 · p. 34

Data availability

The work’s code is available online, and its data are synthetically generated using that code.

Data availability

  • The code for reproducing the work is available online, and all data are generated synthetically using that code.The passage provides a repository address.Source: Data availability · p. 34

CRediT contributions

The contribution statement assigns roles across conceptualization, investigation, methodology, project administration, and writing, with additional roles in data curation, resources, software, validation, and visualization.

CRediT contributions

  • VD, AH, SM, and BM are credited with conceptualization, investigation, methodology, project administration, and both stages of writing.The statement also credits BM with data curation, software, validation, and visualization, and BM and SM with resources.Source: CRediT contributions · p. 34

Appendix A. Software implementation

The implementation reduces FBPINN solution-evaluation cost by evaluating only subdomain networks that contain each collocation point, and parallelizes subdomain-network outputs on the GPU.

Appendix A. Software implementation

  • A precomputed mapping identifies which subdomains contain each collocation point, so training evaluates only networks that contribute to that point.This avoids naively summing over every subdomain network and level at every point.Source: Appendix A · p. 35
  • Because each subdomain network outputs zero outside its subdomain, only networks containing a collocation point contribute to the solution sum.The implementation exploits this property to reduce summation cost.Source: Appendix A · p. 35
  • The implementation uses JAX vmap to compute subdomain-network outputs in parallel on the GPU.This helps use GPU parallelism for the small networks tested, which would not fully utilize it if evaluated sequentially.Source: Appendix A · p. 35

Appendix B. Finite difference solver for Helmholtz equation

The Helmholtz reference solution is computed with a finite-difference solver on a uniform mesh. The discretized system is solved using UMFPACK.

Appendix B. Finite difference solver for Helmholtz equation

  • A 5-point stencil discretizes the Laplacian on a 320 × 320 uniformly spaced mesh over the problem domain.This discretization turns the Helmholtz equation into a linear system.Source: Appendix B · p. 35
  • The finite-difference system is solved with the scipy.sparse.linalg sparse direct solver UMFPACK.Source: Appendix B · p. 35
  • The solver computes a reference solution for the Helmholtz problem studied in section 3.4.3.Source: Appendix B · p. 35

Appendix C. Definition of Fourier features

Fourier features transform neural-network inputs using trigonometric functions parameterized by a frequency matrix. The frequency scale is hand-selected based on test accuracy, and PINN convergence is highly sensitive to that scale and the learning rate.

Appendix C. Definition of Fourier features

  • Fourier features transform network inputs using trigonometric functions before they enter the network.Source: Appendix C · p. 35
  • The frequency matrix Γ has shape k × d, where k is the number of features and x is the input vector.Its values represent feature frequencies and are typically sampled from a univariate Gaussian distribution.Source: Appendix C · p. 36
  • With µ = 0, the authors hand-select σ based on test accuracy; PINN convergence is highly sensitive to σ and the learning rate.Source: Appendix C · p. 36

Appendix D. Definition of self-adaptive PINNs (SA-PINNs)

SA-PINNs learn a separate weight for each collocation point alongside the network parameters. Gradient ascent increases weights for points with persistently high PDE residuals; the experiments use fixed initialization and optimizer settings.

Appendix D. Definition of self-adaptive PINNs (SA-PINNs)

  • SA-PINNs assign and learn a separate weight for each collocation point in the PINN loss.Training combines gradient descent on network parameters with gradient ascent on the weights.Source: Appendix D · p. 36
  • Weights increase for collocation points with consistently high PDE residuals, giving those points greater loss weight.Source: Appendix D · p. 36
  • The experiments set m(λ_i) = λ_i, initialize all λ_i to 1, and use Adam with a 1 × 10−3 learning rate for both updates.Source: Appendix D · p. 36

Appendix E. FBPINN individual level contributions

In the five-level FBPINN, the finest level roughly learns the highest frequencies, while coarser levels learn lower-frequency contributions.

Appendix E. FBPINN individual level contributions

  • The finest level roughly learns the highest frequencies, while coarser levels learn lower-frequency contributions.This behavior is expected given neural-network spectral bias and the individual subdomain normalization used in the FBPINN.Source: Appendix E · p. 37
Loading 2306.05486v3…