PaperlayerView exact revision on arXiv

Source-linked AI summary

PINNacle: A Comprehensive Benchmark of Physics-Informed Neural Networks for Solving PDEs

Zhongkai Hao, Jiachen Yao, Chang Su, Hang Su, Ziao Wang, Fanzhi Lu, Zeyu Xia, Yichi Zhang, Songming Liu, Lu Lu, Jun Zhu

arXiv:2306.08827v2cs.LGcs.NAmath.NAphysics.comp-ph

TL;DR

PINN methods lack comprehensive comparisons across diverse PDEs, despite the variety and complexity of the problems. PINNacle benchmarks more than 10 methods on challenging PDE tasks; its experiments report where methods help and identify nonlinear and high-dimensional problems as ongoing challenges.

  • Problem

    Comprehensive comparisons of PINN methods across diverse PDEs using challenging, real-world datasets are lacking.

    Source: §1 · pp. 1–2

  • Method

    PINNacle combines more than 20 challenging PDE problems with over 10 PINN variants and standardized evaluation.

    Source: §1 · p. 2

  • Results

    The analysis reports benefits from domain decomposition on complex-geometry problems and identifies high-dimensional and nonlinear problems as pressing challenges.

    Source: §5 · pp. 9–10

  • Takeaways & Limitations

    The benchmark’s analysis points to domain decomposition and loss reweighting as approaches for multi-scale or complex-geometry problems, while nonlinear problems remain a research challenge.

    Source: §1 · p. 2, §5 · p. 10

    The selected PDE problems span mathematical properties and are intended not to favor a specific PDE type.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

While significant progress has been made on Physics-Informed Neural Networks (PINNs), a comprehensive comparison of these methods across a wide range of Partial Differential Equations (PDEs) is still lacking. This study introduces PINNacle, a benchmarking tool designed to fill this gap. PINNacle provides a diverse dataset, comprising over 20 distinct PDEs from various domains, including heat conduction, fluid dynamics, biology, and electromagnetics. These PDEs encapsulate key challenges inherent to real-world problems, such as complex geometry, multi-scale phenomena, nonlinearity, and high dimensionality. PINNacle also offers a user-friendly toolbox, incorporating about 10 state-of-the-art PINN methods for systematic evaluation and comparison. We have conducted extensive experiments with these methods, offering insights into their strengths and weaknesses. In addition to providing a standardized means of assessing performance, PINNacle also offers an in-depth analysis to guide future research, particularly in areas such as domain decomposition methods and loss reweighting for handling multi-scale problems and complex geometry. To the best of our knowledge, it is the largest benchmark with a diverse and comprehensive evaluation that will undoubtedly foster further research in PINNs.

1 INTRODUCTION

PINNacle addresses the lack of comprehensive comparisons of PINN methods across diverse, challenging PDEs. It combines a broad task dataset, a toolbox of methods, and standardized evaluation, then uses experiments to identify strengths and remaining challenges.

1 INTRODUCTION

  • The benchmark responds to the absence of rigorous comparisons using challenging datasets drawn from real-world problems.Existing comparisons had focused on sampling or reweighting methods, while PDE variety and complexity make comprehensive analysis difficult.Source: §1 · p. 2
  • PINNacle combines more than 20 challenging PDE problems, over 10 PINN variants, and standardized evaluation in a single benchmark.The tasks cover complex geometry, multi-scale phenomena, nonlinearity, and high dimensionality.Source: §1 · p. 2
  • The benchmark’s task set spans domains including heat conduction, fluid dynamics, biology, and electromagnetics.Its problems are selected to represent several challenges encountered by PINNs.Source: §1 · p. 2
  • Loss reweighting and domain decomposition could improve performance on multi-scale and complex-geometry problems, while variational formulation performs better on inverse problems.The analysis also identifies nonlinear problems as an area few methods adequately address.Source: §1 · p. 2

2 RELATED WORK

Related work spans scientific machine-learning benchmarks, PDE-solving software, and PINN variants, but the paper identifies gaps in comprehensive PINN comparisons and unified variant toolboxes.

2.1 BENCHMARKS AND DATASETS IN SCIENTIFIC MACHINE LEARNING

  • Scientific machine-learning benchmarks cover neural operators, latent Newtonian mechanics, and domain-specific fields, but comprehensive comparisons of PINN approaches remain absent.Prior PINN studies compared components such as collocation sampling and loss reweighting, while PDEBench and PDEArena include multiple scientific machine-learning methods.Source: §2.1 · p. 3

2.2 SOFTWARES AND TOOLBOXES

  • Existing PDE neural-network software includes DeepXDE, but the paper identifies a gap in unified implementations of advanced PINN variants.PINNacle is presented as a flexible interface for implementing and evaluating variants across problems.Source: §2.2 · p. 3

2.3 VARIANTS OF PHYSICS-INFORMED NEURAL NETWORKS

  • PINN variants address challenges through loss reweighting or resampling, optimizers, novel losses, and architectures such as domain decomposition and adaptive activations.The paper selects representative methods from these categories for evaluation on its benchmark dataset.Source: §2.3 · p. 3

3 PINNACLE: A HIERARCHICAL BENCHMARK FOR PINNS

PINNacle organizes PDE tasks, PINN methods, a toolbox, and evaluation into a benchmark for comparing methods across varied problem challenges. Its design links PINN training and known limitations to diverse tasks, representative methods, and multiple error measures.

3 PINNACLE: A HIERARCHICAL BENCHMARK FOR PINNS

  • The benchmark’s design introduces PINN preliminaries, datasets, methods, toolbox structure, and evaluation metrics.These elements are presented as the components needed to compare PINN variants.Source: §3 · p. 3

3.1 PRELIMINARIES OF PHYSICS-INFORMED NEURAL NETWORKS

  • PINNs approximate PDE solutions with neural networks trained using a loss, but imbalanced losses, nonlinear or stiff PDEs, and multi-scale or high-dimensional functions pose challenges.The loss includes PDE and boundary-condition terms and may include data loss when data are available.Source: §3.1 · p. 4

3.2 DATASETS

  • The dataset contains 22 PDE cases selected to span mathematical properties and challenges including complex geometry, multi-scale behavior, nonlinearity, and high dimensionality.The authors describe these challenges as frequent in real-world applications and use FEM data for complex geometry and spectral methods for chaotic problems.Source: §3.2 · pp. 4–5, Table 1 · p. 5

3.3 METHODS AND TOOLBOX

  • The methods and toolbox section organizes PINN variants by changes to loss functions, architecture, and optimization.The selected methods are linked to challenges represented in the dataset, including complex geometry, multi-scale phenomena, and losses with multiple components.Source: §3.3.1 · p. 5

3.3.1 METHODS

  • The evaluated methods include loss reweighting and resampling, MultiAdam, variational and regularized losses, adaptive activations, and domain decomposition.FBPINNs divide the domain into subdomains and train subnetworks there; the selected method applies domain-specific normalization to stabilize training.Source: §3.3.1 · pp. 5–6

3.3.2 STRUCTURE OF TOOLBOX

  • The toolbox provides reusable classes and functions for training, custom PDEs, metrics, visualizations, and result recording, alongside adaptive multi-GPU training.The parallel framework balances training loads across GPUs and supports larger and more complex tasks.Source: §3.3.2 · p. 6

3.4 EVALUATION

  • Evaluation compares PINN predictions with numerical-solver references using L2RE, L1RE, max error, MSE, and Fourier error, with mean L2RE reported in Table 3.L2RE is also tracked over time for time-dependent problems, while additional metrics and results are provided in the appendix.Source: §3.4 · pp. 6–7, Table 3 · p. 7

4 EXPERIMENTS

The experiments show that vanilla PINNs solve only a subset of the benchmark tasks, while PINN variants have different strengths across PDE types. Results also show that parameter choices and training hyperparameters affect performance.

4.1 MAIN RESULTS

  • Using an L2RE of 10% as the success threshold, vanilla PINNs solve 10 of 22 tasks, while variants show task-specific strengths.Loss reweighting and resampling improve some complex-geometry and multi-scale cases; variational formulations excel on inverse problems, while nonlinear problems remain difficult.Source: §4.1 · p. 7, Table 4 · p. 8
  • Loss reweighting and resampling improve some complex-geometry and multi-scale cases, variational formulations excel on inverse problems, and architectural changes help some complex or multi-scale systems.Examples include 1.43% on Poisson-2d-CG for reweighting/resampling, 1.19% on HInv for vPINN, 7.99% on chaotic GS for FBPINN, and 2.39% on Heat-2d-CG for LAAF.Source: Table 4 · p. 8
  • Some methods mitigate challenges in 2D Heat and Poisson cases, but all methods fail on higher-dimensional Poisson3d-CG and longer-time Heat2d-LT.Nonlinear, long-time Burgers, Navier–Stokes, and KS problems challenge most methods; the Wave equation is also particularly difficult.Source: §4.1 · p. 8

4.2 PARAMETERIZED PDE EXPERIMENTS

  • Across six parameterized PDE classes with five examples each, mean L2RE is usually higher than in corresponding non-parameterized cases, and PINN-NTK achieves the best result in three classes.The authors suggest difficult parameter settings may account for higher errors and speculate that PINN-NTK's automatic loss-weight balancing helps on parameterized tasks.Source: §4.2 · p. 8

4.3 HYPERPARAMETER ANALYSIS

  • Larger batch sizes generally improve results, although GS and Poisson2d-C saturate beyond a batch size of 2048.Figure 2 plots batch size, measured as collocation points, on the left and training epochs on the right.Source: §4.3 · p. 9, Figure 2 · p. 9
  • Increasing training epochs reduces overall error, but gains saturate beyond 20k or 80k epochs.The authors report that PINNs approach convergence around 20–80 epochs.Source: §4.3 · p. 9, Figure 2 · p. 9
  • Learning-rate effects vary by problem; rates of 10^-2 often produce error spikes, whereas 10^-5 slows convergence.Rates of 10^-3 or 10^-4, or a step-decay schedule, tend to yield more stable performance; Figure 3 shows convergence curves for three PDEs.Source: §4.3 · p. 9, Figure 3 · p. 9

5 CONCLUSION AND DISCUSSION

PINNacle combines a PDE dataset, a toolbox of over 10 PINN methods, and evaluations on more than 20 challenging problems. The analysis identifies method strengths, hyperparameter sensitivity, and high-dimensional and nonlinear problems as challenges for future work.

5 CONCLUSION AND DISCUSSION

  • PINNacle evaluates over 10 PINN methods on more than 20 challenging PDE problems, with experiments and hyperparameter ablations.The authors plan to add state-of-the-art methods and more practical problem scenarios.Source: §5 · p. 9
  • Domain decomposition benefits complex geometries, and PINN-NTK is a strong method for balancing loss weights.Larger batches and appropriate loss weights or Adam betas may reduce error, but the best hyperparameters usually vary by PDE.Source: §5 · pp. 9–10
  • High-dimensional and nonlinear problems remain pressing challenges, and PINN performance is not yet on par with traditional numerical methods.The authors identify the loss landscape in these settings as underexplored and suggest combining neural networks with preconditioning, weak formulations, or multigrid.Source: §5 · p. 10

A OVERVIEW OF APPENDICES

  • The appendices provide PDE descriptions, hyperparameters and training/testing settings, detailed experiment results, and visualizations.Appendix B includes mathematical descriptions and visualizations of the PDEs; Appendix E reports L2RE, L1RE, MSE, and runtime results.Source: A OVERVIEW OF APPENDICES · p. 14, B DETAILS OF PDES · p. 14

B.1 DEFINITIONS FOR PDES IN MAIN EXPERIMENTS

This appendix specifies the PDE cases and experimental setups used in the main experiments, spanning benchmark equations with varied geometries, coefficients, subdomains, and temporal or multiscale conditions, as well as noisy inverse and parametric tests across flow distributions, domain scales, frequencies, and dimensions. It also records model architecture and primary optimization settings.

1. One-dimensional Burgers Equation (Burgers1d)

  • The cited passage specifies periodic boundary conditions, equating u at opposite x- and y-boundaries.Source: §1 · p. 14

3. Poisson 2D Classic (Poisson2d-C)

  • Poisson2d-C uses a rectangular domain with four circular regions removed.The rectangle is [−0.5, 0.5]²; the four circles are specified in the appendix.Source: §3 · p. 15

4. Poisson-Boltzmann (Helmholtz) 2D Irregular Geometry (Poisson2d-CG)

  • On the domain [−1, 1]^2 with several circles removed, u = 0.2 on ∂Ωrec and u = 1 on ∂Ωcircle; parameters are µ1 = 1, µ2 = 4, k = 8, A = 10.Source: §4 · p. 15

5. Poisson 3D Complex Geometry with Two Domains (Poisson3d-CG)

  • In the two-domain setup, μ and k take values μ1 and k1 on Ω1, and μ2 and k2 on Ω2; related examples use piecewise-linear a(x) in many small squares.The square domain is [−10, 10]², divided into many small squares, with piecewise-linear a(x) in each. Another setup uses [0, 1]² × [0, 5] and an exponential-GRF a(x).Source: §5 · pp. 15–16

14. Basic 1D Wave Equation (Wave1d-C)

  • The basic 1D wave case is utt − 4uxx = 0 on [0, 1] × [0, 1].The appendix also indicates that boundary conditions and an analytical solution are specified.Source: §14 · p. 18

18. Kuramoto-Sivashinsky Equation (KS)

  • The Kuramoto-Sivashinsky case is defined on [0, 2π] × [0, 1], with a warning that error may increase rapidly for chaotic problems.Its governing equation is ut + αuux + βuxx + γuxxxx = 0.Source: §18 · p. 19

19. N-Dimensional Poisson equation (PNd)

  • The N-dimensional Poisson implementation sets n = 5.Source: §19 · p. 19

20. N-Dimensional Heat Equation (HNd)

  • The N-dimensional heat case uses a unit-sphere domain with dimension d = 5 and identifies g(x, t) as the exact solution.Source: §20 · pp. 19–20

21. Poisson inverse problem (PInv)

  • The Poisson inverse problem uses 2,500 uniformly sampled solution values with Gaussian noise N(0, 0.1) to reconstruct the diffusion coefficient.The passage identifies the ground-truth coefficient but does not provide a readable expression in the supplied text.Source: §21 · p. 20

22. Heat (Diffusion) inverse problem (HInv)

  • The heat diffusion inverse problem uses 2,500 randomly sampled space-time solution values with Gaussian noise N(0, 0.1) to reconstruct diffusion coefficients.The sampling domain is Ω × T.Source: §22 · p. 20

B.2 DEFINITIONS AND DESIGN CHOICES FOR PARAMETRIC PDES

  • The best method’s error rose from 26% to 41% across five Burgers2d-C cases with different GRF-sampled initial values.The parametric suite evaluates average PINN performance on Burgers2d-C, Poisson2d-C, Heat2d-MS, NS2d-C, Wave2d-C, and Heat-Nd; spatial flow-distribution differences can alter difficulty.Source: §B.2 · p. 21

2. Poisson 2d Classic (Poisson2d-C)

  • PINN-NTK, MultiAdam, and FBPINN were relatively robust to Poisson2d-C domain scales L ∈ {1, 2, 4, 8, 16}.The linear PDE’s ground truth was obtained by scaling the L = 0.5 solution; the condition number stays unchanged across domain scales.Source: §2 · p. 21
  • PINN-NTK’s loss reweighting and LAAF’s adaptive activation performed well overall on Heat2d-MS, but both declined as frequency variation increased.Five initial-condition parameter pairs were tested, with reference solutions computed using COMSOL.Source: §2 · p. 21
  • NS2d-C varies its top-boundary speed parameter a from 2 to 32, reaching an eightfold higher Reynolds number at a = 32 than the main experiment’s a = 4.The passage motivates this parameterization by noting that higher Reynolds numbers increase equation conditioning and can yield turbulence or small-scale complex flow.Source: §2 · p. 21

6. N-Dimensional Heat Equation

  • MultiAdam was helpful for high-dimensional heat equations tested at dimensions n ∈ {4, 5, 6, 8, 10}.The dimension sweep was designed to compare PINN methods’ ability to fit high-dimensional PDE solutions.Source: §6 · p. 22

C.1 MODEL ARCHITECTURE

  • The experiments use a five-layer MLP with width 100, trained for 20,000 iterations or epochs.For 2D problems, they use 8,192 collocation points and 2,048 boundary/initial points; for 3D problems, 32,768 and 8,192, respectively.Source: §C.1 · p. 22

C.3 OTHER METHOD-SPECIFIC HYPERPARAMETERS

The appendix records method-specific hyperparameters and outlines a DeepXDE-based toolbox workflow for constructing, training, and evaluating PDE models.

C.3 OTHER METHOD-SPECIFIC HYPERPARAMETERS

  • PINN-w weights PDE loss at 1 and initial and boundary losses at 100, while PINN-LRA uses α = 0.1 and PINN and PINN-NTK need no special hyperparameters.RAR adds points with the greatest residual every 2000 epochs.Source: §C.3 · p. 22
  • MultiAdam groups losses into Dirichlet-boundary and non-Dirichlet losses for Wave1d-C, but into PDE and boundary losses for other cases.Wave1d-C uses 10,000 epochs; gPINN assigns gradient terms weight 0.01 and other terms weight 1, with weights requiring further tuning.Source: §C.3 · pp. 22–23

D HIGH-LEVEL STRUCTURE OF TOOLBOX

  • The toolbox wraps DeepXDE with a PDE class, then configures a model from a network, optimizer, and callbacks before compiling it.Its trainer supports parallel training and evaluation across PDE cases and methods on multiple GPUs.Source: D HIGH-LEVEL STRUCTURE OF TOOLBOX · p. 23

E.1 DETAILED RESULTS OF MAIN EXPERIMENTS.

The appendix analyzes the main experiments across error metrics, frequency components, and runtime. It finds that method rankings vary by metric, while several methods show distinct strengths across those measures.

E.1 DETAILED RESULTS OF MAIN EXPERIMENTS

  • Methods that perform well in L2RE generally also perform well in L1RE, but methods with the best results can differ across metrics for the same PDE.The paper attributes the differing rankings to errors capturing different mismatches between predicted and true solutions.Source: §E.1 · p. 23, Table 5 · p. 31, Table 6 · p. 32
  • hp-VPINN performs very well on max error, followed by LAAF; PINN-LRA and PINN-NTK are optimal on some equations, but less stable.Max error measures worst-case error, unlike the average errors measured by L1RE and L2RE.Source: §E.1 · p. 23, Table 7 · p. 33
  • For mid-low frequency functions, FBPINN performs best in most instances; PINN-LRA and ordinary PINN perform better on low- and high-frequency components, respectively.The Fourier-error analysis excludes equations on irregular geometric domains.Source: §E.1 · pp. 23–24, Table 9 · p. 35, Table 10 · p. 36
  • hp-VPINN is fastest on most problems, while other methods add computational overhead compared with vanilla PINN; gPINN can require about twice the runtime.Runtime results average three runs of 20,000 epochs.Source: §E.1 · p. 24, Table 12 · p. 38

E.2 ABLATION EXPERIMENTS

The ablations examine learning rates, collocation-point batch sizes, training epochs, and Adam momentum parameters. Results favor moderate learning rates, larger batches, more epochs, and (β1, β2) = (0.99, 0.99) on average.

E.2 ABLATION EXPERIMENTS

  • Across four PDE problems, vanilla PINN, PINN-NTK, and PINN-LRA were evaluated at 1e-5, 1e-4, 1e-3, 1e-2 and step decay every 1000 epochs by 0.75; authors call 1e-3, 1e-4, or decay a good choice.Source: §E.2 · p. 39
  • Increasing collocation points continually improves performance, though some cases saturate at 8192 points.The tested sizes were 512, 2048, 8192, and 32768, using vanilla PINN and PINN-LRA.Source: §E.2 · p. 39
  • More training epochs improve performance, but results saturate sooner than with larger batch sizes.The tested epoch counts were 5k, 20k, 80k, and 160k.Source: §E.2 · p. 39, Table 15 · p. 40
  • Adam momentum parameters affect convergence speed, stability, and final error, with (β1, β2) = (0.99, 0.99) best on average.β1 and β2 store approximate first- and second-order momentum, respectively.Source: §E.2 · p. 40, Table 16 · p. 40

Other method-specific parameters

The parameter analyses show that PINN performance depends on problem and domain scale, with trade-offs across optimizer stability, accuracy, and hyperparameter choices. Time-dependent error accumulation also remains challenging.

Loss reweighting

gPINN weight

hp-VPINN parameters

FBPINN parameters

Domain scale

MultiAdam and L-BFGS

Temporal error

F OTHER VISUALIZATION RESULTS AND ANALYSIS

The visualizations show relatively low-error solutions for several problems, while other problems remain difficult to learn. They also identify a divergence case for PINN-LRA.

Visualization results

  • Burgers1d, Poisson2d-C, Poisson2d-CG, and NS2d-C are reported as solvable with relatively low error.The visualizations place predictions on the left and prediction-versus-ground-truth errors on the right.Source: F OTHER VISUALIZATION RESULTS AND… · p. 43, Figure 21 · p. 44, Figure 22 · p. 45
  • Other problems remain difficult to learn, sometimes even in the approximate shape of the solution.The paper visualizes two-dimensional cases because they are easier to display, while its code also supports three-dimensional plot functionals.Source: F OTHER VISUALIZATION RESULTS AND… · p. 43
  • PINN-LRA diverged on NS2d-C.Figure 25 shows predictions and prediction errors for the PINN methods on this problem.Source: Figure 25 · p. 48
Loading 2306.08827v2…