Unveiling the optimization process of Physics Informed Neural Networks: How accurate and competitive can PINNs be?

• The introduction situates PINNs as contenders to classical numerical methods across applications including fluid and solid mechanics, quantum optics, and radiative transfer [7].Source: §1 · p. 3
• PINN research is less than a decade old and often relies on trial and error, with a relative lack of rigorous mathematical analysis and specificity.Source: §1 · p. 3
• PINN training faces challenges from computational expense, uncertain hyperparameter importance, and scaling to large or high-dimensional problems.Source: §1 · p. 4
• The study focuses on forward PDE problems in physics-related applications, while noting that its findings may also be useful for inverse problems.Source: §1 · p. 5
• The paper investigates optimization as a fundamental determinant of PINN performance, examining optimizer choice and precision across diverse physical scenarios.Source: §1 · p. 5
• Appropriate optimizer selection and modest optimization changes can improve accuracy by orders of magnitude while reducing network size and computational overhead.Source: §1 · p. 5

• A PINN approximates the PDE solution with a neural-network surrogate, adjusting its weights and biases to minimize a loss based on residuals at sampled points.Source: §2 · p. 6
• Dirichlet and periodic boundary conditions can be hard-enforced by redefining the solution so the conditions hold by construction, independently of the network output.Source: §2 · pp. 6–7
• For Neumann or Robin conditions, soft enforcement adds boundary residual terms to the loss, but performance depends on boundary-point counts and relative weights.Source: §2 · p. 7
• PINN training commonly uses Adam and BFGS-family optimizers; Adam may precede BFGS or L-BFGS to address saddle points before accelerating convergence.Source: §2 · pp. 7–8

3.1. Brief review of optimization procedures

• Quasi-Newton methods approximate the inverse Hessian using first derivatives and matrix-vector products, offering superlinear convergence and faster iterations than Newton’s method.Line search selects step sizes while preserving positive-definiteness of the inverse-Hessian approximation.Source: §3.1 · pp. 8–9

3.2. Modifications of optimization algorithm

• Self-scaled BFGS and self-scaled Broyden modify the BFGS formula through scaling and updating parameters, and the tested choices improved convergence and precision across the problems explored.Source: §3.2 · pp. 9, 11

3.3. Modifications of the loss function

• The modified loss applies a user-defined monotonically increasing function to the usual MSE, with square-root and logarithmic choices potentially accelerating convergence when loss values are small.Source: §3.3 · p. 11

4.1. Current-free Grad-Shafranov equation (CFGS)

• In the current-free case, the authors compare PINN solutions with an analytical solution, using a dipole-quadrupole configuration before extending to broader force-free cases.The configuration has nonzero dipole and quadrupole coefficients and zero higher-multipole coefficients.Source: §4.1 · pp. 12–13

4.1.1. Impact of the optimization algorithm

• BFGS reaches a loss six orders below Adam, and SSBFGS and SSBroyden reduce it by a further two and three orders, respectively.Near J ∼ 10^-13, BFGS needs about 40,000 iterations, versus about 1,000 for its modifications; the modified methods also yield better-conditioned Hessians.Source: §4.1.1 · pp. 14–15, Figure 2 · p. 17

4.1.2. Impact of the loss function.

• Loss transformations yield roughly two orders of magnitude improvement; Jlog converges slightly faster than J1/2 and has a better-conditioned Hessian than J.The study uses Adam during initial training and applies the transformed loss during the BFGS phase; optimizer changes have a stronger convergence effect, though changing the loss is simpler.Source: §4.1.2 · pp. 16, 18, Figure 1 · p. 16

4.2. Non-linear force-free solutions with higher order multipoles.

• For the nonlinear case, the model uses a toroidal function with parameters controlling field strengths, current-region extent, and nonlinearity, alongside eight surface multipoles.The increased complexity calls for larger networks, but efficient optimization keeps the network relatively small.Source: §4.2 · p. 19
• ϵNN is ~10^-3 with standard BFGS and ~10^-5 with SSBFGS or SSBroyden; it begins to surpass ϵFD near 4500 × 4500 resolution.Table 3 reports approximation errors averaged over multiple runs; the stated crossing is near the machine's memory limit.Source: §4.2 · p. 20, Table 3 · p. 21
• For comparable precision (<10^-4), a second-order finite-difference scheme requires at least a 1000 × 1000 grid in this 2D problem.The paper states that PINN advantages would be more pronounced in higher dimensions.Source: §4.2 · p. 21

4.3. Parameter study

• For 2–3 hidden layers, networks with around 30–40 neurons per layer give precise results; the proposed enhancements reduce error faster than simply adding parameters.The optimizer's effect slightly outweighs loss redefinition, while changing the loss requires only a single line of code; the adjustments have relatively minor effects for one-layer networks.Source: §4.3 · pp. 21–22, Figure 5 · p. 22

2D Helmholtz equation (2DH)

• For low-wavenumber 2D Helmholtz, optimizer and loss modifications reduced loss by ∼2 −3 orders and relative L2 error by 1–2 orders; for high wavenumber, relative L2 error fell roughly one order.The high-wavenumber case required increased network complexity to resolve smaller structures.Source: §5 · pp. 24–26, Figure 6 · p. 25, Figure 8 · p. 26

Non-linear Poisson equation (NLP)

• The nonlinear Poisson test raised the oscillation parameter to k = 4 to challenge the solver, with convergence, error norms, and solution comparisons reported.The cited passages do not provide numerical error values.Source: §5 · p. 27, Figure 10 · p. 28, Figure 11 · p. 28

Non-linear Schrödinger equation (NLS)

• For the time-dependent nonlinear Schrödinger system, the PINN solution’s absolute differences from a spectral reference were of the order of 10^-5.The reference used a spectral Fourier discretization of 3000 modes with EDTRK4 time integration.Source: §5 · pp. 29–30, Figure 13 · p. 30

Korteweg-De Vries equation (KdV)

• The KdV test reproduces a two-soliton solution whose collision creates a significant nonlinear interaction, assessing a problem with a Burgers-like term and a third-order derivative.The section presents convergence, error norms, and solution comparisons for this case.Source: §5 · pp. 31–33, Figure 14 · p. 32, Figure 15 · p. 33

One-dimensional Burgers equation (1DB)

• For the one-dimensional Burgers benchmark, the PINN’s absolute differences from a spectral reference were at most ∼10^-5 and concentrated where the solution is steep.The study also plots convergence against a previously reported best result.Source: §5 · p. 34, Figure 16 · p. 34, Figure 17 · p. 35

Allen-Cahn equation (AC)

• The Allen–Cahn PINN achieved a relative L2 error of 2.2 × 10^-6, approximately one order of magnitude below the best result found in the literature.The comparison used a numerical reference computed with 512 Fourier modes; the solution and absolute difference are also shown.Source: §5 · p. 36, Figure 18 · p. 36, Figure 19 · p. 37

3D Navier-Stokes: Beltrami flow (3DNS)

• The 3D Navier–Stokes Beltrami-flow benchmark is a four-equation system for three velocity components and pressure, solved over the four-dimensional (t, x, y, z) domain.In this flow, vorticity is parallel to velocity.Source: §5 · p. 37

5. Other physics problems

• For 3D Navier–Stokes, the study compares self-scaled Broyden PINN predictions of velocity and pressure with analytical solutions and their absolute differences on a fixed cut.Pressure is specified at one spatial point because adding a function of time leaves the governing system unchanged.Source: §5 · p. 39, Figure 21 · p. 40
• The lid-driven-cavity PINN uses a stream function and pressure output, fixes pressure at one domain point, and reports results at Re = 1000 with boundary-loss weight λ = 10.The stream-function formulation solves for ψ instead of the two velocity components.Source: §5 · p. 41
• For the lid-driven cavity, the estimated error is ϵNN ≈ 1.3 × 10^-4 with self-scaled Broyden, versus ϵNN ≈ 1.0 × 10^-3 for BFGS with standard MSE and ϵNN ≈ 1.6 × 10^-4 with logarithmic MSE.No analytical or high-precision numerical solution is available, so the error is estimated using equation (C.3).Source: §5 · p. 41

5.1. Comparison with the literature

• For identical networks, L-BFGS runs about four times faster, but BFGS converges more quickly and achieves a given accuracy more efficiently in iterations and total training time.The comparison concerns the NLS equation and includes L-BFGS with the smaller network used in this work and the larger model considered in.Source: §5.1 · p. 43, Figure 24 · p. 45
• Our solutions are 1–2 orders of magnitude more precise than literature references despite using significantly smaller networks, with errors averaged across runs and reported as mean ± standard deviation.Table 5 compares the models and their relative L2 errors against analytical or numerical references.Source: §5.1 · p. 43, Table 5 · p. 44

6. Conclusions.

• Appropriate optimization choices can improve PINN accuracy by several orders of magnitude across physical problems, and quasi-Newton methods benefit from a Hessian spectrum centered near unity with minimal dispersion.A modified loss Jg can produce a similar effect to the well-conditioned Hessian.Source: §6 · pp. 45–46
• The authors suggest that optimization-algorithm choice typically has a greater effect on convergence than loss-function adjustments, though tweaking the loss is often easier than changing an optimizer or creating one.Source: §6 · p. 46
• Benchmarks show that problems from the literature can be solved with smaller networks and improved precision, enhancing numerical efficiency relative to classical methods.Table 5 illustrates the benchmark results; the supplied passage does not report its cell values.Source: §6 · pp. 46–47
• All simulations ran on a standard PC or, in some cases, a regular laptop; the authors anticipate stronger optimization effects as problem dimensionality increases.Other hyperparameter changes, including activation functions and network structure, may also help.Source: §6 · p. 47
• L-BFGS is widely used in PINNs and faster per iteration than BFGS, but suffers more from ill-conditioning; future work will examine its optimization-theory modifications.The passage says L-BFGS needs more trainable parameters to reach similar accuracy.Source: §6 · p. 47

Appendix A. Derivation of the Grad-Shafranov equation

• In axisymmetry, the magnetic field B is described using poloidal and toroidal stream functions P and T.Source: Appendix A · p. 47
• Substituting the field expression into the force-free condition (∇×B)×B = 0 implies T = T(P) and equation (28).Source: Appendix A · p. 48
• The magnetic field components can be recovered from the stream functions P and T through the stated relations.Source: Appendix A · p. 48

Appendix B. Efficient computation of the scaling parameter τ (1)

• The scaling-parameter computation avoids inverse matrices, which are O(n^3) and can introduce numerical errors when Hk is ill-conditioned.Source: Appendix B · p. 48
• The proposed calculation uses vector multiplications, giving O(n) computation once the step length and gradient are available at iteration k.Source: Appendix B · p. 48
• The modification acts as a switch between standard BFGS and self-scaled BFGS; τk = 1 is selected in most iterations, yet the change substantially improves optimization.Source: Appendix B · pp. 48–49

Appendix C. Error analysis

• When an analytical or high-fidelity numerical solution exists, the PINN approximation is evaluated using relative error.Source: Appendix C · p. 49
• Without such a solution, the procedure evaluates the PINN on a uniform test grid, approximates derivatives with second-order finite differences, and forms a discretized PDE residual.Source: Appendix C · p. 49
• For an exact PDE solution, the finite-difference residual error decreases with resolution as ∼N_0^-p, where p is the approximation order; for a PINN, its error eventually dominates and the residual levels off.Source: Appendix C · p. 49
• Finite-difference validation errors are measurable and reducible by increasing resolution, using higher-order formulas, or using quadruple precision.The passage contrasts this controlled error with direct simulation comparisons, where benchmark-code errors may be uncertain.Source: Appendix C · p. 50