PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimizing the Optimizer for Physics-Informed Neural Networks and Kolmogorov-Arnold Networks

Elham Kiyani, Khemraj Shukla, Jorge F. Urbán, Jérôme Darbon, George Em Karniadakis

arXiv:2501.16371v6cs.LGcs.AImath.OC

TL;DR

PINN and PIKAN training uses optimizers that can struggle with stiff, nonconvex PDE losses, motivating a systematic evaluation of self-scaled and other quasi-Newton methods. Across challenging PDEs and DeepONets, the study reports stronger convergence, accuracy, and robustness, while identifying batch training as an open applicability question.

  • Problem

    PINN optimization remains difficult because PDE-informed losses are highly nonlinear and nonconvex, while standard optimizers can struggle with convergence and accuracy.

  • Method

    The study compares SSBFGS, SSBroyden, BFGS, and related line-search or trust-region strategies across PINNs, PIKANs, and DeepONets.

  • Results

    SSBroyden consistently outperformed SSBFGS and BFGS across the studied PDEs, with faster convergence and greater robustness on complex problems.

  • Takeaways & Limitations

    SSBroyden achieved comparable accuracy with smaller networks in selected PDEs and extended quasi-Newton benefits beyond physics-informed models to operator learning.

  • Takeaways & Limitations

    Applicability to very large-scale batch training remains open because changing mini-batches can destabilize BFGS-based updates that depend on gradient differences.

Abstract

from arXiv · show

Physics-Informed Neural Networks (PINNs) have revolutionized the computation of PDE solutions by integrating partial differential equations (PDEs) into the neural network's training process as soft constraints, becoming an important component of the scientific machine learning (SciML) ecosystem. More recently, physics-informed Kolmogorv-Arnold networks (PIKANs) have also shown to be effective and comparable in accuracy with PINNs. In their current implementation, both PINNs and PIKANs are mainly optimized using first-order methods like Adam, as well as quasi-Newton methods such as BFGS and its low-memory variant, L-BFGS. However, these optimizers often struggle with highly non-linear and non-convex loss landscapes, leading to challenges such as slow convergence, local minima entrapment, and (non)degenerate saddle points. In this study, we investigate the performance of Self-Scaled BFGS (SSBFGS), Self-Scaled Broyden (SSBroyden) methods and other advanced quasi-Newton schemes, including BFGS and L-BFGS with different line search strategies. These methods dynamically rescale updates based on historical gradient information, thus enhancing training efficiency and accuracy. We systematically compare these optimizers using both PINNs and PIKANs on key challenging PDEs, including the Burgers, Allen-Cahn, Kuramoto-Sivashinsky, Ginzburg-Landau, and Stokes equations. Additionally, we evaluate the performance of SSBFGS and SSBroyden for Deep Operator Network (DeepONet) architectures, demonstrating their effectiveness for data-driven operator learning. Our findings provide state-of-the-art results with orders-of-magnitude accuracy improvements without the use of adaptive weights or any other enhancements typically employed in PINNs.

1. Introduction

PINNs embed PDE constraints in neural-network losses, but this creates difficult optimization landscapes and leaves optimization error as a central challenge. The study motivates advanced quasi-Newton methods to improve training across PDE-solving architectures.

  • PINNs incorporate initial conditions, boundary conditions, and PDE residuals into neural-network loss functions for forward and inverse PDE problems.
  • Standard PINN optimization commonly uses Adam, BFGS, or L-BFGS, although other algorithms can outperform them on many problems.
  • Optimization error remains challenging because PDE terms produce highly nonlinear, nonconvex objectives with degenerate and non-degenerate saddle points.
  • Adding the physics term changes Burgers’ loss landscape from smooth and convex to one containing local minima and saddle points.
  • The study investigates quasi-Newton optimizers for PINN and PIKAN approximations of linear, nonlinear, stiff, and chaotic PDEs.

line-search methods

Line-search and trust-region methods use local quadratic models differently, while quasi-Newton schemes approximate curvature from gradient history. The paper examines Broyden-family updates and extends advanced quasi-Newton optimization to PINNs, PIKANs, and DeepONets.

  • Newton and quasi-Newton methods: Newton’s method can converge quadratically near well-behaved solutions but becomes costly in high-dimensional PINN settings because it requires Hessian computation and inversion.
  • Newton and quasi-Newton methods: Quasi-Newton methods approximate the Hessian using first-order information from previous iterations and update it through the secant equation.
  • Broyden-family updates: BFGS and DFP are Broyden-class special cases, obtained by setting θ_k to 0 and 1, respectively.
  • Self-scaled updates: SSBFGS scales the Hessian approximation before the BFGS update to address ill-conditioning, with scaling motivated by reducing the condition number.
  • Study scope: The study evaluates advanced quasi-Newton methods for PINNs, PIKANs, and DeepONets alongside different line-search and trust-region strategies.

2. Computational Experiments

The experiments compare quasi-Newton optimizers, line-search strategies, and PINN/PIKAN configurations across Burgers and other challenging PDE settings. Double precision and self-scaled methods often improve accuracy, while training cost depends strongly on architecture, iteration budget, and optimizer composition.

  • Experimental setup: The study evaluates L-BFGS, BFGS, SSBFGS, and SSBroyden with Strong Wolfe, backtracking, and trust-region strategies across diverse PDE classes.The experiments include steady-state and time-dependent parabolic, hyperbolic, elliptic, and hyperbolic-parabolic equations.
  • Burgers equation: PINNs: In Burgers PINNs, 50,000 optimizer iterations reduce SSBFGS and SSBroyden errors to 10^-8, while producing no noticeable BFGS error improvement.The same study reports that deeper networks improve error but substantially increase training time.
  • Burgers equation: PIKANs: SSBroyden consistently outperforms BFGS for the evaluated Burgers PIKANs, although 50,000 iterations provide minimal additional improvement.The PIKAN comparisons use Chebyshev polynomial representations with different network sizes and polynomial degrees.
  • Burgers equation: overall comparison: Across seven Burgers cases, SSBroyden achieves the best error in double precision and outperforms BFGS and L-BFGS.PIKANs can approach PINN accuracy with more parameters, but at significantly higher training cost; equal-error comparisons favor PINN training efficiency.
  • Burgers equation: efficiency: At relative L2 errors of 10^-4 and 10^-5, SSBFGS and SSBroyden reach the target accuracy significantly faster than standard BFGS.The fixed-threshold comparison reports iterations, final relative error, and total training time rather than using a fixed iteration budget.
  • Precision and convergence: Double precision removes observed BFGS loss degeneration and reduces relative L2 errors by two orders of magnitude compared with single precision, reaching 2 × 10^-6 in remaining cases.Some 30,000-iteration runs still fail to converge, likely because double precision increases sensitivity to the tolerance limit.
  • Allen-Cahn equation: For Allen-Cahn, PINNs and Chebyshev PIKANs reach relative error 10^-6, but PINNs require 973 s versus 1804 s for PIKANs.BFGS is generally slightly faster than SSBroyden, and Chebyshev representations are more efficient than B-splines in accuracy and training time.

2.4. Kuramoto-Sivashinsky equation

The Kuramoto–Sivashinsky experiments use time-marching PINNs and compare BFGS, SSBroyden, and related architectures across increasingly difficult spatiotemporal settings. SSBroyden generally provides faster convergence and more accurate predictions, while SSBFGS performance depends strongly on line-search strategy.

  • Problem setup: The solution domain [0, 1]×[0, 2π] is divided into 20 windows of Δt = 0.05, with Fourier features encoding periodic spatial boundary conditions.Each window is trained separately, introducing sequential-training overhead; the PINNs use five 30-neuron tanh hidden layers and RAD resampling.
  • PINN optimizer comparison: BFGS struggles near t = 1.0, where its predicted solution shows substantially larger absolute error than SSBroyden’s.The increasing difficulty toward later times is also reflected by higher losses in the final windows.
  • PINN optimizer comparison: SSBroyden consistently outperforms BFGS across all 20 time windows, converging faster and producing more accurate predictions.The loss includes PDE-residual and initial-condition contributions, and each window is trained for 30,000 iterations in the cited comparison.
  • Temporal difficulty: The Kuramoto–Sivashinsky relative error becomes harder to control after t = 0.5 as solution complexity increases over time.This trend is reported for both the five-window [0, 0.5] setup and the 20-window [0, 1] setup.
  • PIKAN configurations: Increasing the PIKAN width from 10 to 20 neurons substantially increases parameters and training time without substantially reducing error.The comparison concerns Chebyshev-polynomial PIKANs with five hidden layers.
  • Prediction accuracy: SSBroyden’s predictions are significantly more accurate than BFGS’s despite both reporting relative errors of approximately 10^-1 in the cited comparison.The visual prediction comparison therefore distinguishes the optimizers more clearly than the tabulated relative-error value alone.
  • Lid-driven wedge flow: SSBFGS with trust-region line search recovers all four wedge-flow eddies, including the weakest with |u| = 10^-8, whereas backtracking resolves only the dominant eddy.Adam does not accurately capture any eddies, and similar mean relative L2 errors do not guarantee uniformly accurate velocity fields.

3. Optimizing Data-Driven DeepONet with Self-Scaled Optimizers

The DeepONet study evaluates quasi-Newton optimizers for data-driven operator learning on displacement and damage fields in a one-dimensional bar. SSBroyden and SSBFGS achieve lower errors than the alternatives, with SSBroyden delivering the best accuracy despite slightly longer training.

  • Architecture: The DeepONet maps sampled input functions and spatial coordinates to output fields through separate branch and trunk subnetworks.The branch network encodes sensor-point function values, while the trunk network encodes evaluation coordinates.
  • Data and evaluation: The bar dataset contains displacement and damage fields sampled at 101 spatial nodes over 50 applied displacements, using 45 steps for training and 5 for testing.The study evaluates predicted displacement u and damage α on the held-out displacement steps.
  • Optimizer comparison: SSBroyden and SSBFGS achieve lower relative errors than Adam and BFGS during DeepONet training.The loss combines contributions from the damage and displacement fields.
  • Optimizer comparison: SSBroyden achieves the best accuracy across the evaluated configurations while requiring slightly longer training times.The comparison covers Case 2 and Case 3 configurations.

4. Summary

The study compares advanced quasi-Newton optimizers for PINNs, PIKANs, and DeepONets across challenging PDE and fracture-learning problems. SSBFGS and SSBroyden achieve strong convergence and accuracy without adaptive loss weighting or other enhancements.

  • The study evaluates BFGS, SSBFGS, SSBroyden, L-BFGS with Wolfe line search, BFGS with backtracking, and trust-region methods across multiple PDEs.The comparison uses PINNs and PIKANs implemented with optax and optimistix in JAX.
  • SSBroyden, SSBFGS, and BFGS with Wolfe line search significantly improve PINN convergence rate and accuracy, especially for complex and stiff PDEs.Across the reported cases, SSBroyden consistently outperformed SSBFGS and BFGS, with faster convergence and greater robustness.
  • The study also evaluates quasi-Newton optimizers for DeepONet crack nucleation in a one-dimensional homogeneous bar using displacement and damage-field errors.The DeepONet results concern a purely data-driven setting with increasing prescribed displacements.
  • 4.19 × 10−8 replaces 8.06 × 10−6 for the Burgers-equation error when using SSBroyden with Wolfe line search instead of SOAP.The study reports lower relative l2 errors than SOAP across all tested benchmark problems.
  • 2.65 × 10−5 is achieved for Kuramoto–Sivashinsky over t ∈ [0, 1], versus SOAP's 3.86 × 10−2 over t ∈ [0, 0.8].Comparable improvements are reported for the Allen–Cahn and Ginzburg–Landau equations.
  • SSBroyden achieves comparable accuracy with smaller networks for Kuramoto–Sivashinsky and Allen–Cahn, reducing training time.The reported configurations use five hidden layers of 30 neurons and three hidden layers of 30 neurons, respectively.

Appendix A. Search Methods and Trust-Region Techniques

This appendix overviews search methods and trust-region techniques used with quasi-Newton algorithms to support efficient and reliable convergence.

  • The appendix discusses search methods and trust-region techniques in the context of quasi-Newton algorithms.Its focus is their role in achieving efficient and reliable convergence.

Appendix A.0.1. Line-search methods

Line-search methods choose a descent direction and then approximate a suitable step size by testing trial lengths until the objective decreases sufficiently. The process repeats from each new iterate until convergence.

  • Line search selects a search direction p_k toward lower function values and a step size α_k through approximate one-dimensional minimization.The exact minimum is often computationally prohibitive, so finite trial step lengths are evaluated.
  • After an acceptable step length is identified, the algorithm updates the current iterate and repeats direction and step-size selection.Iterations continue until convergence criteria are satisfied.

Wolfe conditions

Wolfe conditions define sufficient criteria for selecting optimization step sizes. The strong Wolfe variant combines the two conditions, with c1 = 10−4 and c2 = 0.9 used in all tests.

  • The Wolfe conditions impose two inequalities on a step size to ensure sufficient decrease and exclude unacceptably short steps.The constants satisfy 0 < c1 < c2 < 1.
  • Strong Wolfe conditions combine conditions (A.3) and (A.5), using c1 = 10−4 and c2 = 0.9 in all tests.These are described as standard choices in the optimization literature.

Backtracking

Backtracking selects step lengths using the Armijo condition and can converge faster than a trust-region approach in the illustrated quadratic example.

  • Backtracking chooses the step length using the Armijo condition without requiring an additional condition.
  • Six iterations illustrate changing search directions during minimization of f(x, y) = x2 + y2 + xy.
  • The example reports faster convergence to the minimum level-set value with backtracking than with the trust-region approach.

Appendix A.0.2. Trust-region methods

Trust-region methods jointly determine direction and step length by optimizing a local quadratic model inside a bounded region, adapting the region based on step quality.

  • Trust-region methods define a region around the current iterate and seek a point within it that reduces the objective.
  • The algorithm constructs a local quadratic model and approximately solves a trust-region subproblem.
  • A proposed step is accepted and the region expanded after significant improvement, or rejected and the region reduced after poor results.
  • The gradient norm is used as a convergence criterion, with supplementary checks on objective change, iterate change, and maximum iterations.
  • For the quadratic demonstration, the trust-region radius remains constant because the function is convex, whereas PDE problems use an adaptive radius.

Appendix B. Detailed Case Study: Burgers Equation in Single Precision

The single-precision Burgers case study compares SSBroyden, SSBFGS, BFGS, and optimizer combinations under Wolfe, backtracking, and trust-region strategies.

  • The loss and relative error stabilize after roughly 4,000 iterations, indicating no significant further improvement.
  • An Adam or RAdam warm-up of 11 iterations followed by BFGS improves error by an order of magnitude compared with extended Adam followed by BFGS.
  • Extended Adam followed by BFGS provides no additional advantage in Scenario 1 because single-precision representation degenerates the loss function.
  • Scenario 2 evaluates single-precision convergence with a smaller network, 250 initial and boundary data points, and 50,000 collocation points.

Appendix C. Lorenz system

The Lorenz study uses double-precision PINNs with sequential time-window training and compares BFGS and SSBroyden predictions against the numerical solution.

  • The Lorenz system uses parameters σ = 10, ρ = 28, and β = 8/3 and is solved from (1, 1, 1) through t = 20.
  • The PINN has three hidden layers of 30 neurons with tanh activations, followed by BFGS and SSBroyden using Wolfe line search after Adam training.
  • The time interval [0, 20] is divided into 40 windows of length Δt = 0.5, with each window initialized from the previous final state.
  • Figure C.31 compares BFGS and SSBroyden PINN predictions for x(t), y(t), and z(t) against the numerical solution.
  • Table C.20 reports L2 relative errors and training times for the two optimizers across the 40 time windows.

Appendix D. Multi-dimensional Rosenbrock function

The appendix evaluates quasi-Newton methods on the Rosenbrock objective across dimensions from 2D to 20D. SSBroyden with Strong Wolfe line search performs comparably to SSBFGS and BFGS, while all quasi-Newton methods recover the global minimum to machine precision.

  • The Rosenbrock function is used as a non-quadratic benchmark for optimization algorithms.
  • SSBroyden with Strong Wolfe line search converges to the global minimum with accuracy comparable to SSBFGS and BFGS.The comparison considers Rosenbrock’s narrow, curved valleys and multi-modal landscapes.
  • Figure D.32 compares BFGS, SSBroyden, and SSBFGS with Strong Wolfe line search in 2D, 5D, 10D, and 20D.All optimizers use the same starting point, stopping criterion, initial inverse Hessian approximation, and optimization parameters.
  • All quasi-Newton methods recover the Rosenbrock global minimum to machine precision across the tested dimensions.The table contrasts these results with first-order methods, which can converge slowly or stall at suboptimal values in higher dimensions.
Loading 2501.16371v6…