Challenges in Training PINNs: A Loss Landscape Perspective

Pratik Rathore; Weimu Lei; Zachary Frangella; Lu Lu; Madeleine Udell

Source-linked AI summary

Challenges in Training PINNs: A Loss Landscape Perspective

Pratik Rathore, Weimu Lei, Zachary Frangella, Lu Lu, Madeleine Udell

arXiv:2402.01868v2Feb 2, 2024cs.LGmath.OCstat.ML

TL;DR

PINNs are difficult to train because differential operators induce ill-conditioning in their loss, hindering effective optimization. The paper analyzes this landscape, compares Adam and L-BFGS strategies, and introduces NNCG; it reports improved solutions from combining these methods and theoretically supports second-order optimization.

Problem
PINN losses can be difficult to minimize, causing failures to learn PDE solutions and obscuring whether poor performance reflects optimization or architecture limitations.
Method
The paper studies PINN loss conditioning through differential operators and Hessian-based optimization, comparing Adam, L-BFGS, Adam+L-BFGS, and NNCG.
Results
The paper finds Adam+L-BFGS superior across varied network sizes, while NNCG can significantly improve its returned solution; theory connects ill-conditioned operators to ill-conditioned PINN losses.
Takeaways & Limitations
Combining first- and second-order optimization methods provides a more effective PINN training strategy within the studied settings.
Takeaways & Limitations
NNCG is slower than L-BFGS, taking 5, 20, and 322 times more per iteration on convection, reaction, and wave, respectively.

Abstract

from arXiv · show

This paper explores challenges in training Physics-Informed Neural Networks (PINNs), emphasizing the role of the loss landscape in the training process. We examine difficulties in minimizing the PINN loss function, particularly due to ill-conditioning caused by differential operators in the residual term. We compare gradient-based optimizers Adam, L-BFGS, and their combination Adam+L-BFGS, showing the superiority of Adam+L-BFGS, and introduce a novel second-order optimizer, NysNewton-CG (NNCG), which significantly improves PINN performance. Theoretically, our work elucidates the connection between ill-conditioned differential operators and ill-conditioning in the PINN loss and shows the benefits of combining first- and second-order optimization methods. Our work presents valuable insights and more powerful optimization strategies for training PINNs, which could improve the utility of PINNs for solving difficult partial differential equations.

1. Introduction

PINN training is difficult because differential operators create an ill-conditioned loss landscape that slows optimization and can obscure architecture quality. The paper analyzes this difficulty, compares common optimizers, and introduces NNCG to improve solutions beyond Adam+L-BFGS.

PINNs parameterize PDE solutions with neural networks and minimize losses involving residual, boundary, and initial-condition terms.
Optimization difficulties can prevent PINNs from learning solutions and make architectures appear inferior because of under-optimized losses.
The paper connects slow first-order convergence to ill-conditioned differential operators and studies the resulting PINN loss landscape.
Quasi-Newton methods improve PINN loss conditioning by 1000× or more.
Adam+L-BFGS is reported as superior to Adam and L-BFGS across varied network sizes.
NNCG is introduced as a second-order optimizer that significantly improves solutions returned by Adam+L-BFGS.

2. Problem Setup

The paper formulates PINNs as nonlinear least-squares problems that enforce PDE residuals and boundary or initial conditions. Experiments compare optimizers on challenging convection, wave, and reaction problems using controlled network and evaluation settings.

PINNs model u as a neural network whose weights minimize a nonlinear least-squares objective for PDE and boundary or initial-condition constraints.
The residual term measures PDE violation, while the second loss term measures boundary or initial-condition violation.
Zero loss means the network exactly satisfies the PDE and boundary or initial conditions at training points.
Experiments evaluate convection and wave PDEs plus a reaction ODE under coefficient settings considered challenging in prior literature.
The study compares Adam, L-BFGS, and Adam followed by L-BFGS, varying Adam learning rates and the optimizer-switch iteration.
Experiments use three-hidden-layer tanh MLPs with widths 50, 100, 200, or 400, five random seeds, and L2 relative error evaluation.

3. Related Work

Related work addresses PINN training through alternative architectures, loss strategies, activations, and optimizers. This paper instead analyzes and directly targets ill-conditioning in the vanilla PINN loss, including for finite-width networks.

PINNs solve PDEs by penalizing residual, boundary, and initial-condition deviations without requiring known forward-problem solutions.
Prior methods improve training through loss reweighting, resampling, scale-invariant or natural gradients, adaptive activations, new losses, regularization, and architectures.
The paper directly targets ill-conditioning caused by the differential operator in the vanilla PINN objective, which other strategies generally retain.
Unlike NTK-based analyses requiring infinite width, the paper studies Hessian eigenvalues and obtains conditioning results for finite-width PINN architectures.

4. Good Solutions Require Near-zero Loss

Useful PINN solutions generally require extremely small training losses because lower loss typically corresponds to lower L2 relative error. Ill-conditioning and under-optimization make this accuracy difficult to reach.

PINNs must be trained to near-zero loss to obtain reasonably low L2 relative error.
On the convection PDE, reducing loss from 10^-3 to 10^-5 changes L2RE from around 10^-1 to around 10^-2, a 10× improvement.
The loss–L2RE relationship is typical across many PDEs, although the reaction ODE can show L2RE around 1 despite near-zero loss.
Ill-conditioning and under-optimization make reaching sufficiently accurate PINN solutions difficult.

5. The Loss Landscape is Ill-conditioned

The PINN loss is ill-conditioned, largely because the residual term contains an ill-conditioned differential operator. Hessian spectral analyses show that L-BFGS preconditioning substantially improves conditioning across PDEs and loss components.

5.2. The Ill-conditioning is Due to the Residual Loss: The PINN loss is ill-conditioned mainly because the residual loss contains the differential operator.The residual component is more ill-conditioned than the initial- and boundary-condition components.
5.1. The PINN Loss is Ill-conditioned: Ill-conditioned losses have steep and flat directions, making first-order optimization methods progress slowly.The loss condition number is the ratio of the largest to smallest-magnitude Hessian eigenvalues.
5.1. The PINN Loss is Ill-conditioned: Hessian spectral density is used to examine the conditioning of PINN losses after Adam+L-BFGS training across convection, reaction, and wave problems.The same analysis is applied to the full loss and its residual, initial-condition, and boundary-condition components.
5.3. L-BFGS Improves Problem Conditioning: L-BFGS preconditioning reduces eigenvalue magnitudes and condition numbers by at least 10^3 across all three PDEs.The preconditioner also improves the conditioning of each individual loss component.
5.3. L-BFGS Improves Problem Conditioning: Figure 2 shows that lower final loss generally corresponds to lower final L2RE across network widths, optimizers, and random seeds.The figure plots final L2RE against final loss for each experimental combination.

6. Adam+L-BFGS Optimizes the Loss Better Than Other Methods

Adam+L-BFGS consistently achieves lower PINN loss and L2RE than Adam or L-BFGS alone across the evaluated settings. The paper explains this advantage as combining saddle-point avoidance with improved conditioning and faster local convergence.

6.1. Adam+L-BFGS vs Adam or L-BFGS: Adam+L-BFGS consistently provides smaller loss and L2RE than Adam or L-BFGS alone.The comparison covers convection, reaction, and wave problems across network widths.
6.1. Adam+L-BFGS vs Adam or L-BFGS: Adam+L-BFGS delivers the lowest loss and L2RE across each network width in the difficult coefficient settings, with one reaction-problem exception.Adam outperforms the combined method on reaction loss at width 100 and L2RE at width 200.
6.1. Adam+L-BFGS vs Adam or L-BFGS: 14.2× smaller L2RE than Adam is attained by Adam+L-BFGS on convection, while 6.07× smaller L2RE than L-BFGS is attained on wave.These values summarize the best optimizer performance in Table 1.
6.2. Intuition From Optimization Theory: Adam avoids saddle points, whereas L-BFGS can be attracted to them because quasi-Newton methods ignore negative curvature.The paper uses this distinction to motivate running Adam before L-BFGS.
6.2. Intuition From Optimization Theory: First-order methods converge slowly on ill-conditioned problems, while Newton-type methods can reduce the condition number and accelerate convergence.For PINNs, the condition number near a solution is often greater than 10^4.
6.2. Intuition From Optimization Theory: Adam+L-BFGS combines saddle-point avoidance from Adam with condition-number reduction and faster local convergence from L-BFGS.The paper gives faster convergence on the wave equation as an example.
6.2. Intuition From Optimization Theory: Adam can perform comparably on reaction because its largest Hessian eigenvalue is around 10^3, versus around 10^4 for convection and 10^5 for wave.The paper interprets reaction as less ill-conditioned than the other problems.

7. The Loss is Often Under-optimized

Adam+L-BFGS can stop before reaching a critical point, leaving useful gradient information and an under-optimized loss. NNCG continues improving the loss, gradient norm, and PINN solution, but its higher per-iteration cost motivates using it after Adam+L-BFGS.

Adam+L-BFGS may terminate early without reaching a critical point because its line search cannot find a positive step size.Several runs use a zero step size, causing early stopping while the gradient norm remains nonzero.
The remaining gradient contains useful information for further reducing the loss after L-BFGS stops.
NNCG significantly improves both loss and gradient norm after Adam+L-BFGS, whereas gradient descent makes no comparable progress.Figure 4 reports a loss reduction greater than 10× for NNCG in all instances.
NNCG further improves pointwise absolute error and L2RE after Adam+L-BFGS, while gradient descent improves neither loss nor L2RE.
NNCG should follow Adam+L-BFGS rather than replace it entirely because NNCG is substantially slower per iteration.NNCG takes 5, 20, and 322 times longer per iteration than L-BFGS on convection, reaction, and wave, respectively.

8. Theory

The theory links ill-conditioned differential operators to ill-conditioned PINN objectives, explaining why first-order optimization can be slow. It also supports hybrid training that switches from gradient descent to damped Newton steps for condition-number-independent local convergence.

Ill-conditioned operators: Ill-conditioned differential operators make first-order methods slow to reach high-precision PINN solutions.The condition number of the PINN objective grows with the number of residual points under the theorem’s spectral assumptions.
Hybrid convergence: GDND switches from gradient descent to damped Newton’s method after a fixed number of iterations.The algorithm uses gradient descent to approach a minimizer before applying second-order updates.
Scope of analysis: The analysis uses gradient descent rather than Adam, so GDND differs from the Adam+L-BFGS+NNCG method recommended in practice.The paper attributes this choice to existing Adam analyses not mirroring empirical performance.
Ill-conditioned operators: The PINN objective’s condition number is connected to the operator A◦K∞(w⋆), with A defined as the Hermitian square of D.When its spectrum decays polynomially, the condition number grows with nres, which typically ranges from 10^3 to 10^4.
Hybrid convergence: GDND achieves linear convergence independent of the condition number once gradient descent reaches a sufficiently suitable point.The result is local and depends on entering a neighborhood where the stated PŁ⋆ condition holds.

9. Conclusion

The conclusion identifies PINN loss landscapes as ill-conditioned and presents optimization strategies combining Adam, L-BFGS, and NNCG. The theory connects differential-operator conditioning to loss conditioning and supports second-order methods.

Conclusion: The paper demonstrates that ill-conditioning in PINN loss landscapes hinders effective training.It compares Adam, L-BFGS, and Adam+L-BFGS while introducing NNCG as another optimization approach.
Conclusion: The theory connects ill-conditioned differential operators to ill-conditioned PINN losses and supports second-order methods over first-order methods.

Impact Statement

The impact statement frames PINNs as neural-network approaches for solving PDEs without discretization or meshing. The paper studies convection, reaction, and wave problems as experimental settings.

Experimental scope: The experiments include one-dimensional convection, reaction, and wave problems.The passages describe convection as a hyperbolic PDE, reaction as a nonlinear ODE, and wave as a hyperbolic PDE.
Experimental scope: The convection and wave experiments use analytical solutions, with β = 40 for convection and β = 5 for wave.

B. Why can Low Losses Correspond to Large L2RE?

Low PINN losses can correspond to trivial solutions that satisfy residual terms while failing to match exact solutions or boundary conditions. The paper explains how L-BFGS preconditioning and NNCG address the resulting optimization difficulties.

Trivial solutions: Near-zero PINN losses can coexist with L2RE close to 1 because residual minimization can favor trivial solutions.For convection, any constant function has zero residual; for reaction, constant u = 0 or u = 1 does.
Trivial solutions: Figure 6 shows effectively constant PINN solutions that learn initial conditions but fail to learn boundary conditions.The solutions remain close to 0 across much of the domain while differing from the exact solution at boundaries.
Preconditioning: L-BFGS improves optimization by transforming the objective into a space with more favorable Hessian conditioning.Its inverse-Hessian approximation preconditions the objective, enabling larger-progress updates than unpreconditioned gradient descent.
Preconditioning: Figure 7 compares Hessian spectral densities before and after L-BFGS preconditioning for reaction and wave problems.The caption states that each loss component is ill-conditioned and that L-BFGS improves its conditioning.
Optimizer comparison: Adam+L-BFGS typically performs best across network widths on both loss and L2RE.
Optimizer comparison: L-BFGS can terminate early when strong Wolfe line-search conditions are not met, even when acceptable step sizes exist.For reaction, termination can instead occur when the line-search slope falls below 10^-9.
NNCG: NNCG uses Newton steps computed with preconditioned conjugate gradients and Hessian-vector products, but costs more per iteration than L-BFGS.The per-iteration cost is especially high for wave because its Hessian-vector products involve second derivatives.

F.3. Gr(w) Concentrates Around G∞(w)

This section develops finite-sample control of the empirical Gauss–Newton matrix through effective dimension and ridge-leverage coherence, then connects spectral decay to ill-conditioning of the PINN loss. It also reports empirical evidence that the loss condition number grows polynomially with the number of residual points.

F.3. Gr(w) Concentrates Around G∞(w): Finite-sample approximation uses γ-effective dimension and γ-ridge leverage coherence to control the empirical Gauss–Newton matrix around its population counterpart.The argument applies an intrinsic-dimension matrix Bernstein inequality to the centered empirical residual matrices.
F.3. Gr(w) Concentrates Around G∞(w): If the population kernel spectrum decays as λj(A ◦K∞(w)) ≤ Cj^-2α with α > 1/2, then the empirical Gauss–Newton spectrum also decays with high probability.The result requires a sufficient number of residual points and holds with probability at least 1 −δ.
F.3. Gr(w) Concentrates Around G∞(w): An ill-conditioned differential operator produces an ill-conditioned PINN loss, so gradient descent may require many iterations to reach high precision.The theorem implies that first-order methods alone may not provide sufficient accuracy in this setting.
F.3. Gr(w) Concentrates Around G∞(w): The empirical condition-number estimate grows polynomially with the number of residual points near a minimizer, supporting the theoretical prediction.Figure 10 estimates the Hessian condition number using the ratio λ1(HL)/λ129(HL), a tractable lower bound.
F.3. Gr(w) Concentrates Around G∞(w): The condensed PINN loss has Jacobian JF(w), tangent kernel KF(w) = JF(w)JF(w)^T, and Gauss–Newton matrix G(w) = JF(w)^TJF(w).These matrices provide the finite- and parameter-space objects used in the conditioning analysis.

G.3. Fast Local Convergence of Damped Newton’s Method

This section establishes that damped Newton’s method converges rapidly once gradient descent reaches a neighborhood of a minimizer. The proof maintains iterates in a local region where the damped Hessian supplies both smoothness and a PŁ⋆ condition.

G.3. Fast Local Convergence of Damped Newton’s Method: The local convergence region is defined around a minimizer where damped Hessians remain positive definite and comparable across nearby points.For γ ≥ µ, the damped Hessian satisfies spectral sandwich bounds controlled by ε.
G.3. Fast Local Convergence of Damped Newton’s Method: Within the local region, tangent kernels and Gauss–Newton matrices stay close, the damped Hessian approximates the damped Gauss–Newton matrix, and JF(w) has full row rank.These properties connect the Hessian-based iteration to the residual Jacobian structure.
G.3. Fast Local Convergence of Damped Newton’s Method: The local loss is uniformly smooth and satisfies a preconditioned PŁ⋆ condition with respect to the damped Hessian.The resulting descent bound uses the inverse damped Hessian as the local preconditioner.
G.3. Fast Local Convergence of Damped Newton’s Method: Starting from a sufficiently close point, damped Newton iterates remain in the local convergence region.The invariance argument combines one-step evolution with induction and uses γ = µ and η = (1 + ε)^-1.
G.3. Fast Local Convergence of Damped Newton’s Method: Theorem G.12 formalizes fast local convergence for the iteration using (HL(ŵk) + µI)^-1∇L(ŵk), after gradient descent has reached the prescribed neighborhood.The global-to-local theorem applies this Phase II result under the stated smoothness, PŁ⋆, and initialization assumptions.