When and why PINNs fail to train: A neural tangent kernel perspective

Sifan Wang; Xinling Yu; Paris Perdikaris

Source-linked AI summary

When and why PINNs fail to train: A neural tangent kernel perspective

Sifan Wang, Xinling Yu, Paris Perdikaris

arXiv:2007.14527v1Jul 28, 2020cs.LGmath.NAstat.ML

TL;DR

PINNs can fail to train, especially when loss components converge at different rates, but the mechanisms governing this behavior are not well understood. This paper derives the PINN NTK and uses its spectrum to analyze training, showing a deterministic infinite-width kernel and proposing adaptive gradient descent; the theory and algorithms are evaluated numerically.

Problem
The training behavior of PINNs under gradient descent, including why they sometimes fail to train, is insufficiently understood.
Method
The paper derives the NTK of PINNs, analyzes training through its limiting spectrum, and proposes an adaptive gradient-descent strategy.
Results
The PINN NTK converges to a deterministic kernel that remains nearly constant at infinite width, while different loss components exhibit discrepant convergence rates.
Takeaways & Limitations
NTK-based analysis enables adaptive training algorithms that improve PINN trainability, predictive accuracy, and robustness.
Takeaways & Limitations
The theory is not yet established for multilayer extensions, nonlinear equations, alternative architectures, or inverse problems.

Abstract

from arXiv · show

Physics-informed neural networks (PINNs) have lately received great attention thanks to their flexibility in tackling a wide range of forward and inverse problems involving partial differential equations. However, despite their noticeable empirical success, little is known about how such constrained neural networks behave during their training via gradient descent. More importantly, even less is known about why such models sometimes fail to train at all. In this work, we aim to investigate these questions through the lens of the Neural Tangent Kernel (NTK); a kernel that captures the behavior of fully-connected neural networks in the infinite width limit during training via gradient descent. Specifically, we derive the NTK of PINNs and prove that, under appropriate conditions, it converges to a deterministic kernel that stays constant during training in the infinite-width limit. This allows us to analyze the training dynamics of PINNs through the lens of their limiting NTK and find a remarkable discrepancy in the convergence rate of the different loss components contributing to the total training error. To address this fundamental pathology, we propose a novel gradient descent algorithm that utilizes the eigenvalues of the NTK to adaptively calibrate the convergence rate of the total training error. Finally, we perform a series of numerical experiments to verify the correctness of our theory and the practical effectiveness of the proposed algorithms. The data and code accompanying this manuscript are publicly available at \url{https://github.com/PredictiveIntelligenceLab/PINNsNTK}.

1 Introduction

PINNs have achieved broad empirical success, but fully connected models can train unstably and struggle with high-frequency or multiscale solutions. This work uses NTK theory to analyze these training dynamics and motivate adaptive algorithms.

PINNs have produced notable results across computational science and engineering problems, including fluids, bio-engineering, metamaterials, uncertainty quantification, and high-dimensional PDEs.
Fully connected PINNs often fail to train stably or predict accurately when PDE solutions contain high-frequency or multiscale features.
Multi-scale interactions among PINN loss terms can create stiffness in gradient-flow dynamics and impose stringent learning-rate stability requirements.
Fully connected networks exhibit spectral bias and are theoretically and empirically limited in learning high-frequency functions.
The paper studies fully connected PINNs through their NTK to determine when training succeeds or fails and to develop improved training algorithms.

2 Infinitely Wide Neural Networks

The paper formulates fully connected networks under NTK parameterization and analyzes their infinite-width limit. At initialization and throughout fixed training intervals, the NTK converges to a deterministic kernel, making training equivalent to kernel regression.

A fully connected network with L hidden layers is defined recursively using weight matrices, bias vectors, and a smooth coordinate-wise activation function.
The NTK parameterization treats all network weights and biases as the trainable parameter collection θ.
With i.i.d. standard-normal initialization and hidden widths tending to infinity, hidden-layer coordinates converge to centered Gaussian processes with recursively defined covariances.
The NTK is defined while parameters evolve under gradient descent and converges in probability at random initialization to a deterministic kernel as widths grow.
For suitable conditions and fixed training time, the tangent kernel remains close to its limiting kernel, so sufficiently wide networks trained by gradient descent are equivalent to deterministic-kernel regression.

3 Physics-informed Neural Networks (PINNs)

PINNs approximate PDE solutions with neural networks and train them using a composite loss combining boundary-condition and PDE-residual information. The paper derives coupled evolution equations and defines the PINN NTK from parameter gradients.

The PINN problem consists of a differential equation L[u](x) = f(x) on a bounded domain and a boundary condition u(x) = g(x) on its boundary.
For time-dependent problems, time is treated as an additional coordinate, and the initial condition is included as a Dirichlet boundary condition.
A deep neural network u(x, θ) approximates the latent solution, while the PDE residual r(x, θ) is used to construct the training objective.
The boundary and residual data use batch sizes Nb and Nr, which may be randomly sampled at each gradient-descent iteration.
Under infinitesimal-learning-rate gradient flow, the solution and PDE residual evolve through coupled kernel blocks Kuu(t), Kur(t), Kru(t), and Krr(t).
The PINN NTK is the matrix K(t) formed from parameter gradients, with the associated kernel matrices positive semidefinite for any differential operator and network architecture.

4 Analyzing the training dynamics of PINNs through the lens of their NTK

For linear PDEs, sufficiently wide fully connected PINNs admit Gaussian-process and deterministic-NTK descriptions. Under boundedness, smoothness, and infinitesimal-learning-rate assumptions, the NTK remains nearly constant during training, linking training dynamics to kernel regression while exposing practical degeneracy.

Model problem: A one-hidden-layer PINN for the one-dimensional Poisson problem approximates u(x) with a fully connected network and analyzes both function values and PDE residuals.The model uses a smooth activation and explicitly represents the network parameters, including weights and biases.
Infinite-width behavior: At infinite width, the network output and its second derivative converge to centered Gaussian processes under suitable smoothness assumptions.The result extends by induction to higher-order differential operators, multiple hidden layers, and linear PDE operators under appropriate regularity conditions.
PINN NTK: At initialization, the PINN NTK converges in probability to a deterministic limiting kernel as the network width tends to infinity.The limiting kernel is expressed through blocks associated with function values and residuals.
Training dynamics: Under uniformly bounded parameters and loss, bounded activation derivatives through fourth order, and infinitesimal learning rates, the NTK stays asymptotically constant during training.The proof intuition is that small parameter movement in sufficiently wide networks leaves output gradients, and therefore the kernel, nearly unchanged.
Kernel-regression correspondence: If the limiting NTK is invertible, infinitely wide PINNs for the model problem are equivalent to kernel regression after gradient descent.The authors caution that their experience finds PINN NTKs often degenerate, limiting casual use of kernel-regression predictions in practice.

5 Spectral bias in physics-informed neural networks

The NTK spectrum explains spectral bias in PINNs: rapidly decaying eigenvalues make high-frequency target components converge slowly. A further imbalance between residual and boundary-condition kernels can make the residual fit much faster than the boundary conditions.

Spectral bias: Spectral bias is the pathology in which deep fully connected networks struggle to learn high-frequency functions, and the section tests whether PINNs share it.The analysis uses the NTK to connect kernel spectra with PINN learning behavior.
Spectral bias: NTK eigenvalues determine componentwise error decay at approximately e^(-λ_i t), so larger-eigenvalue components are learned faster.For fully connected networks, higher-eigenvalue eigenvectors generally correspond to lower frequencies.
Spectral bias: Rapidly decaying PINN NTK eigenvalues produce extremely slow convergence of high-frequency target components, demonstrating spectral bias.The eigenvalue distribution is reported to decay rapidly, with many eigenvalues near zero in the numerical example.
Componentwise convergence: The total training-error convergence depends jointly on Kuu and Krr, whereas separate function and residual errors depend on their corresponding kernel blocks.The paper defines one kernel as dominating another when its average convergence rate is much larger.
Componentwise convergence: When Krr has much larger eigenvalues than Kuu, the PDE residual converges faster than boundary-condition fitting, potentially preventing the correct solution.The authors identify this discrepancy as a fundamental reason PINNs can fail to train accurately, while noting that Kuu can dominate in some cases.

6 Practical insights

The practical strategy is to calibrate loss weights using NTK spectral information so boundary and residual components converge at comparable rates. The analysis also relates weights and batch sizes to stability and convergence in general PDE settings.

Practical setup: For general PDEs, PINNs approximate the latent solution with a multilayer fully connected network and minimize a composite loss over boundary and residual terms.The loss includes tunable weights and may use full batches or randomly sampled mini-batches.
Stability and general PDEs: Gradient-descent stability requires the learning rate to be at most 2/λ_max of the effective kernel, while larger batches can provide an alternative stability mechanism.For nonlinear PDEs, singular values rather than eigenvalues characterize training-error convergence because the effective kernel need not be positive semidefinite.
Adaptive calibration: Increasing a loss weight changes the corresponding NTK singular values and improves convergence of its associated training-error component.Changing a boundary or residual weight is equivalent, in convergence-rate terms, to changing the corresponding batch size.
Adaptive calibration: The proposed calibration adjusts λ_b and λ_r so boundary and PDE-residual components have similar convergence-rate magnitudes.The method fixes batch sizes and changes weights because very large batches may impose extra computational costs.
Implementation: Adaptive updates may occur every iteration or at a user-specified frequency, and traces can replace full eigenvalue sums to reduce computation.The NTK is computed using the training points supplied at each iteration.
Relation to prior weighting: The NTK-based weighting approach supplies theoretical justification and training-dynamics insight that earlier gradient-statistics weighting lacked.The comparison is explicitly made with Wang et al.’s empirical approach for tuning boundary and residual weights.

7 Numerical Experiments

Numerical experiments test NTK convergence and adaptive loss weighting across Poisson and wave equations. The adaptive scheme substantially improves solution accuracy by balancing loss-component convergence.

Experimental setup: The experiments use fully connected PINNs with NTK parameterization and hyperbolic tangent activations, generally trained by stochastic gradient descent.The study spans one-dimensional Poisson and wave-equation problems with specified network depths, widths, collocation points, and training schedules.
7.1 Convergence of the NTK of PINNs: As network width increases, the relative changes in weights and the NTK decrease, supporting an almost fixed NTK during training for one-hidden-layer PINNs.For width 500, eigenvalues are compared at initialization and after 10,000 gradient-descent steps; the observed behavior is consistent with the stated convergence theory.
7.1 Convergence of the NTK of PINNs: Three-hidden-layer networks show similar weight, kernel, and eigenvalue behavior, although multiple-hidden-layer PINNs are not covered by the current theory.The similarity motivates a conjecture extending deterministic, training-invariant NTK behavior beyond the proved setting.
7.2 Adaptive training for PINNs: Adaptive boundary weighting improves the Poisson relative L2 error from 2.40e −01 to 1.63e −03 after 40,000 iterations.With λr = 1, the theoretically suggested boundary weight is approximately λb = 100; errors decrease toward a local minimum near that value and then oscillate.
7.3 One-dimensional wave equation: For the one-dimensional wave equation, standard PINNs exhibit faster convergence of residual and temporal-derivative terms than boundary conditions, producing severe loss-component imbalance.The eigenvalues of Ku, Kut, and Kr move outward initially and then become nearly static, with Kr and Kut dominating Ku.
7.3 One-dimensional wave equation: Algorithm 1 reduces the wave-equation relative L2 error to 1.73e −3 after 80,000 iterations and yields a stable overall training process.The adaptive weights are updated every 1,000 iterations, adding negligible computational cost relative to standard PINNs.

8 Discussion

The work develops an NTK-based understanding of PINN training dynamics and proposes dynamically weighted loss terms to address unequal convergence rates. It also identifies open questions concerning architectures, equations, inverse problems, optimizers, and the limits of loss weighting.

The paper derives and analyzes the limiting NTK of PINNs to provide a theoretical understanding of their training dynamics.
Fully connected PINNs exhibit spectral bias and unequal convergence rates among loss components contributing to total training error.
The proposed algorithm dynamically updates loss coefficients to balance the average convergence rates of different error components.
The theory raises open questions about extensions to deeper or nonlinear networks, alternative architectures, inverse problems, and momentum-based optimization.
Adjusting loss weights improves PINN performance in practice but cannot change the NTK eigenvalue distribution or directly address its associated limitations.

A Proof of Lemma 3.1

The proof formulates PINN gradient-flow dynamics as coupled boundary and residual error equations governed by block components of the time-dependent NTK.

The proof introduces the corresponding gradient flow for the PINN outputs.
The dynamics are rewritten using boundary and residual output components together with the corresponding NTK blocks.
The equations couple boundary errors through Kuu and Kur, and residual errors through Kru and Krr.

B Proof of Theorem 4.1

The proof establishes Gaussian-process convergence for network outputs and controls the required derivatives under bounded smooth-activation assumptions.

With independently initialized Gaussian weights and biases, the network output converges in distribution to a centered Gaussian process as width tends to infinity.
The proof assumes bounded second derivatives of the activation function to obtain uniform integrability of uxx(x, θ).
The argument concludes after establishing the required convergence properties for the differentiated network output.

C Proof of Theorem 4.3

The proof computes the infinite-width limits of the PINN NTK blocks using Gaussian-process covariance recursions, activation derivatives, and symmetry between cross-blocks.

The proof first computes Kuu(0) and its infinite-width limit using the established fully connected network kernel.
The network is parameterized by weights, biases, and the activation applied to the first hidden layer.
Law-of-large-numbers limits connect the network covariance and derivative covariance to the recursively defined quantities Σ(1)(x, x′) and ˙Σ(1)(x, x′).
The proof combines these limits to obtain the Kuu block and confirms consistency with the earlier kernel formula.
The residual-residual block Krr(0) is computed next using derivatives of the activation function up to third order.
The cross-block Kur is obtained from its defining relation to Kru and the corresponding infinite-width limit.

D Proof of Theorem 4.4

The proof of Theorem 4.4 establishes the required derivative and Jacobian bounds through a sequence of lemmas, assumptions, and finite-data arguments. It then uses these bounds to conclude the theorem in the infinite-width limit.

Assumptions: The argument assumes a bounded domain and a smooth activation function whose derivatives through fourth order are uniformly bounded.The proof explicitly invokes the boundedness of σ^(k) for k = 0, 1, 2, 3, 4.
Auxiliary lemmas: The proof first establishes auxiliary lemmas under Theorem 4.4’s setting for the loss function and its derivative-related quantities.These lemmas are combined through estimates involving the model problem and gradient-descent loss dynamics.
Conclusion: The proof concludes after combining the assumptions and lemmas to establish the stated result of Theorem 4.4.Several intermediate quantities are decomposed and bounded before the final conclusion.
Main theorem: The proof controls the Jacobians of u(x_b, θ(t)) and u_xx(x_r, θ) with respect to the parameters θ.It uses Lemma D.1 to obtain boundedness and repeats the calculation for the residual Jacobian J_r.
Main theorem: Because the training data are finite, the convergence argument reduces to considering pairs of inputs and applying Cauchy–Schwarz together with the preceding uniform bounds.The resulting estimates are taken as N →∞ and are stated uniformly for t ∈ [0, T].