Understanding and mitigating gradient pathologies in physics-informed neural networks

Sifan Wang; Yujun Teng; Paris Perdikaris

Source-linked AI summary

Understanding and mitigating gradient pathologies in physics-informed neural networks

Sifan Wang, Yujun Teng, Paris Perdikaris

arXiv:2001.04536v1Jan 13, 2020cs.LGmath.NAstat.ML

TL;DR

Physics-informed neural networks can struggle with PDE approximation because stiff gradient-flow dynamics produce unbalanced back-propagated gradients during training. This paper analyzes that failure mode and proposes adaptive learning-rate annealing plus a less-stiff architecture, reporting consistent 50-100x predictive-accuracy improvements across computational-physics problems.

Problem
Physics-informed neural networks often struggle to accurately approximate partial differential-equation solutions, motivating analysis of stiffness and gradient imbalance during training.
Method
The paper uses PINNs as a test-bed, analyzes gradient-flow stiffness, and proposes adaptive learning-rate annealing alongside a neural architecture with less stiffness than fully connected networks.
Results
50-100x: the proposed ideas consistently improve PINN predictive accuracy across a range of computational-physics problems.
Takeaways & Limitations
The developments provide new insights into training constrained neural networks and motivate more stable optimization discretizations for PINNs.
Takeaways & Limitations
For velocity-pressure representations, the formulation may fail to enforce prescribed boundary conditions or incompressibility, potentially causing non-uniqueness.

Abstract

from arXiv · show

The widespread use of neural networks across different scientific domains often involves constraining them to satisfy certain symmetries, conservation laws, or other domain knowledge. Such constraints are often imposed as soft penalties during model training and effectively act as domain-specific regularizers of the empirical risk loss. Physics-informed neural networks is an example of this philosophy in which the outputs of deep neural networks are constrained to approximately satisfy a given set of partial differential equations. In this work we review recent advances in scientific machine learning with a specific focus on the effectiveness of physics-informed neural networks in predicting outcomes of physical systems and discovering hidden physics from noisy data. We will also identify and analyze a fundamental mode of failure of such approaches that is related to numerical stiffness leading to unbalanced back-propagated gradients during model training. To address this limitation we present a learning rate annealing algorithm that utilizes gradient statistics during model training to balance the interplay between different terms in composite loss functions. We also propose a novel neural network architecture that is more resilient to such gradient pathologies. Taken together, our developments provide new insights into the training of constrained neural networks and consistently improve the predictive accuracy of physics-informed neural networks by a factor of 50-100x across a range of problems in computational physics. All code and data accompanying this manuscript are publicly available at \url{https://github.com/PredictiveIntelligenceLab/GradientPathologiesPINNs}.

1 Introduction

The paper studies how neural networks can incorporate prior knowledge through soft loss penalties, using physics-informed neural networks as a test-bed. It identifies gradient-flow stiffness as a training failure mode and proposes adaptive optimization and architectural strategies to improve predictive accuracy.

1 Introduction: Soft penalties incorporate domain knowledge by constraining neural-network outputs through regularized empirical-risk objectives.In physics-informed neural networks, the regularization functional encourages approximate satisfaction of specified partial differential equations.
1 Introduction: Physics-informed neural networks provide the paper’s test-bed for analyzing constrained neural networks trained with regularized loss functions.
1 Introduction: The analysis identifies stiffness in gradient-flow dynamics as a fundamental failure mode that creates unstable imbalance among back-propagated gradients.
1 Introduction: The paper proposes adaptive learning-rate annealing to balance data-fit and regularization terms during training.
1 Introduction: A novel neural-network architecture is proposed to reduce stiffness relative to conventional fully connected networks.
1 Introduction: 50-100x improvements in predictive accuracy are reported across a range of computational-physics problems.

2 Methods

The paper analyzes PINN failures caused by stiffness and imbalanced gradients, then proposes adaptive loss-term weighting and an architecture designed to reduce these pathologies.

Gradient pathologies: Standard PINNs can fail to approximate accurate solutions even for simple linear elliptic equations such as the Helmholtz equation.A four-layer, 50-neuron-per-layer model trained for 40,000 steps provides the benchmark setting.
Gradient pathologies: Monitoring individual loss terms reveals sharply imbalanced back-propagated gradients between the boundary and residual losses.The analysis tracks gradients of Lub(θ) and Lr(θ) separately across hidden layers rather than only the aggregate loss.
Gradient pathologies: Larger values of the Poisson-equation constant C produce larger residual-loss gradients and increasingly pronounced gradient imbalance.The imbalance biases training toward neglecting the boundary data-fit term.
Stiffness: Stiff gradient-flow dynamics can make gradient descent unstable, with residual-loss oscillations associated with very large Hessian eigenvalues.The residual-loss Hessian eigenvalues reach up to 10^5, while boundary-loss eigenvalues remain relatively small.
Learning-rate annealing: The proposed learning-rate annealing algorithm automatically tunes loss-term weights from back-propagated gradient statistics to balance data-fit and regularization.The procedure is intended to generalize to multiple loss terms, with small additional computational overhead and low sensitivity to α within [0.5, 0.9].
Neural architecture: The proposed architecture combines multiplicative interactions between input dimensions with residual connections and is designed to be less stiff than conventional fully connected networks.Its forward pass introduces transformer projections and point-wise multiplication while adding relatively small computational and memory overhead.

3 Results

The numerical studies identify severe gradient imbalance and show that adaptive loss weighting and an improved architecture substantially improve PINN accuracy on benchmark PDEs. Results vary by problem: the combined method performs best on Helmholtz and Klein–Gordon, while the tested cavity formulation remains unsuccessful.

Experimental setup: The studies compare four fully connected PINN approaches across PDE benchmarks using Adam optimization and standardized training settings.The models use hyperbolic tangent activations, mini-batches of 128, single precision, Glorot initialization, and no additional regularization.
Helmholtz equation: 8-31% relative L2 errors show that conventional model M1 lacks stable accuracy on the Helmholtz benchmark.The proposed models M2–M4 are more robust across architectures, and M3 improves accuracy over conventional fully connected networks.
Klein–Gordon equation: 17.9% relative L2 error accompanies M1’s failure to fit Klein–Gordon initial and boundary conditions when boundary and initial-condition gradients collapse near zero.The residual gradient remains comparatively flat while the boundary and initial-condition gradients form sharp peaks around the origin.
Klein–Gordon equation: A healthy gradient signal from all loss terms is observed for M2 after adaptive learning-rate annealing.The gradient distributions do not sharply peak around zero across hidden layers.
Klein–Gordon equation: 0.28% relative L2 prediction error is achieved by M4 on Klein–Gordon, about two orders of magnitude below M1 and one order below M2 and M3.M4 combines the proposed learning-rate annealing algorithm with the improved neural architecture; its maximum absolute error is 0.004.
Lid-driven cavity: None of the tested cavity models produces a reasonable approximation because the residual-loss formulation may not exactly enforce boundary conditions or incompressibility.The authors identify potential non-uniqueness and motivate a velocity representation based on a scalar potential.

4 Summary and Discussion

The paper frames PINN training failures as consequences of stiff gradient-flow dynamics and proposes adaptive weighting to mitigate unbalanced gradients. It also identifies unresolved questions about stiffness, approximation, generalization, and robust optimization.

Summary and Discussion: PINNs can struggle to approximate PDE solutions because stiff gradient-flow dynamics produce unbalanced back-propagated gradients.The paper quantifies and analyzes this stiffness to explain difficulties encountered during gradient-descent training.
Summary and Discussion: The learning-rate annealing algorithm uses training-time gradient statistics to adaptively assign weights to different terms in the PINN loss.Its stated aim is to address gradient imbalance and improve predictive accuracy.
Summary and Discussion: The authors state that rigorous understanding of PINN capabilities and limitations remains at an early stage.Open questions concern relations between PDE and gradient-flow stiffness, stiffness reduction, stable optimization, approximation error, and generalization error.

5 Appendix

The appendix presents proof steps and numerical procedures for fluid-flow computation, including stream-function and vorticity updates, boundary treatments, and convergence testing.

Proof: The proof proceeds by fixing neural-network parameters, rewriting the residual loss, integrating by parts, and combining the resulting expressions.The appendix includes the corresponding derivation steps for the loss-function result.
Numerical algorithm: The lid-driven-cavity solver uses stream function and vorticity variables to update the flow field iteratively.The Navier–Stokes formulation is recast using these variables before the numerical algorithm is applied.
Discretization: Spatial derivatives are computed on a uniform [0, 1] × [0, 1] grid using a second-order discretization scheme.Grid points are defined by xi = i/N and yi = j/N, with h = 1/N.
Boundary conditions: Boundary stream-function and vorticity conditions are derived from velocity conditions and Wood’s formula.The appendix defines local and adjacent stream-function/vorticity values and normal and tangential velocity components used in the boundary relation.
Numerical algorithm: The algorithm initializes interior stream-function and vorticity values, advances interior vorticity and stream function, updates boundary vorticity, and then updates velocity.If convergence is not reached, the procedure increments time and repeats the update cycle.
Numerical algorithm: A 128-point grid, time step δt = 1 × 10^3, and convergence criterion ε = 1 × 10^-4 define the simulation settings.The computation terminates when the maximum velocity error falls below ε.