Characterizing possible failure modes in physics-informed neural networks

Aditi S. Krishnapriyan; Amir Gholami; Shandian Zhe; Robert M. Kirby; Michael W. Mahoney

Source-linked AI summary

Characterizing possible failure modes in physics-informed neural networks

Aditi S. Krishnapriyan, Amir Gholami, Shandian Zhe, Robert M. Kirby, Michael W. Mahoney

arXiv:2109.01050v2Sep 2, 2021cs.LGcs.AImath.NAphysics.comp-ph

TL;DR

PINNs impose physical knowledge as soft PDE constraints, but the paper shows that this setup can fail on moderately complex problems despite sufficient network expressivity. Through analyses of representative PDEs, it identifies optimization-related failure modes and tests curriculum regularization and sequence-to-sequence learning, which achieve substantially lower errors than regular PINN training.

Problem
The paper asks why PINNs can fail to learn relevant physical phenomena for moderately complex PDE problems despite incorporating governing equations into the loss.
Method
The paper analyzes PINNs on convection, reaction, and diffusion-related problems, examines their loss landscapes, and evaluates curriculum regularization and sequence-to-sequence learning.
Results
Up to 1-2 orders of magnitude lower error is achieved with curriculum regularization and sequence-to-sequence learning than with regular PINN training.
Takeaways & Limitations
PINN failures can arise from optimization difficulties caused by soft PDE constraints rather than from insufficient neural-network expressivity.
Takeaways & Limitations
The tested approaches are promising, but more sophisticated timestep prediction, sequence-to-sequence methods, and regularization tuning may improve performance further.

Abstract

from arXiv · show

Recent work in scientific machine learning has developed so-called physics-informed neural network (PINN) models. The typical approach is to incorporate physical domain knowledge as soft constraints on an empirical loss function and use existing machine learning methodologies to train the model. We demonstrate that, while existing PINN methodologies can learn good models for relatively trivial problems, they can easily fail to learn relevant physical phenomena for even slightly more complex problems. In particular, we analyze several distinct situations of widespread physical interest, including learning differential equations with convection, reaction, and diffusion operators. We provide evidence that the soft regularization in PINNs, which involves PDE-based differential operators, can introduce a number of subtle problems, including making the problem more ill-conditioned. Importantly, we show that these possible failure modes are not due to the lack of expressivity in the NN architecture, but that the PINN's setup makes the loss landscape very hard to optimize. We then describe two promising solutions to address these failure modes. The first approach is to use curriculum regularization, where the PINN's loss term starts from a simple PDE regularization, and becomes progressively more complex as the NN gets trained. The second approach is to pose the problem as a sequence-to-sequence learning task, rather than learning to predict the entire space-time at once. Extensive testing shows that we can achieve up to 1-2 orders of magnitude lower error with these methods as compared to regular PINN training.

1 Introduction

The paper identifies failure modes in PINNs that arise when PDE knowledge is imposed as a soft constraint, even though the neural network may be expressive enough to represent accurate solutions. It analyzes these optimization difficulties and proposes curriculum regularization and sequence-to-sequence learning as alternatives.

PINN formulation: PINNs incorporate PDE knowledge by adding a soft residual constraint to a data-fit loss, but this does not guarantee that governing equations are obeyed.The residual penalty is controlled by λF, while initial and boundary conditions contribute separate misfit terms.
PINN formulation: PDE-based regularization can be ill-conditioned and differs fundamentally from simple L1 or L2 regularization because it represents structured physical quantities.Approximate satisfaction of a physical constraint may produce qualitatively different or unrealistic solutions than exact satisfaction.
Failure modes: Vanilla PINNs work only in easy parameter regimes and can reach almost 100% error on moderately challenging convection, reaction, and reaction-diffusion problems.These failures occur even when the problems have simple closed-form analytical solutions and after extensive hyperparameter tuning.
Failure modes: Increasing PDE soft-constraint regularization makes the loss landscape more complex and harder to optimize, while reducing it produces high-error solutions that violate the PDE.The difficulty is especially pronounced for problems with non-trivial coefficients.
Failure modes: The failures are attributed to optimization difficulties in the PINN setup rather than insufficient neural-network expressivity.The network architecture has the capacity to find a good solution.
Proposed remedies: Curriculum regularization and sequence-to-sequence learning reduce error by up to 1-2 orders of magnitude compared with regular PINN training.Curriculum regularization progressively increases PDE difficulty, while sequence-to-sequence learning predicts smaller time segments and can better capture sharp solution features.

2 Related work

Related work combines machine learning with PDEs and physical constraints, including regularization-based PINNs and architectures that encode conservation or multiscale structure. Prior studies also report failures involving stiff dynamics, heterogeneous media, fluid flows, high-frequency targets, and gradient imbalance.

Machine learning and PDEs: Machine-learning approaches increasingly combine data-driven methods with PDE knowledge, including regularization terms based on governing equations.Some approaches can train using data generated by the governing equations, with additional data optionally included.
Machine learning and PDEs: PINN-related formulations have shown weaknesses for stiff chemical-kinetics ODEs, heterogeneous media, and certain fluid-flow problems.These examples extend reported difficulties beyond the specific PDE settings analyzed in this paper.
Machine learning and PDEs: Neural-tangent-kernel analyses identify failures for targets with high-frequency features and motivate solutions based on that perspective.Other work attributes some problems to imbalanced back-propagated gradients and proposes learning-rate annealing.
Physical priors and constraints in NNs: Physical priors and constraints are commonly imposed in scientific machine learning to enforce properties such as energy conservation, momentum conservation, or multiscale structure.These approaches include methods that embed specialized constraints directly into neural networks.

3 Possible failure modes for physics-informed neural networks

Across convection and reaction-diffusion examples, vanilla PINNs learn only easy parameter regimes and can fail to capture relevant physical behavior in moderately challenging settings. These failures occur despite analytical solutions and extensive training, with errors becoming very large as PDE coefficients increase.

Failure across physical systems: Vanilla PINNs learn simple problems with small convection, reaction, or diffusion coefficients but fail in moderately challenging physical regimes.The study examines linear and nonlinear systems with varied PDE coefficients and periodic boundary conditions.
Failure across physical systems: Almost 100% error can occur for vanilla PINNs on non-trivial parameter regimes, even after extensive hyperparameter tuning.The reported errors are measured against analytical solutions using L2 relative and absolute errors, averaged across repeated runs.
3.1 Learning convection: For 1D convection, relative error reaches almost 100% when β > 10, although the network can fit the boundary conditions.The PINN performs well for small convection coefficients but fails as β increases.
3.2 Learning reaction-diffusion: For reaction-diffusion with ρ = 5 and ν = 5, the PINN achieves 93% relative error and misses both reaction and diffusion behavior.At ν = 2, relative error is 50%, with sharper transitions remaining unresolved.

4 Diagnosing possible failure modes for physics-informed NNs

The paper diagnoses PINN failures as optimization difficulties caused by PDE-based soft regularization. Increasing the regularization or using more challenging coefficients produces more complex loss landscapes, while reducing regularization can yield high-error solutions that violate the PDE constraint.

Underlying optimization difficulty: PDE-based soft regularization makes the PINN loss landscape harder to optimize and can produce ill-conditioning.The analysis links this difficulty to regularizing with differential operators.
Loss-landscape analysis: At β = 1 the convection loss landscape is relatively smooth, whereas larger β values produce complex, non-symmetric landscapes with high-loss local minima.The optimizer becomes stuck in a local minimum for large β values.
Effect of regularization strength: Increasing the soft-regularization weight makes the loss landscape increasingly complex and harder to optimize, while tuning it does not resolve the problem.Reducing regularization can alleviate landscape complexity but leads to poor solutions that do not satisfy the PDE constraint.

5 Expressivity versus optimization difficulty

The paper argues that vanilla PINN failures arise from optimization difficulty rather than insufficient neural-network expressivity. It presents curriculum regularization and sequence-to-sequence learning as alternatives that substantially reduce error on challenging PDE regimes.

Expressivity versus optimization difficulty: The neural network can represent solutions that vanilla PINNs fail to learn, implicating optimization difficulty rather than architectural capacity.The proposed alternatives are designed to make optimization easier or to decompose the prediction task.
Curriculum regularization: Curriculum regularization warm-starts training at low β or ρ values, then progressively increases the coefficients toward the target problem.Weights from an easier problem reinitialize training for the next, more difficult problem.
Curriculum regularization: Almost two orders of magnitude lower relative error is achieved by curriculum regularization for the convection example with β = 30.The improvement is also reported across other regimes where regular PINNs fail, with reduced error variance.
Sequence-to-sequence learning: Seq2seq learning predicts one time step at a time and feeds each predicted state into the next segment as its initial condition.The comparison keeps the total number of collocation points the same as whole-space-time PINN training.
Sequence-to-sequence learning: Almost two orders of magnitude lower error is obtained with seq2seq learning for reaction and reaction-diffusion cases than with regular PINNs.The approach recovers solutions that regular PINNs fail to capture, including sharp or diffusive features.
Open directions: Further improvements may come from more sophisticated timestep prediction, seq2seq methods, and tuning the regularization parameter.These are identified as promising directions beyond the initial results.

6 Conclusions

The paper concludes that soft PDE regularization can make PINNs fail on moderately challenging convection, reaction, and diffusion problems despite the promise of combining machine learning with physical constraints. It attributes these failures partly to ill-conditioned, complex optimization landscapes and evaluates curriculum and sequential prediction as remedies.

Conclusions: The paper frames curriculum regularization and seq2seq learning as two ways to address failure modes caused by difficult PINN optimization.Both methods alter how the PDE-constrained learning problem is presented to the network.
Conclusions: Vanilla PINNs work only in easy parameter regimes and can approach 100% error on moderately challenging convection, reaction, and reaction-diffusion problems.These failures occur even for problems with simple closed-form analytical solutions and after extensive hyperparameter tuning.
Reaction example: The reaction example shows that PINN error quickly reaches 100% as the reaction coefficient ρ increases.At failure, the model predicts a mostly homogeneous solution close to zero rather than the target structure.
Conclusions: Soft PDE regularization can make PINN optimization ill-conditioned because its differential operator is structurally different from ordinary norm-based regularization.The PDE operator may produce unstable numerical behavior, large gradients, and poor convergence.
Conclusions: For diffusion and convection, the regularization operator’s condition number scales as O(νN^2)^2 and O(βN)^2, respectively.The paper notes that condition number is only one factor alongside function complexity, non-convexity, and optimizer limitations.

B.1 Approximate condition number scaling for PDE regularization in PINNs

The PINN regularization operator has an approximate condition number whose scaling depends on the PDE type, with diffusion becoming more sensitive to grid size than convection. These estimates are approximate because the nonlinear regularization depends on the state variable and its derivatives.

Approximate condition number derivation: The condition-number derivation is approximate because the nonlinear regularization makes its behavior depend on the state variable and its derivatives.The condition number measures how much an operator’s output changes when its input changes.
Convection problem: For convection, the condition number scales as O(βN), reflecting contributions from temporal variation and the β-weighted spatial derivative.The temporal contribution scales as O(δt^-1), while the spatial contribution scales as O(βh^-1) with h = 1/N.
Diffusion problem: For diffusion, the condition number scales as O(νN^2), because the diffusion contribution scales as O(νh^-2).The L2 loss quadratically scales the corresponding output change.
Comparison: Diffusion error increases more rapidly with ν than convection error with β, consistent with diffusion scaling as N^2 rather than N.The reported comparison concerns the approximate condition-number estimates and observed error behavior.

E.1 Extra results for loss landscapes when varying the λ parameter

For the 1D convection example at β = 30, increasing the regularization weight λ makes the loss landscape more complex while error remains consistently high. Error decreases only slightly as λ increases.

Loss landscape: At β = 30, increasing λ makes the loss landscape more complex as the PDE regularization term grows.β = 30 is identified as a point at which the error is high.
Error: Error remains consistently high across λ values, although it decreases slightly as λ increases.The figure reports this behavior for the 1D convection equation.

E.2 Extra curriculum regularization results

Additional experiments show that curriculum regularization improves both solution accuracy and error stability relative to regular PINN training. It also smooths the loss landscape and helps capture sharp reaction features.

Convection: Curriculum regularization achieves significantly lower error and lower error variance across 10 preset random seeds than regular training.The comparison uses the lowest error per seed.
Loss landscape: Curriculum regularization produces a much smoother loss landscape than regular PINN training.
Reaction: For 1D reaction, curriculum training captures sharp solution features and yields 1-2 orders of magnitude lower error than regular training.The sharp features are described as traditionally hard for PINNs to capture.

E.3 Extra sequence-to-sequence learning results

The additional sequence-to-sequence experiments compare predicting the entire state space at once with discretized-state-space learning. Sequence-to-sequence learning achieves lower error for both tested time steps in convection and reaction.

Convection: For 1D convection, sequence-to-sequence learning achieves lower error than predicting the entire state space at once for Δt = 0.05 and Δt = 0.1.The experiment uses 10000 collocation points for both regular PINNs and sequence-to-sequence learning.
Reaction: For 1D reaction, sequence-to-sequence learning achieves lower error than predicting the entire state space at once for Δt = 0.05 and Δt = 0.1.