PaperlayerView exact revision on arXiv

Source-linked AI summary

Mitigating Propagation Failures in Physics-informed Neural Networks using Retain-Resample-Release (R3) Sampling

Arka Daw, Jie Bu, Sifan Wang, Paris Perdikaris, Anuj Karpatne

arXiv:2207.02338v3cs.LGcs.AI

TL;DR

PINNs can fail to converge on complicated PDEs, while the connection between failure modes and sampling strategies has been insufficiently understood. This paper proposes R3 sampling, which targets high-residual regions, and a causal extension for time-dependent PDEs. Across benchmark PDEs, the methods mitigate propagation failures and achieve correct solutions with smaller sample sizes and negligible computational overhead.

  • Problem

    The connection between PINN failure modes and collocation-point sampling has been largely overlooked, despite failures to converge to correct solutions for complicated PDEs.

    Source: §1 · p. 1

  • Method

    R3 adaptively accumulates collocation points in high-residual regions, while Causal R3 extends the approach to encode causality for time-dependent PDEs.

    Source: §1 · p. 2, §4 · p. 5, §4.2 · p. 6

  • Results

    R3 and Causal R3 mitigate propagation failures and converge to correct solutions with significantly smaller sample sizes than baselines and negligible computational overhead.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across benchmark PDEs, R3 empirically demonstrates better performance and mitigates propagation failures.

    Source: §1 · p. 2, §6 · p. 9

    Prior methods such as RAR require dense collocation sets, making residual evaluation expensive and training costs grow as the training set expands.

    Source: §3 · p. 4

Abstract

from arXiv · show

Despite the success of physics-informed neural networks (PINNs) in approximating partial differential equations (PDEs), PINNs can sometimes fail to converge to the correct solution in problems involving complicated PDEs. This is reflected in several recent studies on characterizing the "failure modes" of PINNs, although a thorough understanding of the connection between PINN failure modes and sampling strategies is missing. In this paper, we provide a novel perspective of failure modes of PINNs by hypothesizing that training PINNs relies on successful "propagation" of solution from initial and/or boundary condition points to interior points. We show that PINNs with poor sampling strategies can get stuck at trivial solutions if there are propagation failures, characterized by highly imbalanced PDE residual fields. To mitigate propagation failures, we propose a novel Retain-Resample-Release sampling (R3) algorithm that can incrementally accumulate collocation points in regions of high PDE residuals with little to no computational overhead. We provide an extension of R3 sampling to respect the principle of causality while solving time-dependent PDEs. We theoretically analyze the behavior of R3 sampling and empirically demonstrate its efficacy and efficiency in comparison with baselines on a variety of PDE problems.

1. Introduction

PINNs can fail on complicated PDEs, and the role of collocation-point sampling in these failures has been largely overlooked. The paper frames one failure mode as interrupted propagation from initial or boundary points to interior points, motivating R3 sampling.

1. Introduction

  • PINNs can get stuck at trivial solutions despite zero PDE residuals, while prior work has largely overlooked how collocation-point sampling relates to failure modes.Earlier studies focused mainly on network architectures or loss balancing; some sampling approaches incur large computational costs or fail to converge to correct solutions.Source: §1 · p. 1
  • The propagation hypothesis holds that correct solutions must move from initial or boundary points into the interior to prevent convergence to trivial solutions.When propagation is hindered, PINNs can become stuck at trivial interior solutions that are difficult to escape.Source: §1 · p. 1
  • Propagation failures are characterized by highly imbalanced PDE residual fields, making high-residual regions difficult to represent adequately among collocation points.This motivates dynamically focusing collocation points on those regions during training.Source: §1 · p. 2
  • R3 accumulates collocation points in high-residual regions, retaining persistent ones, releasing resolved ones, and maintaining uniform resampling across the domain.The paper also extends R3 with Causal R3 to encode causality when solving time-dependent PDEs.Source: §1 · p. 2
  • R3 and Causal R3 mitigate propagation failures and converge to correct solutions with significantly smaller sample sizes than baselines, with negligible computational overhead.The paper also theoretically proves R3's Retain, Resample, and Release properties.Source: §1 · p. 2

2. Background and Related Work

The background introduces PINNs as neural networks trained against initial or boundary data and PDE residuals at interior collocation points. It reviews prior explanations of PINN failures, the limited attention to sampling, and a simple ODE example of solution propagation.

2. Background and Related Work

  • PINNs use a neural network to infer a PDE solution, enforcing the PDE over the spatio-temporal domain through residuals at collocation points.The overall loss combines PDE-residual, boundary-data, and initial-data losses; this paper focuses on forward problems.Source: §2 · p. 2
  • Prior failure studies link poor convergence to imbalanced loss gradients, differing loss-term convergence rates, large PDE coefficients, and attraction to trivial solutions.The effect of collocation-point sampling on failure modes had largely been overlooked.Source: §2 · p. 2
  • In the simple ODE example, the correct solution propagates smoothly from the boundary point into interior points as training iterations increase.The example uses 1,000 equispaced collocation points and the boundary condition u(−π/2) = 0.Source: Figure 1 · p. 3
  • PINNs can be attracted to fixed points—roots of the nonlinear PDE function—and become trapped in local minima, causing premature convergence.The trivial solution u = 0 is a special case of a fixed point.Source: §2 · p. 3

3. Propagation Hypothesis

PINN training is difficult because correct labels are confined to initial and boundary points, while interior points may be distant and residual minimization can admit trivial solutions. The paper hypothesizes that correct solutions must propagate from those points into the interior.

3. Propagation Hypothesis

  • Unlike supervised learning, PINNs lack correct labels for interior points, and minimizing PDE residuals alone can leave a network at a trivial solution.Because PDE residuals depend on gradients with respect to space and time, a collocation point’s solution is affected by nearby points.Source: §3 · p. 3
  • The propagation hypothesis holds that PINNs need the correct solution to spread from initial or boundary points to interior points during training.The authors draw an analogy to finite-difference methods that iteratively propagate PDE solutions from initial or boundary points.Source: §3 · p. 3
  • Sharp solution discontinuities at boundaries between correct and trivial-solution regions create narrow, high-residual zones that the authors use to diagnose propagation failure.They describe these zones as potentially disrupting information flow and causing errors to cascade.Source: §3 · p. 3
  • For β=50, skewness and kurtosis rise significantly while relative error stays flat despite decreasing PDE loss; at β=10, both metrics remain relatively small and error decreases.The figures compare residual-field imbalance and solution behavior for the two convection-equation settings.Source: §3 · pp. 3–4, Figure 2 · p. 4

4. Proposed R3 Sampling Algorithm

R3 adaptively retains collocation points in high-residual regions, replaces other points with uniform resamples, and maintains a fixed population without a dense search set. Its theoretical properties describe retention, resampling, and release, while Causal R3 adds time-aware loss and sampling for time-dependent PDEs.

4. Proposed R3 Sampling Algorithm

  • R3 identifies high-residual regions using only a small set of collocation points per iteration, avoiding a dense collocation set and adding little to no sampling overhead.Source: §4 · p. 5
  • R3 retains collocation points whose absolute PDE residual exceeds the population mean, replacing lower-residual points with uniform-domain resamples while keeping population size fixed.The threshold is the expected residual over the current population.Source: §4 · p. 5

4.1. Theoretical Validations of R3 Sampling properties

  • For a fixed residual function, the expected residual of the retained population approaches the maximum as iterations increase, establishing the Retain property.The theorem applies to a fixed Lipschitz-continuous objective.Source: §4.1 · p. 6
  • The resampled population is always non-empty, preserving uniform-distribution support and preventing the collocation population from collapsing entirely onto high-residual regions.Source: §4.1 · p. 6
  • Points are released when their residual falls below the iteration's mean-residual threshold, freeing the population to focus on other regions.Source: §4.1 · p. 6

4.2. Causal Extension of R3 Sampling (Causal R3)

  • Causal R3 uses a time-dependent gate to weight PDE loss, initially revealing a small portion of the time domain and gradually revealing more as training proceeds.For γ ≥ 1.5, the entire time domain is revealed.Source: §4.2 · pp. 6–7
  • Causal R3 modifies the sampling residual by multiplying its absolute PDE residual by the gate value at the collocation point's time.Source: §4.2 · p. 7
  • The gate shifts forward more slowly when causally weighted PDE loss is large, and at convergence it reveals the entire time domain.Source: §4.2 · p. 7

5. Results

Across the benchmark PDEs, R3 and Causal R3 report the lowest errors, while R3 also shows stronger sampling efficiency at small collocation counts. The results include residual trends, accumulation behavior, and Eikonal reconstruction quality.

Comparing Prediction Performance

  • R3 and Causal R3 consistently show the lowest errors across the benchmark PDEs, while several baselines fail on most PDEs.PINN-dynamic still does not converge to low errors for convection with β = 50.Source: §5 · pp. 7–8
  • For convection with β = 50, R3 and PINN-dynamic maintain low maximum residuals, while R3 reduces mean residuals further.Figure 6 compares maximum and mean PDE residuals over training iterations for PINN-fixed, PINN-dynamic, and R3.Source: Figure 6 · p. 7, §5 · p. 8

Accumulation Behavior of R3 Sampling

  • As training progresses, R3's retained-population expected loss moves from L2 toward higher-order Lp norms and approaches L∞.The accumulation demonstration uses two fixed objective functions.Source: §5 · p. 8, Figure 7 · p. 8

Sampling Efficiency

  • At Nr = 100, R3 and Causal R3 maintain low errors and show two orders of magnitude better sampling efficiency than baselines.Baselines perform similarly when Nr exceeds 10K; Nr is directly related to training compute and memory requirements.Source: §5 · p. 8, Figure 8 · p. 9
  • On the regular KS equation, R3 and Causal R3 improve over CPINN at Nr = 128, and Causal R3 performs better as collocation points increase.The comparison is presented in Figure 9.Source: §5 · p. 8, Figure 9 · p. 9

Solving Eikonal Equations

  • For the Eikonal equation, R3 captures fine SDF details on both geometries and has significantly higher mIOU than the baselines.PINN-fixed performs poorly, while PINN-dynamic captures most of the outline with some details missing.Source: §5 · pp. 8–9

6. Conclusions And Future Work Directions

The conclusion presents R3 as a way to mitigate propagation failures, and the accompanying analyses relate residual-focused sampling to loss weighting, uniform dynamic sampling, and R3's evolving collocation set.

6. Conclusions And Future Work Directions

  • The paper frames propagation failures as a PINN failure perspective and presents R3 as a mitigation method with better empirical performance across benchmark PDEs.The conclusion identifies theoretical foundations of the propagation hypothesis and potential application to neural implicit representation learning as future directions.Source: §6 · p. 9, Figure 10 · p. 9

A. Connections Between Lp Norm and Sampling

  • Residual-based sampling and Lp PDE loss are related by a parameter-dependent scaling term, so optimizing one is not directly equivalent to optimizing the other.The discussion concerns sampling q(xr) ∝ |Rθ(xr)|^k and Lp loss.Source: A. Connections Between Lp Norm and… · pp. 11–12

B. Analyzing PINN-Dynamic: A Strong Baseline

  • Dynamic uniform sampling makes it increasingly likely to sample a persistent high-residual region across iterations, but low-residual regions can still dominate each iteration's loss and gradient.The discussion treats high-residual regions as much smaller in area than low-residual regions.Source: B. Analyzing PINN-Dynamic: A Strong… · p. 12

C. Analysis of R3 Sampling

  • The R3 analysis examines how its collocation points evolve over iterations.The supplied passage identifies the analysis scope without stating a specific result.Source: C. Analysis of R3 Sampling · p. 12

C.1. Retain Property of R3.

The analysis establishes R3's accumulation behavior for a fixed objective and describes how resolved points are released, alongside implementation choices for causal gates and a baseline comparison.

C.1. Retain Property of R3.

  • For a fixed, k-Lipschitz objective, R3's retained-population expected value approaches within kε of the maximum as iterations tend to infinity.The analysis extends the accumulation interpretation to changing residuals when the highest-error regions persist over iterations.Source: §C.1 · pp. 12, 14

C.2. Release of Collocation Points from High PDE Residual Regions.

  • R3 releases a point once its residual is at or below the population mean threshold, replacing it with a point resampled uniformly over the domain.The paper treats this as sufficient minimization and the mechanism that restores uniform density.Source: §C.2 · p. 15

D.1. Preventing Abrupt Causal Gate Movement.

  • The causal-gate shift is updated from the PDE loss and clipped to a maximum update magnitude; the experiments typically set that maximum to 0.1.The gate learning rate and tolerance control the shift rate and the loss level required before the gate advances.Source: §D.1 · p. 15

D.2. Choice of Other Gate Functions.

  • An alternative ReLU-tanh gate imposes stricter zeroing after a time cutoff, but the experiments use the tanh gate.A causal gate is required to be continuous in time, monotonically decreasing, and parameterized by a shift that raises gate values.Source: §D.2 · p. 15

E. Comparison of Baselines

  • The baseline comparison is organized around how methods satisfy the work's motivating properties.Table 2 is identified as the comparison visual.Source: E. Comparison of Baselines · p. 15

F. Computational Complexity Analysis of R3 Sampling vs its baselines

The complexity analysis contrasts R3's fixed-size residual evaluations with RAR-based methods that add points and evaluate a dense candidate set. The accompanying appendix passages specify benchmark PDE settings.

F. Computational Complexity Analysis of R3 Sampling vs its baselines

  • Table 2 compares baseline methods by their ability to meet the work's motivating properties.The supplied caption states the comparison scope, not its outcomes.Source: Table 2 · p. 16
  • R3 evaluates residuals on a fixed population, whereas RAR-based methods grow the training set and incur dense-set evaluation costs that can make the cost difference grow.The dense candidate set is much larger than the initial collocation set, and frequent resampling can add substantial cost.Source: F. Computational Complexity Analysis… · pp. 16–17

G.1. Convection Equation

  • The convection case studies use h(x) = sin(x) and periodic boundary conditions while varying the convection coefficient β.The passage identifies the equation as a model of transport phenomena.Source: §G.1 · p. 17

G.2. Allen-Cahn Equation

  • The Allen-Cahn setup is identified as a one-dimensional equation for phase separation in multicomponent alloy systems.The supplied equation passage gives a domain x ∈ [−1, 1], t ∈ [0, 1].Source: §G.2 · p. 17

G.3. Eikonal Equation

  • The Eikonal setup uses the zero-contour constraint as the initial-condition loss and a positive-boundary constraint as the boundary loss.It is used to calculate a signed distance function.Source: §G.3 · p. 18

G.4. Kuramoto–Sivashinsky Equation

  • The KS setup uses periodic boundary conditions and reports regular and chaotic parameter configurations.For the regular setting, α = 5, β = 0.5, and γ = 0.005.Source: §G.4 · p. 18

H. Details on Skewness and Kurtosis Metrics

Skewness and kurtosis characterize different features of value distributions; for PDE residuals, large positive values indicate asymmetric high residuals and narrow regions of high residuals, respectively.

H. Details on Skewness and Kurtosis Metrics

  • The implementation uses the adjusted Fisher-Pearson coefficient for skewness and Fisher’s definition for kurtosis.Skewness is zero for symmetric distributions, while kurtosis is zero for a Normal distribution under these conventions.Source: H. Details on Skewness and Kurtosis… · p. 18
  • Skewness measures distribution asymmetry; in PDE residuals, a large positive value indicates asymmetrically high residual values.Positive skew means more values lie to the right of the center than to the left.Source: H. Details on Skewness and Kurtosis… · p. 18
  • Kurtosis measures tail weight; in PDE residuals, a large positive value indicates high residuals concentrated in narrow regions of the spatio-temporal domain.Positive kurtosis indicates more values in the tails than expected under a Normal distribution.Source: H. Details on Skewness and Kurtosis… · p. 18

I. Hyper-parameter Settings and Implementation Details

This appendix gives implementation settings and additional visualizations of propagation failures, with diagnostic results across convection and Kuramoto-Sivashinsky cases.

I. Hyper-parameter Settings and Implementation Details

  • Baseline settings are tabulated by PDE, and all models for a given PDE share architecture and other hyperparameters; training used one Nvidia Titan RTX 24GB GPU.The settings table lists loss weights, optimizer, learning rate, and scheduler; Causal R3 uses α = 5 and gate learning rate ηg = 1e-3.Source: §I · p. 19, Table 3 · p. 19

J.1. Visualizing Propagation Failure for Different Settings of β

  • For convection with β = 30, 50, and 70, relative errors remain high as PDE residual loss decreases, and growing skewness and kurtosis indicate greater propagation-failure risk.For β = 30, the abrupt increase in skewness and kurtosis around 50K iterations coincides with a sudden relative-error increase.Source: §J.1 · pp. 19–20, Figure 12 · p. 20

J.2. Visualizing Propagation Failure for Kuramoto-Shivashinksy Equation

  • For the Kuramoto-Sivashinsky equation, residual loss decreases while skewness and kurtosis grow, with visualized narrow high-error regions after 100K iterations.The authors hypothesize that sharp boundaries or small regions of high error reflect propagation failure.Source: §J.2 · p. 20, Figure 13 · p. 20, Figure 14 · p. 20

J.3. Additional Results for Sample Efficiency of R3 Sampling

  • The appendix reports sample-efficiency results for convection and Allen-Cahn in Tables 4 and 5, averaged across three random runs.The supplied passages identify the tables and averaging procedure but do not state their results.Source: §J.3 · p. 20

J.4. Convergence Speed of R3 Sampling

The appendix compares convergence and sampling behavior across R3, Causal R3, and RAR-based methods, including how their collocation points evolve during training.

J.4. Convergence Speed of R3 Sampling

  • Causal R3 converges faster to low-error solutions than the baselines on convection and Allen-Cahn, while some baselines fail at low sample counts.cPINN-fixed and PINN-dynamic do not converge for convection at Nr = 1K; at Nr = 20K for convection and on Allen-Cahn, both reach lower errors than PINN-fixed.Source: §J.4 · pp. 20–21, Figure 15 · p. 21

J.5. Sensitivity of RAR-based Methods

  • Adding more collocation points almost always improves RAR-G and RAR-D, but smaller K or larger m also increases the final point-set size.The sensitivity experiments hold the initial set size fixed and vary resampling period K and added-point count m.Source: §J.5 · p. 21, Figure 16 · p. 22

J.6. Visualizing the Evolution of Collocation Points in R3 sampling

  • R3 retains points in high-residual regions and resamples uniformly, reducing PDE loss without admitting high residual imbalance.The retained population selectively focuses on high-residual regions as training progresses.Source: §J.6 · pp. 21–22

J.7. Visualizing the Evolution of Collocation Points in Causal R3

  • Causal R3 samples later times only after earlier-time residuals are minimized; by 90K iterations, its gate reveals the full time domain.At intermediate epochs, gate values are close to 1 over only small portions of the time domain.Source: §J.7 · p. 22

J.8. Visualizing the Dynamics of Collocation Points for RAR-Based Methods

  • RAR-G, RAD, and RAR-D fail to converge to the correct solution even after 100K iterations in the illustrated case.The authors interpret these dynamics as showing that the methods do not mitigate propagation failures.Source: §J.8 · p. 22

J.9. Kuramoto-Sivashinsky (KS) Equations

Across the tested Kuramoto-Sivashinsky cases, R3 methods are broadly comparable to CPINN, while the Eikonal discussion illustrates their performance on complex geometries and the appendix introduces optimization-function experiments.

J.9. Kuramoto-Sivashinsky (KS) Equations

  • On regular KS, R3 performs similarly to CPINN and Causal R3 is significantly better; CPINN is slightly better in the first chaotic case, while all methods struggle in the extended case and R3 methods are somewhat better.The authors characterize R3 and Causal R3 as comparable to CPINN on these benchmark settings; their KS experiments use substantially fewer iterations and average 50–60% less time than CPINN.Source: §J.9 · pp. 22–23

J.10. Additional Discussion and Visualization for the Eikonal Equation

  • For complex Eikonal geometries, R3 performs better than fixed and dynamic PINNs, resolving high-residual regions and producing fewer incorrect sign flips in the gear case.The task requires propagating boundary conditions to a solution that is negative inside and positive outside; small propagation errors can produce large regions with opposite signs.Source: §J.10 · p. 23

K. Optimization Characteristics of R3 Sampling on Test Optimization Functions

K.1. Auckley Function

  • The two-dimensional Ackley function has multiple local maxima in a near-flat region and one large peak at its center.Source: §K.1 · p. 24

K.2. Bohachevsky Function

These appendix passages describe six two-dimensional benchmark functions and show R3 sampling visualizations alongside comparisons across three KS-equation benchmarks.

K.2. Bohachevsky Function

  • The Bohachevsky function is bowl-shaped and has one global maximum.Source: §K.2 · p. 24

K.3. Drop-Wave Function

  • The Drop-Wave function is multimodal and highly complex.Source: §K.3 · p. 25

K.4. Egg-Holder Function

  • The Egg-Holder function is difficult to optimize because it has multiple local maxima.Source: §K.4 · p. 25

K.5. Holder-Table Function

  • The Holder-Table function has many local maxima and four global maxima at its corners.Source: §K.5 · p. 26

K.6. Bukin Function

  • The Bukin function has many local maxima, all lying on a ridge.Source: §K.6 · p. 26

K.7. Michalewicz Function

  • The Michalewicz function has multiple ridges and steep valleys.Source: §K.7 · p. 26
  • Table 6 compares the relative L2 errors (%) of CausalPINN, R3 sampling, and Causal R3 across three KS-equation benchmarks.The caption identifies the comparison scope but does not provide the table values.Source: Table 6 · p. 27
  • Figures 22–24 visualize R3 sampling, Causal R3, and CausalPINN on regular, chaotic, and extended chaotic KS-equation cases.The captions identify the methods and cases, not the outcomes of the comparison.Source: Figure 22 · p. 28, Figure 23 · p. 28, Figure 24 · p. 29
Loading 2207.02338v3…