Source-linked AI summary
Neither Precision Nor Architecture Alone: Controlled Tests of Failure Remedies for Physics-Informed Neural Networks
Jinyuan Zhang, Peng He, He Hu, Yin Yuan, ShengShuo Jiao
TL;DR
PINNs often fail on stiff or advection-dominated PDEs, while competing remedies target numerical stopping or architectural simplicity. This paper tests both through matched, seed-paired studies and finds that their benefits occupy different regime-and-seed slices. The results support joint evaluation of precision, stopping, backbone, and alignment with per-seed reporting.
Problem
The sufficiency and transferability of numerical precision/stopping and architecture/alignment remedies had not been tested together under matched controls.
Method
The study uses matched PDE–seed controls across a pre-registered 144-run study and an independent 85-run convection/wave study, varying precision, tolerance, backbone, and alignment.
Results
The two remedies act on different regime-and-seed slices: alignment recovers hard-convection failures, while precision lowers error but rarely converts them to successes and does not substitute for alignment.
Takeaways & Limitations
Precision, stopping, backbone, and alignment should be evaluated jointly, with outcomes reported per seed.
Takeaways & Limitations
The diagnostic proxies are descriptive because two registered tests are absent and 41/87 records lack L-BFGS logs.
Abstract
from arXiv · showhide
Physics-Informed Neural Networks (PINNs) frequently fail on stiff or advection-dominated PDEs, and two recent accounts offer competing remedies: switching from FP32 to FP64 to repair an L-BFGS stopping artifact, or replacing the MLP with a state-space-model (SSM) backbone plus sub-sequence alignment to counter architectural simplicity bias. We test both under matched, seed-paired controls in a pre-registered 144-run study spanning convection, reaction, and wave, plus an independent 85-run convection/wave study; success is relative $\ell_2$ error below $0.05$. The two remedies act on disjoint regime-and-seed slices: neither substitutes for the other. On hard convection ($β{=}50$), alignment recovers 2/5 seeds in FP32 and 3/5 in FP64, where the unaligned SSM succeeds on 0/5 seeds at either precision and the vanilla MLP moves only from 0/5 to 1/5 across the precision switch---the recoveries trace to the alignment objective, not the backbone. On reaction the backbone alone already succeeds on 3/5--4/5 seeds, so each remedy covers a regime the other does not. Responses are also seed-specific: the same precision switch flips individual seeds in opposite directions and, on wave, lowers median error with no statistically significant success gain. Tightening the inner L-BFGS tolerance in an independent repeated-step runner likewise lowers median error at a large runtime cost, with success counts unchanged. Precision, stopping, backbone, and alignment must therefore be evaluated jointly and reported per seed.
1 Introduction
PINNs can fail on stiff or advection-dominated PDEs, motivating competing numerical and architectural remedies. This study proposes matched, seed-controlled comparisons that vary these interventions jointly.
- PINNs break down on stiff or advection-dominated PDEs, with failures documented across convection, reaction, and wave equations.
- Two concurrent studies propose different remedies: FP64 targets an L-BFGS stopping artifact, while an SSM backbone plus alignment targets MLP simplicity bias.
- The remedies had not been tested together under controlled slices, leaving their sufficiency and transferability across regimes unresolved.
- The local controlled study pools 144 runs, while an independent validation study adds 85 convection and wave runs.
- Each PDE–seed cell shares its collocation grid, budget, and optimizer across variants, with outcomes reported per seed.
- The study varies precision, inner L-BFGS tolerance, backbone, and alignment within matched PDE–seed comparisons.
2 Related Work
Prior work offers multiple explanations for PINN failures and remedies spanning objectives, architectures, arithmetic, and optimization. This paper addresses the unresolved question of which mechanisms bind in which regimes through matched, seed-level controls.
- Diagnosing PINN failure modes: PINN failures have been attributed to complex loss landscapes, gradient imbalance, Neural Tangent Kernel eigenspectra, and training schedules that violate temporal causality.
- Two remedies proposed in 2025: PINNMamba adds an SSM backbone and sub-sequence contrastive alignment, whereas FP64 is proposed to resolve an L-BFGS tolerance artifact.
- Two remedies proposed in 2025: The two 2025 studies used different protocols, reported one seed per configuration, and left their compound remedies unablated.
- Loss weighting and training schedules: Sub-sequence alignment belongs to objective-shaping methods because it adds an agreement term rather than changing the optimizer or number format.
- The optimizer and precision view: The precision view predicts a lower arithmetic stopping floor, but whether this produces success depends empirically on regime and seed.
- Study design: The study complements prior work by jointly controlling precision, stopping, backbone, and alignment, comparing them seed by seed under registered rules.
3 Method
The study formalizes two remedies as interventions on different parts of PINN optimization and evaluates them in matched, seed-paired PDE comparisons. Its protocol combines controlled precision, tolerance, backbone, and alignment sweeps with explicit evidence-coverage and diagnostic limitations.
- Problem setup: The objective combines residual, boundary, and initial-condition losses, while the tested PDE regimes are difficult because residual gradients dominate data gradients and the objective is ill-conditioned.The paper frames both remedies as attempts to restore minimizability, but along different axes.
- Remedy mechanisms: Precision targets the L-BFGS stopping test, whose resolvable progress is bounded by the larger of tolerance_change and the arithmetic roundoff floor.The stated floor is max(τ, u_p|L_k|), with FP32 and FP64 having substantially different unit roundoffs.
- Remedy mechanisms: Alignment adds a sub-sequence agreement term that imposes equality constraints and shrinks the feasible set, removing bump-type spurious solutions under the stated independence condition.The trained objective becomes L + αL_align, and the paper characterizes this intervention as acting on the hypothesis-class axis.
- Experimental protocol: The local controlled study pools 144 runs, while the independent validation study contributes 85 runs; effective sample sizes are reported per research question because slices overlap.The studies vary one factor at a time within matched comparisons and do not form one complete factorial block.
- Experimental protocol: Each matched comparison shares the PDE instance, collocation grid, training budget, and optimizer configuration, reuses seeds across variants, and defines success as unrounded relative ℓ2 error below 0.05.The registered evaluation uses 5,000 Adam steps followed by up to 1,000 outer L-BFGS steps, with disabled TF32.
4 Experiments
Matched experiments test whether precision, stopping tolerance, backbone choice, and alignment provide interchangeable remedies across PDE regimes and seeds. Results instead show regime-specific and seed-specific effects, with alignment driving hard-convection recoveries and tighter tolerance improving error without increasing success counts.
- Matched remedy comparison: 2/5 FP32 and 3/5 FP64 hard-convection seeds succeed with alignment, while the unaligned SSM succeeds on 0/5 at either precision.The vanilla MLP improves only from 0/5 to 1/5 when switching from FP32 to FP64.
- Matched remedy comparison: 3/5–4/5 reaction seeds already succeed with the SSM backbone, while alignment reaches 5/5 at both precisions.Thus the backbone and alignment contribute differently across reaction and hard convection.
- Alignment contribution: Without alignment, the SSM has rL2 above 1.0 in every reported hard-convection cell, attributing recoveries to alignment rather than the backbone alone.This isolates alignment’s contribution under the hard-convection protocol.
- Alignment contribution: 0/5, 0/5, 0/5, 2/5, and 3/5 seeds succeed as α increases through {0, 100, 300, 1000, 3000}, with a threshold-like response.Median rL2 falls from 0.972 at α=300 to 0.801 at α=1000 and 0.041 at α=3000, but the threshold is not precisely calibrated.
- Protocol comparison: PINNsFormer records no successful convection seed, while both protocols solve every reaction cell; this is a protocol-level contrast with unequal methodological controls.The comparison has unequal seeds, a blank alignment field, and differences beyond backbone class.
- Tolerance response: 1/5 success remains unchanged for both precisions as tolerance tightens, while median error improves and runtime rises sharply in the repeated-step runner.FP64 median runtime rises from 120.8 to 2,573.3–2,569.6 seconds, and FP32 from 17.0 to 278.1–275.7 seconds.
- Seed and evaluation robustness: FP64 moves paired hard-convection seeds in opposite directions, while wave success changes from 4/10 to 6/10 without statistically significant improvement.On wave, FP64 lowers median rL2 from 0.125 to 0.045; McNemar p=0.727 and paired Wilcoxon p=0.375.
- Seed and evaluation robustness: The aggregate regime ordering remains stable when the success threshold varies from 0.03 to 0.07, although individual near-threshold seeds change status.This supports robustness of the rankings beyond the rL2=0.05 boundary.
5 Discussion
Matched controls show that precision and architectural remedies remain valid but operate on different regimes and seeds. Their effects are seed-specific, motivating joint evaluation and per-seed reporting within the study’s limited scope.
- Revisiting the two diagnoses: 1/5 hard-convection successes from precision alone make FP64 a complement to, not a substitute for, the architectural remedy.FP64 lowers error in every controlled cell, but rarely converts hard-convection failures into successes.
- Revisiting the two diagnoses: The SSM with alignment recovers hard convection, while the unaligned SSM matches or underperforms the vanilla MLP.The ablation attributes recovery to the alignment objective rather than the Mamba backbone.
- Why seed-level reporting matters: 3/5 hard-convection cell-level success hides opposite seed movements: FP64 rescues seed 2 but loses seed 1.The paper therefore characterizes reversals through their distribution and reports outcomes per seed.
- Implications: Neither “just use FP64” nor “just change the backbone” qualifies as general advice across the tested configurations.The authors recommend comparing precision and stopping jointly, tuning alignment, and reporting per-seed error distributions.
- Limitations: The corpus covers three one-dimensional PDE families at selected parameter values rather than a broad scientific-computing benchmark.The PINNMamba–PINNsFormer comparison also has unequal seeds, no parameter or tuning match, and no identical initialization tensors across architectures.
- Limitations: 41/87 records lack L-BFGS logs, the archived proxy classifier omits two registered conditions, and causal taxonomy validation still awaits rescue experiments.The independent validation also uses a repeated-step runner, while its wave-PINNMamba coefficient differs from the public wave protocol.
6 Reproducibility
The study reports seeds and applies the success threshold to unrounded values, while separating provenance across evidence sources and releasing regeneration artifacts.
- 6 Reproducibility: All random seeds are reported, and rL2 < 0.05 is applied to unrounded values.This preserves seed-level outcomes and avoids threshold changes caused by rounding.
- 6 Reproducibility: The release includes manifests, registered analysis files, proxy data, independent-validation materials, implementation, and figure-regeneration scripts.Pinned upstream code and the environment accompany the submission.
- 6 Reproducibility: Provenance separates preliminary, registered, follow-up, and independent evidence.This distinguishes the study stages used in the analysis.
7 Conclusion
Under matched controls, precision and tolerance differ from architecture and alignment in the PDE and seed slices they affect. The conclusion is to evaluate all four factors jointly and report every outcome per seed.
- 7 Conclusion: Precision and tolerance act on different PDE and seed slices than architecture and alignment, so the remedy classes are not interchangeable.The matched-control comparison supports treating numerical and architectural interventions as distinct.
- 7 Conclusion: Alignment recoveries, tolerance costs, and seed reversals motivate joint evaluation of precision, stopping, backbone, and alignment.The practical recommendation is to vary these factors together rather than draw conclusions from one configuration.
- 7 Conclusion: Every outcome should be reported per seed.Seed-level reporting exposes reversals that aggregate success counts can hide.
A.1 Proof of Proposition 1 (precision-bounded stopping floor)
The proof models floating-point error in the L-BFGS iterate-difference test and derives a precision-dependent stopping floor. FP64 lowers that floor far below the default tolerance, restoring sensitivity to smaller progress.
- Proposition and proof: The proposition bounds the L-BFGS stopping test’s resolvable progress by max(τ, u_p|L_k|).Here u_p is the unit roundoff for precision p, and the iterate difference is |L_{k+1} − L_k|.
- Proposition and proof: Floating-point evaluation decomposes the computed loss difference into true progress plus roundoff error.The proof writes fl(L_k) = L_k + e_k with |e_k| ≤ u_p|L_k|.
- Proposition and proof: Near convergence, roundoff contributes approximately 2u_p|L_k| and masks decreases smaller than that noise floor.The stopping criterion therefore resolves progress only down to max(τ, 2u_p|L_k|), equivalent up to a factor of two.
- Precision comparison: The default tolerance_change is approximately 1.19 × 10^-7.This is the tolerance against which the precision-dependent floor is compared.
- Precision comparison: FP32’s floor is approximately 10^-7–10^-5, whereas FP64’s is approximately 10^-15–10^-13.For the studied loss magnitudes, FP32 can sit at or above the default tolerance, while FP64 places the floor nine orders below it.
- Precision comparison: FP64 relaxes the stopping floor imposed by FP32 without changing the model.The arithmetic intervention improves the stopping test’s ability to detect progress of size τ.
A.2 Proof of Proposition 2 (alignment as over-determination)
The proposition formalizes sub-sequence alignment as an over-determining constraint system that reduces the collocation-feasible hypothesis space and excludes bump-type spurious solutions.
- Dimension reduction: The agreement term imposes m = (k−1)|χ| equality constraints, reducing the feasible-set dimension from dim Fχ∗ to dim Fχ∗−m when constraints are independent.Each constraint matches predictions from sub-sequences that share a collocation point.
- Dimension reduction: At each shared point, alignment penalizes disagreement among predictions issued by all sub-sequences containing that point.
- Dimension reduction: The independence assumption is generic because agreement constraints couple distinct sub-sequence outputs at disjoint points.
- Removal of bump-type spurious solutions: A propagating reference solution renders shared points identically across sub-sequences, whereas a bump-type spurious solution can drift across them.The propagating solution satisfies the cross-sub-sequence agreement that collocation-only spurious solutions do not.
- Removal of bump-type spurious solutions: With enough agreement constraints relative to the bump construction’s local degrees of freedom, bump-type spurious solutions are excluded and the propagating reference becomes dominant.