PirateNets: Physics-informed Deep Learning with Residual Adaptive Networks

1 Introduction

• PINNs use tailored loss functions as soft physical constraints for forward and inverse PDE problems, while broader PIML methods incorporate physical knowledge through multiple parts of the learning pipeline.Prior approaches also encode physical principles through architectures, and PINN research addresses issues such as spectral bias, unbalanced gradients, and causality violation.Source: §1 · pp. 1–2
• The paper argues that PINN PDE-residual minimization depends on network-derivative capacity, and reports that conventional Glorot or He initialization can make MLP derivatives poorly trainable.The paper identifies problematic derivative initialization as a potential source of unstable PINN training.Source: §1 · p. 2
• For second-order linear elliptic and parabolic PDEs, convergence in training error leads to convergence in the solution and its derivatives.This result supports the paper's argument that derivative behavior matters to PINN training.Source: §1 · p. 2
• PirateNets target the limited use of deep PINNs with an architecture and physics-informed initialization designed to address pathological initialization and support stable, efficient scaling.The paper also presents the initialization as a way to integrate physical priors at the model initialization stage.Source: §1 · p. 2
• Experiments report consistent improvements in accuracy, robustness, and scalability across benchmarks.The paper describes comprehensive numerical experiments evaluating PirateNets.Source: §1 · p. 2

2 Physics-informed neural network

• A PINN approximates the unknown solution with a neural network, and automatic differentiation provides gradients needed to define the interior PDE residual.The PDE is posed on a spatial-temporal domain, with a differential operator and initial and spatial boundary conditions.Source: §2 · p. 3
• The boundary residual applies the boundary operator to the network approximation, while initial and boundary conditions specify the solution constraints.The boundary operator can represent Dirichlet, Neumann, Robin, or periodic conditions.Source: §2 · p. 3
• PINN training minimizes a composite empirical loss using training points that may come from a fixed mesh or be sampled at each gradient-descent iteration.The section notes that the formulation reviewed for parabolic equations can also apply to elliptic and hyperbolic equations.Source: §2 · p. 3

• PirateNets’ relative L2 error falls with depth, while MLP error rises toward 100% at 18 hidden layers and ResNet error also rises, though more slowly.Figure 1 averages results over five random seeds for each architecture.Source: Figure 1 · p. 4, §3 · p. 4
• For second-order linear elliptic and parabolic PDEs, training-error convergence leads to convergence of the solution and its derivatives under the stated estimates.The elliptic discussion relates small empirical loss to local H2 approximation and, in one dimension, pointwise derivative approximation.Source: §3 · pp. 4–5
• For hyperbolic equations such as wave and Burgers, propagating singularities can make L2 residual loss insufficient to control higher derivatives; first-derivative convergence uses an H1 temporal-boundary residual.Source: §3 · p. 6
• At initialization, MLP derivatives behave like deep linear networks: all layer weights contribute, while the derivative is independent of input coordinates and vulnerable to vanishing or exploding gradients.Source: §3 · p. 7
• Wider MLPs have derivatives closer to a constant, which can make the network resemble a trivial zero solution, particularly for homogeneous PDEs.The analysis focuses on first derivatives and Tanh, while numerical evidence suggests similar insights for higher-order derivatives and other common activations.Source: §3 · p. 7
• Derivative variance decays exponentially with width for Tanh, GeLU, and sine activations, and second- and fourth-order derivatives show a similar trend.In depth-varying tests, higher-order derivatives of deeper MLPs tend to have larger relative L2 errors than the MLP output, indicating worse trainability and a proxy for harder, less stable PDE-residual optimization.Source: §3 · p. 7

Architecture

• Coordinate embeddings feed two dense encoding maps that act as gates in each residual block; random Fourier features are the primary embedding used here.The cited work motivates these gates as a way to enhance MLP trainability and PINN convergence; Fourier embeddings have been validated for reducing spectral bias and approximating high-frequency solutions.Source: §4 · p. 8
• A PirateNet with L residual blocks has depth 3L and a trainable-parameter count comparable to an MLP of the same depth.Each block has three dense layers, two gating operations, and an adaptive residual connection.Source: §4 · p. 8

Adaptive residual connections

• Initializing each α(l) to zero makes every block an identity map; training can learn the nonlinearity needed to deepen the model and restore expressivity.The model initially represents a linear combination of basis functions, and the authors say this avoids deep-network initialization pathologies.Source: §4 · p. 8

Data-informed initialization

• PirateNets can initialize their final linear layer by least squares from solution measurements, yielding an initial guess optimal in the L2 sense for the available data.Initialization data may come from measurements, initial and boundary conditions, surrogate solutions, or approximations to a linearized PDE; randomly initialized bases may limit data-fitting expressivity.Source: §4 · pp. 8–9

Connections to prior work

• PirateNets differ from Bresson et al.’s Residual Gates by initializing α to zero for identity mappings rather than one for an original ResNet.The authors also distinguish their placement of activation before addition, which permits an exact identity mapping at initialization.Source: §4 · p. 9
• Unlike PIELM, PirateNets use least squares to initialize an approximation that is subsequently corrected through training, rather than solving the PDE in one step.The authors describe the linear least-squares initialization as globally minimizing its objective with minimal computational resources.Source: §4 · p. 9

5 Results

• The experiments use a shared training pipeline and compare PirateNet with a baseline under the same hyper-parameter settings; ablations cover depth, adaptive residual connections, gating, and physics-informed initialization.PirateNet projects coordinates with random Fourier features and uses adaptive residual blocks; its training cost is reported as comparable to an equivalent-depth Modified MLP.Source: §5 · pp. 9–10, Figure 3 · p. 10

5.1 Allen-Cahn equation

• On Allen–Cahn, a 3-block PirateNet achieves relative L2 error 2.24 × 10−5 and the authors report the best result then reported for this example.Physics-informed initialization improves both architectures, and PirateNet improves with depth while deeper Modified MLPs show increasing error; initial-condition and linearized-PDE fits perform comparably.Source: §5.1 · p. 11, Figure 5 · p. 12

5.2 Korteweg–De Vries equation

• On KdV, a 3-block PirateNet reaches relative L2 test error 4.27 × 10−4 and outperforms the cited state of the art [60] by approximately an order of magnitude.PirateNet converges faster than Modified MLP; initialization from initial conditions or the linearized PDE performs similarly, and the architecture retains its depth scalability.Source: §5.2 · p. 13, Figure 7 · p. 14

5.3 Grey-Scott equation

• For Grey–Scott, the final-time relative L2 errors are 3.61 × 10−3 for u and 9.39 × 10−3 for v, with stable PirateNet convergence for Tanh and swish.Swish gives better accuracy in this example, while Modified MLP training is unstable; initializing α = 0 improves accuracy by one or two orders of magnitude over α = 1, and gating further reduces error.Source: §5.3 · p. 15, Figure 8 · p. 15, Figure 9 · p. 16

5.4 Ginzburg-Landau equation

• For the complex Ginzburg–Landau equation, PirateNet’s L2 errors are 1.49 × 10−2 for u and 1.90 × 10−2 for v, versus Modified MLP’s 3.20 × 10−2 and 1.94 × 10−2.The ablations report stable training and improved accuracy with α initialized to 0 as depth increases, while α = 1 collapses deeper models; gating also enhances accuracy.Source: §5.4 · p. 17, Figure 11 · p. 18, Figure 12 · p. 18

5.5 Lid-driven Cavity flow

• At Re = 3200, PirateNet’s relative L2 error is 4.21 × 10−2, versus 1.58 × 10−1 reported for JAXPI; training remains stable as depth increases, with saturation beyond 20.The velocity profiles closely align with Ghia et al., and learned α increases during transitions to higher Reynolds numbers.Source: §5.5 · p. 19, Figure 14 · p. 19

6 Discussion

• Figure 15 compares relative L2-error convergence across PirateNet depths under shared hyperparameters and tracks nonlinearities across six residual blocks.The six-block model contains 18 hidden layers.Source: Figure 15 · p. 20
• Standard initialization schemes produce pathological MLP derivatives that impair PINN trainability and robustness, motivating the focus on network derivatives during initialization and training.For linear elliptic and parabolic PDEs, the authors state that under mild conditions training-error convergence implies convergence of the solution and its derivatives.Source: §6 · pp. 19–20
• PirateNets use adaptive residual connections to start as a linear combination of a chosen basis and recover approximation capacity through trainable α parameters during training.The final layer can be initialized from a least-squares solution to tailor the network to a dataset.Source: §6 · p. 20
• The authors identify significant computational resource requirements and time-consuming training as continuing limitations of PINNs.They suggest acceleration as future work and mention optimizing coordinate embeddings for specific PDEs.Source: §6 · p. 20

A Proofs

• For the elliptic case, the proof represents solution error through interior and boundary residuals, then uses interior regularity and a Dirichlet map to obtain a well-posed boundary problem.The proof constructs a smooth solution matching the boundary residual before applying well-posedness.Source: A Proofs · p. 25
• For the parabolic case, the proof tracks initial and spatial-boundary residuals and uses parabolic regularity estimates and a semigroup representation.The semigroup is generated by −D.Source: A Proofs · p. 25
• The proof of Proposition 3 uses the chain rule and considers a linear regime in which the activation derivative is approximately one.The displayed excerpt begins to simplify the derivative expression under that regime.Source: A Proofs · p. 26
• A separate derivative argument proceeds by induction on network depth, with the one-layer case and the L+1 case forming the induction steps.The excerpt also invokes Chebyshev’s inequality in a bound.Source: A Proofs · p. 26

B Allen-Cahn equation

• The Allen–Cahn equation is solved with periodic boundary conditions from u0(x) = x^2 cos(πx) through final time T = 1.The solution is generated using conventional spectral methods.Source: B Allen-Cahn equation · p. 26
• Validation data use a 512-mode spectral Fourier discretization and ETDRK4 time stepping with step size 10^-5.The data are generated with Chebfun.Source: B Allen-Cahn equation · p. 26
• The solution is recorded every Δt = 0.005, producing a validation dataset with resolution 200 × 512.Table 4 lists the hyperparameter configuration for reproducing Figure 4.Source: B Allen-Cahn equation · p. 26, Table 4 · p. 27

C Korteweg–De Vries equation

• The Korteweg–De Vries equation is solved with periodic boundary conditions from u0(x) = cos(πx) through final time T = 1.The solve uses conventional spectral methods.Source: C Korteweg–De Vries equation · p. 27
• Validation data use a 512-mode spectral Fourier discretization and ETDRK4 time stepping with step size 10^-5.The data are generated with Chebfun.Source: C Korteweg–De Vries equation · p. 27
• The solution is recorded every Δt = 0.005, yielding a validation dataset with resolution 200 × 512.The stated resolution is in the temporal and spatial domain.Source: C Korteweg–De Vries equation · p. 27

• The Grey-Scott equation is integrated from its initial condition to T = 2 using conventional spectral methods with periodic boundary conditions.Source: D Grey-Scott equation · p. 27
• Validation data uses a 200 × 200 Fourier discretization and fourth-order ETDRK4 time stepping with step size 10−3.The passage identifies Chebfun and ETDRK4.Source: D Grey-Scott equation · p. 27
• Sampling every ∆t = 0.02 yields a validation dataset with resolution 100 × 200 × 200 across temporal and spatial dimensions.Source: D Grey-Scott equation · p. 27

• The Ginzburg-Laudau equation is integrated from its initial condition to T = 1 using conventional spectral methods with periodic boundary conditions.Source: E Ginzburg-Laudau equation · p. 27
• The validation solver uses a 200 × 200 Fourier discretization and fourth-order ETDRK4 time stepping with step size 10−3.The passage identifies Chebfun and ETDRK4.Source: E Ginzburg-Laudau equation · p. 27
• Recording the solution every ∆t = 0.01 yields validation data with resolution 100 × 200 × 200 across temporal and spatial dimensions.Source: E Ginzburg-Laudau equation · p. 27
• Table 5 gives the Korteweg–De Vries equation hyper-parameter configuration for reproducing Figure 6.Source: Table 5 · p. 28

• The results are compared against.Source: F Lid-driven Cavity flow · p. 28
• Table 6 gives the Grey-Scott equation hyper-parameter configuration for reproducing Figure 8.Source: Table 6 · p. 29
• Table 7 gives the Ginzburg-Laudau equation hyper-parameter configuration for reproducing Figure 11.Source: Table 7 · p. 29
• The lid-driven cavity flow hyper-parameter configuration is provided for reproducing Figure 14.Source: Table 8 · p. 30