When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?

Zheyuan Hu; Ameya D. Jagtap; George Em Karniadakis; Kenji Kawaguchi

Source-linked AI summary

When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?

Zheyuan Hu, Ameya D. Jagtap, George Em Karniadakis, Kenji Kawaguchi

arXiv:2109.09444v7Sep 20, 2021cs.LGcs.NAmath.DSmath.NAstat.ML

TL;DR

The paper addresses the lack of theory explaining when XPINNs generalize better than PINNs. It derives prior and posterior bounds for multilayer networks and examines domain decomposition’s tradeoff, finding that XPINNs’ advantage depends on simpler subdomain targets outweighing the overfitting risk from fewer local training points.

Problem
The theoretical advantages and disadvantages of XPINNs over PINNs, especially for multilayer networks, remain insufficiently understood.
Source: §1 · p. 1
Method
The authors derive prior bounds from target-function complexity and posterior bounds from trained network matrix norms for multilayer PINNs, then extend the bounds to XPINNs by subdomain.
Source: §1 · p. 2
Results
Across five PDE experiments, XPINN performance varies by task; theory predicts an advantage when reduced subdomain complexity outweighs overfitting from fewer local training points.
Source: §1 · p. 2
Takeaways & Limitations
The bounds can help assess XPINN generalization during optimization and diagnose whether limited subdomain training data contributes to overfitting.
Source: §6 · p. 19
The posterior L2-error bound relies on a PDE norm-constraint assumption linking boundary and residual errors to L2 error.
Source: §3.3 · p. 8

Abstract

from arXiv · show

Physics-informed neural networks (PINNs) have become a popular choice for solving high-dimensional partial differential equations (PDEs) due to their excellent approximation power and generalization ability. Recently, Extended PINNs (XPINNs) based on domain decomposition methods have attracted considerable attention due to their effectiveness in modeling multiscale and multiphysics problems and their parallelization. However, theoretical understanding on their convergence and generalization properties remains unexplored. In this study, we take an initial step towards understanding how and when XPINNs outperform PINNs. Specifically, for general multi-layer PINNs and XPINNs, we first provide a prior generalization bound via the complexity of the target functions in the PDE problem, and a posterior generalization bound via the posterior matrix norms of the networks after optimization. Moreover, based on our bounds, we analyze the conditions under which XPINNs improve generalization. Concretely, our theory shows that the key building block of XPINN, namely the domain decomposition, introduces a tradeoff for generalization. On the one hand, XPINNs decompose the complex PDE solution into several simple parts, which decreases the complexity needed to learn each part and boosts generalization. On the other hand, decomposition leads to less training data being available in each subdomain, and hence such model is typically prone to overfitting and may become less generalizable. Empirically, we choose five PDEs to show when XPINNs perform better than, similar to, or worse than PINNs, hence demonstrating and justifying our new theory.

1 Introduction

The paper addresses the lack of theory explaining when XPINNs generalize better than PINNs. It develops bounds for both methods and identifies a tradeoff between simpler subdomain targets and fewer training points per subdomain.

1 Introduction

Earlier PINN bounds do not establish the advantages and disadvantages of multilayer XPINNs, and some rely on two-layer networks or difficult-to-compute quantities.The introduction notes that Holder continuity regularization is not widely adopted in practice.Source: §1 · p. 1
XPINN domain decomposition trades lower target complexity within subdomains against greater overfitting risk from fewer local training points.The paper uses analytical examples and experiments on five PDEs to examine when XPINNs perform better, similarly, or worse than PINNs.Source: §1 · p. 2
The paper develops prior and posterior generalization bounds for multilayer PINNs and extends them to XPINNs by applying the bounds to each subdomain.The prior bound uses target-function complexity; posterior bounds use capacity controls based on spectral and (2,1) matrix norms.Source: §1 · p. 2

2 Preliminaries

The preliminaries specify the PDE setting and introduce PINNs, XPINNs, neural networks, and the function spaces used to analyze their approximation and generalization properties.

2 Preliminaries

The preliminaries establish notation for vectors, matrices, and their norms, including spectral and l_p,q matrix norms.They also define the infimum of a set.Source: §2 · p. 2

2.1 PDE Problem

The PDEs are posed on a bounded domain, with a differential operator, forcing and boundary data, and an unknown solution.The domain is Ω = (−1, 1)^d.Source: §2.1 · p. 2

2.2 PINN and XPINN

PINNs approximate PDE solutions by minimizing boundary and residual losses, while XPINNs assign subdomains to subnetworks and maintain continuity through interface losses.The XPINN population loss excludes interface losses; interface continuity can let one subnetwork use samples from another subdomain for regularization.Source: §2.2 · pp. 2–3

2.3 Neural Networks

A depth-L network is a composition of linear maps and element-wise nonlinearities, with input derivatives used in PINN residual losses.The setting excludes ReLU because it is not differentiable.Source: §2.3 · p. 4

2.4 Generalized Barron Space

The tree-like function space contains finite multilayer networks under bounded intermediate 1,∞ norms, and approximation complexity is controlled by target Barron norms.The functional-analysis construction supports approximating target functions and their derivatives with low complexity, which the paper uses for its prior bound.Source: §2.4 · p. 5

3 Theory

The theory develops prior and posterior generalization bounds for multi-layer PINNs and XPINNs, then uses them to compare the models. Under a PDE norm assumption, it also connects boundary and residual losses to L2 generalization error.

3 Theory

The analysis assumes a linear second-order PDE operator with bounded, Lipschitz coefficients, conditions used to estimate PINN Rademacher complexity.The paper also assumes the PDE norm constraint that underpins the link between residual and boundary losses and L2 error.Source: §3 · p. 6, §3.3 · p. 8

3.1 A Prior Generalization Bound (Theorem 3.1)

The prior bound combines empirical loss, model complexity, and a statistical term, with generalization controlled by the target solution’s Barron norm under path-norm regularization.A larger Barron norm corresponds to a larger bound; the result is presented for solutions in the tree-like function space.Source: §3.1 · p. 6

3.2 A Posterior Generalization Bound (Theorem 3.2)

The posterior bound controls generalization using trained-network matrix norms and a statistical term, making it easier to compute numerically than the prior bound.The matrix-norm quantities are linked to Rademacher complexity and reflect the complexity of the learned network.Source: §3.2 · pp. 7–8

3.3 Posterior L2 Error Generalization Bound

Under Assumption 3.2, Theorem 3.3 connects the L2 generalization error to boundary and residual losses.The assumption is described as a well-posedness condition that justifies minimizing those losses.Source: §3.3 · p. 8

3.4 Comparing XPINN and PINN by Theorem 3.1

The prior-bound comparison evaluates PINN and subdomain-wise XPINN bounds, weighting the latter by each subdomain’s share of prediction data and assuming similar empirical losses.The model with the smaller resulting quantity is considered more generalizable.Source: §3.4 · pp. 8–9

3.5 Comparing XPINN and PINN by Theorem 3.2

The posterior-bound comparison computes each model’s bound from trained-network quantities, then weights the XPINN subnetwork bounds by their subdomains’ shares of residual samples.The resulting bounds can be measured from trained networks; the smaller bound indicates greater generalizability.Source: §3.5 · p. 9

3.6 Comparing XPINN and PINN by Theorem 3.3

Theorem 3.3 reduces the L2-error comparison to comparing the PINN and XPINN boundary and residual bounds.The paper describes this comparison as computing the respective theoretical bounds.Source: §3.3 · p. 8, §3.6 · p. 9

4 Analytical Examples Based on Theorem 3.1

Analytical examples use Barron-norm bounds to show that XPINN generalization depends on a tradeoff: decomposition can simplify subdomain targets, while fewer training points can increase overfitting. Which effect dominates determines whether XPINN outperforms, matches, or underperforms PINN.

4 Analytical Examples Based on Theorem 3.1

The examples use analytical Barron-norm expressions to compare when XPINN is better than, similar to, or worse than PINN.They are designed to clarify the generalization tradeoff associated with domain decomposition.Source: §4 · p. 9

4.1 Case where XPINN Outperforms PINN

XPINN generalizes better when decomposition lets it learn a simple subdomain with a simpler network while reserving complexity for the harder region.In the example, the target’s Barron norms are 1 and 2 on the subdomains, versus 3 over the whole domain.Source: §4.1 · p. 10

4.2 Case where XPINN is Worse Than PINN

XPINN performs worse when the gain from simplifying one subdomain is outweighed by overfitting from fewer training samples per subdomain.In the example, the bound comparison is 15.625 for PINN versus approximately 16.70 for XPINN.Source: §4.2 · p. 10

4.3 Illustration of a Tradeoff in XPINN generalization

The tradeoff favors PINN when reduced-data effects dominate, XPINN when decomposition’s simplicity effect dominates, and similar performance when the effects balance.In the example, q < 0.683 favors PINN, while q > 0.683 favors XPINN.Source: §4.3 · pp. 10–11, Figure 1 · p. 10

5 Computational Experiments

Across five PDE experiments, XPINN performance varies with a tradeoff: domain decomposition simplifies local targets but leaves less training data in each subdomain. The experiments show XPINNs performing worse than, similarly to, or better than PINNs.

5.1.2 Results

PINN (6.899e-1) has slightly lower KdV test relative L2 error than XPINN (6.955e-1); XPINN's bound is 121.08% of PINN's.The authors conclude that simpler subdomain targets and overfitting from reduced data have similar effects, yielding similar overall performance.Source: Table 1 · p. 12, §5.1.2 · p. 12

5.1.1 Setup

For KdV, XPINN assigns the fluctuating right region to XPINN-R and the smoother left region to XPINN-L, partitioning at x = -0.74.The experiment uses two 10-layer, 20-hidden-unit sine networks for XPINN; the PINN uses the same architecture.Source: §5.1.1 · p. 12

5.2.2 Results

PINN (1.778e-3) outperforms XPINN (4.490e-3) in heat-equation relative L2 error, and their bounds are 100.00% and 243.22%, respectively.The authors report that limited subdomain data makes overfitting more pronounced than the benefit of simpler targets; XPINN-B also has larger errors in bottom-domain corners.Source: Table 2 · p. 13, §5.2.2 · p. 14

5.2.1 Setup

For the heat equation, XPINN divides the domain at t = 0.5, assigning the trigonometric source below and the hyperbolic source above.This split separates the two heterogeneous heat sources, which the passage describes as complex near t = 0 and t = 1.Source: §5.2.1 · p. 13

5.3.1 Setup

For advection, XPINN-LMR splits the domain into three continuous regions where the target is constant, unlike the whole discontinuous solution.The two outer regions have u = 0 and the middle region has u = 1.Source: §5.3.1 · p. 14

5.3.2 Results

The authors report XPINN performs better than PINN on advection; XPINN's bound is 66.59% versus PINN's 100%, with smaller errors near the discontinuity.Each XPINN subdomain targets a constant function, and the authors say the simplicity benefit outweighs overfitting from fewer available data.Source: Table 3 · p. 15, §5.3.2 · p. 15

5.4.2 Results

PINN (5.553e-2) outperforms XPINN3 (1.108e-1), which outperforms XPINN2 (1.387e-1) and XPINN1 (4.022e-1) in Poisson relative L2 error.The authors attribute XPINNs' weaker generalization to limited subdomain data outweighing reductions in target complexity; their sub-networks are also more complex than PINN's.Source: Table 5 · p. 16, §5.4.2 · p. 16

5.5.2 Results

XPINN-AM generalizes better than PINN on compressible Euler equations, while a top-bottom XPINN partition does not generalize well compared with PINN.The top and bottom XPINN sub-net complexities are 42.37% and 131.26%, respectively; the authors present norms and bounds as indicators for evaluating decompositions.Source: Table 6 · p. 18, §5.5.2 · p. 18

6 Conclusion

The paper develops prior and posterior generalization bounds for PINNs and XPINNs and identifies domain decomposition as a tradeoff between simpler subproblems and less training data per subdomain. XPINNs outperform, match, or underperform PINNs depending on which effect dominates, and the bounds can inform adaptive domain decomposition.

6 Conclusion

The paper provides prior and posterior generalization bounds, developing multi-layer Barron-space theory for the prior analysis and norm-based Rademacher complexity for the posterior analysis.The bounds are used to investigate when and how XPINNs improve generalization.Source: §6 · pp. 18–19
Domain decomposition trades reduced target-function complexity in each subdomain against less training data, which can raise empirical Rademacher complexity and promote overfitting.XPINNs outperform PINNs when the complexity reduction dominates, underperform when data scarcity dominates, and perform similarly when the two effects balance.Source: §6 · p. 19
The bounds can help diagnose generalization after a domain decomposition is initialized and guide restructuring subdomains to include more training points when data scarcity drives overfitting.The paper also offers a partial explanation for observations that XPINNs, but not PINNs, are applicable in long-time integration of several PDEs: whole-solution complexity may be decomposed into less complex sub-solutions.Source: §6 · p. 19

A Preliminary: Functional Analysis

This preliminary section collects function-space and compactness results used in the paper’s analysis. It defines several function spaces and states how equicontinuity and compactness support uniform-convergence arguments.

A Preliminary: Functional Analysis

The section defines a multi-index α of order k for derivatives of functions in C^k(Ω), and defines the Holder space C^k,γ(Ω) using a norm.The supplied passage does not specify the norm’s formula.Source: A Preliminary: Functional Analysis · p. 20
A metric space is complete when every Cauchy sequence has a limit in the space, while relative compactness means that the closure of a subspace is compact.The section also states that C^k,γ(Ω) is a Banach space.Source: A Preliminary: Functional Analysis · p. 20
The Arzela-Ascoli theorem states that a class H contained in C(Ω) is relatively compact in C(Ω) if and only if it is equicontinuous.A preceding lemma states that an equicontinuous sequence of continuous functions that converges pointwise has a continuous limit and converges uniformly.Source: A Preliminary: Functional Analysis · p. 20
The Sobolev space W^k,p(Ω) contains locally summable functions whose weak derivatives through order k belong to L^p(Ω); for p = 2, it is a Hilbert space.The section also states a trace theorem for bounded C1 domains.Source: A Preliminary: Functional Analysis · p. 21

B.1 Proof of Theorem 2.2

The appendix proofs establish properties of the generalized Barron space and derive a neural-network approximation result. They use functional-analytic arguments for Theorem 2.2, definitions for Theorem 2.1, and finite-element approximation and induction for Theorem 2.3.

B.1 Proof of Theorem 2.2

The proof of Theorem 2.2 establishes that B_X,Ω is a Banach space and embeds continuously into C^2,1(Ω).It also argues that the unit ball is closed in the C^2(Ω) topology.Source: §B.1 · pp. 23–24

B.2 Proof of Theorem 2.1

The proof of Theorem 2.1 follows directly from the definition of generalized Barron space and the neural-network definition.Source: §B.2 · p. 24

B.3 Proof of Theorem 2.3

The proof of Theorem 2.3 uses a finite-element approximation lemma for functions in a closed convex hull and constructs deeper networks by induction.It merges the trees for the approximating networks into one tree, increasing each layer’s width by a factor of m and adding an outer layer of width m.Source: §B.3 · pp. 24–25

C Proofs of Rademacher Complexity

This appendix develops spectral-norm-based complexity bounds for PINNs and their derivatives using second-order derivative and covering-number arguments. It also establishes bounds for tree-like neural networks and relates tree-like function bounds to Barron-space norms.

C Proofs of Rademacher Complexity

The appendix provides detailed calculations for the second-order derivatives of neural networks.The derivation applies the chain rule and tracks the dimensions of the resulting derivative terms.Source: C Proofs of Rademacher Complexity · p. 26

C.1 Spectral Norm for Complexity

A covering-number approach connects function-class Rademacher complexity to image-set covering numbers and yields bounds under spectral- and matrix (2,1)-norm constraints.The theorem considers networks evaluated on a data matrix, with specified layerwise norm bounds.Source: §C.1 · pp. 26, 30
The passages state that an unspecified subject satisfies a Rademacher complexity bound and introduce a covering-number bound, without identifying the subject or giving bound values or conditions.Source: §C.1 · p. 30
The spectral-norm analysis bounds the Rademacher complexity of function classes formed by all first- and second-order PINN derivatives.The derivation constructs covers of intermediate network outputs; the second-order analysis accounts for the additional Ω term.Source: §C.1 · pp. 35, 38

C.2 Tree-Like Function Space

The spectral-norm-based bound connects to Barron space, where path norms or related (1, ∞) norms upper-bound a quantity governing Rademacher complexity.The stated norm equivalence uses a constant depending on maximal width h.Source: §C.2 · p. 38
The appendix states Rademacher complexity bounds for both tree-like neural networks and PINNs.The accompanying lemmas specify bounds for each class over any set of n points, with maximal network width h and class-dependent constants.Source: §C.2 · p. 39

D.1 Proof of Theorem 3.1

The proof bounds the generalization of PINN boundary and residual losses using Rademacher complexity. It applies a Lipschitz-loss argument for boundary prediction and controls residual complexity through network weight norms.

D.1 Proof of Theorem 3.1

For boundary prediction, truncating the network output to [−1, 1] makes the MSE loss c-Lipschitz, enabling a Rademacher-based generalization bound.Source: §D.1 · p. 40
For residual prediction, the proof bounds the Rademacher complexity of the differentiated network and uses ∥W_l∥_1,∞ ≤ 1 for l < L, leaving the final-layer norm as the complexity control.Source: §D.1 · p. 40
The proof obtains separate generalization bounds for PINN boundary and residual losses.The residual bound uses the complexity of the differentiated network.Source: §D.1 · p. 40

D.2 Proof of Theorem 3.2

The proof derives generalization bounds for PINN losses using classes of composed loss and multilayer network functions. Its probability bounds use layerwise integer controls chosen from the trained network's matrix norms.

D.2 Proof of Theorem 3.2

The proof allocates the confidence parameter across integer-control choices so the resulting bound holds with probability 1 − δ.Source: §D.2 · pp. 40–41
The proof establishes probability bounds over the training dataset for classes combining the MSE loss with multilayer neural networks.It treats boundary and residual losses, with the residual case following a similar argument.Source: §D.2 · pp. 40–41
The bound holds with probability 1 − δ(M, N) for any pair (θ, M, N), provided θ satisfies layerwise integer choices based on its weight matrices’ spectral norms and spectral-to-mixed-norm ratios.Specifically, M(l) = ⌈∥W_l∥_2⌉ and N(l) = ⌈∥W_l∥_{2,1}/∥W_l∥_2⌉.Source: §D.2 · p. 41

D.3 Proof of Theorem 3.3

The paper positions its analysis as the first to establish generalization results for PINNs and XPINNs and to examine when XPINNs perform better. It extends prior theory to multilayer networks and motivates Barron space as a way to measure target-function complexity.

D.3 Proof of Theorem 3.3

Theorem 3.3’s proof invokes Assumption 3.2 and uses a bound involving residual and boundary losses.Source: §D.3 · p. 42

E.1 Physics-Informed Neural Networks

PINNs approximate PDE solutions with neural networks trained to satisfy the PDE’s physical rule, while XPINNs use domain decomposition; the reason for reported XPINN gains was unknown.CPINN is also domain-decomposition-based but is limited to conservation laws and does not allow general spatio-temporal decomposition.Source: §E.1 · p. 42
The paper presents its work as the first proof of generalization for PINNs and XPINNs and the first analysis of when and how XPINNs outperform PINNs.Source: §E.1 · p. 42

E.2 Rademacher Complexity of Neural Networks

The study adopts norm-based controls of neural-network Rademacher complexity, while prior approaches discussed include Frobenius-norm bounds and covering-number bounds.The cited discussion notes that some earlier bounds grow exponentially with depth, while another has polynomial depth dependence.Source: §E.2 · pp. 42–43

E.3 Theory on PINNs

Earlier PINN theory included two-layer Barron-space bounds and multilayer analyses using PDE stability or Holder continuity; this work extends results to multilayer networks and uses Barron and spectral norms.Source: §E.3 · p. 43

F Why Barron Space?

Barron space measures target-function complexity through its Barron norm, and the paper argues that trained network complexity can be controlled by target complexity.The paper motivates this space as a function-space framework for analyzing PDE solutions and generalization.Source: F Why Barron Space? · p. 43

G.1 Comparison of Boundary Loss via Theorem 3.1

The comparison applies Theorem 3.1 to PINNs and to each XPINN subdomain, then aggregates subdomain errors. Under the stated simplifications, generalization is compared through a quantity for which smaller values indicate better generalization.

G.1 Comparison of Boundary Loss via Theorem 3.1

The PINN bound uses the number of boundary training points, while each XPINN subdomain bound uses its own boundary-point count.Source: §G.1 · pp. 43–44
The comparison applies Theorem 3.1 to each XPINN subdomain and weights its generalization error by the proportion of testing data it predicts.Source: §G.1 · pp. 43–44
Assuming similar empirical losses and omitting the last term, the comparison reduces to a quantity where the smaller value indicates better generalization.Source: §G.1 · p. 44

G.2 Comparison of Boundary Loss via Theorem 3.2

The subsection compares PINN and XPINN boundary-loss generalization bounds under Theorem 3.2. The XPINN bound combines subnet bounds according to their shares of the testing data, and the smaller bound indicates greater generalizability.

G.2 Comparison of Boundary Loss via Theorem 3.2

The comparison assumes PINN and XPINN subnets use networks of the same depth and width, while their bounds account for residual training samples in the full model or each subdomain.The passage also refers to layer-matrix norms in the bounds, but their definitions are incomplete here.Source: §G.2 · p. 44
Theorem 3.2 compares the PINN testing-loss bound with a testing-share-weighted XPINN bound; the smaller quantity indicates the more generalizable model.The XPINN bound aggregates the subnet bounds according to each subnet’s share of testing data.Source: §G.2 · p. 44