PaperlayerView exact revision on arXiv

Source-linked AI summary

Augmented Physics-Informed Neural Networks (APINNs): A gating network-based soft domain decomposition methodology

Zheyuan Hu, Ameya D. Jagtap, George Em Karniadakis, Kenji Kawaguchi

arXiv:2211.08939v3cs.LGmath.DSmath.NAstat.ML

TL;DR

PINN domain decomposition can face interface errors and data-allocation trade-offs, motivating a more flexible alternative. APINN uses soft, trainable gating with shared parameters and weighted subnet outputs, and experiments and theory report improvements across PINN and XPINN settings. Its performance depends on decomposition initialization, while its broader data and parameter sharing can increase cost.

  • Problem

    XPINN can exhibit larger interface errors, while decomposed models face generalization trade-offs involving function simplicity and available training data.

  • Method

    APINN uses a trainable gate for soft domain decomposition, weight-averages subnet outputs, shares parameters across subnets, and is analyzed through generalization bounds.

  • Results

    APINN improves PINN and XPINN across reported PDE experiments, including cases where XPINN is weaker than PINN, comparable to it, or already better.

  • Takeaways & Limitations

    Gate optimization and initialization can help identify a potentially better decomposition, and different initial decompositions can produce substantially different APINN performance.

  • Takeaways & Limitations

    APINN can be more expensive than XPINN because it shares more data points and parameters, and its generalization bound relies on an assumption stated for a broad class of PDEs.

Abstract

from arXiv · show

In this paper, we propose the augmented physics-informed neural network (APINN), which adopts soft and trainable domain decomposition and flexible parameter sharing to further improve the extended PINN (XPINN) as well as the vanilla PINN methods. In particular, a trainable gate network is employed to mimic the hard decomposition of XPINN, which can be flexibly fine-tuned for discovering a potentially better partition. It weight-averages several sub-nets as the output of APINN. APINN does not require complex interface conditions, and its sub-nets can take advantage of all training samples rather than just part of the training data in their subdomains. Lastly, each sub-net shares part of the common parameters to capture the similar components in each decomposed function. Furthermore, following the PINN generalization theory in Hu et al. [2021], we show that APINN can improve generalization by proper gate network initialization and general domain & function decomposition. Extensive experiments on different types of PDEs demonstrate how APINN improves the PINN and XPINN methods. Specifically, we present examples where XPINN performs similarly to or worse than PINN, so that APINN can significantly improve both. We also show cases where XPINN is already better than PINN, so APINN can still slightly improve XPINN. Furthermore, we visualize the optimized gating networks and their optimization trajectories, and connect them with their performance, which helps discover the possibly optimal decomposition. Interestingly, if initialized by different decomposition, the performances of corresponding APINNs can differ drastically. This, in turn, shows the potential to design an optimal domain decomposition for the differential equation problem under consideration.

1 Introduction

APINN introduces a soft, trainable alternative to XPINN’s hard domain decomposition, combining flexible subnet structure with shared parameters. Theory and experiments support improved generalization and performance across cases where XPINN is weaker than, comparable to, or better than PINN.

  • APINN uses a trainable gate network to mimic XPINN’s hard decomposition while allowing the partition to be fine-tuned.
  • Its gate removes interface losses, lets every subnet use all training samples, and supports partial parameter sharing across decomposed functions.
  • APINN’s generalization analysis derives bounds for fixed and trainable gates and supports soft, trainable domain and function decomposition.
  • Experiments show APINN can significantly improve PINN and XPINN when XPINN performs similarly to or worse than PINN.
  • When XPINN already outperforms PINN, APINN can still slightly improve XPINN, while initialization by different decompositions can produce markedly different performance.

2 Related Work

Related work develops PINN domain-decomposition methods, theoretical analyses, and parallel variants, while FBPINN provides an interface-free but fixed and non-sharing alternative. These approaches frame APINN’s emphasis on trainable decomposition and parameter sharing.

  • PINN solves forward and inverse PDE problems with neural networks trained to satisfy governing physical laws and data constraints.
  • CPINN and XPINN introduced domain decomposition, with XPINN using sub-PINNs and interface losses to enforce continuity across space-time subdomains.
  • Parallel PINN trains decomposed CPINN and XPINN models across multiple GPUs or CPUs, while Parareal PINN divides long time domains into shorter subdomains.
  • FBPINN removes interface conditions through overlapping subdomains but fixes the decomposition, disallows parameter sharing, and can become costly in multiple dimensions.
  • Prior work analyzes PINN and XPINN generalization, including theoretical results for prototypical nonlinear PDEs such as Navier–Stokes equations.

3 Problem Definition and Background

PINN approximates PDE solutions by minimizing boundary and residual losses, whereas XPINN decomposes the domain into subdomains and adds interface losses to maintain continuity. The background also identifies practical difficulties with higher-order interface conditions and states APINN’s partition-of-unity structure.

  • PINN formulation: PINN approximates the ground-truth PDE solution by minimizing a training loss containing boundary and residual terms.
  • PINN formulation: The boundary term learns boundary conditions, while the residual term enforces the physical laws described by the PDE.
  • XPINN formulation: XPINN partitions the domain into subdomains, applies a sub-PINN to each, and combines their losses with interface terms enforcing continuity.
  • XPINN formulation: XPINN interface loss includes solution and residual continuity terms based on points shared by neighboring subdomains.
  • XPINN limitations: Residual continuity is difficult to enforce accurately for PDEs with higher-order derivatives, and XPINNv2 adds first-order derivative continuity to address this issue.
  • APINN structure: APINN’s gate weights subnetwork outputs and satisfies a partition-of-unity condition, with the weights summing to one.

4 Augmented PINN (APINN)

APINN combines a shared network, multiple sub-nets, and a gating network that weight-averages sub-net outputs for soft, potentially trainable domain decomposition. Its parameter sharing improves efficiency, while its interpretable decomposition supports visualization and initialization-based refinement.

  • 4.1 Parameterization of Augmented PINN: APINN uses a shared network h, m sub-nets E_i, and a gating network G mapping inputs to simplex weights.The gating network weight-averages the sub-net outputs to produce the APINN solution.
  • 4.1 Parameterization of Augmented PINN: APINN is a universal approximator because ordinary multilayer networks are included as a special case when all sub-nets are identical.Choosing E_1 = E_2 = ··· = E_m makes the weighted model degenerate to a vanilla multilayer network.
  • 4.1 Parameterization of Augmented PINN: A trainable gate can fine-tune a pretrained XPINN-like decomposition, while a fixed gate yields the soft version of XPINN.The gate can be initialized from hard upper/lower or inner/outer partitions and then optimized when trainable.
  • 4.2 Parameter Sharing and 4.3 Difference in the position of h: APINN shares parameters across sub-nets to improve efficiency while preserving an interpretable weighted decomposition of the learned function components.Unlike the alternative placing h outside the weighted average, this formulation allows each E_i ◦ h to be visualized.
  • 4.1 Parameterization of Augmented PINN: Gating functions support upper/lower, inner/outer, L-shaped, and high-dimensional decompositions through appropriately chosen spatial weight functions.The examples use exponential functions to concentrate sub-net weights in selected regions.

5 Theoretical Analysis

The analysis explains APINN generalization through soft decomposition, data reuse, interface-loss removal, and function decomposition, with trainable gates adding their own complexity. Proper initialization can reduce this added complexity, while APINN recovers XPINN’s bound under hard-like gating.

  • 5.2 XPINN Generalization: XPINN generalization balances simpler decomposed targets against increased complexity and overfitting from limited subdomain data.XPINN can outperform PINN when target simplification dominates, but PINN can win when data scarcity and overfitting dominate.
  • 5.3 APINN with Non-Trainable Gate Network: For a fixed gate, APINN’s generalization bound contains training loss, model Rademacher complexity, and a probability term over sub-network parameters.The fixed gate contributes through its bounded weights rather than trainable complexity.
  • 5.4 Explain the Effectiveness of APINN via Theorem 5.1: APINN’s benefits include soft domain decomposition, no interface losses, general function decomposition, and all-data access for every sub-PINN.Using all training data is identified as preventing overfitting.
  • 5.4 Explain the Effectiveness of APINN via Theorem 5.1: When the gate mimics hard XPINN decomposition, APINN obtains the XPINN bound while avoiding complex interface losses and interface-near errors.The hard-decomposition reduction is stated explicitly, and the resulting training loss is usually smaller than XPINN’s.
  • 5.6 Numerical Results: For Burgers’ equation, APINN improves XPINN near the steep interface region by removing interface losses and improving parameter efficiency.The steep solution change near x = 0 is difficult for PINNs, and XPINN performs poorly near its interface.
  • 5.4 Explain the Effectiveness of APINN via Theorem 5.1: APINN can outperform PINN when the weighted complexity of simpler sub-PINNs is smaller than the complexity of a single PINN.Each sub-PINN uses all training samples, while the model complexity is a weighted average of sub-PINN complexities.
  • 5.5 APINN with Trainable Gate Network: With a trainable gate, both the gate and sub-PINNs contribute complexity to the generalization bound.The trainable gate contributes R_i(G), unlike a fixed gate, whose contribution is represented through its infinity norm.
  • 5.5 APINN with Trainable Gate Network: Good XPINN-inspired initialization can keep gate parameters near initialization, reduce complexity terms, and improve the generalization bound.The argument links smaller parameter deviations to smaller R_i(G) and a lower right-hand side in Theorem 5.2.

6 Computational Experiments

Across PDE experiments, APINN improves PINN and XPINN by avoiding interface losses, using trainable soft decompositions, and leveraging flexible initialization. Results also show that gate initialization and decomposition strongly affect performance.

  • Burgers equation: The Burgers equation has a steep region near x = 0 that is difficult for PINNs and causes large XPINN errors near the interface.APINN improves interface accuracy by removing interface losses and improving parameter efficiency.
  • Burgers equation: APINN-X performs best on Burgers, while APINN-M performs worse because its MPINN initialization is less suitable for this problem.APINN-X models the steep region without XPINN-style interface conditions.
  • Burgers equation: Fixed-gate APINN-X-F performs slightly worse than trainable APINN and PINN but still significantly outperforms XPINN, supporting soft partitioning and trainable decomposition.The comparison separates the benefit of soft domain partitioning from the additional benefit of gate fine-tuning.
  • Gating-network behavior: Optimized gates preserve initialization characteristics, and different initializations can produce substantially different gate weightings and relative L2 errors.APINN-X gates retain similar left-right decompositions, whereas APINN-M runs can weight the two subnets very differently.
  • Comparative results: Across additional PDE cases, APINN outperforms PINN and XPINN when XPINN is weak, while still slightly improving cases where XPINN already reduces PINN relative L2 error by 27%.In one case APINN-M is best; in another, APINN remains best despite XPINN already outperforming PINN.
  • Comparative results: Using four subnets does not necessarily improve performance, because the number of training samples can trade off against target-function complexity and parameter efficiency.The best APINN-M error remains concentrated near steep regions where the solution changes rapidly.

7 Summary

APINN combines soft, trainable domain decomposition with partial parameter sharing to improve on XPINN and PINN. Its gate network removes interface losses, uses all training samples across sub-networks, and supports decomposition refinement and generalization.

  • Method: APINN uses a trainable gate network to softly partition the domain, mimic XPINN decomposition, and fine-tune the partition.The gate satisfies partition of unity and averages multiple sub-networks as the APINN output.
  • Advantages: APINN avoids complicated interface losses because soft gating maintains continuity between sub-networks.The paper associates this design with better convergence and lower training loss.
  • Advantages: The gate can decompose a complicated target function into simpler parts, reducing sub-network complexity and improving generalizability.This gives APINN a decomposition advantage while retaining the ability to mimic XPINN.
  • Advantages: Trainable gating can refine both domain and function decompositions to discover simpler components and improve generalization.The paper relates this benefit to the PINN generalization theory of Hu et al. [2021].
  • Advantages: Partial parameter sharing captures common components across sub-networks because they learn parts of the same target function.Sharing is presented as an efficiency mechanism for the sub-networks.
  • Advantages: Each APINN sub-network uses all domain training samples, whereas XPINN sub-networks use only samples from their subdomains.The paper presents this distinction as a way to prevent over-fitting.
  • Theory and evidence: APINN benefits are supported empirically and theoretically, including generalization bounds for fixed and trainable gate networks.Certain gate networks can recover PINN and XPINN, while general domain and function decomposition reduces the

A.1 Preliminary

The preliminary material defines the complexity tools used in the theoretical analysis, including Rademacher complexity, covering numbers, and truncated neural-network classes.

  • Complexity tools: The proof framework uses Rademacher complexity and covering numbers to analyze neural-network function classes.These quantities are introduced before the theorem proofs.
  • Rademacher complexity: Rademacher complexity measures a function class on a dataset using independent random signs taking values in {−1, 1}.The definition is stated for a dataset containing n samples.
  • Covering numbers: Matrix covering number N(U, ϵ, ∥·∥) is the smallest cardinality of a subset that covers U at scale ϵ under the specified norm.The covering condition bounds the distance from every element of U to a selected cover element.
  • Complexity tools: A lemma relates Rademacher complexity to function-class images on the dataset under boundedness assumptions.The stated hypothesis class takes values in [0, 1] and contains the zero function.
  • Truncated classes: The analysis uses truncated neural networks so the function class is bounded in [−1, 1], matching the assumed bound on the target function.Its Rademacher complexity is bounded by the sum of the complexities of the positive and negative truncated classes.

A.2 Proof of Theorem 5.1

The proof of Theorem 5.1 bounds the complexity of APINN-style function classes by combining sub-class complexity, gate regularity, and known generalization relationships.

  • Proof strategy: The proof decomposes the APINN function class into component classes associated with multilayer networks and composed mappings.Each F_j represents a function class corresponding to a component such as E_j ◦ h.
  • Complexity bound: Rademacher complexity is bounded using subadditivity and the Lipschitz property of the gate multiplication term.The multiplication by G(x)_j is controlled by its maximum boundary norm.
  • Conclusion: The proof concludes by combining the derived inequality with the relationship between Rademacher complexity and generalization error and Theorem 3.2 of Hu et al. [2021].

A.3 Proof of Theorem 5.2

The proof of Theorem 5.2 derives covering-number bounds for products of neural-network function classes and connects them to Rademacher complexity and generalization.

  • Proof strategy: The proof seeks a covering number for the product class (F · G)_S from the covering numbers of F_S and G_S.The component covering numbers are assumed to have logarithmic forms involving the respective network complexities and ϵ.
  • Covering bound: An ϵ-cover for each factor yields a product cover through the inequality ∥fg − f̂ĝ∥ ≤ ∥g∥·∥f − f̂∥ + ∥f̂∥·∥g − ĝ∥.Under the truncated-network assumptions, the product covering number is bounded by the product of the factor covering numbers.
  • Conclusion: The resulting covering-number inequality is combined with a Rademacher-complexity bound, generalization-error relations, and Theorem 3.2 of Hu et al. [2021].

B Optimization trajectory of the gating network for all subnets in the Boussinesq-Burger equation

The gating network optimization trajectory for all subnets in the Boussinesq-Burger equation is visualized through successive training snapshots. The snapshots span epochs 0 through 50.

  • The visualization tracks the gating network optimization trajectory for all subnets in the Boussinesq-Burger equation.
  • Figure 28 presents the snapshots from top to bottom as the optimization progresses.
Loading 2211.08939v3…