PaperlayerView exact revision on arXiv

Source-linked AI summary

Auto-PINN: Understanding and Optimizing Physics-Informed Neural Architecture

Yicheng Wang, Xiaotian Han, Chia-Yuan Chang, Daochen Zha, Ulisses Braga-Neto, Xia Hu

arXiv:2205.13748v2cs.LG

TL;DR

PINN architecture and hyperparameter selection remains an open challenge, despite its importance to solver performance and the labor involved in manual tuning. Auto-PINN uses pre-experiments to guide decoupled hyperparameter search, and it outperforms baseline strategies in accuracy and stability with fewer trials.

  • Problem

    PINN architecture configurations affect performance, but their relationships are underexplored and manual architecture and hyperparameter tuning is laborious.

    Source: §1 · pp. 1–2

  • Method

    Auto-PINN uses pre-experiments to guide a step-by-step decoupled search over PINN architecture and hyperparameters.

    Source: §1 · p. 2

  • Results

    Across seven PDE benchmarks, Auto-PINN outperforms other optimization strategies in accuracy and stability with fewer search trials.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Pre-experiments indicate that PINN training loss is correlated with error and is a desirable search objective across PDEs.

    Source: §1 · p. 2

Abstract

from arXiv · show

Physics-informed neural networks (PINNs) are revolutionizing science and engineering practice by bringing together the power of deep learning to bear on scientific computation. In forward modeling problems, PINNs are meshless partial differential equation (PDE) solvers that can handle irregular, high-dimensional physical domains. Naturally, the neural architecture hyperparameters have a large impact on the efficiency and accuracy of the PINN solver. However, this remains an open and challenging problem because of the large search space and the difficulty of identifying a proper search objective for PDEs. Here, we propose Auto-PINN, the first systematic, automated hyperparameter optimization approach for PINNs, which employs Neural Architecture Search (NAS) techniques to PINN design. Auto-PINN avoids manually or exhaustively searching the hyperparameter space associated with PINNs. A comprehensive set of pre-experiments using standard PDE benchmarks allows us to probe the structure-performance relationship in PINNs. We find that the different hyperparameters can be decoupled, and that the training loss function of PINNs is a good search objective. Comparison experiments with baseline methods demonstrate that Auto-PINN produces neural architectures with superior stability and accuracy over alternative baselines.

1 Introduction

PINN architecture choices matter, but their performance relationships and appropriate search strategies remain underexplored. Auto-PINN uses pre-experiments to guide a decoupled search and reports better accuracy and stability than baseline strategies with fewer trials.

1 Introduction

  • PINN architecture configurations such as depth, width, and activation affect performance, but existing findings are limited or inconclusive across PDE problems.The cited width-and-depth study covered one PDE and a small search space; studies of activation functions did not establish a preferred choice across PDEs.Source: §1 · p. 1
  • Manual architecture and training-hyperparameter tuning is laborious, while the mixed search space is large and PDE search objectives are unclear.Many PDEs may lack exact solutions, so error values are unavailable as a natural search objective.Source: §1 · p. 2
  • Pre-experiments suggest hyperparameters can be decoupled step by step, while PINN training loss is correlated with error and offers a search objective across PDEs.Dominant choices, such as the activation function, can be selected first and fixed while searching other hyperparameters.Source: §1 · p. 2
  • Auto-PINN outperforms other optimization strategies in accuracy and stability with fewer search trials across seven PDE benchmarks.The pre-experiments guide its step-by-step decoupling strategy.Source: §1 · p. 2

2 Preliminaries

The preliminaries specify the PDE benchmarks and the architecture search space used in the study.

2.1 Benchmarking PDEs.

  • Four representative PDEs—two diffusion equations, a wave equation, and Burgers’ equation—support pre-experiments; formal comparisons add two advection equations and a reaction equation.The benchmarks include different differential operators and boundary or initial conditions, with varied sampling schemes.Source: §2.1 · p. 2

2.2 Search Space

  • The architecture search space includes width, depth, four activation functions, and five discrete Changing Point values for the Adam-to-L-BFGS switch.Changing Point is the proportion of total epochs trained with Adam; the remainder uses L-BFGS.Source: §2.2 · p. 3
  • Learning rates and training epochs are excluded from the initial search because later results indicate they have a small effect on final architecture search results..Source: §2.2 · p. 3

3 Pre-Experiments and Observations

Pre-experiments identify hyperparameter patterns that can guide PINN search, while showing that structure-performance relationships vary across PDEs. They also find a strong relationship between training loss and error, supporting loss as a search objective.

3 Pre-Experiments and Observations

  • A dominant activation function can be identified for each PDE by searching a small subset, with median error serving as a selection metric.Tanh is given as the best choice for Heat_0.Source: §3 · p. 3
  • Under the dominant activation function, larger changing points perform better or comparably to smaller values.Source: §3 · p. 3

3.1 Structure-Error Relationship

  • The “wider and deeper PINNs are better” rule does not hold for every PDE, including Wave.Further structure experiments find width regions where the best-performing network is not the deepest, including for Heat_0 and Wave.Source: §3 · p. 3, §3.1 · p. 5

3 Pre-Experiments and Observations

  • The error at the minimum training loss is usually close to the lowest error reached during training, although this is only a local comparison within each configuration.Source: §3 · p. 3

3.1 Structure-Error Relationship

  • Good-performing width regions can be identified before searching for the optimal depth, since the best depth depends on width.Source: §3.1 · p. 5

3.2 Loss-Error Relationship

  • Strong linear correlation between log-loss and log-error across configurations in a PDE problem makes training loss a suitable performance-assessment objective.The study reports this relationship across the configuration space, not just within individual heatmap cells.Source: §3.2 · p. 5

4 Auto-PINN: Automated Architecture Optimization for PINNs

Auto-PINN uses pre-experiment findings to search PINN hyperparameters in a decoupled sequence, using training loss to guide the search. Its default settings sharply reduce the number of trials needed to find architectures.

4.1 Methodology

  • Auto-PINN searches one hyperparameter at a time while fixing the others to one or a few options, reducing the search-space scale.Source: §4 · p. 5
  • The search uses the smallest training loss values as its objective, first selecting an activation function with median error and then narrowing width regions and searching depth.The selected activation function is fixed for subsequent steps; promising width intervals are used to identify candidate structure configurations.Source: §4.1 · pp. 5–6
  • The procedure searches for a changing point for each candidate, then retrains each selected architecture five times with different initializations and reports median performance.Source: §4.1 · p. 6

4.2 Complexity Analysis

  • At most 261 trials find the best architectures under Auto-PINN’s default settings, versus 10,080 possible configurations (2.59% of the search space).Source: §4.2 · p. 6

5 Experiments

Across seven PDE benchmarks, Auto-PINN is reported to outperform Random Search and HyperOpt in stability and accuracy with fewer search trials. Experiments also examine training hyperparameters and data sampling, finding distinct sampling-related architecture preferences across PDEs.

5.1 Experimental Settings

  • Experiments use seven PDE benchmarks and compare Auto-PINN with Random Search and HyperOpt, which each receive 300 samples.Implementation uses a learning rate of 1e −5 and 10000 training epochs.Source: §5.1 · p. 6

5.2 Comparison Results

  • Auto-PINN outperforms Random Search and HyperOpt in stability and accuracy with fewer search trials.The concentrated error distributions contrast with frequent large performance variances for the baselines; median Auto-PINN errors match or exceed baseline performance in most cases.Source: Figure 4 · p. 7, §5.2 · p. 7

5.3 Influence of Learning Rates and Training Epochs

  • Searched architectures cluster across different learning rates and epoch settings, while PDEs show distinct preferences for network width and depth.Heat_0 requires wider structures but is insensitive to depth; Wave is insensitive to width but prefers shallower PINNs.Source: Figure 5 · p. 7, §5.3 · p. 7

5.4 Influence of Data Sampling

  • The structure–sampling relationship differs by PDE, so searched architectures are not appropriate to reuse when the sampling strategy changes.The experiments compare random and uniform sampling at two densities for collocation, boundary, and initial points.Source: Figure 6 · p. 8, §5.4 · p. 8

6 Conclusion and Future Work

The conclusion presents Auto-PINN as a decoupled search approach using PINN loss as its objective. It reports stability and effectiveness, notes sensitivity to sampling strategy, and identifies more sophisticated sampling methods as future work.

6 Conclusion and Future Work

  • Auto-PINN searches neural architectures and hyperparameters for different PDEs using step-by-step decoupling and PINN loss as the search objective.The authors describe it as the first systematic approach of this kind for PINNs.Source: §6 · p. 8
  • The comparison experiments demonstrate Auto-PINN’s stability and effectiveness, while the best configuration depends on the adopted sampling strategy.The authors report performance is not sensitive to learning rates and training epochs.Source: §6 · p. 8
  • Future work will incorporate more sophisticated data sampling strategies into Auto-PINN’s search space to pursue better performance.The authors also hope the observations motivate future PINN research and that Auto-PINN serves as a strong baseline.Source: §6 · p. 8

Appendix A Benchmarking PDEs and Data Samplings

Appendix A introduces the PDEs and data sampling strategies used in the experiments.

Appendix A Benchmarking PDEs and Data Samplings

  • Appendix A presents the PDEs and data sampling strategies used in the experiments.Source: Appendix A · p. 11

A.1 Benchmarking PDEs

The benchmark set spans heat, wave, Burgers, advection, and reaction PDEs, with varied differential operators and boundary or initial conditions. These standard problems are used to illustrate regular PINN behavior.

A.1 Benchmarking PDEs

  • The seven benchmarks include two heat equations, a wave equation, Burgers’ equation, two advection equations, and a reaction equation.The benchmark set is described as using standard PDEs.Source: §A.1 · p. 11
  • Heat_0 uses Dirichlet boundary conditions, with zero values at x = −1 and x = 1 and initial condition u(x, 0) = sin πx.The heat equation describes temperature distribution over time.Source: §A.1 · p. 11
  • Heat_1 uses Neumann boundary conditions on its heat-equation benchmark.Its displayed solution is u(x, t) = 1 + t + e^−4π²t cos 2πx + x sin t.Source: §A.1 · p. 11
  • The wave benchmark models oscillation propagation and combines Dirichlet and Neumann conditions.The cited passage describes mechanical and electromagnetic waves as examples.Source: §A.1 · p. 11
  • Burgers’ equation is used for applications including shock flows, combustion-chamber wave propagation, and vehicular traffic, with an accurate approximate solution.Its boundary values are zero at x = −1 and x = 1; its initial condition is u(x, 0) = −sin πx.Source: §A.1 · p. 11
  • Advection_0 describes a scalar field transported by a known velocity field, while Advection_1 differs from it in boundary conditions.The reaction equation describes chemical reactions.Source: §A.1 · pp. 11–12

A.2 Data Sampling Methods

The appendix describes the sampling schemes used in the experiments and introduces mathematical expressions for the activation functions in the search space.

A.2 Data Sampling Methods

  • The experiments use random and uniform sampling methods, with details provided in Table 1.Random denotes pseudo-random sampling, while uniform sampling uses a grid in the spatio-temporal domain; numeric suffixes distinguish sampling densities.Source: §A.2 · p. 12

Appendix B Activation Functions

  • Appendix B presents mathematical expressions for the four activation functions in the search space.Source: Appendix B · p. 13

C.1 Additional Pre-Experiment Results

The appendix supplies further pre-experiment visualizations and details of Auto-PINN’s search procedure, trial complexity, and selected architectures.

C.1 Additional Pre-Experiment Results

  • The additional error heatmaps cover Heat_0, Heat_1, Wave, and Burgers, with rows ordered by the sampling methods in Table 1.The main-paper heatmaps are included for reference.Source: §C.1 · p. 14, Figure 7 · p. 14, Figure 8 · p. 15
  • Figures 11 and 12 provide additional structure-error and loss-error relationship results, respectively.Source: §C.1 · p. 17

C.2 More Details about Auto-PINN

  • The default procedure fixes changing points, selects an activation function, searches width and depth, tunes changing points for five candidates, then verifies selected architectures.The activation choice is selected using the median search objective; top width intervals and candidate structures are also selected using that objective.Source: §C.2 · pp. 17–18
  • Auto-PINN uses at most 261 trials, or 2.59% of the 10,080-configuration default search space, excluding Step 4 verification.The analysis breaks this down across Steps 1, 2.1, 2.2, and 3.Source: §C.2 · p. 18
  • Table 2 lists Auto-PINN’s best architectures for each PDE and sampling method, selected by smallest L2 error among the top five results.The appendix states that experiments use a learning rate of 1e−5 and 10,000 training epochs.Source: Table 2 · p. 18, §C.2 · p. 18

Related papers

Loading 2205.13748v2…