PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning the solution operator of parametric partial differential equations with physics-informed DeepOnets

Sifan Wang, Hanwen Wang, Paris Perdikaris

arXiv:2103.10974v1cs.LGmath.NAstat.ML

TL;DR

DeepONets require costly paired observations and may produce outputs inconsistent with physical laws. The paper introduces physics-informed DeepONets, which impose governing equations through automatic differentiation, improving accuracy and data efficiency while enabling rapid parametric-PDE solution.

  • Problem

    DeepONets typically require large, expensive paired input-output datasets, and their learned operators may not remain consistent with the physical laws underlying observed data.

  • Method

    Physics-informed DeepONets use automatic differentiation to penalize governing-equation residuals and bias DeepONet outputs toward physical consistency.

  • Results

    Physics-informed DeepONets improve predictive accuracy and data efficiency, solve parametric PDEs without paired input-output observations, and predict O(10^3) time-dependent PDE solutions in a fraction of a second.

  • Takeaways & Limitations

    The framework supports unsupervised parametric-PDE solution and can accelerate scientific modeling across engineering, Earth system science, and computational biology.

  • Takeaways & Limitations

    The optimal feature embedding or network architecture for a given parametric PDE remains an open question, particularly for multi-scale operators and PDEs.

Abstract

from arXiv · show

Deep operator networks (DeepONets) are receiving increased attention thanks to their demonstrated capability to approximate nonlinear operators between infinite-dimensional Banach spaces. However, despite their remarkable early promise, they typically require large training data-sets consisting of paired input-output observations which may be expensive to obtain, while their predictions may not be consistent with the underlying physical principles that generated the observed data. In this work, we propose a novel model class coined as physics-informed DeepONets, which introduces an effective regularization mechanism for biasing the outputs of DeepOnet models towards ensuring physical consistency. This is accomplished by leveraging automatic differentiation to impose the underlying physical laws via soft penalty constraints during model training. We demonstrate that this simple, yet remarkably effective extension can not only yield a significant improvement in the predictive accuracy of DeepOnets, but also greatly reduce the need for large training data-sets. To this end, a remarkable observation is that physics-informed DeepONets are capable of solving parametric partial differential equations (PDEs) without any paired input-output observations, except for a set of given initial or boundary conditions. We illustrate the effectiveness of the proposed framework through a series of comprehensive numerical studies across various types of PDEs. Strikingly, a trained physics informed DeepOnet model can predict the solution of $\mathcal{O}(10^3)$ time-dependent PDEs in a fraction of a second -- up to three orders of magnitude faster compared a conventional PDE solver. The data and code accompanying this manuscript are publicly available at \url{https://github.com/PredictiveIntelligenceLab/Physics-informed-DeepONets}.

1 Introduction

Parametric PDE learning requires solution operators that map varying functions, geometries, parameters, and initial or boundary conditions to PDE solutions. Physics-informed DeepONets address expensive data requirements and physical inconsistency by enforcing PDE constraints during training.

  • Parametric PDEs require learning operators between infinite-dimensional function spaces rather than ordinary finite-dimensional functions.
  • Traditional solvers require independent simulations for different geometries, parameters, or initial and boundary conditions, creating substantial computational cost.
  • Large paired input-output datasets can require expensive experiments or high-fidelity simulations, motivating training from PDE knowledge and initial or boundary conditions alone.
  • Physics-informed DeepONets use automatic differentiation to regularize outputs toward satisfying governing PDE constraints, even without latent-output training data.
  • 1–2 orders of magnitude reduction in predictive errors and up to 100% reduction in required examples are reported for physics-informed DeepONets.
  • The paper develops and evaluates the approach through subsequent architectural, technical, and numerical studies across parametric PDE settings.

2 Learning operators with DeepONets

DeepONets represent solution operators with branch and trunk networks whose latent representations are combined to produce continuous output functions. Their approximation can nevertheless violate governing equations, motivating physics-informed constraints and evaluation on an anti-derivative problem.

  • Parametric PDEs allow domain shapes, initial or boundary conditions, coefficients, and source terms to vary over specified ranges.
  • A DeepONet uses a branch net for sampled input functions and a trunk net for continuous coordinates, combining their feature embeddings through a dot product.
  • Training data pair sampled input functions and output evaluations across multiple coordinates, producing tensors with dimensions determined by samples, locations, sensors, and output dimension.
  • The anti-derivative experiment trains on 10,000 Gaussian-process input functions and corresponding RK45 solutions, using 100 sensors and testing 1,000 random samples.
  • Automatic differentiation computes derivatives of the inferred solution with respect to input coordinates so residuals can be compared with the input function.
  • A conventional DeepONet can fit the solution while approximating its derivative residual with step functions, producing large approximation error.
  • Similar poor predictions with ReLU and tanh indicate that universal operator approximation does not ensure compatibility with the ground-truth solution operator.

3 Physics-informed DeepONets

Physics-informed DeepONets incorporate PDE residual penalties into DeepONet training, using collocation points to enforce physical constraints. In the anti-derivative example, they preserve prediction accuracy while improving data efficiency and satisfying the underlying ODE constraint.

  • Physics-informed DeepONets extend DeepONets by minimizing PDE residuals alongside the operator-learning loss.The approach uses automatic differentiation to impose physical constraints through regularization.
  • The trained physics-informed DeepONet closely matches the exact solution and derivative, while satisfying the underlying ODE constraint.The result is reported over a representative input and supported by mean relative L2 errors over 1,000 test realizations.
  • Collocation points are randomly sampled from each output domain to approximately enforce the governing physical constraints.
  • The anti-derivative demonstration uses five-layer branch and trunk networks with 50 hidden units per layer and tanh activations.Training minimizes the composite loss for 40,000 Adam iterations.
  • Conventional DeepONets require more than 10x training data to achieve the same accuracy as their physics-informed counterpart.This comparison uses mean relative L2 error across different numbers of input-function samples.

4 Numerical results

The numerical studies evaluate physics-informed DeepONets across parametric ODEs and PDEs, including settings with no paired solution observations. Results show accurate predictions, benefits from Fourier features and loss weighting, and rapid inference for many time-dependent PDEs.

  • 4.1 The anti-derivative operator: The anti-derivative example trains a physics-informed DeepONet using input-function measurements and a zero initial condition, without paired input-output data.The model samples 10,000 input functions and uses the zero initial condition s(0)=0 as the available solution information.
  • 4.1 The anti-derivative operator: 40,000 training iterations produced excellent agreement between physics-informed DeepONet predictions and ground-truth solutions for test inputs.The conventional DeepONet instead learned a degenerate map that fit s(0)=0 but produced erroneous predictions for x>0.
  • 4.1 The anti-derivative operator: Fourier feature embeddings improved high-frequency solution learning, whereas a conventional fully connected trunk failed to accurately capture high-frequency oscillations.The Fourier-feature model used σ=50 and was trained for 300,000 Adam iterations.
  • 4.1 The anti-derivative operator: 7.12e-03 was the relative L2 prediction error averaged over 1,000 out-of-distribution test examples sampled with length scale l=0.2.Training inputs for this case used length scale l=0.01.
  • 4.2 Diffusion-reaction systems: In the diffusion-reaction benchmark, physics-informed DeepONet achieved an average relative L2 error of ∼0.45%, compared with ∼1.92% for conventional DeepONet.The conventional model used 100 randomly selected solution measurements per input function, while the physics-informed model used the PDE constraints.
  • 4.3 Burgers’ equation: O(10^3) Burgers-equation solutions can be predicted in a fraction of a second, with inference taking ∼10ms and reaching up to three orders of magnitude speedup over a conventional spectral solver.The inference is trivially parallelizable.

5 Summary and Discussion

Physics-informed DeepONets improve predictive accuracy and data-efficiency while enabling unsupervised solution of parametric PDEs. The paper also identifies open questions around architectures, loss weighting, training, accuracy, and robustness.

  • Physics-informed DeepONets reduce predictive errors by up to 1-2 orders of magnitude while improving generalization performance.
  • Physics-informed DeepONets can reduce the number of training examples required by up to 100%.
  • The framework can solve parametric PDEs without paired input-output observations, using initial or boundary conditions instead.
  • Open questions include optimal feature embeddings, network architectures, loss weights, and algorithms for training accuracy and robustness.

A Nomenclature

This section identifies the tables that summarize the paper’s notation and nomenclature.

  • Table 4 summarizes the main symbols and notation used throughout the work.

B Hyper-parameter settings

The benchmark models use specified activation functions, optimization settings, and architecture tables as default hyper-parameter references.

  • Most benchmarks use Tanh activations in both branch and trunk networks, while the Eikonal airfoil benchmark uses ELU activations.
  • Physics-informed DeepONets are trained with mini-batch Adam optimization using batches of 10,000 and exponential learning-rate decay.The decay rate is 0.9 every 1,000 iterations.
  • Table 5 provides default hyper-parameter settings for each benchmark unless otherwise stated.
  • Tables 6 and 7 list the physics-informed and conventional DeepONet architectures used for each benchmark.

C Computational cost

Physics-informed DeepONets require more training time than conventional DeepONets, but their trained models provide highly rapid parallel inference for many PDEs.

  • Training: Physics-informed DeepONet training is generally slower than conventional DeepONet training because it requires additional computations.
  • Training: Table 8 reports computational costs in hours for training DeepONet and physics-informed DeepONet models on a single NVIDIA V100 GPU.
  • Inference: ~10ms is sufficient for a trained physics-informed DeepONet to predict the entire spatio-temporal Burgers solution.
  • Inference: O(10^3) PDEs can be solved in a fraction of a second, with up to three orders of magnitude speedup over a traditional spectral solver.Figure 14 compares inference timing for trained physics-informed DeepONets with conventional spectral-solver timing.

D Anti-derivative

The figures document physics-informed DeepONet training and predictions for anti-derivative, parametric ODE, irregular-input ODE, and diffusion-reaction problems. These studies include settings without paired input-output data and evaluate both in-distribution and out-of-distribution examples.

  • 40,000 Adam iterations compare training-loss convergence for conventional DeepONets with different activations and physics-informed DeepONets using Tanh activations.
  • A physics-informed DeepONet solves a 1D parametric ODE without paired input-output data, using only the initial condition during training.
  • Predicted solutions and corresponding ODE residuals are shown across three 1D parametric ODE test examples.
  • For irregular input functions, training-loss convergence is compared between conventional fully connected and Fourier feature architectures over 300,000 Adam iterations.
  • The irregular-input ODE experiments report predicted solutions and ODE residuals for four in-distribution examples with each architecture, plus four out-of-distribution Fourier-feature examples.
  • For a parametric diffusion-reaction system, training-loss convergence is compared with and without paired input-output data, and predictions are shown for three test examples.

F Effect of the batch-size

The diffusion-reaction study examines how batch size affects physics-informed DeepONet accuracy and training-loss convergence.

  • Relative L2 prediction error is averaged over 1,000 test examples for physics-informed DeepONets trained with different batch sizes.
  • Training-loss convergence is compared across physics-informed DeepONets trained using different batch sizes.

G.1 Modified full-connected neural network

The modified fully connected architecture combines two transformed input branches through learned element-wise gates. Its use is illustrated for Burgers’ and parametric Eikonal equations.

  • G.1 Modified full-connected neural network: The modified MLP forward pass is introduced as the architecture used in the subsequent formulation.
  • G.1 Modified full-connected neural network: U and V are independently computed by applying the activation function to affine transformations of the network inputs X.
  • G.1 Modified full-connected neural network: At each layer, Z(k) is computed from H(k), then gates an element-wise mixture of U and V to produce H(k+1).
  • G.1 Modified full-connected neural network: The architecture adds transformer-network weights and biases to the parameters otherwise used in a standard fully connected network.
  • Burgers’ equation: A modified MLP architecture is used to predict solutions for three Burgers’ equation test examples.
  • Parametric Eikonal equation: For the parametric Eikonal equation, training-loss convergence is shown with Tanh and ELU activations, and signed distance functions are predicted for three test examples.
Loading 2103.10974v1…