PaperlayerView exact revision on arXiv

Source-linked AI summary

Reliable extrapolation of deep neural operators informed by physics or sparse observations

Min Zhu, Handi Zhang, Anran Jiao, George Em Karniadakis, Lu Lu

arXiv:2212.06347v1cs.LGphysics.comp-ph

TL;DR

Deep neural operators can fail when inference inputs fall outside the training support, motivating reliable extrapolation methods. This paper measures DeepONet extrapolation complexity with the 2-Wasserstein distance, develops physics- and sparse-observation-informed methods, and reports effective extrapolation across diverse PDE problems. It also provides method-selection guidance based on available information, accuracy, and inference speed.

  • Problem

    DeepONets and other neural networks are usually accurate for inputs within training support but may produce large errors when extrapolation is required.

  • Method

    The paper quantifies extrapolation using the 2-Wasserstein distance and uses PDE-informed fine-tuning, sparse-observation fine-tuning, or multifidelity learning.

  • Results

    The proposed methods demonstrate excellent extrapolation capability for diverse PDE problems, while extrapolation error increases with function-space distance and shows a U-shaped dependence on model capacity.

  • Takeaways & Limitations

    Method choice can be guided by whether physics or sparse observations are available, together with desired accuracy and inference speed.

  • Takeaways & Limitations

    The study considers either complete physics or sparse observations, leaving combined sparse observations with partial physics for future method development.

Abstract

from arXiv · show

Deep neural operators can learn nonlinear mappings between infinite-dimensional function spaces via deep neural networks. As promising surrogate solvers of partial differential equations (PDEs) for real-time prediction, deep neural operators such as deep operator networks (DeepONets) provide a new simulation paradigm in science and engineering. Pure data-driven neural operators and deep learning models, in general, are usually limited to interpolation scenarios, where new predictions utilize inputs within the support of the training set. However, in the inference stage of real-world applications, the input may lie outside the support, i.e., extrapolation is required, which may result to large errors and unavoidable failure of deep learning models. Here, we address this challenge of extrapolation for deep neural operators. First, we systematically investigate the extrapolation behavior of DeepONets by quantifying the extrapolation complexity via the 2-Wasserstein distance between two function spaces and propose a new behavior of bias-variance trade-off for extrapolation with respect to model capacity. Subsequently, we develop a complete workflow, including extrapolation determination, and we propose five reliable learning methods that guarantee a safe prediction under extrapolation by requiring additional information -- the governing PDEs of the system or sparse new observations. The proposed methods are based on either fine-tuning a pre-trained DeepONet or multifidelity learning. We demonstrate the effectiveness of the proposed framework for various types of parametric PDEs. Our systematic comparisons provide practical guidelines for selecting a proper extrapolation method depending on the available information, desired accuracy, and required inference speed.

1 Introduction

DeepONets extend neural networks to operators between infinite-dimensional function spaces, but extrapolation beyond training support remains difficult. The paper studies this problem and proposes physics- or observation-informed remedies.

  • Motivation: Neural networks can approximate nonlinear operators, enabling DeepONets to serve as surrogate PDE solvers after training.DeepONets take sampled input functions through a branch network and evaluation locations through a trunk network.
  • Motivation: Neural networks often perform accurately within training support but can incur very large errors outside it.In the sine-regression example, two networks trained on [0, 1] perform well there but fail for x < 0 or x > 1.
  • Problem setting: DeepONet inputs are generated from Gaussian random fields whose correlation length controls sampled-function smoothness.Larger correlation lengths produce smoother functions, creating distinct function spaces for training and testing.
  • Contributions: The paper quantifies extrapolation complexity with the 2-Wasserstein distance and studies effects of model capacity, dataset size, and activation functions.The analysis is intended to characterize how DeepONet extrapolation error changes across these factors.
  • Contributions: For extrapolation reduction, the workflow uses either governing PDEs or sparse output observations as additional information.The proposed approaches fine-tune a pretrained DeepONet or use multifidelity learning with low-fidelity predictions and high-fidelity observations.

2 Extrapolation of deep neural operators

The paper formulates DeepONet operator learning through branch and trunk networks, then evaluates extrapolation on antiderivative and diffusion-reaction operators. Training and testing use Gaussian random fields with different correlation lengths.

  • Operator learning via DeepONet: DeepONet learns an operator G mapping input functions v to output functions u using branch and trunk subnetworks.The branch network processes sampled values of v, while the trunk network processes the output location ξ; their outputs are combined by an inner product.
  • Experiment setup: The experiments include an antiderivative operator defined by an ODE with initial condition u(0) = 0.DeepONet is used to learn the corresponding solution operator.
  • Experiment setup: A second experiment uses a diffusion-reaction equation with zero initial and boundary conditions and maps source terms v(x) to solutions u(x, t).The coefficient settings include k = 0.01 and D, as described in the experiment setup.
  • Experiment setup: Training and testing functions are sampled from Gaussian random fields, while reference solutions come from numerical solvers.The ODE reference uses Runge-Kutta(4, 5), and the PDE reference uses a second-order finite-difference method on a 101 × 101 mesh.

2.3 Interpolation and extrapolation regions

Extrapolation is defined by differing training and testing correlation lengths, with error depending on direction and distance between function spaces. The 2-Wasserstein distance provides a quantitative measure of extrapolation complexity.

  • Interpolation and extrapolation regions: Different training and testing correlation lengths define extrapolation, while equal lengths define interpolation.The study denotes ltrain < ltest as Ex.− and ltrain > ltest as Ex.+.
  • Interpolation and extrapolation regions: When ltrain = ltest, the L2 relative error is smaller than 10^-2, whereas the bottom-right region with ltrain > ltest has larger error.The reported comparison comes from the ODE and diffusion-reaction testing-error plots.
  • Extrapolation complexity: Ex.+ error grows with a polynomial rate as the 2-Wasserstein distance between training and test spaces increases.Figure 2 reports mean errors over 10 runs with one-standard-deviation error bars across the ODE and diffusion-reaction problems.
  • Extrapolation complexity: A larger 2-Wasserstein distance represents a greater difference between Gaussian random-field spaces.The metric is therefore used to quantify extrapolation complexity between training and testing distributions.
  • Extrapolation complexity: Convergence rates differ across problems and are significantly different for the reported diffusion-reaction cases.The 95% confidence intervals are [2.01, 2.25] and [1.79, 1.92], with a two-sided T-test p-value of 0.0001.

2.5 Understanding the extrapolation error

Extrapolation error follows a U-shaped bias-variance pattern, with earlier overfitting and larger errors than interpolation. Increasing training data helps, while activation choice—especially adaptive activations—affects accuracy and overfitting.

  • Extrapolation test error has a U-shaped curve as network size or training iterations increase, with an earlier underfitting-to-overfitting transition than interpolation.
  • Smaller testing correlation lengths produce larger extrapolation errors throughout training and earlier transitions to overfitting.
  • The study does not observe double-descent behavior, possibly because the tested model capacity is not large enough.
  • Larger training datasets reduce extrapolation test error, although the improvement diminishes as dataset size becomes large.
  • ReLU and Hat achieve similar results and outperform tanh, SiLU, and GELU for the tested activation-function setting.
  • L-LAAF with its optimal scaling factor slightly outperforms non-adaptive activations, while L-LAAF-Hat and L-LAAF-ReLU best alleviate overfitting.

3 Reliable learning methods for safe extrapolation

The workflow first determines whether a new input is interpolative or extrapolative, then uses physics, sparse observations, or multifidelity learning to improve extrapolative predictions. These methods adapt a pre-trained DeepONet while addressing practical risks such as overfitting and catastrophic forgetting.

  • Reliable extrapolation methods require additional information: governing physics or sparse observations of the output function.
  • The workflow predicts with a pre-trained DeepONet, evaluates physics or observation mismatch, and applies extrapolation methods when the mismatch exceeds a user-specified threshold.
  • Physics-informed fine-tuning fixes the new input in the branch net and minimizes PDE-residual and initial/boundary-condition losses.
  • Pre-training provides a better initialization than random-start PINNs, making physics-based fine-tuning faster, especially when extrapolation complexity is small.
  • Fine-tuning with sparse observations adapts a pre-trained DeepONet, but using very few observations can cause overfitting and catastrophic forgetting.
  • Multifidelity learning treats sparse new observations as high-fidelity data and complements them with lower-fidelity predictions.

4 Extrapolation results

Across parametric PDE examples, physics-informed fine-tuning and observation-based fine-tuning generally reduce extrapolation errors, while the strongest method depends on the available information, observation count, noise, and problem complexity.

  • 4.1 Antiderivative operator: 1.52% average L2 relative error was achieved by physics fine-tuning for the antiderivative operator, improving on the pre-trained DeepONet’s 11.6%.With more than five sparse observations, FT-Obs-T, MFGPR, and MFNN achieved about 2% accuracy, while FT-Obs-A had larger error.
  • 4.2 Diffusion-reaction equation: Fine-tuning with physics achieved 0.32% L2 relative error for the diffusion-reaction equation, while FT-Obs-A and FT-Obs-T generally outperformed multifidelity methods with 20–100 observations.With 200 observations, GPR and MFGPR became competitive because the equation has a smooth solution.
  • Fine-tuning behavior: FT-Phys reached about 7% error versus about 100% for PINNs at small iteration counts, consistent with the benefit of initializing from a pre-trained DeepONet.The comparison attributes the difference to random PINN initialization versus fine-tuning pre-trained parameters.
  • 4.4 Advection equation: For advection, physics fine-tuning reached 1.42% L2 relative error, while FT-Obs-A and FT-Obs-T outperformed MFGPR and MFNN, with FT-Obs-T performing best among proposed methods.Fine-tuning the entire network, trunk, or last trunk layer produced roughly 1% error across learning rates from 0.01 to 0.0001.
  • Extrapolation severity: Increasing extrapolation by decreasing the testing correlation length increased error, but all proposed methods outperformed DeepONet and GPR; FT-Phys remained at 3.22% under the strongest tested extrapolation.With 100 measurements, FT-Obs-T performed best among fine-tuning methods, while MFNN improved relative to MFGPR as extrapolation increased.
  • Noise robustness: Observation noise increased L2 relative error; FT-Obs-T was best below 5% noise, whereas MFGPR performed best as noise increased to 10%.The comparison tested robustness while keeping other settings unchanged.
  • 4.6 Lid-driven cavity flow: In lid-driven cavity flow, FT-Obs-A and FT-Obs-T improved velocity-field predictions over the pre-trained DeepONet and GPR, while multifidelity methods had larger errors near the central vortex and moving lid.Adaptive weighting further slightly improved FT-Obs-T accuracy.

5 Conclusions

The paper characterizes DeepONet extrapolation and proposes methods using physics or sparse observations, with practical guidance for method selection and acknowledged theoretical and computational limitations.

  • 5 Conclusions: Extrapolation complexity is quantified by the 2-Wasserstein distance between function spaces, and extrapolation error shows a U-shaped relationship with model capacity.Extrapolation has larger error and an earlier transition point than interpolation and Ex.−; larger training datasets help both interpolation and extrapolation.
  • 5 Conclusions: The proposed workflow first determines whether an input is interpolative or extrapolative, then uses PDE-loss fine-tuning or observation-based fine-tuning and multifidelity learning.The methods use either complete governing physics or sparse observations as additional information.
  • 5 Conclusions: The paper provides a guideline for choosing methods according to available information, desired accuracy, and inference speed.Physics fine-tuning can achieve very high accuracy, while observation fine-tuning generally outperforms multifidelity learning; faster methods can trade accuracy for speed.
  • 5 Conclusions: The study considers complete physics or sparse observations separately, while combined sparse observations and partial physics remain an open practical problem.The authors also identify the need for further theoretical and computational work on extrapolation error.

A Abbreviations and notations

This appendix lists the abbreviations and notations used throughout the paper.

  • A Abbreviations and notations: Table 8 provides the main abbreviations and notations used throughout the paper.The appendix serves as a reference for the paper’s terminology and symbols.

B Hyperparameters

The appendix records the architectures, optimizers, iteration counts, and regularization settings used for pre-training, fine-tuning, and multifidelity learning.

  • B Hyperparameters: Table 9 specifies the DeepONet architectures and pre-training hyperparameters, including trunk and branch depths, activations, and shared width.The first and second depth and activation subcolumns correspond to the trunk and branch networks.
  • B Hyperparameters: Physics fine-tuning uses Adam, with the number of iterations listed in Table 10.The specific FT-Phys settings are provided in the accompanying hyperparameter table.
  • B Hyperparameters: FT-Obs-A uses 500 L-BFGS iterations for most problems, while FT-Obs-T uses 3000 Adam iterations at learning rate 0.001 in most cases.The antiderivative problem uses Adam for 1000 iterations in FT-Obs-A and FT-Obs-T.
  • B Hyperparameters: Multifidelity learning uses dataset sizes from Table 11; MFGPR is limited to at most 400 low-fidelity points, while MFNN uses SiLU and 10000 Adam iterations.L2 regularization is also applied, with strengths varying by problem.

C Layer-wise locally adaptive activation function

The layer-wise locally adaptive activation function uses a tunable scaling factor whose selected value depends on the activation function.

  • C Layer-wise locally adaptive activation function: L-LAAF tunes the scaling factor n separately for each activation function using candidate values 1, 2, 5, and 10.The candidate set is evaluated for tanh, SiLU, GELU, ReLU, and Hat activations.

D Fine-tune with physics

Fine-tuning with physics evaluates several learning rates for a pre-trained DeepONet across three parametric PDEs: diffusion-reaction, Burgers’, and advection.

  • Fine-tuning with physics uses a pre-trained DeepONet as the starting model.
  • Detailed results are reported for diffusion-reaction, Burgers’, and advection equations.

E Comparisons of different values of λ for FT-Obs-T methods

The study examines how λ affects test errors for FT-Obs-T under varying numbers of observed points and testing correlation lengths in the diffusion-reaction equation.

  • The effect of λ on test errors is evaluated for the diffusion-reaction equation.
  • The comparison varies both the number of observed points and ltest.
  • Results cover 12 cases combining the tested observation counts and testing correlation lengths.

F MFGPR for the Lid-driven cavity flow

For lid-driven cavity flow, MFGPR performs poorly with the RBF kernel but better with the Matern kernel with ν = 1.5. The corresponding figure compares predictions and errors for both kernels.

  • MFGPR with the RBF kernel has a large error in the cavity flow problem.
  • MFGPR with the Matern kernel with ν = 1.5 performs better than the RBF-kernel version for cavity flow.
  • The Matern kernel uses a correlation length l, a modified Bessel function Kν(·), and the gamma function Γ(·).
  • Figure 21 presents prediction and error results for the RBF and Matern kernels, with panels A and B corresponding to the two kernels.
Loading 2212.06347v1…