PaperlayerView exact revision on arXiv

Source-linked AI summary

Physics-Informed Neural Operator for Learning Partial Differential Equations

Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, Anima Anandkumar

arXiv:2111.03794v4cs.LGmath.NA

TL;DR

PINO addresses the difficulty of learning high-fidelity PDE solution operators from limited or unavailable data and the optimization failures of purely physics-informed methods. It combines coarse-resolution training data with higher-resolution PDE constraints in a neural-operator framework, improving generalization and enabling high-resolution predictions across PDE families.

  • Problem

    Data-driven neural operators need representative training data and can fail to approximate high-resolution operators from coarse data, while PINNs face optimization challenges on multi-scale dynamic PDEs.

  • Method

    PINO combines training data with higher-resolution PDE losses for operator learning, then optionally fine-tunes a learned operator on a specific PDE instance.

  • Results

    PINO achieves 6.22% error on Kolmogorov flow without data and improves generalization as low-resolution data or additional PDE instances are added.

  • Takeaways & Limitations

    PINO provides high-fidelity solution-operator approximations across resolutions while retaining neural-operator inference speed and supporting settings with little or no training data.

  • Takeaways & Limitations

    Higher-resolution fine-tuning increases computational and memory costs and may destabilize optimization, while anchor losses can relieve this issue.

Abstract

from arXiv · show

In this paper, we propose physics-informed neural operators (PINO) that combine training data and physics constraints to learn the solution operator of a given family of parametric Partial Differential Equations (PDE). PINO is the first hybrid approach incorporating data and PDE constraints at different resolutions to learn the operator. Specifically, in PINO, we combine coarse-resolution training data with PDE constraints imposed at a higher resolution. The resulting PINO model can accurately approximate the ground-truth solution operator for many popular PDE families and shows no degradation in accuracy even under zero-shot super-resolution, i.e., being able to predict beyond the resolution of training data. PINO uses the Fourier neural operator (FNO) framework that is guaranteed to be a universal approximator for any continuous operator and discretization-convergent in the limit of mesh refinement. By adding PDE constraints to FNO at a higher resolution, we obtain a high-fidelity reconstruction of the ground-truth operator. Moreover, PINO succeeds in settings where no training data is available and only PDE constraints are imposed, while previous approaches, such as the Physics-Informed Neural Network (PINN), fail due to optimization challenges, e.g., in multi-scale dynamic systems such as Kolmogorov flows.

1 Introduction

PINO combines data-driven neural operators with higher-resolution PDE constraints to learn solution operators for PDE families. It addresses limited-data and optimization challenges while improving high-resolution generalization and supporting inverse problems.

  • Neural operator learning: Neural operators learn mappings from input functions, such as initial and boundary conditions, to solution functions for PDE families.They are universal approximators for continuous operators and converge under mesh refinement.
  • Limitations of data-driven learning: Data-driven neural operators can follow unseen frequencies but cannot perfectly match ground truth beyond the training resolution when only coarse data are available.Training data may be expensive, unavailable, or limited to low-resolution observations.
  • PINO: PINO combines training data with a PDE loss imposed at higher resolution to improve solution-operator fidelity and extrapolate to unseen frequencies.Its neural operator architecture computes function outputs and their derivatives at arbitrary query points.
  • Reported outcomes: PINO improves generalization and physical validity while requiring fewer or no training data, with average relative error 7% lower on transient and Kolmogorov flows.It retains the 400x speedup of data-trained FNO over a GPU-based pseudo-spectral solver.
  • Inverse problems: PINO supports inverse problems through forward-operator optimization or direct inverse-operator learning, with the latter 3000x faster than accelerated MCMC solvers for Darcy flow.The PDE loss enforces physical validity in both approaches.
  • Related approaches: Neural operators require representative data, whereas PINNs can fail on multi-scale dynamic PDEs because of difficult optimization and information-propagation challenges.PINNs also solve individual PDE instances and require re-optimization for other instances.

2 Preliminaries and problem settings

The paper formulates stationary and dynamical PDE solution operators and motivates neural operators as discretization-convergent surrogates. It identifies data dependence and optimization difficulties as central limitations motivating physics-informed operator learning.

  • Problem settings: Stationary and dynamical PDE systems define solution operators mapping coefficients or initial conditions to unknown solution functions.Stationary systems use PDE parameters, while dynamical systems use initial conditions and evolve over time.
  • Operator-learning objective: Operator learning typically minimizes data loss over sampled input-output pairs and can additionally use an operator PDE loss.Computing spatial and temporal derivatives of the neural operator is non-trivial and motivates specialized derivative methods.
  • Neural operators: Neural operators approximate solution operators by composing pointwise maps, linear operators, nonlinear activations, and kernel integral operators.The architecture lifts input functions into higher-dimensional representations, applies L layers, then projects back to the output dimension.
  • Neural operators: Fourier neural operators restrict kernel integration to convolution and use the Fast Fourier Transform for efficient computation.This produces a fast architecture for PDE problems and supports Fourier-layer implementations on suitable meshes.
  • Challenges: Purely data-driven operator learning depends on representative training data and may generalize poorly across coefficients, geometries, or difficult PDE settings.The paper contrasts this dependence with PINN difficulties on high-frequency, multi-scale, turbulent, or long-range problems.
  • Discretization convergence: Neural operators are discretization-convergent, allowing evaluation at different resolutions without retraining under mesh refinement.The paper notes that FNO can extrapolate to unseen frequencies, although coarse training data may prevent perfect super-resolution accuracy.

3 Physics-informed neural operator (PINO)

PINO uses one neural operator for operator learning and instance-wise equation solving, combining data supervision with PDE constraints and optional fine-tuning. Its implementation addresses derivative computation while exposing accuracy, resolution, and computational trade-offs.

  • PINO framework: PINO uses one neural operator for both operator learning and equation-solving problems.The framework has operator learning and instance-wise fine-tuning phases.
  • Operator learning: Operator learning can combine data loss and PDE loss, including unlimited virtual PDE instances sampled from coefficient or initial-condition distributions.PDE constraints can therefore reduce or remove the requirement for a supervised dataset.
  • Instance-wise fine-tuning: Instance-wise fine-tuning uses the learned operator output as an ansatz and optimizes PDE loss with an optional anchor loss.The anchor loss keeps the fine-tuned model near the learned operator when the PDE loss is difficult to optimize.
  • Optimization: The operator is updated using Lpde + αLop, with optional layer fine-tuning and progressive training toward finer test-time grids.These choices support additional adaptation beyond the initial operator-learning phase.
  • Derivative computation: Neural-operator derivatives can be computed by autograd, function-wise differentiation, or numerical differentiation methods such as finite differences and Fourier differentiation.Function-wise differentiation explicitly applies the chain rule through operator layers, while numerical methods offer fast, memory-efficient alternatives.
  • Trade-offs: Numerical differentiation has grid and smoothness requirements, and derivative errors can be amplified in the output solution.Higher resolution reduces truncation error but increases computation and memory and may destabilize optimization.
  • Derivative computation: Autograd is general and exact but is usually slower and more memory-consuming than numerical methods.The paper notes that numerical methods can be significantly faster when the parameter count exceeds grid size.

4 Experiments

Experiments show that PINO combines PDE constraints with operator learning to improve resolution generalization, data efficiency, and solution accuracy across several PDE settings. It also supports instance-wise fine-tuning, transfer across conditions, inverse problems, and non-periodic flows.

  • Operator learning with physics constraints: PINO learns Burgers solution operators without data and achieves 0.38% error, compared with 1.38% for PI-DeepONet.
  • Operator learning with physics constraints: On chaotic Kolmogorov flow, PINO achieves 6.22% error without data, while additional low-resolution data consistently improves optimization and accuracy.The result is reported for T = 0.125 and evaluated on high-resolution test instances.
  • Solve equation using operator ansatz: For long temporal transients, PINO with 4800 training data achieves 2.87% error and 400x speedup, improving to 1.84% with test-time optimization.PINO from scratch and PINN both exceed 50% error in this setting.
  • Solve equation using operator ansatz: PINO outperforms PINN, LAAF-PINN, and SA-PINN with 20x smaller error and 25x speedup in the reported Kolmogorov-flow comparison.The learned operator ansatz enables function-wise optimization and faster convergence.
  • Solve equation using operator ansatz: PINO also transfers across Reynolds numbers, handles lid-cavity flow with non-periodic boundaries and multiple outputs, and improves inverse-problem performance with physics constraints.The lid-cavity experiment reaches 14.52% relative error in two minutes; the inverse model reports 2.29% output error and 97.10% classification accuracy.

5 Conclusion and future work

PINO bridges physics-informed optimization and data-driven neural operator learning through operator learning and instance-wise fine-tuning. The paper identifies higher-dimensional scalability and fine-tuning convergence as remaining challenges while highlighting broad transfer potential.

  • Conclusion: PINO combines an operator ansatz over multiple PDE instances with instance-wise fine-tuning for faster and more accurate queried solutions.The fine-tuning scheme uses the learned neural operator ansatz to solve the solution function for a specific instance.
  • Limitations: PINO’s current FNO backbone makes extension to higher-dimensional problems difficult.The limitation is attributed to the use of the Fast-Fourier transform.
  • Limitations: Gradient-descent fine-tuning converges more slowly than using a finer grid.The paper points to further optimization techniques as a future need.
  • Future work: PINO’s extrapolation property supports applications across broad conditions, including transfers across Reynolds numbers.The paper also proposes pre-trained model libraries and studying transfer across geometries as future directions.

A Implementation details

The implementation uses a Fourier neural operator backbone with fixed architectural defaults and GeLU activations. Experiments run on Nvidia V100 and A100 GPUs.

  • Implementation defaults: The default backbone uses width 64, mode 8 or 12, four layers, and GeLU activations.These settings apply unless an experiment specifies otherwise.
  • Hardware: Numerical experiments are performed on Nvidia V100 and A100 GPUs.
  • Implementation defaults: The implementation specifies Fourier neural operator as the default backbone for the experiments.

A.1 Burgers Equation

The experiments train PINO solution operators using sampled initial conditions or coefficient conditions. Reported results include low relative error for Burgers and a separate Darcy-flow training setup on a V100 GPU.

  • Burgers equation: PINO is trained on 1000 sampled initial conditions for the solution operator.The initial conditions follow a Gaussian measure, and training uses width 64, mode 15, GeLU activation, Adam, and 500 epochs.
  • Burgers equation: 0.38% relative l2 error is achieved by PINO averaged over 200 testing instances.PINO-DeepONet achieves 1.38% under the reported comparison.
  • Darcy flow: The Darcy-flow setup trains on 1000 coefficient conditions with zero boundary conditions enforced by a sinusoidal mollifier.Training takes around 1 hour on a single Nvidia V100 GPU.

A.3 Long temporal transient flow.

The long-transient experiment studies flow evolution from near-zero initial velocity to an ergodic state over a fixed time horizon. It compares FNO with PINO variants differing in instance-wise fine-tuning.

  • Experiment setup: The flow evolves from an initial condition with near-zero velocity to a final state reaching ergodicity.
  • Experiment setup: The experiment uses T = 50 and l = 1 with periodic boundary conditions.
  • Compared methods: FNO is compared with PINO without and with instance-wise fine-tuning.The forcing is fixed as f(x) = 0.1(sin(2π(x1 + x2)) + cos(2π(x1 + x2))).

A.4 Chaotic Kolmogorov flow.

The chaotic Kolmogorov-flow experiments evaluate operator generalization using finite-resolution training data, additional initial conditions, and comparisons with PINN-based approaches.

  • Experimental setup: The experiments use Reynolds number 500 and report average relative L2 error across 300 test instances.Training data include 4,000 initial-condition/solution pairs at spatial resolution 64 × 64 and temporal resolution 65.
  • Baselines: The comparison includes PINO, PINN, LAAF-PINN, SA-PINN, and GPU-based pseudo-spectral solver baselines.The supplied passages identify baseline implementations and report comparisons involving instance-wise fine-tuning.
  • Experimental setup: Training on additional initial conditions boosts the operator’s generalization ability.

A.5 Transfer learning across Reynolds numbers

The transfer-learning study tests whether operator ansatzes trained at one Reynolds number improve instance-wise fine-tuning at another Reynolds number in Kolmogorov flow.

  • Results: Fine-tuning a source operator gives better convergence accuracy than learning from scratch for the higher-Reynolds-number problem Re = 500.
  • Results: Fine-tuning the source operator consistently improves convergence speed across the tested Reynolds numbers.The results are averaged over 40 instances, with rows representing test cases and columns representing source operators.
  • Evaluation: Figure 8 reports relative L2 error versus update step for Re = 500 and T = 1, averaged over 40 test instances.
  • Evaluation: The study frames the comparison as an accuracy-complexity trade-off among PINO, PINN, and a GPU-based pseudo-spectral solver.

B.1 Additional baselines

Additional baseline experiments compare PINO with adaptive PINN variants and demonstrate PINO on lid-cavity flow and non-periodic geometries.

  • Additional baselines: LAAF-PINN and SA-PINN converge much faster than original PINN but remain substantially behind PINO.LAAF-PINN adds learnable activation parameters, while SA-PINN adds weights for collocation points.
  • Additional baselines: PINO’s contribution is orthogonal to adaptive activation and self-adaptive-loss techniques, which can be transferred into the PINO framework.
  • Lid-cavity flow: PINO predicts lid-cavity velocity fields with 14.52% relative error after two minutes at resolution 65 × 65 × 50.This example uses instance-wise fine-tuning without an operator-learning phase and employs velocity-pressure formulation.
  • Fourier continuation: Fourier continuation lets FNO handle arbitrary geometries by zero-padding inputs and generating a smooth output extension on a periodic padded domain.The loss is computed on the original space during training, and Fourier continuation is used for non-periodic boundary conditions.
  • Fourier continuation: Fourier continuation also supports efficient and accurate numerical Fourier gradients except in the presence of shocks.
  • Inverse problems: In inverse problems, data constraints match observed outputs while PDE constraints restrict recovered input-output pairs to the PDE solution manifold.

E Discretization convergence and representation equivalence

PINO and FNO are discretization-convergent, enabling higher-resolution predictions, whereas representation-equivalent models cannot generate frequencies outside their fixed representation space.

  • Discretization convergence: Resolution convergence means convergence to an infinite-dimensional continuum operator as mesh refinement increases.A discretization-convergent operator may have higher error at coarse discretization but can be evaluated across resolutions.
  • Conceptual distinction: Representation equivalence requires invariance under any discretization, a goal distinct from resolution convergence.The distinction is explicit because FNOs can be resolution-convergent without satisfying representation equivalence.
  • Conceptual distinction: FNOs can reintroduce aliasing error because pointwise physical-space nonlinearities generate Fourier modes larger than the current grid.
  • Discretization convergence: FNO and PINO retain the expressive power to approximate the underlying operator as resolution approaches infinity.
  • Higher-resolution prediction: PINO trained on 64x64 data extrapolates to unseen higher frequencies, while representation-equivalent models cannot generate new frequencies.The fixed representation space of those models introduces irreducible approximation error tied to its predefined size.
Loading 2111.03794v4…