PaperlayerView exact revision on arXiv

Source-linked AI summary

A comprehensive and fair comparison of two neural operators (with practical extensions) based on FAIR data

Lu Lu, Xuhui Meng, Shengze Cai, Zhiping Mao, Somdatta Goswami, Zhongqiang Zhang, George Em Karniadakis

arXiv:2111.05512v1physics.comp-ph

TL;DR

The paper asks how DeepONet and FNO compare for realistic neural-operator applications and develops extensions to make both suitable for complex settings. Across 16 benchmarks, the operators perform similarly in many problems, but FNO is extremely sensitive to noise whereas DeepONet remains more robust; the paper also compares their theoretical error estimates.

  • Problem

    The paper addresses the need for a fair comparison of DeepONet and FNO in realistic applications involving complex geometries, noisy data, and other challenging data characteristics.

  • Method

    The authors develop practical extensions for both operators, evaluate them on 16 benchmarks, and compare their approximation error estimates theoretically.

  • Results

    Across the benchmarks, DeepONet and FNO show similar performance in most problems, but FNO fails under 0.1% Gaussian noise while DeepONet remains satisfactory at relatively large noise levels.

  • Takeaways & Limitations

    DeepONet is more flexible in problem settings and datasets, while FNO’s extreme noise sensitivity may prevent reliable generalization within the data distribution.

  • Takeaways & Limitations

    Network-size estimates may not be optimal because solution operators and corresponding numerical methods can be approximated in many ways.

Abstract

from arXiv · show

Neural operators can learn nonlinear mappings between function spaces and offer a new simulation paradigm for real-time prediction of complex dynamics for realistic diverse applications as well as for system identification in science and engineering. Herein, we investigate the performance of two neural operators, and we develop new practical extensions that will make them more accurate and robust and importantly more suitable for industrial-complexity applications. The first neural operator, DeepONet, was published in 2019, and the second one, named Fourier Neural Operator or FNO, was published in 2020. In order to compare FNO with DeepONet for realistic setups, we develop several extensions of FNO that can deal with complex geometric domains as well as mappings where the input and output function spaces are of different dimensions. We also endow DeepONet with special features that provide inductive bias and accelerate training, and we present a faster implementation of DeepONet with cost comparable to the computational cost of FNO. We consider 16 different benchmarks to demonstrate the relative performance of the two neural operators, including instability wave analysis in hypersonic boundary layers, prediction of the vorticity field of a flapping airfoil, porous media simulations in complex-geometry domains, etc. The performance of DeepONet and FNO is comparable for relatively simple settings, but for complex geometries and especially noisy data, the performance of FNO deteriorates greatly. For example, for the instability wave analysis with only 0.1% noise added to the input data, the error of FNO increases 10000 times making it inappropriate for such important applications, while there is hardly any effect of such noise on the DeepONet. We also compare theoretically the two neural operators and obtain similar error estimates for DeepONet and FNO under the same regularity assumptions.

1. Introduction

The paper presents a systematic comparison of DeepONet and FNO across realistic benchmarks, extending both operators to address practical application requirements. It reports comparable performance in simpler settings while targeting complex geometries, differing input/output dimensions, noisy data, and training efficiency.

  • Problem setting: The paper frames operator regression as learning mappings from infinite-dimensional input functions to infinite-dimensional output functions, unlike ordinary function regression.This setting motivates comparing neural operators as simulation and system-identification tools.
  • Comparison scope: The study compares DeepONet and FNO systematically across 16 benchmarks designed around complex geometries, non-smooth solutions, unsteadiness, and noisy data.The benchmarks are intended to expose both advantages and limitations in realistic applications.
  • DeepONet extensions: DeepONet is extended with extra branch and trunk features, hard Dirichlet and periodic boundary constraints, POD-DeepONet, and a scaling designed to improve accuracy.POD-DeepONet uses training-data POD modes as the trunk basis, while the branch network learns their coefficients.
  • Efficiency and training: The authors introduce a faster DeepONet implementation with computational cost comparable to FNO and apply input/output normalization to both operators.The study also demonstrates the effect of normalization and analyzes a new DeepONet scaling.
  • FNO extensions: FNO is extended through dFNO+ for different input/output domains and gFNO+ for complex-geometry domains.These extensions enable comparisons in settings where the original FNO formulation is less directly applicable.
  • Theory and reproducibility: The comparison includes theoretical error estimates for DeepONet and FNO alongside computational evaluations and aims to make codes and data accessible under FAIR principles.The theoretical analysis emulates Fourier methods for both neural operators.

2. Operator learning

The paper frames operator learning as approximating mappings from input functions to output functions in scientific and engineering systems. It emphasizes the need for reliable learning from limited, diverse, and FAIR data.

  • Operator learning approximates mappings from input functions to output functions, such as forcing terms or initial conditions to PDE solutions.
  • Scientific applications may lack enough data to ensure accurate and reliable deep-learning models.
  • The study situates scientific data within FAIR principles: findability, accessibility, interoperability, and reusability.
  • The data challenge includes multimodal inputs from sources such as images, videos, velocimetry, medical imaging, and simulated data.
  • The paper states that its codes and data will be available on Github upon publication.

3. Operator regression networks

The section introduces DeepONet and its practical extensions, including feature-based inductive biases, hard boundary-condition constraints, POD-based bases, rescaling, and approaches for differing output dimensions. It also describes FNO-related extensions for nonmatching domains and complex geometries.

  • DeepONet: DeepONet uses branch and trunk networks: the branch processes a discretized input function, while the trunk takes output coordinates without discretizing the output function.
  • DeepONet: DeepONet is architecture-flexible because its branch network can use FNN, ResNet, CNN, RNN, or GNN structures suited to the input discretization.
  • DeepONet extensions: Known output or input features can be added to the trunk or branch nets to encode prior knowledge, including oscillatory behavior and historical data.
  • DeepONet extensions: Dirichlet and periodic boundary conditions are imposed as hard constraints through modified outputs and Fourier feature expansions.
  • DeepONet extensions: POD-DeepONet computes a proper-orthogonal-decomposition basis from mean-subtracted training data and uses the branch network to learn its coefficients.
  • DeepONet extensions: Scaling by O(1/√p) improves POD-DeepONet accuracy in the reported experiment, while the analysis applies only to He initialization.
  • FNO extensions: The proposed dFNO+ strategy unfolds boundary data into a one-dimensional function when zero-padding an input subset would be inefficient.

4. Theoretical comparison

The theoretical comparison analyzes universal approximation and error estimates for DeepONet and FNO, including their Burgers’ equation solution operators. Under comparable constructions and regularity assumptions, both architectures can achieve matching asymptotic accuracy and network-size estimates.

  • Universal approximation: Both DeepONet and FNO are compared through universal approximation theorems for nonlinear operators.The comparison includes theorem statements for both architectures and their operator classes.
  • DeepONet: DeepONet represents operator evaluations using branch and trunk networks, whose architectures may use fully connected, residual, or convolutional networks.The branch receives sampled input-function values, while the trunk evaluates output locations.
  • FNO: FNO uses Fourier interpolation and Fourier-domain convolution blocks to process discretized functions on a common periodic domain.Its blocks combine pointwise linear terms with Fourier-transformed convolution coefficients.
  • Approximation limits: Both architectures can suffer from the curse of dimensionality when ReLU or tanh networks approximate Lipschitz operators with high-dimensional sampled inputs.The cited discussion attributes this limitation to the approximation capacity of the underlying networks.
  • Burgers’ equation: For the periodic 1D Burgers’ equation, the solution operator is analyzed using Fourier interpolation, rational-function approximation, and ReLU-network emulation.The construction first approximates the operator numerically and then emulates that approximation with neural networks.
  • Burgers’ equation: Both FNO and DeepONet of size O(m3 ln(m)) can achieve accuracy O(m−1) in the analyzed Burgers’ equation setting.The authors note that these network-size estimates may not be optimal because alternative numerical approximations are possible.

5. Numerical results

The numerical study evaluates DeepONet, FNO, and practical extensions across 16 problems using repeated L2 relative-error measurements. Performance is similar in simpler settings, while DeepONet variants are more accurate in the reported complex Darcy geometries.

  • Evaluation protocol: The study evaluates 16 problems using mean and standard deviation of L2 relative error from five independent training trials.Dataset sizes and DeepONet network sizes are reported separately in supplementary tables.
  • Burgers’ equation: For Burgers’ equation, output normalization yields errors of 2.15±0.09% for DeepONet and 1.93±0.04% for FNO.DeepONet changes only slightly with normalization, whereas FNO is reported as more sensitive to it.
  • Rectangular Darcy flow: For rectangular Darcy flow, errors are around 2% for piecewise-constant data and around 1% for continuous data across the methods.POD-DeepONet is smallest for piecewise-constant data, while FNO is slightly better for continuous data without a significant difference from POD-DeepONet.
  • Complex Darcy geometries: In the pentagram-with-hole Darcy problem, DeepONet and POD-DeepONet are about 2% more accurate than dgFNO+ on the reported comparison.A 50 × 50 dgFNO+ grid reaches 2.78±0.01% error but remains less accurate than both DeepONet variants.
  • Complex Darcy geometries: For triangular Darcy flow, POD-DeepONet achieves the best reported accuracy, with an L2 relative error of 0.18%.Both DeepONet variants are reported as more accurate than dgFNO+ in this setting.
  • Complex Darcy geometries: For triangular Darcy flow with a notch, errors are 2.64±0.02% for DeepONet, 7.82±0.03% for dgFNO+, and 1.0±0.00% for POD-DeepONet.POD-DeepONet ranks first, followed by DeepONet, among the three approaches.

5.3. Multiphysics electroconvection problem

The electroconvection benchmark maps a 2D electric-potential field to a 2D cation-concentration field across unseen boundary conditions. All methods are accurate, but POD-DeepONet performs best, while FNO is hindered by the output’s stiff boundary; related advection tests show DeepONet’s stronger robustness to noisy inputs and dimensionality changes.

  • Problem setup: The benchmark couples flow, electric potential, and ion concentrations in a 2D electroconvection problem.The networks learn φ(x, y) → c+(x, y) over a regular 101 × 51 mesh.
  • Data generation: Fifteen boundary-condition settings train the networks, while ΔΦ = 13.4 and 62.15 provide unseen test conditions.The cases are generated with ΔΦ ranging from 5 to 75 in increments of 5.
  • Electroconvection results: POD-DeepONet outperforms the other methods, followed by normalized DeepONet, although all testing errors remain very small.The output resolution is 101 × 51, and the stiff boundary occurs at y = 0 where c+ drops from 2 to approximately zero.
  • Representation flexibility: DeepONet reaches 0.49±0.04% accuracy from 800 random output measurements, whereas FNO requires a regular output mesh.This illustrates DeepONet’s flexibility when output observations are not supplied on a regular grid.
  • Noise robustness: With 0.1% Gaussian input noise, FNO fails while DeepONet predicts accurately; at 5.0% noise, DeepONet’s L2 relative error is 9.28%.The corresponding figure reports 270% for FNO at 0.1% noise and 0.36% for DeepONet at the same noise level.
  • Advection dimensionality: For 1D-to-2D advection, DeepONet variants retain errors of order 0.1%, while brute-force FNO has approximately 50% relative error.FNO cannot directly map the 1D initial condition to the 2D solution without an extension.
  • Advection dimensionality: dFNO+2 is difficult to train: four of ten experiments become trapped with L2 relative error above 20%, despite FFT-mode changes.Its training cost is also much higher than that of 2D FNO.
  • Instability waves: In instability-wave prediction, DeepONet with Fourier features reports 8.90±0.60% error, outperforming POD-DeepONet at 20.8±1.12% and FNO.Input and output functions have different mesh resolutions, requiring interpolation for FNO.

5.5. Predicting surface vorticity of a flapping airfoil

The flapping-airfoil benchmark maps a time-dependent angle of attack to surface vorticity, a 1D-to-2D operator-learning task. A modified DeepONet uses historical vorticity states as trunk features, while FNO requires input-dimension augmentation.

  • Problem setup: The benchmark predicts NACA0012 surface vorticity ω(s, t) from the normalized angle-of-attack signal α(t).The airfoil surface is discretized using 152 points at Re = 2500.
  • Data preparation: Training uses independent two-period signals with different phases, splitting approximately 36.6 time units for training from the remaining data.The full time-dependent signal spans about 56.4 time units.
  • Modified DeepONet: The modified DeepONet replaces the trunk’s time coordinate with several historical states of the predicted vorticity.During prediction, the network’s prior outputs are recursively concatenated and supplied to the trunk net.
  • Results: Because the mapping is 1D-to-2D, FNO needs dimension augmentation; the brute-force version yields 16.20% error and fails to predict correctly.The table reports testing L2 relative errors for DeepONet and FNO variants.

5.6. Navier-Stokes equation in the vorticity-velocity form

The paper evaluates operator learning on Navier–Stokes vorticity prediction and regularized cavity flows. Performance is comparable in the former, while cavity-flow accuracy depends strongly on temporal features and the case considered.

  • 5.6. Navier-Stokes equation in the vorticity-velocity form: The Navier–Stokes benchmark learns vorticity at T = 20 from the first ten time steps on a fixed 64 × 64 spatial grid.The viscosity is ν = 0.001, with periodic boundary conditions and randomized initial fields.
  • 5.6. Navier-Stokes equation in the vorticity-velocity form: DeepONet and FNO have comparable accuracy after normalization, with errors of 1.78% and 1.81%, respectively.POD-DeepONet performs best at 1.71%, while unnormalized networks exceed 2% error.
  • 5.7. Regularized cavity flows: The cavity-flow benchmark uses boundary conditions as inputs and predicts converged velocity fields over Reynolds numbers from 100 to 2080 in Case A.Ten unseen Reynolds numbers within the same range form the test set.
  • 5.7. Regularized cavity flows: Case B fixes Re = 1000 and uses 100 velocity snapshots from the final 10,000 LBM iterations to study unsteady prediction.Unsteady I trains on 90 snapshots, whereas Unsteady II trains on 10 and tests on the final 10.
  • 5.7. Regularized cavity flows: The networks receive sin(5t) as an additional feature to improve predictions for the time-dependent cavity boundary condition.The feature is concatenated to DeepONet’s branch input and added as a dFNO+ channel.
  • 5.7. Regularized cavity flows: Figure 10 shows the LBM velocity components and their time series at a representative cavity location, using normalized velocity and time axes.The case is Re = 1000 at T = 1,500,000dt.
  • 5.7. Regularized cavity flows: For Case A, dFNO+ is slightly more accurate than DeepONet, while POD-DeepONet achieves the best accuracy.This comparison is reported for the regularized cavity-flow results.
  • 5.7. Regularized cavity flows: For Case B, the baseline networks are inaccurate without feature expansion, whereas expanded DeepONet and dFNO+ have similar performance.The passage reports that adding the extra feature significantly changes predicted accuracy.

6. Summary

The study compares DeepONet and FNO theoretically and across 16 PDE benchmarks, introducing extensions for realistic mappings. Both perform similarly in many settings, but DeepONet is more flexible and substantially more noise-robust.

  • Overall comparison: DeepONet does not discretize the output and supports arbitrary branch architectures, making it more flexible than FNO for problem settings.FNO uses a fixed architecture and discretizes the output.
  • Theoretical comparison: Continuous FNO can be viewed as a DeepONet subcase with a specialized branch network and discrete trigonometric trunk basis.The theoretical comparison also finds equal accuracy for same-sized operators emulating the Cole-Hopf transformation.
  • Computational comparison: Across 16 PDE problems, both operators achieve good accuracy with proper extensions and similar performance in most problems.The conclusion summarizes the computational comparison across diverse applications.
  • Noise robustness: With only 0.1% Gaussian test noise, FNO can fail completely on advection and instability-wave problems, whereas DeepONet remains satisfactory at relatively larger noise levels.The authors associate FNO’s sensitivity with a potentially unstable learned operator.
  • Limitations and future work: Training can be expensive, and physics-informed losses may reduce data needs, but hybrid physics-data training is reported as more effective for realistic problems.The authors identify architecture design sustaining favorable convergence for larger data sizes as future work.

S1. Proof of Theorem 4.3

The proof combines kernel identities, integration by parts, regularity estimates, and Fourier interpolation bounds to control two terms. The resulting constant depends on problem parameters and may become large when κ is small.

  • The proof uses ∂xK(x, y, t) = −∂yK(x, y, t) and integration by parts to decompose the expression into I1 and I2.
  • Regularity of u0 and heat-kernel properties establish that u belongs to W 1,∞.
  • Fourier interpolation error estimates are used to bound the two decomposed terms.
  • The estimates give ∥I1∥L2(−π,π) ≤Ch and ∥I2∥L2(−π,π) ≤Ch for sufficiently small h, equivalently large m.
  • C > 0 depends on t, M0, M1, and κ, and may be very large when κ is small.

S3. Architectures of DeepONets

The supplementary material lists DeepONet and POD-DeepONet architectures separately for each benchmark example.

  • DeepONet and POD-DeepONet architectures are listed for each example in Tables S2 and S3.

S4. Data generation for Darcy flows

Darcy-flow datasets are generated with PDEtoolbox on unstructured meshes across several geometries, then organized for operator-network training and testing. The supplementary tables document dataset sizes and network architectures.

  • The PDEtoolbox in Matlab solves the Darcy-flow equation for different geometries.
  • Rectangular domain: Rectangular-domain simulations use 1,893 unstructured meshes and interpolate solutions to a 20 × 20 uniform grid.
  • Table S1 reports dataset sizes, while Tables S2 and S3 report DeepONet and POD-DeepONet architectures.
  • Pentagram with a hole: Pentagram-with-hole simulations use 1,938 unstructured meshes and generate 2,000 solutions from different boundary conditions, with 1,900 for training.
  • Triangular domain: Triangular-domain simulations use 861 unstructured meshes and generate 2,000 solutions, including 1,900 training samples and 100 testing samples.
  • Triangular domain with a notch: Triangular domains with a notch use 1,084 unstructured meshes to generate 2,000 solutions, split into 1,900 training and 100 testing samples.

S5. Data generation for regularized cavity flow

Regularized cavity-flow data are generated with MRT-LBE on a 256 × 256 uniform grid for steady and unsteady cases. The unsteady dataset uses velocity fields from the final 10,000 iterations.

  • Both steady and unsteady simulations use the multi-relaxation-time lattice Boltzmann equation model on a 256 × 256 uniform grid.
  • Non-equilibrium extrapolation is used for all boundary conditions, with dt = dx = L/N.
  • Steady case: For the steady case, iteration terminates when the stated convergence criterion is met.
  • Unsteady case: For the unsteady case, the maximum iteration number is 1,500,000, and velocity fields from the last 10,000 iterations form the DeepONet/FNO dataset.
  • Figure S1 presents two Darcy-flow datasets, while Figure S2 shows unstructured meshes for different Darcy-flow geometries.
Loading 2111.05512v1…