PaperlayerView exact revision on arXiv

Source-linked AI summary

On universal approximation and error bounds for Fourier Neural Operators

Nikola Kovachki, Samuel Lanthaler, Siddhartha Mishra

arXiv:2107.07562v1math.NA

TL;DR

The paper asks whether FNOs can approximate nonlinear operators between infinite-dimensional spaces both universally and efficiently. It proves universality, develops Ψ-FNOs using Fourier-based differentiation and neural approximation, and derives efficient approximation bounds for Darcy-type elliptic and incompressible Navier–Stokes operators. The authors also identify rectangular-domain and fixed-basis limitations in comparison with DeepOnets.

  • Problem

    It is unclear whether neural operators such as FNOs are universal for nonlinear infinite-dimensional operators and whether universality implies computational efficiency.

  • Method

    The paper proves FNO and Ψ-FNO universality and uses Ψ-FNO Fourier layers for exact derivatives together with σ-layers that approximate continuous functions.

  • Results

    For Darcy-type elliptic and incompressible Navier–Stokes operators, Ψ-FNO size grows polynomially and sub-linearly in the error under suitable hypotheses.

  • Takeaways & Limitations

    FNOs can efficiently approximate operators arising from these two widely used PDEs, providing a theoretical justification for their use.

  • Takeaways & Limitations

    Generic Lipschitz operators can require extremely large FNOs in the worst case, while FNO efficiency is clearest on rectangular domains and their fixed trigonometric basis may be less adaptable than learned DeepOnet trunk nets.

Abstract

from arXiv · show

Fourier neural operators (FNOs) have recently been proposed as an effective framework for learning operators that map between infinite-dimensional spaces. We prove that FNOs are universal, in the sense that they can approximate any continuous operator to desired accuracy. Moreover, we suggest a mechanism by which FNOs can approximate operators associated with PDEs efficiently. Explicit error bounds are derived to show that the size of the FNO, approximating operators associated with a Darcy type elliptic PDE and with the incompressible Navier-Stokes equations of fluid dynamics, only increases sub (log)-linearly in terms of the reciprocal of the error. Thus, FNOs are shown to efficiently approximate operators arising in a large class of PDEs.

1. Introduction

The paper addresses whether neural operators, especially FNOs, can universally approximate nonlinear operators between infinite-dimensional spaces and do so efficiently for PDE-related operators. It proves FNO universality and develops Ψ-FNO error bounds for representative PDEs.

  • Motivation: Neural networks traditionally approximate finite-dimensional functions, whereas operator-learning tasks map infinite-dimensional inputs to outputs such as PDE solution trajectories.This makes the applicability of standard universality results to operator learning unclear.
  • Fourier Neural Operators: FNOs use convolution-based integral kernels evaluated in Fourier space, with numerical experiments showing efficient approximation of operators arising in PDE simulation.This Fourier-space evaluation is presented as a computational advantage over other neural-operator frameworks.
  • Research gap: The paper investigates both universality and computational efficiency because universality alone does not rule out exponential network-size growth with increasing accuracy.The authors therefore ask whether FNOs can efficiently approximate broad classes of operators, including those from parametric PDEs.
  • Contributions: FNOs are shown to approximate a very large class of continuous nonlinear operators to desired accuracy.The result is positioned as an operator-learning analogue of universal approximation for finite-dimensional neural networks.
  • Contributions: For Darcy-type elliptic and incompressible Navier–Stokes operators, Ψ-FNO size is proved to grow polynomially, specifically sub-linearly in the error, under suitable hypotheses.The paper derives explicit error bounds for this concrete computational realization of FNOs.

2. Approximation by Fourier Neural Operators

This section defines FNOs as Fourier-parameterized neural operators and establishes their universal approximation on compact subsets, including Lipschitz domains and Ψ-FNO variants.

  • Neural operators and FNOs: FNOs approximate operators between Banach spaces by combining local lifting and projection maps with nonlinear operator layers.The layers use pointwise affine maps, nonlinear activations, and nonlocal integral operators; FNO kernels are translation-invariant and evaluated through Fourier transforms.
  • Fourier Neural Operators: FNOs specialize neural operators by using convolution kernels whose Fourier coefficients define nonlocal linear mappings.On periodic domains, the convolution is represented using the Fourier transform and inverse transform, with Hermitian symmetry imposed for real-valued outputs.
  • Fourier Neural Operators: Any finite-dimensional neural network can be represented as an FNO by setting its Fourier multipliers to zero and applying the network pointwise.This embeds standard neural networks into the FNO framework as a special case without nonlocal Fourier interactions.
  • Universal approximation: FNOs are universal: for any continuous G: Hs(Td; Rda) → Hs′(Td; Rdu), compact K, and ϵ > 0, an FNO approximates G to the prescribed accuracy.The proof approximates component operators of a Fourier-domain decomposition and composes the resulting FNOs.
  • Universal approximation: For Lipschitz domains, a continuous linear extension into a periodic domain allows an FNO’s restricted output to approximate any continuous operator.The construction lets FNOs learn a suitable periodic output representation automatically rather than requiring it to be designed explicitly.
  • Ψ-FNOs: Ψ-FNOs approximate arbitrary FNOs and inherit universal approximation for continuous operators on compact subsets when input regularity satisfies s > d/2.Their construction uses pseudo-spectral Fourier projections, with the approximation theorem requiring sufficient regularity for those projections to be well-defined.

3. Approximation of PDEs by Ψ-FNOs

The paper explains why universality alone does not guarantee efficient FNO approximation, then constructs Ψ-FNOs that emulate pseudospectral methods for Darcy and Navier–Stokes operators. Explicit bounds show substantially better accuracy–size scaling for these PDE solution operators than for generic Lipschitz operators.

  • Universality alone can coexist with super-exponential FNO size growth in the desired accuracy, motivating explicit PDE-specific complexity bounds.The worst-case bound can make generic operator approximation impractical.
  • Ψ-FNOs efficiently emulate pseudospectral methods, providing a mechanism for approximating solution operators of a broad class of PDEs.The construction composes σ- and Fourier layers to reproduce the computational structure of pseudospectral schemes.
  • 3.1. Stationary Darcy Flow: For Darcy flow, Ψ-FNOs approximate the coefficient-to-solution operator through a pseudospectral discretization followed by neural emulation.The operator maps a coefficient a to the solution u, and the approximation targets H1 error below ǫ.
  • 3.1.2. Ψ-FNO approximation of the Darcy equations: For Darcy flow, the required Ψ-FNO size scales sub-log-linearly in reciprocal accuracy when k > d, with uniformly bounded lifting dimension.The stated approximation guarantee is sup Y∈[−1,1]N ∥G(a(·,Y))−N(a(·,Y))∥H1 < ǫ.
  • 3.2. Incompressible Navier-Stokes equations: For Navier–Stokes, Ψ-FNOs emulate a pseudospectral method mapping initial data u0 to the final-time solution u(·,T).The construction uses efficient neural approximations of the algorithm’s nonlinearities.
  • 3.2.2. Approximation of algorithm 3.9 by Ψ-FNOs: For physically relevant dimensions d = 2, 3, the first-order Navier–Stokes construction has at most sub-quadratic accuracy scaling, while a second-order time scheme yields sub-linear scaling for r ≥2d.The first-order scheme’s low temporal accuracy requires many time steps and affects overall complexity.

4. Comparison of FNOs with DeepOnets

The paper relates Ψ-FNOs to DeepOnets by fixing the trunk to a trigonometric basis and constraining the branch architecture, yielding a more parsimonious representation.

  • DeepOnet architecture: DeepOnets use separate branch and trunk neural networks, mapping sensorized input functions and spatial coordinates into the output representation.The branch and trunk nets are ordinary neural networks trained concurrently in the DeepOnet framework.
  • Architectural relationship: A Ψ-FNO can be viewed as a DeepOnet whose trunk approximates a trigonometric basis and whose branch architecture is specifically constrained.The corresponding sensor points are equi-distributed Cartesian grid points.
  • Size comparison: The Ψ-FNO’s fixed trigonometric trunk and constrained branch provide a more parsimonious representation than a canonical dense-layer DeepOnet.The paper compares the total size of the two approximations through their architectural degrees of freedom.
  • Size comparison: For problems efficiently approximated by FNOs, the estimates suggest lift(N) ≪ width(N) and size(N) ≪ size(β, τ).These inequalities indicate that the Ψ-FNO can require substantially fewer architectural degrees of freedom than the corresponding DeepOnet emulator.

5. Summary and Discussion.

The paper proves universality for FNOs and Ψ-FNOs, argues that Ψ-FNOs efficiently represent PDE operators, and establishes efficient approximation results for Darcy flow and incompressible Navier–Stokes operators.

  • Universality: FNOs and Ψ-FNOs can approximate any continuous operator to desired accuracy.The proof uses approximation of the Fourier transform and its inverse together with finite-dimensional neural-network approximation of the Fourier-conjugate operator.
  • Universality: In the worst case, approximating a general Lipschitz continuous operator may require FNO size that grows super-exponentially with reciprocal error.The paper contrasts this worst-case behavior with more favorable PDE-specific estimates for Ψ-FNOs.
  • PDE approximation: Ψ-FNOs can approximate the nonlinearities and differential operators defining PDEs efficiently, motivating their interpretation as data-adapted pseudospectral methods.Their computational realization is adapted and optimized based on training data.
  • PDE approximation: For stationary Darcy flow and incompressible Navier–Stokes equations, the constructed Ψ-FNO size grows polynomially, in fact sub-linearly, with reciprocal error.These rigorous estimates support efficient approximation of the corresponding nonlinear solution operators.
  • Comparison with DeepOnets: Ψ-FNOs can be interpreted as a special case of DeepOnets with a trigonometric trunk and equi-distributed Cartesian sensor points.The paper uses this comparison to motivate their more parsimonious architecture.
  • Limitations: FNOs are efficient on rectangular domains, while efficient realization on arbitrary domains through extension and restriction operators remains unclear.General DeepOnets can instead learn trunk representations from data, whereas FNOs fix the trigonometric basis.
  • Limitations: Applying the analysis to other neural operators and to non-scientific settings such as images, text, and speech remains to be investigated.The paper identifies these settings as directions for extending the analysis.

Appendix B. Notation and Technical Preliminaries.

This appendix establishes notation for Fourier analysis on the periodic torus, Sobolev spaces, trigonometric polynomials, projections, and periodic extension operators used in the paper’s approximation arguments.

  • Fourier setting: The paper works primarily on the periodic torus T^d, identified with [0, 2π]^d, and defines Fourier transforms for square-integrable functions.Fourier coefficients and inverse transforms provide the representation used throughout the analysis.
  • Sobolev spaces: H^s(T^d) consists of L2 functions whose Fourier coefficients have finite Sobolev norm, with H^0(T^d)=L2(T^d).The equality follows from Parseval’s identity.
  • Sobolev spaces: Homogeneous Sobolev spaces consist of Sobolev functions with zero mean, and trigonometric polynomials are represented using finitely many Fourier modes.The notation distinguishes full and zero-mean spaces used in the PDE analysis.
  • Projections: The pseudo-spectral projection I_N is defined by interpolation on a regular periodic grid and maps continuous functions into finite-dimensional trigonometric polynomial spaces.The grid points have the form x_j = 2πj/(2N + 1).
  • Projections: For s > d/2, Sobolev embedding gives a compact embedding H^s(T^d) into C(T^d), ensuring that the pseudo-spectral projection is well-defined.The appendix also recalls approximation error estimates for this projection.
  • Fourier setting: The discrete Fourier transform is defined on regular-grid values indexed by J_N and associated with Fourier wavenumbers K_N.These finite-dimensional transforms support the pseudo-spectral constructions used by the FNO analysis.
  • Extension operators: A bounded Lipschitz domain admits a continuous linear extension operator into a surrounding hypercube while preserving the function on the original domain.The construction uses a smooth windowing function that vanishes, together with its derivatives, near the outer boundary.

Appendix C. Non-linear lifting and projection operators

The appendix shows that nonlinear lifting and projection maps can approximate the corresponding linear maps on compact sets while retaining controlled Lipschitz behavior.

  • Approximation lemma: A neural operator with Lipschitz interior layers can replace its linear lifting and projection maps with single-hidden-layer neural networks on compact input sets.The approximating lifting and projection networks have widths bounded by 2d_v.
  • Operator construction: The composed approximating operator bN_h = bQ_h ◦ L ◦ bR_h(a) is formed by replacing the original lifting and projection around the fixed Lipschitz interior composition.The intermediate image of the compact input set remains compact under the interior mapping.
  • Proof structure: The proof proceeds through three claims controlling projection Lipschitz constants, compact-set approximation, and the corresponding lifting approximation.These claims together establish the lemma’s approximation result.
  • Projection approximation: The Lipschitz constant of the approximating projection bQ_h remains bounded independently of h.This bound is used to control approximation errors on compact sets.
  • Projection approximation: The approximating projection has the expansion bQ_h(v(x), x) = Qv(x) + hR(h; Qv(x)), with a uniformly bounded remainder.The resulting L2 error is bounded by C∥v∥L∞∥v∥L2.
  • Compact-set control: The proof controls approximation on compact sets by introducing a cut-off operator that maps L2 functions into L2 ∩ L∞.A sufficiently large cutoff level controls the truncated tail, while a sufficiently small h controls the neural approximation remainder.

D.1. Proof of Lemma 2.6.

Lemma 2.6 extends FNO approximation from L2-valued outputs to continuous operators with Sobolev-valued outputs by composing an approximator with a single-layer FNO.

  • Setup: The proof starts with a continuous operator G: Hs → Hs′ on a compact set K and targets arbitrary accuracy ǫ.Compactness of G(K) provides a finite Fourier-mode approximation.
  • Construction: A universal FNO eN: Hs → L2 first approximates G, but may not map into Hs′ when s′ > s.This output-regularity mismatch motivates the additional layer.
  • Construction: A single-layer FNO eL: L2 → Hs′ is composed with eN to form N := eL ◦ eN: Hs → Hs′.The construction uses a compact projected image and Lemma D.1 to approximate the identity on that image.
  • Error estimate: The approximation error is split into the initial L2 approximation error and the single-layer correction error.The Sobolev projection estimate contributes the factor CN^s′ to the first term.
  • Error estimate: Choosing δ so that (CN^s′ + 1)δ ≤ ǫ/2 yields an FNO Hs → Hs′ approximating G to the desired accuracy.The proof concludes by combining the two error bounds.

D.2. Proof of Lemma 2.7.

Lemma 2.7 constructs FNOs that approximate Fourier-based mappings by encoding products with sine and cosine modes, then combining the resulting constant outputs.

  • Construction: An ordinary neural network approximates the products between Fourier coefficients and the trigonometric features.The approximation is uniform over bounded coefficient and trigonometric inputs.
  • Fourier-mode encoding: The resulting FNO layer produces approximations to PNv(x) cos(⟨k,x⟩) and PNv(x) sin(⟨k,x⟩) for the retained modes.The construction uses Fourier multipliers to extract the relevant zero-th Fourier modes.
  • Guarantee: The constructed mapping outputs constant functions and achieves the desired accuracy on inputs bounded in L2.The proof concludes because ǫ is arbitrary.
  • Output assembly: A projection combines the mode-wise outputs into a scalar-valued L2 function, yielding an FNO approximation of the target mapping.The composition remains an FNO because both component maps admit FNO representations.

D.4. Proof of the Universal Approximation Theorem 2.5.

The universal approximation proof reduces the target operator to finite Fourier coordinates, approximates each component with FNOs, and controls the resulting composition error.

  • Reduction: Compactness of K and continuity of G provide finite-dimensional Fourier truncations and uniform continuity on the relevant compact sets.This reduces the approximation problem to controlled finite-mode representations.
  • Decomposition: The operator is decomposed into input projection, Fourier-coordinate transformation, finite-dimensional approximation, and reconstruction.Each component is approximated separately before composing the resulting FNOs.
  • Component approximation: The Fourier-coordinate map is approximated by a finite-dimensional neural network, while projection and reconstruction maps are represented by FNOs with constant outputs where needed.The construction identifies finite-dimensional maps with local FNO layers.
  • Conclusion: The constructed FNO approximates GN to arbitrary accuracy, and restricting it to Hs yields the universal approximation result for general Sobolev outputs.The special L2-output case is extended to s′ ≥ 0 through Lemma 2.6.
  • Error control: Smooth bias approximations, Sobolev regularity, Hölder continuity, and uniformly bounded pseudo-spectral projections control errors across the multilayer composition.The proof uses a descending sequence of Sobolev indices above d/2 and a telescoping error estimate.

D.6. Proof of Lemma 2.16.

Lemma 2.16 transfers universal approximation from L2 outputs to Hs′ outputs by smoothing the approximating FNO and controlling Fourier projection errors uniformly.

  • Setup: The proof starts from an L2-output Ψ-FNO approximation of G on a compact input set K with error δ.The target is an Hs′-valued approximation for sufficiently large Fourier cutoff N.
  • Regularization: The approximating image of K is compact, and finite-dimensional norm equivalence allows it to be treated as compact in a higher Sobolev space Hm.Here m is chosen strictly larger than both s′ and d/2.
  • Regularization: Lemma D.1 supplies a single-layer FNO that approximates the identity on the compact projected image in Hm.The new Ψ-FNO inserts this layer after the original approximation and includes the pseudo-spectral projection.
  • Error estimate: The resulting error bound combines the original L2 error, projection error, and identity-approximation error, with constants independent of N and δ.The constant depends only on the domain and s′.
  • Conclusion: Choosing δ sufficiently small gives an Hs′-valued Ψ-FNO approximation for every N above a threshold N0.The threshold depends on the operator, compact set, and selected δ.

D.7. Technical Results on the structure of Ψ-FNOs.

These technical results establish how Ψ-FNOs represent, compose, and approximate neural-network layers while controlling their depth, width, and lifting dimensions.

  • Representation: Any composition of σ-layers can be identified with a Ψ-FNO.This connects ordinary neural-network components to the Ψ-FNO architecture.
  • Linear approximation: A linear Ψ-FNO layer can be approximated arbitrarily accurately by a Ψ-FNO with width O(N^d), constant depth, and constant lift.The activation must be twice continuously differentiable and non-constant.
  • Composition: A composition of p Ψ-FNOs is representable by one Ψ-FNO whose depth is bounded by the sum of the component depths.The construction pads layers with zeros to use a uniform lifting dimension.
  • Replacement: For Lipschitz activations, replacing each continuous operator in a composition by an approximating Ψ-FNO yields a Ψ-FNO approximation of the full composition.The resulting network inherits depth, width, and lift bounds from the component approximations.
  • Replacement: The replacement argument controls intermediate outputs by bounding the first approximated operator within a larger ball.For example, the estimate gives ∥N1(u)∥L2 ≤ 2M1.

Appendix E. Technical Results and Proofs for Section 3.1

The appendix proves that the discretized Darcy fixed-point map is well behaved under Sobolev regularity and coercivity assumptions, with sufficiently many Fourier modes.

  • Contraction estimate: For sufficiently large N, the discretized coefficient remains coercive and the mapping FN is a contraction.The threshold N0 depends on s, d, ∥a∥Hs, and λ.
  • Contraction estimate: The contraction proof combines the coercivity bound with spectral or pseudo-spectral approximation estimates for the coefficient.The argument establishes aN ≥ λ/2 for N ≥ N0.
  • Convergence setup: The appendix then prepares the convergence proof for the algorithm approximating the Darcy equation.The next theorem uses the contraction result to analyze the algorithm.

E.1. Proof of Theorem 3.3.

The proof combines Fourier-Galerkin error estimates, Picard iteration, and Ψ-FNO approximations of nonlinear operations to establish Darcy-solver convergence and network emulation.

  • Error estimate: Combining discretization and iteration estimates yields an error bound for the algorithm under Sobolev regularity assumptions on a and f.The constant is independent of N and depends on s, k, d, λ, ∥a∥Hs, and ∥f∥Hk−1.
  • Picard iteration: The output of Algorithm 3.2 is generated by K-fold Picard iteration of the contraction mapping FN.For N ≥ N0, the contraction factor satisfies Lip(FN) ≤ 1 − λ/2.
  • Nonlinear approximation: Neural networks with bounded width and depth can approximate the quadratic nonlinearities needed by the Darcy update.The construction approximates squares and products using finite differences and parallelized subnetworks.
  • Ψ-FNO emulation: The Darcy algorithm is written as a composition of alternating operators, with K bounded logarithmically in N, enabling Ψ-FNO emulation.The composition lemma transfers the component approximation bounds to the full mapping.

F.4. A second-order in time accurate pseudo-spectral method for approximating the Navier-Stokes equation (3.12) and its emulation by Ψ-FNOs.

This section develops a second-order-in-time pseudospectral Navier–Stokes scheme and proves that its solution operator admits an efficient Ψ-FNO approximation under regularity and time-step conditions.

  • Scheme: The proposed scheme is second-order accurate in time for approximating the Navier–Stokes equations.It requires two starting values and uses a reduced time step to initialize the second-order method.
  • Scheme: The method uses fixed-point iteration to solve the implicit operator equation in each time update.The algorithm is stated as Algorithm F.6.
  • Convergence: For a suitable time step τ ∼ N^-r/2 and δ ≤ C N^-r, the number of time steps scales as nT ∼ N^r/2.The theorem assumes r ≥ d + 2 and a CFL-type restriction.
  • Solution operator: The Navier–Stokes solution operator maps initial data at time zero to the solution at time T over a bounded regularity class.The class consists of solutions with bounded L2 norm and specified Sobolev-time regularity.
  • Ψ-FNO emulation: For any N, a Ψ-FNO approximates this solution operator with width O(N^d), depth O(N^r/2 log N), and constant lift.The approximation is stated for the solution class under the theorem’s assumptions.

Appendix G. Proof of Theorem 4.2

The proof reduces each FNO layer to affine mappings followed by componentwise activation, then represents the resulting mapping with an ordinary neural network. A modified output layer converts grid values into coefficients in a real trigonometric basis with the required span.

  • Each FNO layer is the composition of an affine mapping and componentwise application of the activation function σ.
  • The Ψ-FNO mapping can be represented by an ordinary neural network eβ mapping grid-encoded inputs to grid-valued outputs.
  • Modifying eβ’s linear output layer and re-indexing its components yields a network β whose outputs correspond to coefficients in a real trigonometric basis.
  • Because the real trigonometric basis has the same span as the complex exponential basis, the proof reduces the claim to establishing existence of a suitable neural network.
Loading 2107.07562v1…