PaperlayerView exact revision on arXiv

Source-linked AI summary

Spectral Neural Operators

V. Fanaskov, I. Oseledets

arXiv:2205.10573v2math.NAcs.NA

TL;DR

Neural-network outputs are difficult to analyze, and sampling restricts represented functions while FNO activations can introduce aliasing. The paper introduces SNO over Chebyshev or Fourier coefficients; benchmarks report it performs better than or comparably to FNO on many problems, with low-viscosity viscous Burgers as an exception.

  • Problem

    Neural-operator outputs are neural networks that are difficult to analyze, while sampling restricts the represented function class and FNO activations can introduce aliasing.

    Source: §1 · p. 1

  • Method

    SNO separates interpolation from the mapping between function spaces and represents input and output functions using finite Chebyshev or Fourier series.

    Source: §1 · p. 2

  • Results

    Across benchmarks, SNO performs better than or comparably to FNO except on low-viscosity viscous Burgers, while basic DeepONet performs much worse, especially for high-frequency targets.

    Source: §5.2 · p. 9

  • Takeaways & Limitations

    The results position SNO as a competitive candidate for further investigation.

    Source: §6 · p. 9

    The current SNO applies only to smooth input and output data because of the Gibbs phenomenon.

    Source: §6 · p. 9

Abstract

from arXiv · show

A plentitude of applications in scientific computing requires the approximation of mappings between Banach spaces. Recently introduced Fourier Neural Operator (FNO) and Deep Operator Network (DeepONet) can provide this functionality. For both of these neural operators, the input function is sampled on a given grid (uniform for FNO), and the output function is parametrized by a neural network. We argue that this parametrization leads to 1) opaque output that is hard to analyze and 2) systematic bias caused by aliasing errors in the case of FNO. The alternative, advocated in this article, is to use Chebyshev and Fourier series for both domain and codomain. The resulting Spectral Neural Operator (SNO) has transparent output, never suffers from aliasing, and may include many exact (lossless) operations on functions. The functionality is based on well-developed fast, and stable algorithms from spectral methods. The implementation requires only standard numerical linear algebra. Our benchmarks show that for many operators, SNO is superior to FNO and DeepONet.

1 Introduction

FNO and DeepONet use sampled inputs and neural-network outputs, which the paper argues can obscure output behavior and, for FNO, introduce aliasing. The paper proposes Spectral Neural Operators (SNOs) based on Chebyshev and Fourier series, and compares them with FNO and DeepONet across 16 problems.

1 Introduction

  • Neural-operator outputs are neural networks that are difficult to analyze, unlike classical parametrizations that allow controlled extraction of function and derivative information.For ReLU networks, the piecewise-linear partition can have exponentially many subintervals in the number of layers, making it difficult to find.Source: §1 · p. 1
  • About 25% of the L2 norm is the average discrepancy between FNO outputs evaluated on grids h and 2h for 750 functions, attributed to activation-generated aliasing.The figure illustrates the discrepancy caused by aliasing of high harmonics produced by activation functions.Source: Figure 1 · p. 2, §1 · p. 2
  • Sampling restricts the represented function class; in FNO, nonlinear activations can introduce high frequencies that alias and distort lower frequencies.DeepONet is defined on trigonometric polynomials of degree at most N when sampling uses 2N + 1 points.Source: §1 · p. 2
  • SNO separates interpolation from the mapping between function spaces and represents both inputs and outputs with finite Chebyshev or trigonometric series.This fixes the highest possible resolution and represents functions as finite coefficient vectors.Source: §1 · p. 2
  • Series representations support lossless function operations, arbitrarily accurate fine-mesh interpolation, and access to global information such as bounds on functions and derivatives.The listed operations include integration, differentiation, and shifts.Source: §1 · p. 2
  • The paper compares SNO, DeepONet, and FNO on 16 problems and includes analyses of FNO aliasing and neural-operator generalization and super-resolution tests.The paper also introduces SNO based on Chebyshev and Fourier series.Source: §1 · p. 2

2 Chebyshev and trigonometric polynomials

The paper represents functions as finite Chebyshev or trigonometric series, using their coefficients as a computational representation. Fast transforms support coefficient recovery and interpolation, while spectral operations and smooth-function compression make these bases useful for computation.

2 Chebyshev and trigonometric polynomials

  • Functions are represented on [−1, 1]^3 as finite series in Chebyshev polynomials or trigonometric polynomials.The Chebyshev polynomials are specified by a recurrence, while the trigonometric basis uses harmonics exp(iπkx).Source: §2 · p. 3
  • DCT-I and FFT recover coefficients and interpolate on Chebyshev and uniform grids with O(N log(N)) complexity, alongside adaptive algorithms that truncate series to meet accuracy requirements.Coefficient recovery also requires N evaluations of the function.Source: §2 · p. 3
  • For functions with m continuous derivatives, the nth Chebyshev or Fourier coefficient is O(n^-m), supporting compression of smooth functions.Source: §2 · p. 3
  • Integration, differentiation, Fourier shifts, and multiplication of series can be performed with machine accuracy.Source: §2 · p. 3
  • The properties of these bases support PDE spectral methods and adaptive lossless function computations; Chebfun is presented as a role model for neural operators.The paper notes that it uses only a small part of Chebfun's functionality.Source: §2 · p. 3

3 Consequences of spectral representation

The section traces how finite spectral representations expose aliasing in neural operators and proposes tests for its effects, including generalization to unseen frequencies. It reports that FNO can acquire grid-dependent bias and that the tested neural operators show no super-resolution.

3.1 Activation functions and aliasing

  • A finite Fourier representation can gain higher frequencies after activation, which a grid cannot distinguish from lower frequencies; FNO’s global operations make this aliasing problematic.The analysis focuses mainly on Fourier series and treats sufficiently smooth functions as well represented by a reasonable number of terms.Source: §3.1 · p. 3, §3 · p. 3
  • Aliasing produced an approximately two-fold error increase for integration and a five-fold increase for differentiation, with the smoother operator showing the milder effect.The authors caution that these experiments still yielded good approximations for both operators.Source: §3.1 · p. 4
  • Training FNO on a sufficiently fine grid decreases aliasing error, while training on an insufficiently fine grid can lead it to mitigate distortions that disappear at finer resolution.The resulting discrepancy is expected to be milder for operators with smoother outputs.Source: §3.1 · p. 4

3.2 Super-resolution

  • The proposed super-resolution test shifts evaluation to unseen harmonics, and FNO, DeepONet, and SNO show no super-resolution in the reported experiment.The test evaluates on a grid fine enough to resolve the shifted frequencies; the authors note that the test can be meaningless for some operators.Source: §3.2 · p. 5

4 Spectral neural operator

The Spectral Neural Operator maps finite sets of input coefficients to output coefficients while preserving a Chebyshev or Fourier series representation. Its coefficient sequences use slightly unusual linear algebra, and the section points to benchmarks comparing SNO, FNO, and DeepONet.

4 Spectral neural operator

  • Table 2 reports relative L2 test errors for problems across SNO in Chebyshev and Fourier bases, FNO, and DeepONet.For nonperiodic 2D problems, the caption specifies a Chebyshev basis along t.Source: Table 2 · p. 6
  • SNO maps finite input coefficients to finite output coefficients while preserving the structure of truncated Chebyshev or Fourier series.Computing on finer grids requires interpolation; inputs incompatible with an SNO architecture can be smoothed by chopping their series.Source: §4 · pp. 5–6
  • Coefficient sequences can be treated like vectors with trailing zeros, allowing operations such as adding vectors of different lengths or multiplying by nonconformable matrices.In coefficient space, linear operators appropriately become low-rank integral operators.Source: §4 · p. 6
  • The section states that ordinary feedforward networks mapping C^n to C^m can qualify as neural operators under this coefficient representation.The benchmarks include SNO implemented as ordinary feedforward networks, alongside FNO and DeepONet.Source: §4 · p. 6

5 Numerical examples

The numerical examples compare FNO, DeepONet, and Chebyshev- and Fourier-basis SNOs across several operator benchmarks. SNO performs better than or comparably to FNO on many problems, while basic DeepONet performs much worse in these benchmarks.

5 Numerical examples

  • The benchmarks compare FNO, DeepONet, Chebyshev-basis SNO, and Fourier-basis SNO, with additional SNO architectures and initialization sensitivity examined separately.The paper reports results for 16 problems.Source: §5 · p. 6

5.1.1 Integration, shift and differentiation

  • The first three test operators are integration, a shift-product mapping, and differentiation.The supplied passages identify the operators but do not give their benchmark outcomes.Source: §5.1.1 · p. 8

5.1.2 Parametric ODE

  • The parametric ODE benchmark learns a mapping from f(t) to y(t), the solution of a family of ODEs based on modified DETEST problem A4.The provided passages do not specify its benchmark outcome.Source: §5.1.2 · p. 8

5.1.3 Elliptic equation

  • The elliptic benchmarks predict solutions from diffusion coefficients in both one and two dimensions, with smoothly varying coefficients.The one-dimensional coefficient varies from 1 to 20; the two-dimensional problem has zero Dirichlet boundary conditions on the square.Source: §5.1.3 · p. 8

5.1.4 Burgers equation

  • The Burgers benchmarks test one-dimensional prediction at t = 1 and two-dimensional solution prediction over t ∈ [0, 1], at viscosities 0.01 and 0.1.The low-viscosity case is the exception in which both SNO networks fail to achieve reasonable test error.Source: §5.1.4 · p. 8, §5.2 · p. 9

5.1.5 KdV equation

  • The KdV benchmarks cover single-soliton propagation and two-soliton interaction, predicting either a final solution or the full spatio-temporal domain.The two-soliton setup places the faster soliton behind the slower one so the model must predict their interaction dynamics.Source: §5.1.5 · pp. 8–9

5.1.6 Nonlinear Schrödinger

  • For the Kuznetsov-Ma breather, the task is to predict ∥ψ(x, t)∥ on x ∈ [−1, 1] and t ∈ [0, 5] from initial conditions.The single parameter ν is sampled from a uniform distribution on [1.5, 3.5].Source: §5.1.6 · p. 9

5.2 Discussion of results

  • SNO performs better than or comparably to FNO on many problems; low-viscosity Burgers is the exception, and both SNO networks fail to achieve reasonable test error there.Basic DeepONet performs much worse on these benchmarks, especially when target functions contain high frequencies; more elaborate DeepONet versions can have training error comparable to FNO.Source: §5.2 · p. 9

6 Conclusion and future work

The conclusion presents SNO as an operator on sequences of Chebyshev or Fourier coefficients and reports competitive benchmark performance. It identifies smooth input and output data as a current applicability constraint and lists non-adaptive bases and a basic architecture as improvement areas.

6 Conclusion and future work

  • SNO operates on sequences of Chebyshev or Fourier coefficients, and the benchmarks show it is competitive with other neural operators.The paper also describes FNO aliasing bias and discusses generalization and super-resolution properties.Source: §6 · p. 9
  • The current SNO version applies only to smooth input and output data because of the Gibbs phenomenon.The authors propose studying special reconstruction procedures to address this constraint.Source: §6 · p. 9
  • The bases used are non-adaptive, and the architecture is basic and does not fully exploit the available operations.The authors propose investigating adaptive global bases and more complex SNO architectures.Source: §6 · p. 9

A Aliasing and ReLU

  • The appendix extends the aliasing-error analysis to refined grids and describes the error as measuring aliasing after spectral interpolation followed by an activation function.The error is computed using a modified sum.Source: A Aliasing and ReLU · p. 12

B Additional comments on super-resolution

The additional super-resolution discussion cautions that relative L2 error can be misleading when high harmonics have little effect. In the ODE example, the solution becomes easy to approximate as the test harmonic band shifts, so the apparent super-resolution result does not establish meaningful high-frequency prediction.

B Additional comments on super-resolution

  • Relative L2 error can give misleading super-resolution conclusions when high harmonics do not matter.The ODE example has trivial high-frequency dynamics: predicting constant output 1 is enough for low relative L2 error, and the elliptic problem is described as having the same property.Source: B Additional comments on… · p. 12
  • As the test band shifts upward, the ODE solution is well approximated because the absolute value of the exponent argument decreases.The neural operators appear to show super-resolution in this test and performance deteriorates slightly as Δk increases, while remaining good overall.Source: B Additional comments on… · p. 12, Table 3 · p. 12

C Details on training and architectures

The appendix specifies FNO, DeepONet, and several SNO architectures, including coefficient-based and grid-based variants. SNO layers use coefficient matrices and combine pointwise operations with low-rank integral operators.

C.1.1 FNO

  • FNO uses four integral-operator layers, with 16 Fourier modes and 64 features in 1D, or 12 modes and 32 features in 2D.Its activation is ReLU.Source: §C.1.1 · p. 13

C.1.2 DeepONet

  • DeepONet uses four dense branch layers and a four-layer trunk; its activation is tanh.The 1D layers have 100 neurons or features each; the 2D branch and trunk specifications differ.Source: §C.1.2 · p. 13

C.1.3 SNO

  • SNO operates on Chebyshev or Fourier series coefficients, with layers stacking functions as columns and series coefficients as rows.Its first architecture resembles FNO but uses parameter sharing in feature space.Source: §C.1.3 · p. 13
  • SNO's N1 and N3 change the number of functions and apply pointwise operations, while N2 adds a low-rank integral operator.For D > 1, N2 applies integral operators separately to each dimension.Source: §C.1.3 · p. 13

xcSNO(Ch), xcSNO(F)

  • SNO variants include Chebyshev-basis and mixed temporal-Chebyshev/spatial-Fourier forms, plus versions operating on grid values.The xcSNO variants combine grid operations with coefficient operations and are used only for D = 1 problems.Source: §C.1.3 · p. 14, xcSNO(Ch), xcSNO(F) · p. 14

C.2 Initialization

  • Biases and linear-operator tensor values are drawn from standard normal distributions, with the latter also multiplied by a normalization factor.The normalization factor's specification is not included in these passages.Source: §C.2 · p. 14

C.3 Details on datasets and training

The benchmarks use 1,000 training and test examples per problem and fix a common spectral resolution. Training schedules vary by dimension, while dataset generation uses known solutions or numerical solvers as described for the problems.

C.3 Details on datasets and training

  • All benchmarks use 100 uniform-grid points, 100 Chebyshev coefficients, and 100 Fourier harmonics per dimension, with 1,000 examples in each train and test set.SNO train and test errors are recomputed on a uniform grid for consistency with networks operating on uniform-grid functions.Source: §C.3 · p. 15
  • Training uses Adam at learning rate 0.001, with exponential decay by 0.5 every 10,000 epochs in 1D and 5,000 in 2D.The 1D batch size is 1,000 for 50,000 epochs; 2D training runs for about 6.7 hours per network, with differing epoch counts.Source: §C.3 · p. 15
  • All training runs use a single NVIDIA Tesla V100 SXM2 16 GB GPU.Source: §C.3 · p. 15
  • Datasets use explicit solutions where available; other problems use fourth-order Runge-Kutta with pseudospectral discretization or a Chebyshev pseudospectral method.The latter methods use 100 Fourier modes or a 100-point Chebyshev grid, respectively.Source: §C.3 · p. 15

D Benchmarks for additional architectures

Additional SNO architectures are generally close to the coefficient-based SNO results in one and two dimensions. The reported exception is poor performance on Burgers with ν = 0.001; xSNO(Ch) is slightly better than FNO on the 2D elliptic equation.

D Benchmarks for additional architectures

  • The main text reports results for SNO(Ch) and SNO(F), neural operators defined in coefficient space, while Tables 4 and 5 report architectures that transform function values on Chebyshev or uniform grids.Source: D Benchmarks for additional… · p. 15
  • For 1D problems, additional architectures are not radically different from SNO(Ch) and SNO(F), with slight improvements on smooth Burgers, elliptic, and parametric ODE problems.For Burgers with ν = 0.001, performance remains poor.Source: D Benchmarks for additional… · p. 15
  • For the 2D elliptic equation, xSNO(Ch) performs slightly better than FNO.Source: D Benchmarks for additional… · p. 15

E Sensitivity to initialization

Across three selected 1D problems, training shows some sensitivity to initialization. The authors report that it is not sufficient to change the conclusions of Section 5.2.

E Sensitivity to initialization

Related papers

Loading 2205.10573v2…