The Cost-Accuracy Trade-Off In Operator Learning With Neural Networks

1 Introduction

• The cost-accuracy curve guides approximation-method choice because it relates computational resources to a desired accuracy.For data-driven operator approximations, the theory is limited and these curves are not analytically understood.Source: §1 · pp. 1–2
• The study focuses on parameterization and finite-data errors, using fixed standard stochastic-gradient-descent settings and leaving discretization and optimizer interactions for future analysis.The authors note that future work could study interactions among discretization, parameterization, and data-volume errors.Source: §1 · pp. 2–3

1.1 Literature Review

• Existing DeepONet–FNO comparisons use fixed parameterizations and do not measure how evaluation cost depends on achieved accuracy, despite differing error rankings across problems.The paper identifies this cost-accuracy question as its primary focus.Source: §1.1 · p. 5

1.2 Our Contributions

• The paper unifies three established operator approximators—PCA-Net, DeepONet, and FNO—with PARA-Net, a pointwise lifting of finite-dimensional neural networks formalized here.Source: §1.2 · p. 5
• The study compares error against data volume and network size, and evaluation cost against error, across four PDE operator-learning problems.The problems cover smooth and discontinuous outputs, and experiments show data-limiting effects and potential over-fitting in some methods.Source: §1.2 · pp. 5–6
• PARA-Net is not competitive on evaluation cost per unit accuracy for the studied problems, while PCA-Net, DeepONet, and FNO are viable in some problem regimes.PARA-Net is also particularly prone to over-fitting, except for the advection problem.Source: §1.2 · p. 6
• The paper examines DeepONet’s learned output range against PCA-Net, motivating a PCA modification to DeepONet introduced in [13].The comparison probes the strengths and weaknesses of the basic DeepONet implementation.Source: §1.2 · p. 6
• The authors caution that their experiments do not establish a best operator approximator for any problem class and call for theory to support complexity comparisons.Their conclusions about preferred methods remain problem-dependent.Source: §1.2 · p. 6

• The target is an operator Ψ† from U to V, approximated by parameterized operators Ψ mapping inputs in U to outputs in V.Source: §2 · p. 7
• The formulation assumes the input and output domains Du and Dv are closed and bounded.Source: §2 · p. 7
• In practice, training uses i.i.d. input samples and their target outputs to define an empirical risk, and θ⋆,N minimizes that empirical objective.Because optimization over θ is non-convex, the attained parameter may only approximate θ⋆,N.Source: §2 · p. 7
• The authors also use an empirically approximated relative-error measure when studying error against data volume, parameter dimension, and evaluation cost.Source: §2 · p. 7
• The examples often use equal input and output domains and scalar-valued functions, while the methodology is presented more generally.Source: §2 · p. 8

3.1 Preliminaries

• Finite-dimensional functionals characterize inputs, while PCA of input and output measures supplies orthonormal bases for representing functions.The framework places the function spaces in a Hilbert-space setting and uses the bases to define finite-dimensional representations.Source: §3.1 · p. 8

3.2 PCA-Net

• PCA-Net maps input PCA coefficients through a neural network to coefficients of a truncated, data-derived output PCA basis.The bases and input functionals are computed from data before neural-network parameter optimization.Source: §3.2 · pp. 8–9

3.3 DeepONet

• DeepONet uses a branch network for the input mapping and a learned trunk network for output-space functions, unlike PCA-Net’s precomputed output basis.In these implementations, both methods use input PCA-basis functionals; the study does not focus on pointwise-input alternatives.Source: §3.3 · pp. 9–10

3.4 PARA-Net

• PARA-Net evaluates a neural network at the input’s PCA coefficients and output location, yielding an input-dependent approximation space.For the studied problems, the advantage of nonlinear approximation spaces is outweighed by the cost of repeated pointwise output reconstruction, especially as spatial dimension grows.Source: §3.4 · p. 11

3.5 Fourier Neural Operator (FNO)

• FNO lifts inputs to channels, applies Fourier Neural Layers using retained Fourier modes, then projects channels to outputs.In the current implementation, the number of retained modes scales exponentially with spatial dimension; wavelets are noted as a possible higher-dimensional alternative.Source: §3.5 · pp. 12–13

4 Numerical Studies

• The study evaluates cost and accuracy as data volume and network size vary across the four test problems.Source: §4 · p. 13
• Four operator-learning tests cover Navier–Stokes forcing to later-time vorticity, Helmholtz wavespeed to disturbance, structural boundary load to interior von Mises stress, and advection initial condition to later solution.The section presents qualitative results for each problem and later discusses cost-accuracy trade-offs across all four.Source: §4 · p. 13
• Outputs are smooth for Navier–Stokes and Helmholtz, while structural-mechanics and advection outputs have discontinuities in the field or its gradients.Source: §4 · p. 13

4.1 Neural Network Architectures

• PCA-Net, PARA-Net, and DeepONet branch and trunk networks use three fixed-width fully connected hidden layers with ReLU and linear outputs; FNO uses three Fourier Neural Layers with GELU.Figure 4.1 schematizes the architectures, with circles indicating layer widths and edges indicating transformations.Source: §4.1 · p. 13, Figure 4.1 · p. 14
• Online evaluation costs are measured in FLOPs, and DeepONet and PCA-Net have the same evaluation cost in these implementations.Their branch and PCA-Net architectures match, and DeepONet trunk basis functions can be precomputed after training.Source: §4.1 · p. 14, Table 4.1 · p. 14
• Parameter complexity is analyzed for all four architectures in terms of input and output dimensions and network-size parameters w or d_f.Source: §4.1 · p. 14, Table 4.2 · p. 15
• Figure 4.2 shows sample input and output functions for the Navier–Stokes problem.Source: Figure 4.2 · p. 15

4.2.1 Navier-Stokes Equation

• For Navier–Stokes, all architectures capture the main vorticity features, but FNO errors are significantly lower and PARA-Net predictions are grainier.The authors attribute FNO’s lower errors to the problem’s fit with a spectral representation and PARA-Net’s graininess to pointwise prediction.Source: §4.2.1 · p. 16, Figure 4.3 · p. 17

4.2.2 Helmholtz Equation

• For Helmholtz median cases, all networks match the disturbance field qualitatively, while PCA-Net and FNO have similar error structures and DeepONet and PARA-Net have smaller-scale, different errors.All four networks share the same worst-case input, where the frequency is near an eigenfrequency; prediction accuracy deteriorates independently of architecture.Source: §4.2.2 · p. 19, Figure 4.5 · p. 19, Figure 4.6 · p. 20

4.2.3 Structural Mechanics Equation

• For structural mechanics, all networks qualitatively capture the stress jump at the material interface, while FNO shows interface oscillations associated with uniform-grid interpolation.The comparisons cover median- and largest-error inputs.Source: §4.2.3 · p. 21, Figure 4.8 · p. 22, Figure 4.9 · p. 23

4.2.4 Advection Equation

• For advection, all networks capture the two discontinuities in median cases, but their worst-case behavior differs on inputs with many closely spaced jumps.PCA-Net and DeepONet show Gibbs phenomena; FNO overshoots near small-scale discontinuities, while PARA-Net avoids Gibbs oscillations but misses the true discontinuities.Source: §4.2.4 · p. 24, Figure 4.10 · p. 24

4.3.1 Accuracy vs. Training Cost

• Smooth problems approach the O(N^-1/2) error-data rate except for the smallest networks, while nonsmooth problems generally fall short and plateau as network expressivity limits accuracy.FNO generally has the lowest error at a given data volume; PARA-Net often needs more data, and PCA-Net and DeepONet behave similarly except on Helmholtz.Source: §4.3.1 · p. 26

4.3.2 Accuracy vs. Network Size/Expressivity

• Overfitting patterns depend on both problem and architecture: Helmholtz shows none at tested sizes, while structural mechanics and advection show broader architecture-specific overfitting.For Helmholtz, N = 2500 suffices for the tested networks; FNO avoids Navier–Stokes overfitting, and DeepONet and PARA-Net are more robust than PCA-Net and FNO on advection.Source: §4.3.2 · pp. 27–28

4.3.3 Accuracy vs. Evaluation Cost

• PARA-Net costs at least an order of magnitude more to evaluate with similar or worse error, whereas FNO generally has the lowest error but higher cost than PCA-Net and DeepONet in two-dimensional problems.FNO’s cost depends on the number of Fourier modes, which scales exponentially with spatial dimension; the paper therefore cannot make definitive relative-merit conclusions across FNO, PCA-Net, and DeepONet.Source: §4.3.3 · pp. 29–30

4.3.4 Out-Of-Distribution (OOD) Generalization

• For Navier–Stokes and structural mechanics, out-of-distribution errors are generally higher and converge more slowly with training data than in-distribution errors.The comparison uses inputs drawn from a Gaussian random field with covariance C′ = 4C; the performance gap remains large for structural mechanics even at smaller data volumes.Source: §4.3.4 · p. 30

4.3.5 PCA-Net vs. DeepONet Output Spaces

• DeepONet’s learned output bases are often local, and 40%–70% of its trained basis functions are zero; PCA-Net instead uses global PCA bases.For Helmholtz, the PCA basis captures high-frequency oscillations that DeepONet’s learned basis misses, explaining PCA-Net’s lower errors; the cause may involve optimization error, since a deeper trunk worsened results.Source: §4.3.5 · pp. 31–32

4.3.6 Role Of Optimization

• Test errors are generally close to or above training errors, with more overfitting at smaller data volumes and larger network sizes; overfitting is more common on discontinuous problems.Training error remains nonzero, and lower training loss may not improve generalization where networks already overfit.Source: §4.3.6 · p. 34

5 Conclusions

• FNO may offer the best cost-accuracy trade-off for the tested 1D and 2D simple-geometry problems, but its cost may be prohibitive in 3D; PARA-Net is not competitive in this setting.DeepONet and PCA-Net may be preferable in ≥3D, while complex geometries were not considered and require further comparison.Source: §5 · p. 34
• PCA-Net generally matches or outperforms DeepONet, whose learned output bases can become highly localized and fail to represent the Helmholtz output space.The results suggest including PCA bases in DeepONet’s output representation for some problems.Source: §5 · pp. 34–35
• The study offers numerical insights but no definitive ranking of PCA-Net, DeepONet, and FNO, whose relative merits are expected to depend on the problem.The experiments motivate further rigorous complexity analysis across accuracy, parameter count, and data volume.Source: §5 · p. 35
• The experiments do not theoretically address linking parameterization to data volume or how to distribute parameters across complex architectures.Examples of unresolved choices include allocating FNO parameters between channels and Fourier modes and DeepONet parameters between trunk and branch networks.Source: §5 · p. 35

A Pointwise Inputs To DeepONet

• Pointwise evaluations are represented as real-valued linear functionals on U, with k indexing the evaluations and functionals.The multi-index k = (ℓ, m) combines evaluation locations and canonical unit vectors; the resulting input dimension is du = di × ℓ′.Source: A Pointwise Inputs To DeepONet · p. 36
• The notation can be relabeled to index the real-valued functionals by k ∈ {1, · · · , du}, aligning this case with the PCA input notation.Here du counts the input functionals, not spatial points alone.Source: A Pointwise Inputs To DeepONet · p. 36
• For this setting, the input space may be viewed as H = L2(Du; R^di) and U = C(Du; R^di), or as a suitably embedded Sobolev RKHS.The passage specifies a Sobolev space of fractional order greater than dx/2 as one such RKHS.Source: A Pointwise Inputs To DeepONet · p. 36

B.1 Network parameter complexity

• PCA-Net has 2w^2 + w(du + dv) + 3w + dv parameters, while PARA-Net has 2w^2 + w(do + du + dy) + 3w + do; each uses an initial nonlinear layer, two internal nonlinear layers, and a final linear layer.Source: §B.1 · pp. 36–37
• DeepONet’s branch and trunk networks together use 4w^2 + w(du + dv + dy + dvdo) + 6w + dv + dvdo parameters.The branch has PCA-Net complexity, while the trunk maps from R^dy to R^(dvdo).Source: §B.1 · pp. 36–37
• The implemented FNO parameter count is df di + df + df do + do + 3df^2 kmax + 3df^2.The total includes lifting and projection layers and three internal Fourier operators; Table B.1 lists counts for the tested problems and network sizes.Source: §B.1 · p. 37, Table B.1 · p. 37

B.2 Network evaluation cost

• A nonlinear layer σ(Ax + b) costs 2mn + n, while a linear layer costs 2mn under the stated activation-cost approximation.The nonlinear estimate includes the matrix-vector product, vector-vector sum, and an activation cost approximated as n.Source: §B.2 · p. 37
• Projecting u onto du PCA bases costs du(2Npdi −1), and recovering v from dv PCA coefficients costs (2dv −1)Npdo.Source: §B.2 · p. 38
• PCA-Net and DeepONet with a precomputed trunk have the same evaluation cost; PARA-Net’s expression accounts for evaluation at Np points.The supplied expressions give PCA-Net and DeepONet cost as du(2Npdi −1) + 2duw + 4w2 + 2dvw + 3w + (2dv −1)Npdo, and PARA-Net cost as du(2Npdi −1) + [2(du + dy)w + 4w2 + 2wdo + 3w]Np.Source: §B.2 · p. 38
• FNO evaluation cost includes lifting and projection layers plus three Fourier layers, with the reported total expression depending on Np, df, di, do, log(Np), and kmax.The total is 2Npdfdi + 3(10dfNplog(Np) + kmax(2df2 − df)) + 2Npdodf.Source: §B.2 · p. 38

C Error vs. cost results

• Across all four test problems and tested training-data volumes, Figure C.1 reports test error versus online evaluation cost; results for each column are similar.Only the third column is extracted and reported in main-text Figure 4.14.Source: Figure C.1 · p. 38, C Error vs. cost results · p. 38