Source-linked AI summary
Bounding the Rademacher Complexity of Fourier neural operators
Taeyoung Kim, Myungjoo Kang
TL;DR
FNO generalization error had not been estimated, even though training uses finite samples. The paper bounds complexity and generalization using group-norm capacities for FNOs with FCN or CNN layers. Experiments report links between capacity, model properties, and empirical error, including dependence on mode count.
Problem
The paper identifies a lack of generalization-error estimates for FNOs, which matters because only finitely many samples are available for hidden-data inference.
Source: §1 · p. 2Method
The authors define group-norm capacities and use them to bound Rademacher complexity and posterior generalization error for FNOs with FCN or CNN layers.
Source: §1.3 · p. 3, §3.1 · p. 6, §3.2 · p. 13Results
Experiments qualitatively indicate that empirical generalization error depends on mode count and that capacity reflects information about p, q, weights, and architecture.
Source: §1.3 · p. 3, §5 · p. 19Takeaways & Limitations
The experiments suggest that the selected group norm determines which weight and architecture information capacity represents, while mode count is relevant to empirical generalization error.
Source: §1.3 · p. 3, §5 · p. 19The stated generalization bounds assume training samples are independently drawn from the data distribution.
Source: §3.2 · p. 13
Abstract
from arXiv · showhide
A Fourier neural operator (FNO) is one of the physics-inspired machine learning methods. In particular, it is a neural operator. In recent times, several types of neural operators have been developed, e.g., deep operator networks, Graph neural operator (GNO), and Multiwavelet-based operator (MWTO). Compared with other models, the FNO is computationally efficient and can learn nonlinear operators between function spaces independent of a certain finite basis. In this study, we investigated the bounding of the Rademacher complexity of the FNO based on specific group norms. Using capacity based on these norms, we bound the generalization error of the model. In addition, we investigated the correlation between the empirical generalization error and the proposed capacity of FNO. From the perspective of our result, we inferred that the type of group norms determines the information about the weights and architecture of the FNO model stored in the capacity. And then, we confirmed these inferences through experiments. Based on this fact, we gained insight into the impact of the number of modes used in the FNO model on the generalization error. And we got experimental results that followed our insights.
1. Introduction
FNOs efficiently model operators between function spaces, but their generalization error had not been estimated despite finite training samples. The paper develops group-norm capacity bounds and uses experiments to examine capacity and mode count in relation to empirical error.
1. Introduction
- The study addresses the missing generalization-error estimates for FNOs, where finite training samples leave hidden-data accuracy to be considered.The introduction distinguishes this estimation question from prior work on FNO approximation properties.Source: §1 · p. 2
1.1 Overview of FNOs
- An FNO lifts input functions, processes them through Fourier layers, and projects the result into an output function space.Each Fourier layer combines an activation with a neural network and a parameterized convolution; the layer network can be an FCN or CNN.Source: §1.1 · p. 2, Figure 1 · p. 2
1.2 Probably approximately correct (PAC) learning
- PAC learning frames the tradeoff between approximation and estimation error through the complexity of the hypothesis class.The introduction identifies Rademacher complexity as one of several quantities related to hypothesis-class complexity.Source: §1.2 · p. 3
1.3 Our Contributions
- The paper defines group-norm-based capacities, bounds Rademacher complexity and posterior generalization error for FCN- and CNN-layer FNOs, and tests these results experimentally.The experiments examine how p and q values and mode count relate to capacity information and empirical generalization error.Source: §1.3 · p. 3
2. Preliminary
The preliminary section sets up discretized FNO inputs and outputs, describes Fourier and CNN layers, and defines the sampled-data losses used for analysis.
Notation
- The notation abbreviates coordinate sequences as x and k, with · marking indices used in the norm calculation.Source: §2 · p. 3
Fourier transform
- On a uniformly discretized domain, the Fourier transform can be implemented as an FFT, using the convolution-to-pointwise-multiplication relation.The analysis selects the domain [0, 2π]^d.Source: §2 · pp. 3–4
FNO definition
- The FNO maps discretized input vectors to output vectors by lifting inputs, applying Fourier layers, and projecting the resulting representation.Only partial frequencies are used, indexed by a set bounded by the maximal modes.Source: §2 · p. 4
CNN layer
- A CNN layer computes local inner products with a kernel; the FNO variant replaces the general linear map with a CNN layer alongside parameterized convolution.Odd-sized kernels and padding preserve the input tensor dimensions.Source: §2 · pp. 4–5, Figure 2 · p. 5
Discretization and loss
- The function domain and input-output data are discretized into finite-dimensional vectors, and sampled pairs define training and distribution losses.Training samples are independently drawn from the stated distribution.Source: §2 · pp. 3, 5
3. Generalization bound for FNOs
This section defines capacity-constrained hypothesis classes for FNOs using group norms, then bounds their Rademacher complexity and generalization error. The results cover Fourier layers with FCN and CNN variants, with posterior estimates under stated data and error conditions.
3.1 Mathematical Setup
- The capacity is defined from layer-weight norms, with p indexing positions, frequencies, and inputs and q indexing outputs for Fourier layers.The CNN kernel norm instead indexes kernels and inputs by p and outputs by q.Source: §3.1 · p. 6
- The hypothesis classes fix depth, maximal Fourier modes, width, input and output sizes, and activation function while varying the admissible FNO weights.The paper also defines a class for FNOs with CNN layers and auxiliary classes ending at a Fourier layer.Source: §3.1 · p. 7
3.2 Main Results
- The proof first bounds the output norm, then bounds sample Rademacher complexity, inductively peeling off Fourier layers.The derivation uses Hölder’s inequality and norm inequalities; the CNN case modifies the induction for its layers.Source: §3.2 · pp. 8–10
- For FCN and CNN Fourier-layer FNOs, the paper gives high-probability generalization error bounds for ERM minimizers under bounded-loss and approximation-error conditions.The posterior estimates additionally assume bounded input norms and training-sample approximation error.Source: §3.2 · p. 13
- For trained FNOs with either FCN or CNN Fourier layers, the paper gives posterior estimates of generalization and expected error with confidence at least 1 − δ.The stated conditions include bounded input norms and bounded error on training samples.Source: §3.2 · p. 14
4. Experiments
Experiments on Burgers-equation data relate proposed capacities to empirical generalization error and examine how that error depends on kmax.
4. Experiments
- The Burgers experiment used 800 training and 200 independently generated test function pairs, with the domain discretized at N=1024.Inputs were initial conditions and targets were solutions at t=0.5.Source: §4 · p. 14
- The initial correlation sweep varied weight decay, kmax, and CNN kernel size over 100 iterations, using two Fourier layers and width 64.The sweep randomly selected kmax from 8, 12, 16, and 20 and kernel sizes from 1, 3, 5, and 7.Source: §4 · p. 14
- 0.89 correlation between capacity and generalization error was maintained at p,q=∞; for wider kmax gaps, higher p,q gave the highest correlations.For models with small kmax gaps, correlation weakened as p and q increased.Source: §4 · p. 15, Figure 4 · p. 16
- For generalization error divided by Fourier-layer weight norms, correlation with 2√kmax rose with p,q then slightly fell at very high p,q; depth-2 dependence was more convex.The reported correlation was with 2√kmax for depth 2.Source: §4 · p. 17
5. Conclusion
The study bounds FNO Rademacher complexity using capacities based on group norms and model architecture, then validates its findings experimentally. The conclusion identifies scope for extending the analysis to parameterized activations and dataset-specific complexity.
5. Conclusion
- The proposed FNO capacity depends on model architecture and weight group norms and is used to bound Rademacher complexity.The study also derives a bound on posterior generalization error.Source: §5 · p. 19
- Experiments indicate that group-norm choices affect what information about FNO weights and architecture the capacity captures.The study reports insights into the impact of p and q values.Source: §5 · p. 19
- Because FNO weights are higher-rank tensors, the analysis may also help study other neural networks with higher-rank tensors.Source: §5 · p. 19
- The study may guide PAC-learning analysis of neural operators, for which detailed analysis had not yet been performed.Source: §5 · p. 19
- The analysis assumes a fixed activation; parameterized activations such as GeLU require modifying the analysis.Source: §5 · p. 19
- The complexity bound lacks specific dependence on each problem, while FNO performance varied across the PDE problems tested.The authors identify extending complexity to include dataset information as future work.Source: §5 · p. 19