Source-linked AI summary
Frequency Bias and OOD Generalization in Neural Operators under a Variable-Coefficient Wave Equation
Runlong Xie, An Luo
TL;DR
Neural operators can accelerate PDE solution prediction, but their behavior under structured distribution shifts is not well understood. This paper compares FNO and DeepONet on a variable-coefficient wave equation with frequency and smoothness shifts. FNO is more accurate in-distribution and under smoothness shifts, while DeepONet is more stable under frequency OOD conditions despite higher overall error.
Problem
Structured responses to frequency and medium-heterogeneity shifts remain less systematically studied than predictive accuracy, leaving neural-operator extrapolation under such changes unclear.
Method
The paper conducts a controlled FNO–DeepONet comparison for terminal-state learning in a one-dimensional variable-coefficient wave equation using structured frequency and smoothness OOD settings.
Results
FNO achieves higher accuracy in-distribution and under smoothness shifts, while DeepONet shows more stable performance under frequency OOD conditions.
Takeaways & Limitations
Predictive accuracy alone is insufficient; structured OOD evaluation and spectral analysis reveal how architectural frequency representations shape generalization.
Takeaways & Limitations
The study’s conclusions are bounded by the tested architectures and indicate that extending training to higher frequencies may improve performance within that range.
Abstract
from arXiv · showhide
Neural operators learn to map initial conditions to the terminal solution of partial differential equations (PDEs), providing a surrogate for the full operator mapping. This enables rapid prediction across different input configurations. While recent neural operator architectures have demonstrated strong performance on diverse PDE tasks, their behavior under structured distribution shifts remains insufficiently understood. To investigate this, we study operator learning in a wave propagation setting governed by a one-dimensional variable-coefficient wave equation, using two representative architectures, the Fourier Neural Operator (FNO) and the Deep Operator Network (DeepONet). To examine their generalization under distribution shifts, we consider structured out-of-distribution (OOD) settings that independently vary input frequency and coefficient smoothness. The results show that under smoothness shifts, both models maintain stable performance, with FNO achieving lower error. In contrast, under frequency shifts, FNO exhibits a sharp increase in error under unseen high-frequency inputs, whereas DeepONet shows milder degradation despite higher overall error. Our analysis reveals that these differences arise from how each architecture represents and responds to variations in frequency structure. Together, these findings highlight a fundamental gap between strong in-distribution performance and generalization under distribution shifts in operator learning, underscoring the role of architectural representation bias in developing more reliable neural operators for physics-based PDE simulations beyond the training distribution.
1. Introduction
Neural operators offer fast surrogates for PDE simulation, but their stability under structured shifts in frequency and medium structure remains unclear. This work compares FNO and DeepONet in a controlled variable-coefficient wave setting to connect OOD behavior with architectural representation.
- Motivation: Neural operators learn mappings between input functions and PDE solutions, enabling rapid prediction across changing configurations.They are motivated by repeated scientific simulation and surrogate-modeling applications.
- Architectural comparison: FNO uses spectral convolution for global frequency interactions, whereas DeepONet combines input-function and coordinate representations through branch–trunk decomposition.These architectures provide contrasting operator representations for studying distribution-shift behavior.
- Research gap: Structured distribution shifts remain insufficiently understood, especially when input frequency content or medium structure differs from training data.The question matters because scientific models are often applied to systematically different physical systems.
- Study design: The study evaluates terminal-state operator learning for a one-dimensional variable-coefficient wave equation under independent frequency and coefficient-smoothness shifts.The setting captures wave propagation through spatially varying material properties.
- Contributions: The work combines controlled FNO–DeepONet comparison, structured OOD testing, and spectral-error analysis to examine degradation across frequency regimes.This framework is intended to isolate architectural effects in physically structured PDE settings.
2. Problem Setup
The paper studies a conservative operator mapping from an initial displacement and spatially varying wave speed to a terminal wave field. Fixed boundaries, zero initial velocity, and controlled frequency and smoothness shifts define the problem setting.
- Operator mapping: The operator maps initial displacement u0(x) and wave-speed function c(x) to the wave field u(x, T) at a fixed terminal time.The domain is one-dimensional, with x ∈[0, 1].
- Distribution shifts: The controlled setting varies input frequency and coefficient smoothness while minimizing other factors such as model scale or pretraining.These shifts are used to study architecture behavior under structured distribution changes.
- Governing equation: The governing equation is a conservative variable-coefficient wave equation, allowing spatially varying propagation speed in heterogeneous media.The conservative form represents local interactions when the medium changes spatially.
- Conditions: Homogeneous Dirichlet boundaries fix u(0, t) and u(1, t), while the initial velocity is set to zero.The resulting problem contains frequency interactions between the input wave field and spatially varying coefficients.
3. Methods
FNO and DeepONet approximate the same operator with contrasting representations: FNO performs truncated Fourier-space convolution, while DeepONet uses coordinate-conditioned basis expansions.
- FNO: FNO lifts the input functions into latent features and updates them through Fourier layers that model global spatial interactions.Fourier transforms, learned spectral weights, and pointwise transformations define the layer updates.
- FNO: FNO retains a fixed number of low-frequency modes, imposing a smoothness-oriented spectral bias while reducing computational cost.The truncation assumes dominant solution structure is primarily encoded in low-frequency components.
- DeepONet: DeepONet combines branch-network coefficients from input functions with trunk-network basis functions conditioned on spatial coordinates.Its output is a learned coordinate-dependent expansion rather than an explicit fixed spectral representation.
- Representation bias: FNO models frequency-component interactions, whereas DeepONet constructs solutions through coordinate-conditioned basis expansions.These distinct inductive biases motivate comparing their behavior under structured distribution shifts.
- Experimental rationale: The controlled comparison focuses on canonical architectures to attribute performance differences more directly to their underlying representation mechanisms.The study avoids additional factors such as pretraining, attention, and multi-resolution designs.
4. Experimental Setting
The experiments generate supervised wave data numerically, partition samples into ID and structured OOD regimes, and train FNO and DeepONet under matched conditions. Evaluation combines relative solution error with spatial-energy and spectral diagnostics.
- Data generation: A conservative finite-difference solver generates terminal solutions from initial displacement and spatially varying wave-speed inputs.Each sample contains u0(x), c(x), and u(x, T).
- Dataset design: Initial conditions use random Fourier combinations, with OOD-frequency samples adding higher-frequency components absent from training.Coefficient fields are constructed through random frequency superposition, while smoothness shifts alter their spatial variability.
- Dataset partition: The dataset includes training, in-distribution testing, and OOD testing subsets formed by systematically modifying selected input characteristics.The OOD scenarios target frequency and coefficient-field smoothness shifts.
- Pipeline: The pipeline trains FNO and DeepONet to map (u0, c) to u(x, T) and evaluates them under ID, frequency-shift, and smoothness-shift conditions.Figure 1 summarizes solver-based data generation, model training, and structured evaluation.
- Training: Both models use identical datasets, batch sizes, training durations, random seeds, and relative L2 loss for a controlled comparison.Optimization uses Adam with a fixed learning rate.
- Metrics: Evaluation uses relative L2 error, an energy-like spatial diagnostic, and Fourier analysis of prediction error.The energy-like quantity is auxiliary rather than a strict conserved energy, while spectral analysis tracks error across frequency components.
5. Results and Analysis
Both models perform stably in-distribution but diverge under unseen frequency inputs: FNO degrades sharply through structured multi-mode spectral errors, while DeepONet degrades more gradually. The results link these contrasting OOD behaviors to architectural representation rather than spectral capacity alone.
- FNO Degrades Sharply under OOD-Frequency: FNO achieves strong in-distribution accuracy but degrades sharply when test inputs contain higher frequencies than those seen during training.Its relative L2 error rises substantially under OOD-frequency compared with Validation and ID evaluation.
- FNO Degrades Sharply under OOD-Frequency: Under OOD-frequency, FNO predictions show pronounced phase shifts and oscillatory distortions concentrated near rapid oscillations or strong local curvature.These structured local errors suggest unresolved or poorly extrapolated high-frequency content rather than random noise.
- Spatial Errors Correspond to Multi-Mode Spectral Deviation: FNO OOD-frequency error increases across multiple mid-to-high Fourier modes, indicating broader spectral disruption rather than failure confined to the highest frequencies.Localized spatial errors align with elevated errors across a band of modes.
- OOD Degradation is Closely Linked to Architectural Bias: Increasing FNO’s retained modes from 8 to 16 to 32 does not reduce OOD-frequency error; instead, error increases while ID performance remains stable.The ablation indicates that additional spectral capacity alone does not guarantee extrapolation beyond training-constrained frequency relationships.
- DeepONet Exhibits Smoother Degradation under OOD-Frequency: DeepONet’s OOD-frequency error increases more gradually, with smoother waveform deviations and more uniformly distributed spectral errors than FNO.Its predictions preserve approximate peak and trough locations, while no single frequency band dominates the error.
- Spatial Errors Correspond to Multi-Mode Spectral Deviation: Across both models, spatial errors concentrate where target solutions have strong oscillations or high local curvature and correspond to structured frequency-domain deviations.The contrast is in how errors propagate across frequency components: FNO shows multi-mode accumulation, whereas DeepONet shows distributed deviations.
- OOD Degradation is Closely Linked to Architectural Bias: Overall, stable in-distribution performance does not ensure stable frequency-shift extrapolation, and the contrasting degradation patterns point to architectural representation bias.The findings identify representation structure as central to generalization under frequency shift.
6. Conclusion
The study finds that neural-operator accuracy alone does not characterize robustness: FNO performs better in-distribution and under smoothness shifts, while DeepONet is more stable under frequency shifts. These degradation patterns are linked to how architectures represent and extrapolate frequency information, motivating structured OOD and spectral evaluation.
- FNO achieves higher accuracy within the training distribution and under smoothness shifts, while DeepONet performs more stably under frequency OOD conditions.
- Predictive accuracy alone is insufficient to characterize neural-operator performance in physics-based settings.
- Structured OOD evaluation and spectral analysis reveal how operator-learning architectures interact with input structure.
- The observed degradation patterns suggest that generalization is strongly influenced by how frequency information is encoded and coupled within the model.
- Future work may explore frequency-aware representations, learnable spectral support, or hybrid global-spectral and localized-spatial architectures.
Impact Statement
Neural operators may accelerate repeated PDE evaluation, but their robustness depends on whether deployment inputs match the frequency structures represented during training. This work therefore supports evaluating learned PDE surrogates beyond standard in-distribution metrics before scientific or engineering deployment.
- Neural operators have the potential to accelerate repeated PDE evaluation in scientific computing, physics simulation, and engineering design.
- Performance can depend strongly on the frequency structures represented during training.
- Models that perform well within the training distribution may still degrade under unseen frequency conditions.
- The solver maps an initial condition and wave-speed function c(x) to the numerical solution at fixed terminal time T.
- The spatial domain uses a uniform grid with spatial step Δx, time step Δt, Nx spatial points, and Nt time steps.
- The spatial discretization uses a numerical-flux finite-difference form to preserve the PDE’s conservative structure.
- The supervised learning samples use a unified finite-difference reference solver for the conservative variable-coefficient wave equation.
B. Dataset Sampling and OOD Construction
The dataset pairs randomized initial displacement and coefficient fields with terminal wave solutions, while OOD splits independently alter input frequency or coefficient smoothness. Training and in-distribution samples share distributions, whereas OOD samples introduce unseen frequency content or smoothness regimes.
- Each sample contains an input pair, u0(x) and c(x), with the corresponding terminal solution u(x,T) generated by a reference solver.
- Initial displacement fields use random Fourier-mode combinations whose frequency cutoff Ku controls oscillatory complexity.
- Training samples contain only low-frequency initial conditions, while the OOD-frequency split introduces higher-frequency components absent during training.
- Coefficient fields are generated by random frequency superposition, with Kc controlling spatial smoothness and regimes categorized as smooth, medium, or rough.
- Training, validation, ID test, and OOD test samples use independent random seeds to avoid data leakage.
C. Model and Training Hyperparameters
The experiments use Adam optimization with a relative L2 supervised loss and report configurations for reproducible comparisons. Supplementary figures extend the evaluation across additional frequency-shift samples and smoothness-shift samples.
- Training: All models are trained with Adam without learning-rate scheduling, using the relative L2 loss as the supervised objective.
- Training: The relative L2 loss compares predicted and reference solutions through their normalized Euclidean error.The supplied passages define the loss as the prediction-reference norm divided by the reference-solution norm.
- Additional evidence: Figure 7 adds high-frequency OOD samples using the same evaluation pipeline as the main results, broadening the view of frequency-shift behavior.
- Additional evidence: Figure 8 adds wave-speed smoothness OOD samples and shows that lower smoothness-shift error is not limited to one selected example.
D.3. Full Modes Ablation Table
The appendix reports the complete retained-mode ablation values used to generate the main-text ablation figures.
- Table 2 reports the complete retained-mode ablation values used to generate the ablation figures in the main text.
D.4. Additional Spectral Error Curves
Additional spectral curves provide a frequency-domain view of prediction error across the main evaluation splits. The curves compare how FNO and DeepONet errors vary across sine modes.
- Figure 8 presents additional OOD-smoothness prediction examples with input fields, terminal predictions, and residual curves following the Figure 7 convention.
- Figure 9 reports spectral error curves across validation, ID, OOD-frequency, and OOD-smoothness evaluation splits.
- The curves provide a frequency-domain view of how prediction error varies across sine modes for FNO and DeepONet.