Source-linked AI summary
Selection, Representation, and Execution in Sparse Fourier Neural Operators
Abdul Qadir Ibrahim, Martin Burger
TL;DR
Sparse FNOs raise the question of whether reductions in representation, parameters, or analytical operations also reduce inference cost. The paper studies these transitions separately across support selection and execution experiments, finding that accuracy-preserving reductions often fail to lower measured latency. It therefore defines useful sparsity as reduced support that preserves accuracy and maps to a genuinely cheaper hardware execution path.
Problem
Sparse representations are expected to reduce model size and inference cost, but FNO sparsity levels need not produce the same computational outcome.
Method
The paper empirically evaluates support selection, parameter and analytical reductions, and measured execution for FNO-like operators using controlled compression experiments.
Results
The mixed dictionary does not outperform the conventional FNO overall, while neither resolution reduction nor an 83% parameter reduction lowers measured latency; residual correction recovers accuracy but removes the expected saving.
Takeaways & Limitations
Useful sparse neural operators must preserve the solution map and translate reduced structure into a cheaper execution path on target hardware.
Takeaways & Limitations
Latency results apply to the tested model sizes, software stack, single NVIDIA A6000 GPU, and ordinary PyTorch eager execution; other settings may differ.
Abstract
from arXiv · showhide
Sparse representations are often expected to make models smaller and also reduce inference cost. For Fourier Neural Operators (FNOs), these objectives are not equivalent or do not always align: removing parts of the learned operator can leave the underlying transforms and dense computations unchanged, while changing the grid on which the model is evaluated can introduce overhead of its own. We therefore distinguish sparsity in the representation, in the stored parameters, in the theoretical operation count, and in measured runtime, and present an empirical study of several routes toward sparse FNOs that tests each transition between them separately. Coarsening the execution grid reduces the theoretical cost without reducing measured latency, and adding a correction term recovers accuracy at the cost of making the model slower. Even an 83\% parameter reduction remains slower than the dense baseline under ordinary execution. These results motivate a stricter definition of useful sparsity: the deployed operator must preserve solution accuracy and map its reduced support to a genuinely cheaper execution path.
1 Introduction
The paper separates support selection, representation reduction, analytical cost, and measured execution, then tests whether sparsity survives each transition. Functional selection improves support quality, but mixed representations and substantial parameter reductions do not necessarily produce faster execution.
- Compression framework: The study distinguishes selection, representation, and execution to test whether removing operator groups preserves accuracy, reduces stored or analytical cost, and accelerates measured hardware execution.The execution criterion requires comparable accuracy and faster measured hardware execution.
- Support selection: At tight budgets, loss-based selection gives lower mean error than parameter-norm and random selection on Darcy and Burgers flows.This indicates that parameter magnitude alone is not always a reliable importance measure.
- Architectural comparison: The mixed Fourier–wavelet representation does not outperform the conventional FNO overall.The conventional FNO remains the main architectural reference, while the mixed model is used to study heterogeneous support selection.
- Architectural comparison: The FNO advantage is concentrated in smooth bulk regions, while functionally selected mixed support is competitive near permeability interfaces.This regional breakdown refines the overall architectural comparison.
- Measured execution: Neither resolution coupling nor an 83% reduction in stored parameters produces lower measured latency under the tested execution paths.These reductions are real at their claimed levels but do not survive the transition to measured execution.
- Measured execution: Useful sparsity requires mapping reduced structure to an execution path that is cheaper on target hardware, including transforms, contractions, and kernels.Nominal sparsity alone does not guarantee computational sparsity.
2 Background and related work
The background defines neural operators as discretization-independent function-space mappings and presents Fourier and wavelet representations as complementary choices. It then distinguishes four sparsity levels because reductions in representation, parameters, or analytical operations need not reduce actual execution cost.
- Neural operators: Neural operators learn mappings between function spaces rather than mappings tied to a fixed discretization.In practice, these mappings are discretized on a grid.
- Fourier representation: An FNO represents learned interactions in Fourier space, where mode truncation reduces retained spectral coefficients but complete transforms and pointwise paths may still execute.The layer combines spatial Fourier transformation, learned channel mixing, inverse transformation, and local mixing.
- Wavelet representation: Wavelet operators complement Fourier operators by representing localized multiscale structure, including scale and orientation subbands.Fourier modes are global, whereas wavelet representations are localized and multiscale.
- Four levels of sparsity: The paper defines basis sparsity as fewer active Fourier shells or wavelet subbands, parameter sparsity as fewer stored trainable scalars, and analytical sparsity as a lower theoretical FLOP count.These are distinct from measured computational sparsity.
- Four levels of sparsity: Computational sparsity means lower measured memory use or wall-clock latency after inactive structure is removed from execution.A reduction at one sparsity level does not necessarily produce a reduction at the next.
- Related compression work: Prior compression work likewise shows that fewer parameters or factorized operators can remain slow when FFTs and additional tensor operations dominate execution cost.This motivates evaluating both model size and runtime rather than treating parameter reduction as computational savings.
3 PDE benchmarks
The paper evaluates mixed Fourier–wavelet representations on Darcy and Burgers benchmarks, which test heterogeneous localized structure and nonlinear evolving gradients. These benchmarks provide complementary settings for assessing whether representations preserve both global and fine-scale solution features.
- Complementary benchmarks: The two benchmarks expose different structural demands: Darcy emphasizes heterogeneous coefficients and localized features, whereas Burgers emphasizes evolving gradients.
- Darcy flow: Darcy flow tests two-dimensional steady flow with sharp coefficient contrasts and localized solution features.The coefficient field is heterogeneous, and localized gradients arise where the coefficient jumps.
- Burgers flow: Burgers flow tests one-dimensional nonlinear transport with gradients evolving across spatial scales.The benchmark uses ν = 0.01 at resolution 256.
- Visual comparison: Representative dense and sparse predictions are visually indistinguishable from ground truth at the displayed scale.The shown predictions come from the mixed dictionary and its functionally selected support.
4 Methods
The methods study sparse FNO-like operators through structured Fourier–wavelet groups, functional and parameter-based support selection, resolution coupling, residual correction, and transformed-channel representations. Each route separates reduced support or theoretical work from the practical execution path and its measured cost.
- Mixed dictionary: The mixed Fourier–wavelet operator combines global Fourier shells and localized wavelet subbands as structured groups for support selection.Groups are removed as complete operator components rather than individual scalar weights.
- Support selection: Functional selection removes candidate groups according to their measured calibration-loss effect and recomputes scores after each removal.The selected support is fixed before retraining from the dense checkpoint.
- Support selection: Parameter-norm selection ranks controlled expert parameters with size-normalized scores, while shared lifting, projection, and local-mixing parameters are excluded.The resulting parameter-matched support can differ from functionally selected support because group size is not directly included in ranking.
- Resolution coupling: Resolution coupling evaluates retained Fourier prefixes on smaller latent grids, including restriction before and interpolation after each layer.The intended FFT and pointwise workload is reduced, but measured runtime includes grid transitions and synchronization.
- Hybrid residual: A coarse branch adds a learned fine-grid residual through a single global gate to recover accuracy lost through coarsening.The gate is initialized at ρ ≈ 0.269 and is optimized through data loss in the reported experiment.
- Transformed-channel basis: The transformed-channel prototype tests whether channel-space block sparsity can execute competitively before adding learned transforms.The phase-0 setup uses identity transforms and distinguishes representation effects from execution limitations.
- Transformed-channel basis: 83.2% parameter reduction separates the diagonal representation from the standard FNO in the reported parameter counts.The standard FNO has 549,569 parameters, while the diagonal model has 92,353.
5 Results
Within the heterogeneous Fourier–wavelet dictionary, functional support selection generally outperforms parameter-norm and random selection, but the conventional FNO remains the stronger dense baseline. Across execution routes, reductions in analytical cost or parameters do not translate into lower measured latency, and accuracy recovery can add overhead.
- Functional selection: At matched group counts, functional selection achieves lower mean test error than parameter-norm selection on both Darcy and Burgers, with a larger gap on Burgers.Matching parameter counts strengthens the separation on Darcy, where parameter-norm selection retains fewer groups but yields considerably higher error.
- Functional selection: Functional selection beats independently sampled random supports in 34 of 36 exact-cardinality comparisons.Operating-point budgets were chosen using a development run and held fixed for confirmation runs.
- Dense architectural comparison: The conventional FNO is smaller and more accurate than the dense mixed dictionary on Darcy and is more accurate with fewer parameters on parameter-matched Burgers.The mixed-model rows use pre-selection checkpoints, whereas dense Table 2 values use additional training.
- Regional error: The FNO’s global Darcy advantage is concentrated in the smooth bulk, where its error is 10–16% lower, while functional mixed support is slightly better near interfaces.The regional comparison uses a different mixed-model checkpoint from the global architectural comparison.
- Execution cost: A 50.9% analytical-cost proxy on Burgers and a 27.2% proxy on Darcy do not reduce latency, which instead rises above dense controls.Reported latency increases from 1.194 to 1.474 ms on Burgers and from 1.631 to 2.070 ms on Darcy; Darcy validation error also rises substantially.
- Execution cost: Adding a fine-scale residual lowers coarse-only error from 0.021322 to 0.011888 but raises latency from 1.653 to 1.892 ms.The residual result is close to the dense development checkpoint’s 0.012107 error but does not improve latency.
- Execution cost: An 83% reduction in real scalar parameters still produces 1.852 ms latency versus 1.397 ms for the standard FNO, while the dense-block transformed representation reaches 3.075 ms.The fixed identity-basis implementation was already slower than the dense baseline, so learning the channel transform was not pursued.
6 Discussion
The study separates support selection, representation reduction, and execution, showing that success at one level does not ensure savings at the next. Its conclusions are conditional on the tested representation, comparisons, hardware, software, and implementation.
- Computational compression requires preserving the solution map, reducing representation or analytical cost, and realizing that reduction through an efficient execution path.
- A conventional FNO remains a stronger dense baseline overall, although heterogeneous supports can be locally competitive near permeability interfaces.
- Latency depends on transforms, memory movement, kernel fusion, batch size, implementation, and whether inactive structures are physically removed from execution.
- The mixed representation is motivated by adaptive-basis work, but selected structures must also admit efficient implementation.
- Comparisons are limited because the WNO experiment covers only a Burgers diagnostic, the Darcy run lacks a valid final result, and some latency measurements use development checkpoints.
- Conclusions about parameter-norm selection concern this particular norm-based control because its score is not invariant to all reparameterizations.
- Latency results apply to the tested model sizes, PyTorch eager execution, single NVIDIA A6000 hardware, and limited experimental runs.
- A fused block-sparse kernel or larger spatial and three-dimensional problems could change the transformed-channel latency balance.
7 Conclusion
The study evaluates ways to reduce Fourier neural-operator representation and computational cost through functional support selection and execution changes. Functional selection preserves accuracy within the tested dictionary, but neither resolution reduction nor 83% parameter reduction lowers measured latency, motivating hardware-aware definitions of useful sparsity.
- Functional support selection preserves accuracy and outperforms size-normalized parameter-norm selection at tested operating points within a heterogeneous Fourier–wavelet dictionary.
- The mixed dictionary does not outperform the conventional FNO baseline overall.
- Neither resolution-based reduction nor an 83% reduction in stored parameters lowers measured latency under the tested execution paths.
- A fine-scale residual recovers much of the accuracy lost through coarsening but removes the expected computational saving.
- Useful sparse representations must preserve the solution map and translate reduced structure into a cheaper target-hardware execution path.
Data and code availability
The manuscript provides a GitHub repository containing the code and an archived manuscript version, while excluding large datasets, checkpoints, and raw training outputs.
- Code is available on GitHub, with the manuscript version archived under the arxiv-v1 tag.
- Large datasets, model checkpoints, and raw training outputs are not included, but experimental protocols and artifact names are documented.
A Supporting figures
The appendix supplies diagnostics for regional error, support selection, retraining, random controls, and parameter–error trade-offs. These figures supplement the main accuracy and latency comparisons.
- Regional Darcy diagnostics compare interface and bulk error distributions against an independent conventional FNO under the same masked metric.
- Support diagnostics show agreement across runs and retained-group distributions across Fourier and wavelet families at the structured-group level.
- Additional checks cover retraining after support fixation and outcome distributions for random-support controls.
- Parameter reduction can coexist with preserved accuracy but does not by itself imply lower inference latency.
- Functional selection changes both the number and type of retained groups, whereas parameter-norm selection allocates groups differently across Fourier and wavelet families.