PaperlayerView exact revision on arXiv

Source-linked AI summary

Sensitivity-Constrained Fourier Neural Operators for Forward and Inverse Problems in Parametric Differential Equations

Abdolmehdi Behroozi, Chaopeng Shen and, Daniel Kifer

arXiv:2505.08740v3cs.LGcs.CE

TL;DR

Neural operators can efficiently approximate differential-equation solutions, but parameter inversion and sensitivity estimation remain difficult, and sensitivities matter for tasks guided by gradients. SC-FNO adds supervision of parameter sensitivities to the operator training objective; across the tested cases, it estimates sensitivities more accurately than FNO, including a 1/25 Jacobian error on PDE4. The paper presents this approach as useful for inversion and optimization, while noting that FNOs still require extensive training data and struggle to generalize beyond training datasets.

  • Problem

    Neural operators are used for forward simulations, but parameter inversion has received less attention and parameter-sensitivity estimation has not been studied for neural operators; sensitivity information also guides gradient-based tasks.

    Source: §1 · p. 2

  • Method

    SC-FNO adds a loss that supervises predicted parameter sensitivities against sensitivities computed by numerical solvers or analytical solutions.

    Source: §2.1 · p. 3

  • Results

    Across tested ODE and PDE cases, SC-FNO estimates sensitivities more accurately than FNO; for PDE4, it generates 1/25 the Jacobian error as FNO.

    Source: §3.2 · pp. 7–8

  • Takeaways & Limitations

    Sensitivity constraints target parameter dependence not captured by PDE-based losses, and the paper presents SC-FNO as suitable for inversion or optimization, including higher-dimensional inputs.

    Source: §2.2 · p. 4, §3.6 · p. 10

    FNOs still require extensive training data, and generalization beyond training datasets remains challenging.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

Parametric differential equations of the form du/dt = f(u, x, t, p) are fundamental in science and engineering. While deep learning frameworks such as the Fourier Neural Operator (FNO) can efficiently approximate solutions, they struggle with inverse problems, sensitivity estimation (du/dp), and concept drift. We address these limitations by introducing a sensitivity-based regularization strategy, called Sensitivity-Constrained Fourier Neural Operators (SC-FNO). SC-FNO achieves high accuracy in predicting solution paths and consistently outperforms standard FNO and FNO with physics-informed regularization. It improves performance in parameter inversion tasks, scales to high-dimensional parameter spaces (tested with up to 82 parameters), and reduces both data and training requirements. These gains are achieved with a modest increase in training time (30% to 130% per epoch) and generalize across various types of differential equations and neural operators. Code and selected experiments are available at: https://github.com/AMBehroozi/SC_Neural_Operators

1 INTRODUCTION

Neural operators use initial conditions and physical parameters to approximate differential-equation solution paths, supporting forward simulation and parameter inversion. The introduction argues that existing operators struggle with inversion and sensitivity estimation, and presents SC-FNO as a framework for addressing these tasks.

1 INTRODUCTION

  • FNOs use frequency-domain representations to learn solution trajectories efficiently, with meshless variants that can produce solutions at different spatiotemporal resolutions.Learning trajectories directly in the time domain would require more weights and extensive training data.Source: §1 · pp. 1–2
  • Parameter inversion and parameter-sensitivity estimation remain less addressed by neural-operator research than forward simulation, and FNO+PINN still produces inaccurate sensitivity estimates.The authors report that their approach can outperform FNO+PINN by up to 30%.Source: §1 · p. 2
  • Sensitivity information guides gradient-based optimization and data assimilation, while inaccurate sensitivities can yield erroneous responses to perturbed parameters or inputs.The passage also connects sensitivity information with engineering design, control, risk assessment, and resource allocation.Source: §1 · p. 2
  • SC-FNO adds parameter-sensitivity learning to FNO to address forward and inverse problems, and the approach is compatible with finite differences or automatic differentiation.The authors report accurate sensitivities with less training data and generalization to other neural operators.Source: §1 · p. 2

2 METHODOLOGY

SC-FNO augments solution-path training with supervision of parameter sensitivities computed from differentiable solvers or finite differences. The methodology describes its FNO-based mapping, an optional PINN equation loss, two ways to prepare gradient data, and an implementation that reuses precomputed sensitivities.

2 METHODOLOGY

  • SC-FNO trains on solution paths and parameter sensitivities, optionally adding a PINN equation loss; sensitivity data can come from differentiable models or finite differences.The framework is also applicable to other neural operators.Source: §2 · p. 3

2.1 SC-FNO: ENHANCING FNO WITH SENSITIVITY

  • FNOs approximate mappings between function spaces using Fourier-domain operations, but extensive training data and generalization beyond training datasets remain challenges.The Fourier transform facilitates applying linear operators, while a learnable frequency-domain function helps encode and decode information.Source: §2.1 · p. 3
  • SC-FNO adds sensitivity-based losses to neural operators, constraining predicted parameter Jacobians against numerical-solver or analytical sensitivities while mapping initial conditions, coordinates, time, and parameters to u.The framework uses finite differences or differentiable solvers to obtain sensitivity data and is also presented for other neural-operator types.Source: §2.1 · pp. 3–4

2.2 THE PINN-LOSS EQUATION AS AN OPTIONAL REGULARIZER

  • The optional PINN equation loss combines PDE, initial-condition, and boundary-condition terms, whereas SC-FNO supervises parameter sensitivities absent from ordinary PDE formulations.PINN-style collocation loss supervises spatial and temporal derivatives; SC-FNO prepares gradient data and does not require optimization during forward simulation.Source: §2.2 · p. 4

2.3 GRADIENT COMPUTATION METHODS FOR DIFFERENTIAL EQUATIONS

  • Training data pair true solution paths with parameter gradients computed either by automatic differentiation through a differentiable solver or by finite differences using perturbed-parameter solves.The computed solution paths and sensitivities are archived for later use, making their preparation a one-time cost per equation.Source: §2.3 · p. 4

2.4 IMPLEMENTATION DETAILS

  • The implementation reuses precomputed true gradients and samples changing subsets of space-time points each epoch to supervise AD-computed SC-FNO gradients without additional solver runs during training.FNO and SC-FNO share the same architecture and inputs; the difference is their loss configuration.Source: §2.4 · p. 5

3 EXPERIMENTS

Across ODE and PDE tests, the experiments compare data-only, equation-loss, and sensitivity-loss neural operators across inversion, prediction, training-data volume, parameter dimension, and gradient-generation methods.

3 EXPERIMENTS

  • The experiments compare data-only FNO, FNO with equation loss, and models adding sensitivity loss across ODEs and PDEs.The tested tasks include parameter inversion, robustness to perturbations, varying training-data volumes, high-dimensional parameter spaces, and alternative gradient calculations.Source: §3 · pp. 5–6

3.1 PARAMETER INVERSION FROM SOLUTION PATHS

  • SC-FNO reached R2 = 0.998 in single-parameter inversion, with less than 1/5 FNO's and 1/4 FNO-PINN's relative L² inversion error.In simultaneous inversion, SC-FNO's relative L² error was 1/6 of FNO's and 1/2.8 of FNO-PINN's.Source: Figure 1 · p. 6, Figure 2 · p. 6, §3.1 · p. 6

3.2 SURROGATE MODEL QUALITY AND ROBUSTNESS TO INPUT PERTURBATION

  • On in-range PDE tests, SC-FNO models predicted parameter sensitivities more accurately than FNO; SC-FNO also captured PDE3 patterns and had 1/25 the PDE4 Jacobian error of FNO.For PDE1, SC-FNO and SC-FNO-PINN sensitivity R2 values were 0.92–0.93; for PDE2, they were 0.91–0.99.Source: §3.2 · pp. 7–8

3.3 MODEL PERFORMANCE ACROSS VARYING TRAINING DATA VOLUMES

  • With only 500 training samples, FNO’s R2 dropped to 0.8, while SC-FNO maintained an R2 around 0.9, comparable to FNO’s performance with 1,000 training samples.SC-FNO and SC-FNO-PINN showed a notably slower rate of decline as training sizes diminished.Source: §3.3 · p. 9

3.2 SURROGATE MODEL QUALITY AND ROBUSTNESS TO INPUT PERTURBATION

  • At 40% perturbation, SC-FNO's solution R2 was 0.912 for PDE1 and 0.933 for PDE2, versus FNO's 0.529 and 0.734, respectively.For PDE1, SC-FNO remained relatively stable as perturbation increased while FNO declined sharply.Source: §3.2 · p. 8, Figure 5 · p. 9

3.4 HANDLING FUNCTIONAL AND HIGH-DIMENSIONAL PARAMETER SPACE

  • With 82 zoned parameters and 500 samples, SC-FNO's solution relative L² error was 0.0073 versus FNO's 0.0282.For Jacobian predictions, the relative L² errors were 0.213 for SC-FNO and 2.46 for FNO; SC-FNO with 100 samples also outperformed FNO with 500 samples on solution error.Source: Table 4 · p. 9, §3.4 · pp. 9–10

3.5 DIFFERENT GRADIENT CALCULATION METHODS

  • SC-FNO trained with either AD- or FD-generated gradients achieved R² > 0.95 for solution paths and R² > 0.9 for sensitivities.AD provided higher accuracy and efficiency, while FD remained applicable to existing model code.Source: Table 5 · p. 10, §3.5 · p. 10

3.6 FURTHER DISCUSSION AND CONCLUSION

  • SC-FNO constrains time-integrated parameter sensitivities computed using differentiable solvers or finite differences, with PDE1 training memory of 764 MB versus FNO's 722 MB.The discussion presents sensitivity regularization as complementary to time-step-free methods such as FNO.Source: §3.6 · p. 10

A SC-FNO ARCHITECTURE AND SENSITIVITY INTEGRATION

SC-FNO combines solution-path learning with sensitivity and, optionally, equation-based regularization. Its architecture accepts equation parameters, spatial and temporal coordinates, and initial conditions.

A SC-FNO ARCHITECTURE AND SENSITIVITY INTEGRATION

B DIFFERENTIAL EQUATION DETAILS

The appendix describes the differential-equation cases, their initial conditions and parameter setups, and the use of analytical solutions to validate solver and sensitivity computations.

B DIFFERENTIAL EQUATION DETAILS

  • The composite harmonic oscillator has an analytical solution used to validate differential-equation solvers and sensitivity computations.Its parameters modulate oscillation frequency and amplitude, and analytical sensitivities are also provided.Source: B DIFFERENTIAL EQUATION DETAILS · p. 17
  • The Duffing oscillator is a nonlinear system with damping, linear and nonlinear stiffness, driving amplitude, and frequency parameters.Its initial conditions are x(0) = ϵ and ˙x(0) = ζ.Source: B DIFFERENTIAL EQUATION DETAILS · p. 17
  • The forced Burgers’ equation models velocity over x ∈ [0, 1.0] and t ∈ [0, π], with advection, viscosity, and periodic forcing.The setup uses a Gaussian initial pulse centered at x0 = 0.5 with width σ = 0.3, a sinusoidal component, and periodic boundary conditions.Source: B DIFFERENTIAL EQUATION DETAILS · pp. 17–18
  • The Navier–Stokes stream-function/vorticity case uses an initial vorticity distribution determined by α and β and sets Re to 1000.The described spatial domain is x, y ∈ [0, 1], with time extending to 3.Source: B DIFFERENTIAL EQUATION DETAILS · p. 18
  • The Allen–Cahn case tests sensitivity analysis in a system where small parameter changes can produce abrupt phase transitions.It uses u(x, 0) = c tanh(ωx) and periodic boundary conditions; the varying parameters include diffusion, linear, cubic, and initial-condition terms.Source: B DIFFERENTIAL EQUATION DETAILS · p. 18
  • The parameter ranges for the simulations are listed in Table B.6, and simulation values were randomly generated from uniform distributions.The passage describes the sampling as giving equal probability to values throughout each specified range.Source: Table B.6 · p. 18, B DIFFERENTIAL EQUATION DETAILS · p. 18

C FNOS HYPERPARAMETERS

The appendix reports case-study FNO hyperparameters and compares training times across model configurations and batch sizes.

C FNOS HYPERPARAMETERS

D.1 PERFORMANCE COMPARISON OF NEURAL OPERATORS

The appendix comparison covers original and sensitivity-constrained neural operators on PDE1 and PDE2, with tables identified for results on each system.

D.1 PERFORMANCE COMPARISON OF NEURAL OPERATORS

  • The appendix compares FNO, WNO, MWNO, and DeepONets with their sensitivity-constrained counterparts on PDE1 and PDE2.The supplied passage states the comparison scope but does not report the table values or identify a performance winner.Source: §D.1 · p. 20
  • Table D.9 reports R2 for PDE1 with 2 × 10^3 training samples.The supplied table caption does not include the metric values.Source: Table D.9 · p. 20
  • Table D.10 reports R2 for PDE2 with 2 × 10^3 training samples.The supplied table caption does not include the metric values.Source: Table D.10 · p. 20

D.2 ADDITIONAL RESULTS FOR THE INVERSION EXPERIMENTS

Sensitivity-constrained models show higher accuracy in simultaneous multi-parameter inversion across the reported PDEs and neural operators.

D.2 ADDITIONAL RESULTS FOR THE INVERSION EXPERIMENTS

  • Sensitivity-constrained versions have uniformly much higher inversion accuracy, with smaller differences between neural operators than between constrained and unconstrained versions.Table D.11 compares PDE1 and PDE2 inversion accuracy with and without gradient supervision.Source: Table D.11 · p. 20, §D.2 · p. 20
  • For PDE1, Figure D.8 presents individual parameter results in simultaneous multi-parameter recovery, comparing FNO and SC-FNO.The figure caption notes that α was presented in Figure 1b.Source: Figure D.8 · p. 21
  • For PDE2, Figure D.9 presents individual parameter results in simultaneous multi-parameter recovery, comparing FNO and SC-FNO.Source: Figure D.9 · p. 21

D.3 VERIFICATION OF DIFFERENTIABLE AND FINITE DIFFERENCE SOLVERS

The appendix compares differentiable and finite-difference solver data preparation, reporting error metrics and runtime for ODE1 alongside dataset-preparation timing details.

D.3 VERIFICATION OF DIFFERENTIABLE AND FINITE DIFFERENCE SOLVERS

  • Figure D.10 shows inversion of parameter α in PDE3 using FNO and SC-FNO models.Source: Figure D.10 · p. 22
  • Table D.12 compares dataset-preparation computation time using an AD solver with and without a Jacobian.Source: Table D.12 · p. 22
  • Table D.13 compares solution-path and parameter-gradient R2 values and runtime for generating 2 × 10^3 samples with differentiable and fourth-order finite-difference solvers.Measurements used a V100 GPU and four Intel Xeon processors.Source: §D.3 · p. 21, Table D.13 · p. 22
  • Table D.13 reports error metrics and runtime for training-data preparation with AD and FD solvers on ODE1.Source: Table D.13 · p. 22

D.4 SURROGATE MODEL QUALITY FOR THE ORDINARY DIFFERENTIAL EQUATIONS (ODES).

The appendix reports test R2 and Relative L² values for solution paths and sensitivities.

D.4 SURROGATE MODEL QUALITY FOR THE ORDINARY DIFFERENTIAL EQUATIONS (ODES).

  • The appendix reports test R2 values and Relative L² for solution paths u and sensitivities.Source: Table D.14 · p. 22
Loading 2505.08740v3…