PaperlayerView exact revision on arXiv

Source-linked AI summary

A Novel Fourier Feature Network for Solving Partial Differential Equations

Qihong Yang, Zhijie Su, Yangtao Deng, Qiaolin He

arXiv:2608.14733v1cs.LGcs.AI

TL;DR

Solving PDEs accurately and efficiently remains challenging for neural networks. The paper introduces Fourier Feature Networks and reports higher accuracy than ELMs, including errors near machine precision in numerical experiments.

  • Problem

    Existing neural-network PDE solvers can be computationally costly or fail to attain machine-level precision, motivating more accurate and efficient approaches.

  • Method

    Fourier Feature Networks use single-hidden-layer basis functions built from cos, sin, or both, with coefficients fitted by least squares and scaling optimized.

  • Results

    6.4756×10^-15 FEN L∞ error versus 1.7599×10^-10 ELM L∞ error; FENs also outperform ELMs on nonlinear Burgers’ equation.

  • Takeaways & Limitations

    FENs provide a simple single-hidden-layer alternative that achieves higher numerical accuracy than ELMs without affine input transformations.

  • Takeaways & Limitations

    FENs depend strongly on scaling-factor selection, and both FENs and ELMs struggle with high-dimensional problems.

Abstract

from arXiv · show

Building on the foundation of single-hidden-layer neural networks, Fourier Feature Networks (FENs) are proposed, which incorporate Fourier features using $\cos$, $\sin$, or a combination of both. Similar to Extreme Learning Machines (ELMs), FENs employ a single-hidden-layer architecture to generate a set of basis functions. The target function is then approximated as a linear combination of these basis functions, with the coefficients determined using the least squares method. However, unlike ELMs, which often rely on affine transformations to improve representational power, FENs can achieve high-precision solutions without requiring such transformations on the input variables. To evaluate the representational capacity of these networks, we search for an optimal scaling factor within a predefined range for the randomly initialized and fixed weights and biases. By adjusting this scaling factor, we ensure a fair comparison between FENs and ELMs using various activation functions, such as $\text{sigmoid}$, $\tanh$, and $\text{swish}$. Our numerical experiments demonstrate that FENs consistently achieve higher accuracy than ELMs.

1. Introduction

The introduction contrasts training-based and randomized neural-network approaches for PDEs, highlighting the former’s accuracy and computational limitations. It motivates improving ELM accuracy without added complexity and introduces Fourier Feature Networks as the paper’s proposed approach.

  • Existing PDE solvers: Training-based PDE methods minimize equation residuals using gradient-based optimizers such as Adam or L-BFGS.DRM, DGM, and PINNs are identified as training-based methods.
  • Limitations: Training-based neural networks often provide limited accuracy while requiring considerable computational time.This limitation motivates interest in alternative randomized neural networks.
  • Randomized methods: Randomized neural networks can solve PDEs efficiently and accurately but present notable implementation challenges.The introduction identifies ELMs as a relevant randomized neural-network approach.
  • Related work: ELMs may produce unsatisfactory results because they rely on Xavier or Kaiming initialization, while related work seeks optimal or near-optimal initialization-related parameters.The differential-evolution approach computes an optimal or near-optimal R_m value for ELMs.
  • Motivation and contribution: The paper targets machine-precision accuracy at low computational cost by improving ELMs without domain decomposition, numerical integration, or increased network complexity.The stated strategy deliberately avoids complexity-increasing operations while retaining a simple ELM framework.
  • Paper organization: The article introduces Fourier Feature Networks and evaluates their efficacy through numerical experiments after presenting ELM architecture and activation functions.The stated activation functions are sigmoid, tanh, and swish.

2. Preliminaries

The preliminaries define neural feature spaces as spans of last-hidden-layer basis functions and describe ELMs, whose hidden parameters are randomized while output weights are directly computed. They also explain affine input normalization as important for ELM-based PDE solving and numerical stability.

  • Neural feature space: Neural networks map d-dimensional inputs through hidden layers, whose outputs define the neural feature space.The feature space is represented by the last hidden layer’s outputs.
  • Neural feature space: For x∈ℝ^d, hidden-layer mappings ϕ_i(x), 1≤i≤M, form basis functions spanning the neural feature space.M denotes the number of outputs from the last hidden layer.
  • Neural network output: With a single-output layer, the neural network represents a function as a linear combination of last-hidden-layer outputs.This output-layer formulation motivates solving for linear coefficients over the neural feature basis.
  • Extreme Learning Machines: ELMs randomly initialize input-to-hidden weights W and biases b, then directly compute hidden-to-output weights w.The activation function acts elementwise on Wx+b, leaving w as the quantity that must be calculated.
  • Normalization: Normalization supports training, generalization, robustness, and numerical stability when neural networks approximate functions or solve PDEs.For ELM-based PDE solving, affine transformations map inputs from a closed domain Ω to [-1,1]^d and are described as critical for superior performance.

3. Methodologies

This section introduces single-hidden-layer Fourier feature networks using cosine, sine, or combined cosine-and-sine activations to improve approximation capacity without affine input transformations. It represents solutions through basis functions, determines coefficients by least squares, and selects initialization scaling factors by enumerating candidates and minimizing error.

  • Fourier feature architectures: Fourier feature networks use single-hidden-layer architectures with cos, sin, or combined cos and sin activations to represent the target function.The section presents separate cosine, sine, and combined Fourier feature constructions.
  • Fourier feature architectures: Unlike sigmoid, tanh, and swish networks, the cosine and sine Fourier feature networks do not require affine transformations of their inputs.The no-affine-transformation property is stated for both the cosine and sine variants.
  • Fourier feature architectures: The combined network adds cosine and sine basis contributions, with M = 2p basis functions and separate weights, biases, and input matrices for each activation.Its output combines weighted cosine and sine terms, while each parameter vector has dimension p.
  • Least-squares solution: Coefficients are obtained by forming the basis-function matrix A at sampled dataset points and solving A w = F by least squares.The vector F contains the target function values at the sampled points.
  • Scaling-factor search: The method initializes weights and biases from a unit-variance uniform distribution, enumerates scaling factors over a constrained range, and selects the factor with the smallest loss.The loss is computed as ||A·w − F|| after solving for the least-squares coefficients at each candidate scale.
  • Scaling-factor search: Adjusting the scaling factor can improve network-parameter applicability, while FENs generally use broader search ranges and achieve lower errors than ELMs.The passage states that this trend appears for FENs as well as the illustrated tanh ELM case.

4. Numerical Experiments · 4.1. Two-dimensional function · 4.2. Helmholtz equation

Numerical experiments assess FEN applicability and accuracy on two-dimensional function approximation and Helmholtz boundary value problems. Across these tasks, FENs achieve substantially lower errors than ELMs and outperform FEM in the Helmholtz comparison.

  • 4. Numerical Experiments: The experiments evaluate FENs on function approximation and PDE solving, using maximum absolute (L∞) and relative L2 errors.Experiments ran on a Debian 12 server with an Intel Xeon Platinum 8358 CPU and NVIDIA A100 GPU.
  • 4.1. Two-dimensional function: For the two-dimensional function, FENs and ELMs are compared on a 101 × 101 training grid with M = 400, 900, 1600, and 2500 basis functions.ELMs use sigmoid, tanh, and swish activations, while FEN scaling factors are optimized by range search.
  • 4.1. Two-dimensional function: 6.4756×10^-15 and 1.4677×10^-15 are the smallest FEN L∞ and L2 errors, compared with 1.7599×10^-10 and 1.8013 × 10^-11 for ELMs.These results indicate substantially higher representational power for FENs in the two-dimensional approximation task.
  • 4.2. Helmholtz equation: For the Helmholtz equation, both methods use 101 × 101 collocation points divided into interior and boundary subsets.The subsets are S_r for interior points and S_b for boundary points.
  • 4.2. Helmholtz equation: FENs achieve lower Helmholtz errors than ELMs across all tested parametric configurations.With sufficient basis functions, FENs reach minimum L∞ and L2 errors of 5.3300 × 10^-14 and 2.7200 × 10^-14, versus 4.9477 × 10^-10 and 2.7595 × 10^-10 for ELMs.
  • 4.2. Helmholtz equation: FEM remains much less accurate than FENs even with a large number of degrees of freedom.FENs use global trigonometric basis functions and collocation for strong-form solutions, whereas FEM uses local basis functions and a variational weak-form approach.
  • 4.2. Helmholtz equation: For a Helmholtz solution related to tanh, FENs outperform ELMs with non-trigonometric activations, particularly when the basis-function count is small.FEN minima are 3.2613 × 10^-15 for L∞ and 2.4689 × 10^-15 for L2, versus 4.1078 × 10^-15 and 3.8454 × 10^-15 for trigonometric ELMs.

4.3. Diffusion equation

The diffusion-equation experiment treats time as a spatial dimension, trains FENs and ELMs on a 101×101 collocation grid, and finds substantially lower errors for FENs.

  • Experimental setup: The simulations use a diffusion coefficient ν=0.01, spatial interval parameters a_1=0 and b_1=5, and final time t_f=1.The source, boundary, and initial functions are selected so that an exact solution is available.
  • Experimental setup: FENs and ELMs solve the converted two-dimensional problem using 101×101 collocation points, with Table 8 specifying parameters and optimal scaling factors.The time-dependent one-dimensional diffusion equation is reformulated by treating time as a spatial coordinate.
  • Results: FENs produce significantly lower L∞ errors than ELMs across the diffusion-equation comparison.Table 9 compares sigmoid-, tanh-, and swish-activated models, while Figure 4 displays the L∞ error curves.
  • Results: 9.2371 × 10^-14 and 5.9186 × 10^-15 are the minimal L∞ and L2 errors achieved by FENs, respectively.These minima are reported across the tested basis-function configurations.
  • Results: 5.7246×10^-9 and 3.8524×10^-10 are the minimal L∞ and L2 errors obtained by ELMs, respectively.The reported ELM minima are higher than the corresponding FEN minima.

4.4. Heat equation

The heat-equation experiment uses a space-time PDE with boundary and initial conditions, trained on a 51 × 51 × 51 collocation grid. FENs achieve substantially lower L∞ and L2 approximation errors than ELMs.

  • 4.4. Heat equation: The functions f(x, y, t), g(x, y, t), and h(x, y, t) are selected so that the exact solution is known.This construction enables approximation-error evaluation against an exact solution.
  • 4.4. Heat equation: 51 × 51 × 51 collocation points are used for training, while Tables 10 and 11 summarize scale-search parameters and L∞/L2 errors.The comparison covers FENs and ELMs with varying numbers of basis functions and sigmoid, tanh, and swish activations.
  • 4.4. Heat equation: 2.6090 × 10^-14 and 1.7562 × 10^-14 are the smallest FEN L∞ and L2 errors, versus 2.4443 × 10^-12 and 1.4761 × 10^-12 for ELMs.The reported error curves and values show consistently lower approximation errors for FENs than ELMs.

4.5. Wave equation

The wave equation is solved on a unit-square spatial domain and unit-time interval using a 51 × 51 × 51 collocation grid. With optimal scaling factors, FENs achieve substantially lower L∞ and L2 errors than ELMs.

  • The wave equation is defined on Ω = (0, 1)2 with temporal domain (0, 1].
  • The experiment uses a uniform grid of 51 × 51 × 51 collocation points and searches for optimal scaling across varying basis-function counts.The corresponding parameters are listed in Table 12.
  • The comparison reports L∞ and L2 errors for FENs and ELMs using sigmoid, tanh, and swish activations.The performance results are presented in Table 13.
  • 2.4425 × 10−15 and 1.7659 × 10−15 are the minimum L∞ and L2 errors achieved by FENs, respectively.
  • 1.4412 × 10−12 and 5.1755 × 10−13 are the minimum L∞ and L2 errors achieved by ELMs, respectively.The comparison establishes higher computational precision for FENs than ELMs in this wave-equation experiment.

4.6. Nonlinear Helmholtz equation

For the one-dimensional nonlinear Helmholtz boundary value problem, FENs and ELMs are evaluated using Picard iteration and scale-optimized parameters. FENs achieve substantially lower errors and higher accuracy than ELMs, although tanh is ELMs’ strongest activation.

  • Experimental setup: The nonlinear Helmholtz experiment uses 3000 uniform collocation points and 100 Picard iterations to linearize the nonlinear term at each step.The initial approximation is generated from randomly initialized coefficients, after which βsin(u₀) is moved to the right-hand side.
  • Experimental setup: Scale search identifies optimal scaling factors for the nonlinear Helmholtz problem, with resulting L∞ and L2 errors reported for FENs and ELMs.Figure 7 compares the corresponding L∞ error curves.
  • Method comparison: ELM with tanh clearly outperforms sigmoid and swish, but FEN error curves remain consistently and significantly lower than ELM curves.The results indicate that activation choice critically affects ELM performance, while FENs retain superior accuracy and robustness.
  • Accuracy results: 2.2427 × 10^-13 is the smallest L∞ error for FENs, versus 3.7986 × 10^-11 for ELMs.These are the lowest approximation errors reported for the two methods on this problem.
  • Accuracy results: 1.3705 × 10^-15 is the smallest L2 error for FENs, compared with 4.4912 × 10^-12 for ELMs.The reported values demonstrate substantially higher solution accuracy for FENs.

4.7. Poisson equation with an oscillating solution

The section evaluates FENs and ELMs on a highly oscillatory one-dimensional Poisson equation whose exact solution superposes sine functions with exponentially increasing frequencies. FENs achieve substantially smaller errors than ELMs, while a cosine FEN nearly overlaps the exact solution.

  • The exact solution is a superposition of sine functions with exponentially increasing frequencies, producing highly oscillatory behavior.
  • Networks are trained with N_x = 3000 uniformly distributed collocation points, while tanh uses a narrower admissible scaling-factor range for numerical stability.The narrower tanh range reflects numerical stability requirements during optimal scaling-factor searches.
  • With M = 900 and ρ_opt = 130, the cosine FEN numerical solution exhibits an almost perfect overlap with the exact solution.This comparison is reported in Figure 8 for the FEN using cosine activation.
  • 6.1572 × 10^-8 and 8.0396 × 10^-8 are the minimum L∞ and L2 errors achieved by ELMs, respectively.
  • 1.3878 × 10^-11 and 3.0395 × 10^-11 are the corresponding minimum L∞ and L2 errors achieved by FENs, respectively.These results demonstrate FENs’ greater suitability for the highly oscillatory Poisson problem.

4.8. Nonlinear Burgers’ equation

For the nonlinear Burgers’ equation with ε=0.01, the networks were trained on 200×200 collocation points using 100 Picard iterations. When M=5000, FENs substantially outperform ELMs, while neither network approximates the solution well for M≤2500.

  • Training setup: The problem uses 200×200 uniform collocation points and 100 Picard iterations for neural-network training.The approximate solution is obtained using FEN, with optimal scaling factors and related search parameters reported separately.
  • Accuracy comparison: When M≤2500, none of the networks approximate the exact solution well.This behavior is observed in the L∞ error curves.
  • Accuracy comparison: At M=5000, FENs outperform ELMs in approximating the exact solution.Despite improved ELM representational ability, FEN errors remain much lower, indicating ELMs are unsuitable for this problem.
  • Accuracy comparison: The L∞ and L2 errors listed in Table 19 further support FENs’ superior approximation performance.The passage states that these tabulated errors reinforce the observed advantage of FENs over ELMs.

4.9. High-dimensional Poisson equation

The study evaluates FENs and ELMs on the Poisson equation in dimensions 5, 7, 10, and 15 using 10,000 basis functions and optimized scaling factors. FENs are substantially more accurate at d=5 and remain competitive at higher dimensions while achieving lower computational time through analytical derivatives.

  • Experimental setup: Both methods use 50,000 interior collocation points, 1000d boundary points, and M=10,000 basis functions for dimensions 5, 7, 10, and 15.The larger basis-function count is chosen to capture the complexity of high-dimensional solution spaces.
  • Accuracy comparison: At d=5, FENs achieve substantially smaller L∞ and L2 errors than ELMs.FEN minima are 1.2396 × 10^-13 and 3.9641 × 10^-14, versus ELM minima of 1.5994 × 10^-12 and 4.5378 × 10^-13.
  • Accuracy comparison: At higher dimensions, ELMs with sigmoid and FENs with sin activation exhibit comparable precision, while ELMs with tanh have the highest error.For d=7, 10, and 15, both methods reach similar error magnitudes of approximately 10^-9 and 10^-10, 10^-6 and 10^-7, and 10^-5 for both metrics, respectively.
  • Computational cost: Searching over an excessively large scaling-factor range or using an excessively small step size can substantially increase computational cost.The optimal-scale search therefore introduces a computational-cost tradeoff.

5. Conclusions

The paper proposes Fourier Feature Networks as single-hidden-layer basis-function models for function approximation and linear or nonlinear PDEs, distinguishing them from ELMs by avoiding indispensable affine transformations. Numerical promise is tempered by sensitivity to scaling-factor choices and difficulty with high-dimensional problems.

  • Contributions: FENs represent target functions as linear combinations of hidden-layer basis functions in a single-hidden-layer neural network.The architecture is proposed for function approximation and solving linear and nonlinear PDEs.
  • Comparison with ELMs: Unlike ELMs, FENs do not require affine transformations to obtain high-precision solutions for certain problems.The passages identify affine transformations as indispensable for ELMs in those cases.
  • Training procedure: Input-to-hidden weights and biases are randomly sampled, fixed during training, and scaled by factors searched over a specified range.Only the output-layer linear-combination coefficients are optimized, with the selected factors minimizing algebraic-equation error.
  • Limitations: Poor scaling-factor choices can prevent high-precision solutions, motivating more effective methods for determining optimal scaling factors.The conclusion explicitly identifies scaling-factor dependence as a remaining research issue.
  • Limitations: Both FENs and ELMs struggle with high-dimensional problems, highlighting the need for an algorithm capable of addressing this limitation.The passage presents high-dimensional problems as an unresolved challenge for both network types.
Loading 2608.14733v1…