PaperlayerView exact revision on arXiv

Source-linked AI summary

Comparison of neural and spline representations for physics-informed learning

R{é}gis Duvigneau

arXiv:2610.06294v1math.APcs.NAmath.NAmath.OC

TL;DR

Physics-informed learning faces training difficulties, including sensitivity to hyperparameters and difficult loss minimization. This paper compares neural networks and B-Splines across four PDE problems, finding different strengths and limitations for the two representations.

  • Problem

    Physics-informed methods are not yet general-purpose solvers, with reported sensitivity to choices such as network architecture, optimizer, and loss weights.

    Source: §1 · p. 2

  • Method

    The study compares neural and B-Spline representations in a shared physics-informed framework across four PDE problems, examining error reduction, sampling, optimization, and arithmetic precision.

    Source: §2.1 · p. 3, §9 · p. 34

  • Results

    B-Splines provide controlled errors, fast loss minimization, and an established convergence rate, while neural networks can approximate solutions with few parameters but have training difficulties that limit accuracy.

    Source: §8 · p. 33, §9 · p. 34

  • Takeaways & Limitations

    The representations have complementary advantages, motivating hybridization as a possible way to combine them, while B-Spline tensorial structure limits scalability.

    Source: §8 · p. 33, §9 · p. 34

    Runtime comparisons are omitted because hardware, implementation, and programming language complicate reproducibility, so the study focuses on error estimates.

    Source: Spline representation · p. 10

Abstract

from arXiv · show

Physics-informed machine learning as recently emerged as a new paradigm for the resolution of partial differential equations, including functionalities for parametric modeling and inverse design, using neural networks to represent the solution fields. However, several difficulties have been reported concerning the training of such networks, resulting in slow convergence, low accurate solutions and sensitivity to the choice of hyper-parameters. In order to better understand the mechanisms underlying these difficulties, the present work proposes a rigourous comparison of neural networks and B-Splines representations within the same physics-informed learning framework. Four problems with closed-form solutions are numerically studied to establish a rigorous analysis of the accuracy and efficiency of the two representations, in terms of asymptotic convergence rates, sampling requirements, loss function minimization and arithmetic precision needed.

1 Introduction

The complexity and computational cost of simulation workflows limit their wider industrial adoption. This work compares neural and B-Spline representations within a shared physics-informed learning framework to examine representation-related difficulties.

1 Introduction

  • Simulation workflows remain difficult to adopt widely because of their implementation complexity and computational cost.The introduction situates this motivation across industrial sectors.Source: §1 · p. 1
  • Physics-informed learning incorporates PDE residuals into training losses, addressing data scarcity and enabling inverse problems, but remains sensitive to hyperparameter choices.The paper notes that these approaches are not yet general-purpose solvers like the finite-element method.Source: §1 · pp. 1–2
  • The study compares neural networks and B-Splines in a similar framework, using four closed-form PDE problems to analyze approximation accuracy and efficiency.The paper examines errors, sampling, loss minimization, and arithmetic precision.Source: §1 · p. 2

2 Physics-informed machine learning

Physics-informed learning represents the solution parametrically and trains it by minimizing sampled PDE and condition residuals. Its loss optimization is challenging, requiring descent methods and higher-order derivatives.

2.1 Principle of residuals minimization

  • A parameterized solution u(θ; x, t) is trained by minimizing a loss aggregating PDE residuals and initial and boundary-condition errors.The framework is intended to be agnostic to the solution representation, provided its space and time derivatives can be evaluated.Source: §2.1 · p. 3
  • Collocation points sample the space-time domain where the PDE must hold, and the PDE loss approximates the squared L2 norm of residuals.Initial and boundary terms use dedicated samples.Source: §2.1 · p. 3

2.2 Training procedure

  • Loss minimization is difficult because the parameter set is large and the loss may be non-convex and highly curved.These optimization properties affect practical implementation, accuracy, and robustness.Source: §2.2 · p. 3
  • Training typically uses descent updates, often beginning with a stochastic first-order method such as ADAM and continuing with a quasi-Newton method such as L-BFGS.Gradients require derivatives with respect to parameters of both the solution and its PDE residuals, including higher-order cross derivatives.Source: §2.2 · p. 4

3 Neural representation

The neural representation uses a multilayer perceptron to map space-time inputs to solution fields. Its trainable weights and biases are optimized, with automatic differentiation supporting derivative evaluation.

3 Neural representation

  • An MLP maps coordinate inputs through fully connected hidden layers to the unknown solution fields.The illustration depicts three hidden layers of five neurons each.Source: §3 · p. 4, Figure 1 · p. 5
  • The input layer receives space-time coordinates, hidden neurons apply nonlinear activations, and the output layer predicts the solution.The network parameters are its connection weights and biases.Source: §3 · pp. 4–5
  • Automatic differentiation through backpropagation is used to evaluate solution derivatives and loss gradients with respect to network parameters.The loss gradients are needed by the descent algorithm.Source: §3 · p. 5

4 Spline representation

B-Splines represent the PDE solution as a combination of basis functions controlled by degrees of freedom. Their analytic structure supports derivative evaluation and boundary-condition enforcement.

4 Spline representation

  • In this framework, B-Spline functions represent PDE solutions rather than geometries, despite their standard role in computer-aided design.The degrees of freedom directly control local function values.Source: §4 · pp. 5–6
  • A knot vector defines the parametric domain and basis functions; clamped knot vectors support interpolation and tangency conditions at the endpoints.The paper illustrates seven quadratic basis functions and notes that their degree, knot count, and function count satisfy l = n+p+1.Source: §4 · pp. 5–6, Figure 2 · p. 6
  • The solution is formed by associating degrees of freedom with basis functions and extending them to multiple variables through tensor products.The solution and its derivatives can be obtained by analytical differentiation.Source: §4 · p. 6
  • B-Spline derivatives can be computed efficiently because differentiating a basis function yields a lower-degree B-Spline.Cartesian-frame derivatives can be more complex when composed with a geometric mapping.Source: §4 · p. 7
  • Clamped knot vectors let boundary values be imposed independently through least-squares projection, yielding a quadratic problem in the degrees of freedom.For Dirichlet data polynomial of degree p2 or less, the described boundary solution is exact; Neumann conditions can similarly use the first interior degree of freedom.Source: §4 · p. 7

5 Properties of the two representations

Neural networks and B-Splines differ in derivative computation, boundary-condition handling, support, and optimization properties. The section also compares their approximation and sampling properties and how their parameters can be initialized.

5 Properties of the two representations

  • B-Spline basis functions have localized influence, unlike the infinite support of MLP functions, so evaluations outside a basis function's support can be avoided.The passage specifies that each B-Spline function is nonzero over p + 1 knot intervals.Source: §5 · pp. 7–8
  • For a one-dimensional Poisson problem with Dirichlet conditions, an adequately collocated B-Spline loss has a positive-definite Hessian and a unique solution, whereas MLP losses are globally non-convex.The stated MLP non-convexity is associated with weight and bias swaps that can produce the same output and several minima.Source: §5 · p. 8
  • MLPs can approximate any continuous function to any desired accuracy with one hidden layer and a non-polynomial activation, while smooth-function B-Spline approximation errors decay at O(h^(p+1)) in L∞ and L2.Here h is the knot interval.Source: §5 · p. 8
  • Analytic B-Spline derivatives are more efficient in computation time and memory than neural-network automatic differentiation, while neural networks can use Cartesian or other convenient coordinate frames.B-Splines use a parametric domain mapped to the physical domain, so Cartesian derivatives also require differentiating that mapping; matching the physical boundary to B-Spline extremities facilitates boundary-condition imposition.Source: §5 · pp. 8–9
  • Clamped B-Spline knot vectors allow initial and boundary conditions to be imposed by projection, reducing training to the PDE residual and allowing boundary degrees of freedom to be removed.For PINNs, these conditions are imposed by penalizing residuals in a multi-objective loss, whose terms can antagonize one another and cause ill-conditioning.Source: §5 · p. 9

Sampling

  • Greville abscissae provide an alternative to random collocation for B-Splines, using n points per direction and yielding convergence orders p for even degrees and p − 1 for odd degrees.This choice supports stable B-Spline interpolation and is similar to isogeometric collocation.Source: Sampling · p. 9
  • Neural-network parameters are usually initialized randomly, requiring multiple runs to assess robustness, whereas B-Spline degrees of freedom allow an intuitive solution guess.The passage links the need for multiple neural-network runs to different solutions arising in practice depending on initialization.Source: Sampling · p. 9

6 Computational Tests

Across four closed-form PDE tests, B-Splines recover their theoretical asymptotic convergence rates, while neural-network errors often plateau; the experiments compare accuracy rather than runtime.

6.1 Numerical setup

  • The common setup solves closed-form problems; accuracy is assessed with RMSE and MaxAE on a shared uniform grid, and error fields are compared near 1,200 degrees of freedom.Runtime is omitted because hardware and implementation differences make reproducible timing comparisons difficult.Source: §6.1 · p. 9, Spline representation · p. 10

Neural representation

  • Neural tests use tanh MLPs, fixed Latin-hypercube collocation points, five runs, and ADAM followed by BFGS; spline tests use degrees 2–6, Greville points, and BFGS.Boundary conditions are penalized for networks and generally imposed by projection for splines.Source: Neural representation · p. 10, Spline representation · p. 10

6.2 Poisson equation

  • For Poisson, neural networks plateau at 10^-4 while B-Splines continue reducing error and recover rates p for even and p −1 for odd degrees.Both reach 10^-4 with about 500 parameters; neural-network error is boundary-localized, indicating an unbalanced loss formulation.Source: §6.2 · p. 11, Figure 6 · p. 14

6.3 Burgers equation with viscosity

  • For Burgers, B-Splines recover asymptotic rates after refinement, while networks approximate well with few parameters except single-layer models but plateau near 10^-4.Spline error concentrates around the shock through the Gibbs phenomenon; neural-network results show greater run-to-run dispersion.Source: §6.3 · p. 15, Figure 9 · p. 18

Spline representation

  • Across Burgers, incompressible Navier–Stokes, and compressible Euler tests, spline error decreases with refinement while neural-network errors show plateaus or larger errors in specified cases.The equations add, respectively, a nonlinear hyperbolic term, a coupled multi-field system, and curved boundaries with slip conditions.Source: §6.3 · p. 15, §6.4 · p. 19, §6.5 · p. 24

6.4 Incompressible Navier-Stokes equations

  • For incompressible Navier–Stokes, B-Splines recover theoretical rates and preserve symmetric errors, whereas networks lose solution symmetry and single-layer networks have higher errors.The neural model couples three output fields, while each spline field has a separate model with three times as many degrees of freedom.Source: §6.4 · p. 19, Figure 12 · p. 23

6.5 Compressible Euler equations

  • B-Splines reduce Euler pressure error below 10^-5 near 1,200 parameters, while neural networks remain around 10^-4 and fail below 5 × 10^-5 across architectures.B-Splines also recover theoretical asymptotic rates; neural-network solutions lose symmetry.Source: §6.5 · p. 24, Figure 15 · p. 28

7 Additional studies

The additional studies test whether sampling, arithmetic precision, and optimizer choice affect the representation comparison. Sampling changes little, double precision lowers neural-network error, and training behavior differs substantially between networks and B-Splines.

7 Additional studies

  • The additional studies examine sampling, precision, and optimizer effects to clarify the role of representation in the earlier comparisons.The sampling and precision checks revisit the Poisson case; optimizer behavior is also examined for that case.Source: §7 · p. 29, §7.1 · p. 29, §7.2 · p. 31

7.1 Sampling

  • Changing neural-network sampling type and size has negligible impact on error, while variability remains similar under deterministic sampling.The neural-network study matches sampling size approximately to parameter count and compares against Greville sampling.Source: §7.1 · p. 29, Figure 16 · p. 29, Figure 17 · p. 30

7.2 Single / double precision

  • One order of magnitude lower error with double precision, but increasing network width from 32 to 64 neurons yields only a small reduction and significantly increases computation time.The comparison uses four-layer networks on the Poisson equation.Source: Figure 18 · p. 31, §7.2 · p. 31, Table 10 · p. 31

7.3 Comparison of training phase

  • B-Splines reach far lower loss in fewer iterations; ADAM alone converges very slowly, while direct BFGS reaches a local minimum and staged BFGS obtains lower loss.The network comparison uses BFGS after 2000 ADAM iterations; similar plots across initial states indicate representation-dependent conditioning.Source: Figure 19 · p. 32, §7.3 · pp. 32–33

8 Discussion

The discussion contrasts compact neural approximations with B-Splines’ error control and convergence behavior, while identifying training variability and optimization as practical neural-network difficulties. B-Splines’ tensorial structure, in turn, constrains scalability as input or output dimensions grow.

Discussion

  • Neural networks can approximate localized Burgers solutions with few parameters, with their lack of compact support permitting degrees of freedom to focus where needed.The passage presents this as a particular observation for the Burgers equation.Source: §8 · p. 33
  • Increasing neural-network parameters does not ensure corresponding error reduction, and the study establishes no asymptotic convergence for the tested cases.This contrasts with the B-Spline convergence behavior described in the discussion.Source: §8 · p. 33
  • Neural-network results vary with initialization: ε2 error typically varies by a factor of 2 to 3, while ε∞ variation can reach one order of magnitude.The variation appears greater for convection-dominant problems and is described as a practical limitation for engineering use.Source: §8 · p. 33
  • Neural-network loss minimization is harder: optimization may converge slowly or to local minima, with optimization error appearing to dominate quadrature and approximation errors.The authors note that this can require long iterative processes and yield deceptive results.Source: §8 · p. 33
  • Double precision can reduce neural-network error by one order of magnitude, while the rate of convergence remains low.The reported precision effect is for the neural-network experiments.Source: §8 · p. 33
  • B-Spline bases provide fast loss minimization and established convergence as degrees of freedom increase, but their tensorial nature limits scalability with more input dimensions or physical fields.The discussion attributes the dominant B-Spline error to approximation error.Source: §8 · p. 33
  • The discussion identifies non-convexity and ill-conditioning as neural-network training bottlenecks; prior remedies include adaptive penalties and preconditioning, while local refinement and low-rank bases are proposed for B-Spline scalability.The authors state that proposed neural-network methods have not established clear convergence as parameter counts increase.Source: §8 · pp. 33–34

9 Conclusion

The paper conducts a rigorous comparison of neural and spline representations across four PDE problems, examining error reduction as degrees of freedom increase and studying sampling, precision, and optimization for one case. Neural networks offer good approximations with few parameters but face training difficulties, whereas B-Splines support error control but scale poorly; the authors suggest hybridization as a possible direction.

Conclusion

  • The comparison studies four PDE problems using l2 and l∞ error norms, focusing on error reduction as degrees of freedom increase, with sampling, precision, and optimizer effects examined for one case.The stated objective is a rigorous comparison of neural and spline representations for physics-informed learning.Source: §9 · p. 34
  • Neural networks can approximate solutions with few parameters but encounter training difficulties that limit accuracy, whereas B-Splines allow rigorous error control but have low scalability due to their tensorial nature.These are the contrasting properties reported by the authors.Source: §9 · p. 34
  • Hybridizing or coupling the representations is proposed as a perspective for combining their respective advantages.The conclusion presents this as a possible direction.Source: §9 · p. 34
Loading 2610.06294v1…