PaperlayerView exact revision on arXiv

Source-linked AI summary

The Finite Neuron Method and Convergence Analysis

Jinchao Xu

arXiv:2010.01458v1math.NAcs.NA

TL;DR

Constructing adequately approximating H^m-conforming finite element spaces for general domains and partitions remains an open problem. The paper introduces the finite neuron method, using neural networks to build conforming piecewise polynomials and analyze their approximation for high-order elliptic problems. Its practical use is limited by the difficulty of solving the associated nonlinear, non-convex optimization problem.

  • Problem

    For general domains and partitions, constructing adequately approximating H^m-conforming finite element spaces remains an open problem.

    Source: §1.2 · p. 3

  • Method

    The finite neuron method uses neural networks to construct H^m-conforming piecewise polynomials for high-order elliptic boundary value problems.

    Source: §6 · p. 27

  • Results

    The paper reports good approximation properties for its H^m-conforming piecewise-polynomial function class.

    Source: §6 · p. 27

  • Takeaways & Limitations

    The method's underlying grid is determined by neuron hyperplanes rather than given a priori, and the paper notes this avoids an issue of approximating curved domain boundaries with a finite element grid.

    Source: §1.3 · p. 5

    The discrete solution requires solving a highly nonlinear, non-convex optimization problem, and more refined algorithms are needed for high-accuracy solutions.

    Source: §6.1 · p. 27

Abstract

from arXiv · show

We study a family of $H^m$-conforming piecewise polynomials based on artificial neural network, named as the finite neuron method (FNM), for numerical solution of $2m$-th order partial differential equations in $\mathbb{R}^d$ for any $m,d \geq 1$ and then provide convergence analysis for this method. Given a general domain $\Omega\subset\mathbb R^d$ and a partition $\mathcal T_h$ of $\Omega$, it is still an open problem in general how to construct conforming finite element subspace of $H^m(\Omega)$ that have adequate approximation properties. By using techniques from artificial neural networks, we construct a family of $H^m$-conforming set of functions consisting of piecewise polynomials of degree $k$ for any $k\ge m$ and we further obtain the error estimate when they are applied to solve elliptic boundary value problem of any order in any dimension. For example, the following error estimates between the exact solution $u$ and finite neuron approximation $u_N$ are obtained. $$ \|u-u_N\|_{H^m(\Omega)}=\mathcal O(N^{-{1\over 2}-{1\over d}}). $$ Discussions will also be given on the difference and relationship between the finite neuron method and finite element methods (FEM). For example, for finite neuron method, the underlying finite element grids are not given a priori and the discrete solution can only be obtained by solving a non-linear and non-convex optimization problem. Despite of many desirable theoretical properties of the finite neuron method analyzed in the paper, its practical value is a subject of further investigation since the aforementioned underlying non-linear and non-convex optimization problem can be expensive and challenging to solve. For completeness and also convenience to readers, some basic known results and their proofs are also included in this manuscript.

1 Introduction

The paper studies numerical methods for high-order elliptic boundary value problems and develops the finite neuron method (FNM), a neural-network-based class of conforming piecewise polynomials. It targets the open challenge of constructing adequately approximating H^m-conforming finite element spaces in general settings.

1.1 Model problem

  • The model is a 2m-th order elliptic boundary value problem in a bounded domain, with assumptions on the leading and zeroth-order coefficients.The paper considers m ≥ 1 and d ≥ 1.Source: §1.1 · p. 2

1.2 A brief overview of existing methods

  • For m ≥ 2 and d ≥ 2, constructing H^m-conforming finite element spaces is complicated, and their general construction in any dimension remains open.Existing constructions can require high polynomial degrees or special grid structures.Source: §1.2 · pp. 3–4

1.3 Objectives

  • The paper aims to connect neural-network methods with finite elements, combining tools from both areas to develop theoretical insights and algorithmic approaches.The method's implicit grid is determined by neuron hyperplanes and the activation function.Source: §1.3 · p. 5
  • The FNM uses neural networks to construct H^m-conforming piecewise polynomials for high-order elliptic problems, extending earlier work on ReLU networks and H^1-conforming linear finite elements.Its underlying grid is implicit rather than specified in advance, and solving for the discrete solution entails nonlinear, non-convex optimization.Source: §1.3 · p. 5

2 Preliminaries

The preliminaries introduce Monte Carlo and stratified sampling identities used in approximation analysis, then develop Barron spectral-space tools for neural-network approximation in Sobolev norms.

2.1 Monte-Carlo and stratified sampling techniques

  • The sampling preliminaries establish identities for Monte Carlo and stratified sampling, with stratification decomposing the approximation error across subsets of the sampling domain.The paper notes that these identities underpin the two sampling approaches, while its presentation avoids random-sampling concepts.Source: §2.1 · pp. 7–8

2.2 Barron spectral space

  • A Fourier representation expresses a function through cosine features, providing a route to neural-network approximation through sampling.The cosine feature is g(x, ω) = cos(ω · x + b(ω)), with ρ(ω) = |û(ω)|.Source: §2.2 · pp. 10–11
  • The section defines a Barron spectral norm using extensions of a function from Ω to R^d and relates Barron spectral and Sobolev norms.It also presents an H^m approximation perspective and cites prior estimates for the spectral-space results.Source: §2.2 · pp. 11–12

3 Finite neuron functions and approximation properties

Section 3 defines finite neuron functions and develops approximation estimates using B-spline and ReLU^k activations. The two approaches differ in their assumptions and approximation properties.

3 Finite neuron functions and approximation properties

  • Finite neuron functions are neural-network functions that form piecewise polynomials of degree k on grids bounded by neuron hyperplanes.The network function class is not a linear vector space.Source: §3 · p. 13

3.1 B-spline as activation functions

  • For 0≤m≤k, Theorems 3.3–3.4 state existence of ωᵢ∈Rᵈ and bᵢ∈R, with Theorem 3.4 requiring positive ε; Theorem 3.5 bounds ∥ω̄ᵢ∥≤1 and |bᵢ|≤T+k+1.The bₖ basis functions are locally supported.Source: §3.1 · pp. 14–16
  • The approximation analysis also extends to compactly supported activation functions in W^{m+1,∞}(R).The stated extension assumes compact support.Source: §3.1 · p. 17

3.2 [ReLU]k as activation functions

  • For ReLU^k, the paper generalizes a positive-part-power representation of complex exponentials and derives approximation estimates for functions in B^{k+1}(Ω).The comparison notes that some ReLU^k estimates can be asymptotically better, but require more regularity than the B-spline results.Source: §3.2 · pp. 17, 20, 21
  • Asymptotically, results for B-spline activation also hold for ReLU^k, while some ReLU^k estimates require stronger regularity assumptions.For example, Theorem 3.3 requires u∈B^m, whereas Theorem 3.9 requires u∈B^{k+1}, even for m=0.Source: §3.2 · p. 21

4 Deep finite neuron functions, adaptivity and spectral accuracy

Section 4 studies deep finite neuron functions, their approximation properties, and their relation to finite element functions. The results include polynomial reproduction, spectral approximation, and representation of linear finite element functions.

4.1 Deep finite neuron functions

  • Deep finite neuron functions use layered networks and can form piecewise polynomials of degree k^2 on grids bounded by quadratic polynomial level sets.The section studies these functions' adaptive and spectral accuracy properties.Source: §4.1 · p. 21, §4 · p. 21

4.2 Reproduction of polynomials and spectral accuracy

  • ReLU^k deep networks reproduce polynomials of degree k, and the paper says the resulting estimate may provide spectral approximation accuracy.Theorem 4.2 gives the estimate underlying this conclusion.Source: §4.2 · p. 22

4.3 Reproduction of linear finite element functions and adaptivity

  • On specified simplicial grids, any linear finite element function has a ReLU-DNN representation with O(d) hidden layers, O(κdN) neurons, and O(dκdN) nonzero parameters.The grid condition is that any union of simplexes sharing a vertex is convex; κ depends on shape regularity.Source: §4.3 · p. 22
  • Because deep finite neuron functions can reproduce linear finite element functions, the paper concludes they can be at least as adaptive as finite element methods.This conclusion is stated in connection with the adaptive feature and capability of finite element methods.Source: §4.3 · p. 22

5 The finite neuron method for boundary value problems

Section 5 formulates elliptic boundary-value problems of order 2m and analyzes finite-neuron approximations under different boundary conditions. It establishes convergence results and supporting regularity estimates.

5.1 Elliptic boundary value problems of order 2m

  • For each k = 0, 1, ..., m−1, Lemma 5.1 gives a bounded linear differential operator Bk^N : H2m(Ω) → L2(∂Ω) of order 2m−k−1; it states that a following identity holds.The lemma can be proved by induction with respect to m. For a proof of a similar identity, the paper refers to (Chapter 2) and.Source: §5.1 · p. 23
  • For f ∈ H^s(Ω), the solution has H^(2m+s)(Ω) regularity when s ≥ −m, with a further estimate stated for any ϵ > 0.These regularity results support the error analysis for the boundary-value problems.Source: §5.1 · p. 25

5.2 The finite neuron method for (1.1) and error estimates

  • For deep-neural-network functions, the solution to (5.28) exists but may not be unique.The analysis also relates formulations with different boundary conditions through stated equivalences and identities.Source: §5.1 · p. 24, §5.2 · p. 26
  • Theorem 5.11 analyzes the finite-neuron solution with penalty parameter δ ∼ N^-1/2-1/d, and the paper identifies convergence analysis for (5.28) and (5.31) as new.The supplied theorem passage gives the parameter scaling but not the theorem's resulting error bound.Source: §5.2 · p. 27

6 Summary and discussions

The paper presents the finite neuron method as a construction of H^m-conforming piecewise polynomials with good approximation properties. It also identifies the optimization required to obtain discrete solutions as a major practical challenge.

6 Summary and discussions

  • The ReLU^k-based finite neuron method constructs H^m-conforming piecewise polynomials and is proved to have good approximation properties.The paper considers elliptic problems of order 2m in any dimension.Source: §6 · p. 27

6.1 Solution of the non-convex optimization problem

  • The method’s practical efficiency remains unsettled because its highly nonlinear, non-convex optimization problem is hard to solve, and more refined algorithms are needed to attain the proved convergence order.The paper discusses several training approaches but says higher-accuracy algorithms still need development.Source: §6.1 · p. 27, §6 · p. 27

6.2 Competitions between locality and global smoothness

The discussion connects the difficulty of constructing H^m-conforming finite elements with the tension between local degrees of freedom and global smoothness. The finite-neuron construction relaxes locality and uses a global construction without an initial grid.

6.2 Competitions between locality and global smoothness

  • Classic finite elements must glue local degrees of freedom into globally H^m-smooth functions, while a global construction without an initial grid makes smooth piecewise-polynomial functions easier to construct.The paper notes that finite-neuron functions still have an implicitly defined grid and proposes exploring methods between classic finite elements and the finite neuron method.Source: §6.2 · p. 28

6.3 Piecewise Pm for Hm(Ω): from finite element to finite neuron method

The finite neuron method relaxes reliance on an a priori grid to construct H^m-conforming piecewise polynomials of degree m. The paper contrasts this with classic finite-element challenges and H^m-nonconforming constructions.

6.3 Piecewise Pm for Hm(Ω): from finite element to finite neuron method

  • Relaxing the a priori grid constraint makes degree-m H^m-conforming piecewise-polynomial construction straightforward in the finite neuron setting.The paper describes the method as mesh-less or vertex-less despite a hidden grid, and raises the prospect of an intermediate approach combining advantages of both methods.Source: §6.3 · p. 28

6.4 Adaptivity and spectral accuracy

The finite neuron method may combine local adaptivity with global spectral accuracy, but these capabilities are tied to different results and may come at the cost of nonlinear, non-convex optimization.

6.4 Adaptivity and spectral accuracy

  • The method is expected to combine finite-element-like local adaptivity with spectral-like global accuracy, although the paper presents these features through separate results.Deep FNM can recover locally adaptive finite element spaces for m = 1; Theorem 4.2 illustrates the spectral feature.Source: §6.4 · p. 28
  • For m = 1, the deep finite neuron method can recover locally adaptive finite element spaces.Source: §6.4 · p. 28
  • Theorem 4.2 illustrates the method’s spectral feature, while its broader adaptive behavior across scales remains conceivable rather than established.The passage describes adaptivity at multiple scales as a possibility.Source: §6.4 · p. 28
  • The potentially highly adaptive features may require solving a nonlinear, non-convex optimization problem.Source: §6.4 · p. 28

6.5 Comparison with PINN

FNM and PINN differ in their objective and scope: PINN is broadly applicable through PDE-residual minimization, whereas FNM relies on special physical principles and naturally enforces Neumann conditions without penalization.

6.5 Comparison with PINN

  • PINN minimizes PDE-residual norms with boundary-condition and other penalizations, while FNM uses minimization grounded in special physical laws.Source: §6.5 · p. 29
  • PINN is described as directly applicable to a wide range of problems, whereas FNM applies only to problems admitting principles such as energy minimization or least action.Source: §6.5 · p. 29
  • FNM naturally enforces Neumann boundary conditions through its underlying minimization problem, without the penalization used by PINN.Source: §6.5 · p. 29

6.6 On the sharpness of the error estimates

The paper’s sharper asymptotic error estimates require greater solution regularity, and the best stated rate does not improve with polynomial degree; the Dirichlet estimate is also weaker than the Neumann estimate.

6.6 On the sharpness of the error estimates

  • The best stated asymptotic estimate may remain non-optimal even for sufficiently regular solutions, and its rate does not improve as polynomial degree k increases.Obtaining better asymptotic estimates than (3.69) remains for further investigation.Source: §6.6 · p. 29
  • The Dirichlet error estimate (5.39) is weaker than the Neumann estimate (5.30), a loss the passage contrasts with Nitsche’s trick in FEM.The authors call the Dirichlet estimate undesirable and possibly non-optimal.Source: §6.6 · p. 29

6.7 Neural splines in multi-dimensions

The section identifies multidimensional neural splines as a possible extension of spline functions, noting that conventional spline generalization can be non-trivial, especially on curved domains.

6.7 Neural splines in multi-dimensions

  • Generalizing the spline functions described in § 3.1 to multiple dimensions is non-trivial, especially when the domain has a curved boundary.Source: §6.7 · p. 29
  • The multidimensional generalization of nN1(bk) is described as straightforward and as yielding a nonlinear space with very good approximation properties.Source: §6.7 · p. 29
  • A neural-network extension of B-splines that is locally polynomial and globally smooth is proposed as a potential direction for CAD and isogeometric analysis.Source: §6.7 · p. 29
Loading 2010.01458v1…