Fully Deterministic Shallow Neural Networks with Localized Activations. Part I: Construction and Optimal Sobolev Approximation

Ran Bi; Weibing Deng

Source-linked AI summary

Fully Deterministic Shallow Neural Networks with Localized Activations. Part I: Construction and Optimal Sobolev Approximation

Ran Bi, Weibing Deng

arXiv:2610.09338v1Oct 7, 2026math.NAcs.NA

TL;DR

The paper targets reproducible feature spaces with quantitative approximation estimates, motivated by PDE residual least-squares and variational methods. It prescribes shallow networks with fixed Gaussian and tanh-difference activations, and proves an optimal Sobolev approximation rate for their feature spaces.

Problem
PDE residual least-squares and variational methods motivate reproducible feature spaces with quantitative approximation estimates.
Source: §1 · p. 1
Method
Explicit deterministic rules prescribe fixed Gaussian and tanh-difference network parameters using spherical cubature directions and Gauss-Legendre or equispaced midpoint offsets.
Source: §1 · p. 1, §2.2 · p. 4
Results
The approximation error is bounded by C N^(-(s-m)/d) in H^m for 0 ≤ m < s, with the rate optimal among target-independent linear spaces.
Source: §1 · p. 2
Takeaways & Limitations
The prescribed networks attain the optimal worst-case Sobolev approximation order among linear spaces chosen independently of the target.
Source: §1 · p. 2
The theoretical node counts are conservative, and adaptive target-dependent spaces would require error estimates and control of offset truncation under refinement.
Source: §6 · p. 28

Abstract

from arXiv · show

We construct shallow neural networks with fixed Gaussian and tanh-difference activations on bounded Lipschitz domains $\Omega\subset\mathbb{R}^d$, $d\ge2$. For each $s>0$, explicit deterministic rules prescribe all hidden parameters independently of the target $u\in H^s(\Omega)$. At frequency resolution $M$, the $N_M\asymp M^d$ prescribed features span a linear space $V_M$. For every sufficiently large integer $N$, we choose $M\asymp N^{1/d}$ with $N_M\le N$ and construct a bounded linear operator $A_N:H^s(\Omega)\to V_M$ such that \[ \|u-A_Nu\|_{H^m(\Omega)} \le C N^{-(s-m)/d}\|u\|_{H^s(\Omega)} \] for every $u\in H^s(\Omega)$ and every integer $0\le m<s$, with $C$ independent of $u$, $N$, and $m$. Sobolev width lower bounds show that this rate is optimal among linear spaces of dimension at most $N$ chosen independently of the target. We use explicit spherical cubature nodes for the directions and Gauss-Legendre nodes or equispaced midpoints for the offsets. We show that the exterior offset margin can shrink to zero as $M\to\infty$ while preserving the approximation rate. A margin beyond $\sup_{x\in\Omega}|x|$ of order $M^{-1}\log M$ for the Gaussian and $M^{-1}(\log M)^2$ for the tanh difference suffices to truncate the ridge integral and remove exterior midpoint nodes. Numerical experiments in two and three dimensions examine $L^2$ and $H^1$ approximation.

1. Introduction

The paper gives fully deterministic shallow networks with fixed Gaussian and tanh-difference activations whose prescribed feature spaces achieve optimal Sobolev approximation order. Explicit direction and offset rules support a shrinking exterior margin while preserving that order.

1. Introduction

The networks achieve an H^m error of order N^(-(s-m)/d) for 0 ≤ m < s, and Sobolev width bounds establish optimality among target-independent linear spaces.The operator takes values in the prescribed feature space.Source: §1 · p. 2
Unlike Fu and Wang’s [17] fixed enlarged offset interval, the construction uses explicit product cubature and a shrinking offset interval.The approach combines Fourier representation with parameter quadrature.Source: §1 · p. 2
Explicit product cubature prescribes about M^(d-1) directions, and Gauss-Legendre or equispaced midpoint offsets yield about M^d features independently of the target.No quasi-uniformity assumption is needed.Source: §1 · p. 3
A single linear operator attains the stated approximation order simultaneously for every integer derivative order 0 ≤ m < s.For real targets, its Fourier formula gives real output coefficients that depend linearly on the target.Source: §1 · pp. 2–3
The sufficient exterior margin shrinks as log(M)/M for Gaussian activations and log^2(M)/M for tanh differences while preserving the approximation rate.The margin allows truncation of the offset integral and removal of exterior midpoint nodes.Source: §1 · pp. 1, 3
Experiments in two and three dimensions examine L2 and full H1 approximation as feature count and target spectral decay vary.The construction is motivated by PDE methods that need reproducible feature spaces with quantitative approximation estimates.Source: §1 · pp. 1, 3

2. Network construction and approximation theorem

The paper prescribes localized Gaussian or tanh-difference features independently of the target and constructs bounded linear approximants with optimal Sobolev rates. Explicit direction and offset rules produce spaces whose approximation order is optimal among target-independent linear spaces of comparable dimension.

2.1. Notation and preliminaries.

The Sobolev spaces are defined by restriction from R^d, using the quotient norm at noninteger orders; a bounded extension operator preserves real-valued functions.The extension is independent of the target function.Source: §2.1 · pp. 3–4

2.2. Deterministic network construction.

The features use a common scale ε_M and localized Gaussian or tanh-difference activations; the exterior offset margin controls truncation error and removal of exterior midpoint nodes.Each ridge feature localizes near its hyperplane, with width determined by ε_M.Source: §2.2 · p. 4
At each resolution, explicit spherical cubature directions and Gauss-Legendre or midpoint offsets prescribe feature parameters independently of the target and derivative order.The resulting feature space has dimension at most N_M = J_M K_M.Source: §2.2 · pp. 4–5

2.3. Approximation theorem.

Theorem 2.1 gives a bounded linear operator into the prescribed feature space, with an H^m error bound of order M^(m-s) for every integer 0 ≤ m ≤ m*.The constants are independent of M, the target, and m.Source: §2.3 · p. 5
For sufficiently large N, choosing M ≍ N^(1/d) yields an H^m error of order N^(-(s-m)/d), optimal among target-independent linear spaces of dimension at most N.Classical Sobolev width bounds establish this optimality.Source: §2.3 · pp. 5–6

3. Ridge representation and offset truncation

The paper converts a frequency-truncated extension into a ridge representation, then controls frequency, coefficient, and offset errors to obtain continuous approximants. For tanh difference, spectral exclusions regularize the inverse multiplier near its nonzero Fourier zeros.

3.1. Frequency cutoff and spectral exclusions.

Frequency truncation gives H^m error of order M^(m-s); for tanh difference, neighborhoods of nonzero Fourier zeros are excluded to control the inverse multiplier.The Gaussian transform has no real zeros, while the tanh-difference transform vanishes at nonzero multiples of π.Source: §3.1 · pp. 6–7

3.2. Ridge representation with nonnegative offsets.

A regularized inverse multiplier yields a ridge representation with nonnegative offsets, whose coefficient and feature are invariant under (ω, b) ↦ (−ω, −b).The representation remains valid after any fixed number of spatial derivatives on a bounded domain.Source: §3.2 · pp. 9–10

3.3. Coefficient estimates.

The ridge coefficient is uniformly bounded over directions and real offsets, and is entire in the offset variable.Its complex-offset derivatives satisfy bounds stated in Lemma 3.3.Source: §3.3 · p. 10

3.4. Continuous offset tail estimates.

Truncating offsets to [0, B_M] gives an H^m error of order M^(m-s) when B = B_M.The continuous offset-tail estimates apply simultaneously for every integer 0 ≤ m ≤ m*; the Gaussian condition allows a_2 = 1.Source: §3.4 · pp. 12–13

4. Discretization of the ridge representation

Section 4 discretizes the truncated ridge representation using positive spherical cubature with Gauss–Legendre offsets, or a midpoint construction based on angular and trigonometric approximation. It controls the exterior-offset tail and combines local, frequency, and offset-tail estimates to establish the stated Sobolev approximation rate for both prescribed offset rules and a bounded linear network map, while also proving exactness properties of midpoint quadrature on an auxiliary period.

4.1. Polynomial approximation of the integrand.

The ridge integrand admits an H^m-valued parameter-polynomial approximation with angular degree at most qM, uniform offset error, and linear dependence on u.For each fixed offset, the approximant is an angular polynomial; the activation polynomial used in its construction is independent of u.Source: §4.1 · pp. 14–15

4.2. Spherical cubature and Gauss-Legendre quadrature.

An explicit positive antipodal spherical cubature rule is exact on Pq(S^{d−1}); paired antipodal nodes have equal weights, and GL offset nodes and weights are positive.The spherical rule is combined with Gauss–Legendre offset quadrature to discretize the truncated representation.Source: §4.2 · p. 16

4.3. Polynomial and trigonometric approximation.

For midpoint offsets, symmetrization and Fourier truncation yield an angular–trigonometric approximant that is even in b, antipodally invariant in ω, and linear in u.For j ≠ 0, the Fourier coefficients satisfy sup_ω ||bY_M,L(j, ω)||_X ≤ M e^(−π|j|η/L) + CB/j^2.Source: §4.3 · pp. 17, 19

4.4. Discrete offset tail estimates.

The exterior-offset contribution to the discrete tail is bounded in Hm(Ω) by C M^(m−s)||u||Hs(Ω).Source: §4.4 · p. 20

4.5. Midpoint quadrature on the auxiliary period.

The auxiliary-period midpoint tensor rule integrates PM,L,m exactly for 0≤m≤m∗, and positivity bounds its error by twice its total mass times the uniform approximation error.The rule has total mass L|Sd−1|.Source: §4.5 · p. 22

4.6. Proof of Theorem 2.1.

For Gauss-Legendre offsets, the three Hm(Ω) error contributions sum to at most C M^(m−s)||u||Hs(Ω) for integers 0≤m≤m∗.Source: §4.6 · p. 22
For sufficiently large N, choosing M = floor((N/C1)^(1/d)) gives NM≤N and ||u−vM[u]||Hm(Ω)≤C N^−(s−m)/d ||u||Hs(Ω) for both offset rules, using the same bounded linear map across m.The nodes, weights, and approximation map are independent of m.Source: §4.6 · p. 23

5. Numerical experiments

The experiments test prescribed feature spaces on squares and cubes using Fourier targets, least-squares coefficients, and L2 or H1 errors. Across the tested resolutions, errors decrease with network width and fitted orders rise with target spectral decay.

5.1. Experimental setup.

The targets are finite Fourier sums with random Gaussian coefficients, so s controls their spectral decay rather than exact Sobolev regularity.Three targets with distinct seeds are generated for each s = 2, 4, 6.Source: §5.1 · pp. 23–24
In two dimensions the direction and offset counts are J = 4M and K = 2M; in three dimensions they are J = 8M 2 and K = 3M.The corresponding feature counts are N = 8M 2 and 24M 3.Source: §5.1 · p. 24
The experiments use weighted least squares to fit output coefficients, with function values for L2 and function values plus all first derivatives for H1.Errors are evaluated on a separate tensor-product Gauss-Legendre grid; fitted orders use a finite range of resolutions.Source: §5.1 · p. 24

5.2. Two-dimensional results.

For Gaussian L2 approximation in two dimensions, fitted orders range from 1.150 to 1.157, 2.099 to 2.109, and 2.967 to 3.251 for s = 2, 4, 6, respectively.The GL and equispaced offset rules give similar orders for s = 2, 4; for s = 6, their orders lie on opposite sides of the reference value 3.Source: Figure 1 · p. 25, §5.2 · p. 25
For two-dimensional H1 approximation, fitted orders range from 0.615 to 0.669, 1.566 to 1.658, and 2.508 to 2.843 for s = 2, 4, 6, respectively.Figure 2 compares Gaussian and tanh-difference activations with equispaced offsets; reference orders are 1/2, 3/2, and 5/2.Source: §5.2 · p. 25, Figure 2 · p. 26

5.3. Three-dimensional results.

Across all eight experiments, median error decreases with N and fitted order increases with spectral decay parameter s over the tested resolutions.The experiments use prescribed feature spaces with least-squares coefficients.Source: §5.1 · p. 24, §5.3 · p. 26
For tanh-difference L2 approximation in three dimensions, fitted orders range from 0.774 to 0.801, 1.493 to 1.512, and 2.171 to 2.178 for s = 2, 4, 6, respectively.GL and equispaced offsets are compared; reference orders are 2/3, 4/3, and 2.Source: §5.3 · p. 25, Figure 3 · p. 27
For three-dimensional H1 approximation, fitted orders range from 0.433 to 0.443, 1.158 to 1.174, and 1.845 to 1.862 for s = 2, 4, 6, respectively.The reference orders are 1/3, 1, and 5/3; Figure 4 compares the activations using GL offsets.Source: §5.3 · pp. 25–26, Figure 4 · p. 27

6. Concluding remarks

The paper presents deterministic prescribed networks with optimal Sobolev approximation rates and reports numerical tests on squares and cubes. It identifies conservative theoretical node-count constants and leaves stable coefficient recovery for further study.

6. Concluding remarks

Positive spherical cubature and Gauss-Legendre or equispaced offset nodes prescribe approximately M^d features, while Fourier integrals define output coefficients.The hidden weights, biases, and common scale are fixed independently of the target.Source: §6 · p. 27
The construction achieves the optimal H^m(Ω) rate for integers 0 ≤ m < s without logarithmic loss, matching worst-case optimality for fixed linear spaces.The proof separately estimates the activation tails and controls real Fourier zeros of the tanh difference through prescribed spectral exclusions.Source: §6 · pp. 27–28
The experiments use the prescribed feature families with least-squares output coefficients, and median errors fall as network width grows while fitted orders rise with target spectral decay.The experiments cover squares and cubes, with node counts chosen separately from sufficient theoretical bounds.Source: §6 · p. 28
The theoretical node-count constants are conservative, and sharper approximation and quadrature estimates could reduce finite-resolution node requirements.Relating continuous Sobolev Gram-matrix condition estimates to sampled least-squares systems and developing preconditioners require further work.Source: §6 · p. 28

Appendix A. Estimates for weighted multiplier integrals

The appendix bounds weighted multiplier integrals by splitting frequency ranges and treating root cells with prescribed exclusions. The estimates include critical exponents and yield the stated Gaussian and tanh-difference bounds.

Appendix A. Estimates for weighted multiplier integrals

Root-cell integration with prescribed exclusions supports the estimates for either sign of p, including p = −1; the argument also covers a frequency band ending inside a cell.The text notes that integrating over root cells gives a δ−1 factor, whereas a supremum bound for the squared reciprocal multiplier would give δ−2 and weaken the estimate.Source: Appendix A · pp. 28–29
For the Gaussian estimates, the argument separates low- and high-frequency contributions and uses e^2TM = (M + 2)^2 to control the high-frequency bound and, for p ≥ −1, the low-frequency contribution.For p < −1, the low-frequency contribution is bounded independently of the resolution.Source: Appendix A · pp. 28–29
The Gaussian estimates in (3.22) follow by substituting a = 2d − 2 and a = d − 1 into the derived bounds.Source: Appendix A · pp. 28–29
The tanh-difference estimates use the two specified values of a in (3.22), with the critical cases identified for ΓM and the associated multiplier bound.Source: Appendix A · pp. 28, 30

Appendix B. Spherical kernel estimates

The appendix estimates spherical kernel masses by exploiting rotational invariance and splitting angular integration into near and far caps. It also controls the inner-ball contribution before applying the angular operator estimate.

Appendix B. Spherical kernel estimates

Rotational invariance makes each kernel mass independent of θ, allowing the kernel integral in θ to be expressed through that same mass.Weighted Cauchy-Schwarz is then applied.Source: Appendix B · p. 30
In the outer region, the cap coordinate u = 1 − ω · θ separates the near-cap estimate from the far-cap estimate, with the latter using zω ≥ A + r/ε.On the near cap, the surface-density bound is C u^ν−1; on the far cap, the density remains integrable for d = 2.Source: Appendix B · p. 30
The Gaussian near-cap bound follows by rescaling v = ru/ε and controlling the resulting integral for both 0 ≤ A ≤ 1 and A ≥ 1.The far-cap Gaussian decay absorbs powers of ε−1 and 1 + A.Source: Appendix B · p. 30
For the tanh-difference kernel, the near cap gives an integral of v^ν−1e−2v, while the far cap gains an e−2r/ε factor.The inner-ball kernel masses are bounded separately, after which the angular operator estimate yields (3.24).Source: Appendix B · pp. 30–31
The cap estimates use an incomplete-gamma bound for λ ≥ 1 and integrability for 0 ≤ λ ≤ 1.The resulting estimates are squared and integrated in r.Source: Appendix B · p. 31

Appendix C. Polynomial approximation with derivatives

The proof builds a polynomial approximation by truncating a Chebyshev expansion, then extends the approximation to derivatives by integrating its highest-derivative error.

Appendix C. Polynomial approximation with derivatives

The proof truncates the Chebyshev expansion of g^(m)(T*t) at degree ℓ and rescales the polynomial to [−T*, T*], using a truncation estimate with ρ = e^η.The resulting polynomial is denoted rℓ.Source: Appendix C · p. 31
For m = 0, the proof sets pℓ = rℓ; for m ≥ 1, it defines a polynomial with real coefficients.The passage introducing the m ≥ 1 definition does not specify its formula.Source: Appendix C · p. 31
The construction matches derivatives at zero for 0 ≤ j < m, integrates the highest-derivative error m − j times, and yields a degree-at-most-ℓ + m polynomial satisfying (4.2) on both sides of the origin.The matching condition is ℓ(0) = g^(j)(0).Source: Appendix C · p. 31