Neural Network Approximation

Motivation

• Neural networks achieve state-of-the-art performance across diverse learning problems, yet solid mathematical analysis explaining these empirical successes remains lacking.The introduction motivates approximation theory as a route toward practical improvements and a priori performance guarantees.Source: §1 · p. 2

Purpose and scope

• The article compares neural-network approximation with classical methods using approximation rates, n–widths, metric entropy, and approximation classes, focusing on nonlinear manifold approximation and especially ReLU networks.ReLU networks are emphasized because they are widely used in numerical settings and are comparatively amenable to analysis.Source: §1 · p. 3

ReLU structure

• ReLU outputs are continuous piecewise linear functions whose allowable partitions and correlations become richer with depth, motivating the paper’s emphasis on deep networks.For a fixed parameter count, the output family depends on the network’s architecture and depth.Source: §1 · p. 3

Key takeaways

• For fixed n, ReLU networks form a rich parametric family of continuous piecewise linear functions with space-filling properties in the error-measuring Banach space.These properties are presented as central to the paper’s account of neural-network approximation.Source: §1 · p. 3
• Space filling may approximate larger function classes with relatively few parameters than classical methods, but it reduces parameter-selection stability and weakens a priori guarantees and uncertainty quantification.The paper frames this as a trade-off between numerical efficiency and stability requiring comprehensive study.Source: §1 · p. 4

Open questions

• Open problems include matching ReLU approximation-rate bounds, identifying function classes that benefit from neural networks, and quantifying how stability constraints affect parameter selection and performance.The exposition presents these questions as unresolved even for ReLU networks.Source: §1 · p. 4

2. What is a Neural Network?

• A feed-forward neural network is a directed acyclic graph with input, output, and hidden nodes, where activation functions, biases, and edge weights define neuron computations.For fixed architecture, varying the trainable parameters produces a family of output functions mapping Ω⊂R^d into R^d′.Source: §2 · pp. 4–5

2.1. Fully Connected Networks

• Fully connected networks arrange inputs, hidden units, and outputs in layers, connect each layer only to the next, and compute hidden-layer transformations with weight matrices and biases.All hidden nodes share one activation σ, while output activations are the identity; layer widths determine the architecture.Source: §2.1 · pp. 5–6

2.2. The Set ΥW,L(σ; d, d′)

• For fixed width W and depth L, the output set ΥW,L is a loosely defined parametric manifold obtained by mapping trainable weights and biases to output functions.The set is closed under output-layer parameter addition but need not be closed under function addition, so it is not generally a linear space.Source: §2.2 · pp. 7–8

2.3. Fundamental Operations with Neural Networks

• Parallelization increases width to implement linear combinations of network outputs, while concatenation combines networks sequentially to implement compositions and related transformations.These constructions support addition, composition, and shifted dilates within appropriately chosen width and depth classes.Source: §2.3 · pp. 9–10

2.4. One Layer Neural Networks

• One-hidden-layer networks are dense in continuous functions on compact domains for continuous sigmoidal activations and ReLU, under the stated sufficient condition.Polynomial activations fail this density property because their outputs lie in a fixed finite-dimensional polynomial space.Source: §2.4 · p. 11

• Every ReLU-network output is a continuous piecewise linear function subordinate to a finite partition into convex polytopes.Each cell is a convex polytope, and the output is affine on every cell of the partition.Source: §3 · p. 12
• The class of CPwL functions subordinate to partitions with at most n cells is nonlinear, so sums of functions from different partitions typically leave the class.The paper uses this class for comparison rather than studying it directly.Source: §3 · p. 12
• A ReLU network with d inputs, one output, and m hidden neurons yields a CPwL output subordinate to a partition with at most 3^m cells.The partition arises from neuron activation patterns, whose regions are convex polytopes.Source: §3 · pp. 12–13
• For a fixed architecture, almost every parameter choice produces an output subordinate to a partition with at most 2^m cells.The exceptional parameter set has measure zero; activation-pattern cells containing zero pre-activations are generically empty or lower-dimensional.Source: §3 · p. 13

3.1. Univariate ReLU Networks

• Univariate ReLU outputs are continuous piecewise-linear functions on partitions of R into intervals, commonly called continuous linear free-knot splines.The univariate specialization sets both input and output dimensions to one.Source: §3.1 · pp. 13–14

3.1.1. Single Layer Univariate ReLU Networks

• ReLU networks generate hat functions that peak at one knot, vanish at two outer knots and outside their interval, and are piecewise linear between these points.The hat function H(0,1/2,1) is realizable by a width-three univariate ReLU network.Source: §3.1.1 · p. 14

3.1.2. Deep Univariate ReLU Networks

• As depth increases, breakpoint characterization becomes more complicated: not every piecewise-linear function with the permitted large breakpoint count belongs to the network class.Linear dependencies among the pieces restrict the realizable functions, and the exact large-depth characterization remains unresolved.Source: §3.1.2 · pp. 15–16

3.1.1. Single Layer Univariate ReLU Networks

• For one-layer univariate networks, outputs have at most W breakpoints, contain all W-interval continuous linear free-knot splines, and are essentially equivalent in approximation power.The network class uses 3W + 1 parameters, compared with 2W for the corresponding spline class, while not containing every spline with W + 1 intervals.Source: §3.1.1 · p. 14, §3.1.2 · p. 15

3.1.2. Deep Univariate ReLU Networks

• Deep univariate ReLU networks can have at most (W + 1)^L breakpoints, and this exponential dependence on depth can be attained by iterated hat-function composition.Composing H(0,1/2,1) with itself L−1 times yields a sawtooth function with 2^L−1 teeth using width W = 2.Source: §3.1.2 · pp. 15–16

3.2.1. Multivariate ReLU Networks with one hidden layer

• One-hidden-layer networks partition R^d into convex polytopes from W hyperplanes, but cannot realize every CPwL function subordinate to that arrangement.They cannot represent nonzero compactly supported functions, and for a fixed arrangement their outputs lie in a linear space of dimension 2W + 1, whereas all subordinate CPwL functions generally have much larger dimension.Source: §3.2.1 · pp. 17–18
• A CPwL function subordinate to a W-hyperplane partition equals a one-hidden-layer ReLU output plus a globally affine function exactly when each hyperplane’s gradient jump is a scalar multiple of its unit normal.The criterion requires the jump across H_j to equal a_jw_j wherever H_j meets no other hyperplane.Source: §3.2.1 · p. 19

3.2.2. Deep Multivariate ReLU Networks

• Deep multivariate ReLU networks use bent hyperplane arrangements whose cell counts can grow exponentially with depth because composition increases the number of linear pieces.Their minimal convex-polytope partition is not simply the cells of a hyperplane arrangement, and the best cell-count bound is unknown; generic bounds improve from 3WL to 2WL.Source: §3.2.2 · pp. 19–21

3.2. Multivariate ReLU Networks

• Multivariate ReLU outputs are CPwL functions on convex-polytope partitions, with many more cells than parameters but constrained affine pieces.For one hidden layer, cell counts can be of order W^d; with depth, they can grow exponentially in L, while sufficiently large width and depth eventually represent every CPwL function on convex-polytopal partitions.Source: §3.2.2 · p. 21

3.3.1. Special networks and their set of outputs Υ

• Special networks reserve source channels to forward inputs and collation channels to aggregate computations, and on bounded domains their outputs remain in ΥW,L.They may omit ReLU at designated nodes; on arbitrary inputs, conversion to true ReLU networks increases width to W0+2d+2k.Source: §3.3.1 · pp. 21–22

3.3.2. Some important properties of deep ReLU networks

• Deep ReLU networks can add m outputs with width W+d+1 and depth L1+···+Lm, while retaining the resulting function in ΥW,L.The construction augments subnetworks with source and collation channels and concatenates them; all functions are viewed on [0,1]^d or a bounded rectangle.Source: §3.3.2 · p. 23
• The pointwise minima and maxima of z1(x), ..., zm(x) belong to ΥW,L(ReLU; d, 1), and applying ReLU to either output places it in ΥW,L+1(ReLU; d, 1).For scalar inputs, the corresponding minima and maxima belong to ΥW,L(ReLU; m, 1).Source: §3.3.2 · pp. 24–25
• The maximum of m network outputs can be implemented with width max{W1+···+Wm, 3·2^⌈log2 m⌉−1} and depth L0+⌈log2 m⌉.The same statement holds for max{S1, . . . , Sm}, with all Sj viewed on [0,1]^d or any bounded rectangle R ⊂ Rd.Source: §3.3.2 · pp. 25–26

3.3.3. General CPwL functions

• Every simplex tent function belongs to ΥW,L with W=3·2^⌈log2(d+1)⌉−1 and L=1+⌈log2(d+1)⌉.The tent function vanishes outside the simplex and equals 1 at its selected interior point, extending the representable class beyond one-layer CPwL outputs.Source: §3.3.3 · p. 27
• Any CPwL function on Rd belongs to ΥW′,L with L=⌈log2(d+1)⌉ and sufficiently large W′.The proof represents CPwL functions as linear combinations of piecewise-linear convex functions and uses min/max constructions plus parallelization.Source: §3.3.3 · p. 27
• General CPwL functions on rectangles belong to ReLU network classes with fixed width W = d + 2 and sufficiently large depth.The construction uses depth at most L = pd for a representation involving p component networks.Source: §3.3.3 · p. 28

3.3.4. Finite element spaces

• Finite-element spaces X(K) are contained in ReLU network classes with width W = 3(n + 1)^d 2^⌈log2 d∗⌉−1 and depth L = 1 + ⌈log2 d∗⌉.Here d∗ = (d + 1)!, and the resulting network uses O((n + 1)^d) parameters.Source: §3.3.4 · p. 29
• The ReLU family contains the chosen finite-element space and infinitely many spaces from alternative partitions, making it broader than FEM and beyond restricted adaptive partitioning.The paper states that these networks achieve provably better approximation rates for various Sobolev and Besov classes than FEM spaces, while requiring near-optimal parameter selection.Source: §3.3.4 · p. 30
• Although NNs may substantially improve approximation efficiency for elliptic problems, whether that advantage survives stable numerical implementation remains unclear.This contrasts the potential error-versus-parameter improvement with the unresolved stability of concrete implementations.Source: §3.3.4 · p. 30

• Deeper networks are generally preferred in practice because composition and minimum operations require increasing depth, whereas addition can be implemented through either width or depth.This motivates comparing width and depth under a fixed parameter budget.Source: §3.4 · p. 30
• For one-dimensional ReLU networks, a sufficiently large fixed width with depth L ≍ n/W_0^2 uses n(W_0,L) ≍ W_0^2L ≍ n parameters, comparable to Υ_n,1.The statement applies when the output functions are viewed on [0,1] or any finite interval [r,e].Source: §3.4 · p. 31
• In this special case, Υ_W0,L is richer than the comparable shallow class because it contains compositions, so depth beats width.Source: §3.4 · p. 31
• The open problem asks whether shallow ReLU networks are contained in fixed-width deep networks with comparable parameter counts, with constants depending at most on d.Without parameter-count control, the inclusion is positive because every such shallow output is a CPwL function and belongs to a sufficiently deep network of width d+2.Source: §3.4 · p. 31

Problem formulation

• The interpolation problem asks for the largest D* such that every data set of size D* can be interpolated by a fixed network class.For finite data {(x(i), yi)}, interpolation requires S(x(i)) = yi, and D* is expected to relate to the number of network parameters.Source: §3.5 · pp. 31–32

One-dimensional ReLU interpolation

• For one-dimensional width-W ReLU networks, any W + 1 ordered data points can be interpolated by infinitely many outputs, whereas alternating-sign data generally cannot be interpolated at W + 2 points.The infinite family arises from freely choosing breakpoints between consecutive data sites; Proposition 3.5 gives the alternating-sign obstruction.Source: §3.5.1 · pp. 32–33

Deep fixed-width ReLU networks

• A ReLU network of width 3 and depth D − 1 can interpolate arbitrary values at any D ordered points in [0, 1].The construction propagates the input, generates hinge terms across layers, and accumulates their weighted sums.Source: §3.5.2 · p. 34

Multidimensional interpolation

• For ReLU networks with input dimension d, interpolation capacity satisfies D*(W, L; ReLU, d) ≥ D*(W, L; ReLU, 1) by projecting data onto a direction with distinct coordinates.The resulting interpolant is a ridge function T(x) = S(v · x), though the construction is not locally adapted to the data.Source: §3.5.2 · pp. 34–35

Local bases and stability

• Locally supported dual bases yield bounded interpolation projections, and finite-element nodal bases in ReLU networks give dist(f, ΥW,L(ReLU; d, 1))C(Ω) ≤ 2 dist(f, X(K))C(Ω).For the Kuhn simplicial decomposition, the nodal projection has norm one, allowing neural-network approximation estimates to follow from finite-element estimates.Source: §3.5.2 · p. 35

3.6. VC dimension of ReLU outputs

• For one-hidden-layer ReLU outputs, VC dimension is W+1 in d=1, at most C0W log2 W for d≥2, at least c0W log2 W for d≥4, and at least W+1 for d=2,3.For d=2,3, the exact VC dimension remains unresolved because the stated upper and lower bounds do not coincide.Source: §3.6 · p. 37
• With fixed sufficiently large width W0 and variable depth L, the parameter count satisfies n(W0,L) ≍ W0^2L, and Theorem 3.9 provides dimension bounds with constants depending only on d.The supplied passage states the theorem and its parameter-count regime but does not include the displayed numerical bounds.Source: §3.6.2 · pp. 37–38

3.6.3. Bit extraction using ReLU networks

• Bit extraction shatters N=n^2 equally spaced points and interpolates prescribed bit data while maintaining |S(t)−S(ti)|≤1 on each interval.The construction is presented for d=1, with full d≥2 implementations cited to Yarotsky 2018 and Shen, Yang & Zhang 2019.Source: §3.6.3 · p. 38
• Although the network uses only Cn parameters, it handles N+1=n^2+1 data points, yielding the quadratic lower bound on VC dimension through encoded sign or bit sequences.For d=1, the argument uses width W0=11; the higher-dimensional case follows by the interpolation construction used in Proposition 3.7.Source: §3.6.3 · p. 38
• The explicit construction concatenates piecewise-linear subnetworks for index extraction, encoded values, bit decoding, and error correction into width 11 and depth 15n+2.It uses a continuous ReLU surrogate for quantization that agrees with the true bits on the required representations; the authors note the width-depth construction is probably not optimal.Source: §3.6.3 · pp. 40–41
• The construction modifies ˜S only on small complementary intervals, using upper and lower envelopes combined through min and max to obtain the corrected function S.The correction keeps ˜S unchanged outside those intervals and constrains the corrected values on them between 0 and 1.Source: §3.6.3 · p. 44
• The resulting ReLU network has width W=11 and depth L = 15n + 2 after concatenating the networks for ˜S, U, and ˆU and adding min-max layers.The component networks for U and ˆU each have depth 4n + 1, while the network for ˜S has depth 7n − 2.Source: §3.6.3 · pp. 44–45
• The construction relies on N = n^2 + 1 equally spaced univariate points, while analogous multidimensional interpolation requires points whose projections onto some vector equal the prescribed locations.The passage notes that related constructions establish interpolation on equally spaced tensor-product grids, but the admissible point patterns remain to be clarified.Source: §3.6.3 · p. 45
• The authors note that the construction cannot increase the number of interpolation points significantly because of the upper bound in (3.27).They also identify determining other point patterns supporting the construction as an open issue.Source: §3.6.3 · p. 45

4. Classical model classes: smoothness spaces

• Classical model classes are compact subsets of Banach spaces, commonly represented by unit balls of Lipschitz, Hölder, Sobolev, and Besov spaces on Ω = [0, 1]^d.The definitions extend to more general domains such as Lipschitz domains in R^d.Source: §4 · p. 45

4.1. Lp spaces

• Lp(Ω) spaces provide the underlying integrability framework, with norms for p ≥ 1, quasi-norms for 0 < p < 1, and continuous-function or essentially bounded variants at p = ∞.The section assumes familiarity with their standard properties.Source: §4.1 · p. 46

4.2. Sobolev spaces

• Sobolev spaces generalize C^r by replacing classical derivatives with weak derivatives D^αf required to belong to Lp for |α| = r.Their norms combine the derivative seminorm with the Lp norm.Source: §4.2 · p. 46

4.3. Besov spaces

• Besov spaces measure smoothness of arbitrary order s > 0 in Lp, with q providing a finer gradation through the decay of moduli of smoothness.They generalize Lipschitz and Hölder spaces and include these as special cases for suitable parameters.Source: §4.3 · pp. 46–47
• Besov embeddings relate smoothness, integrability, and compactness: spaces above s = d(1/τ − 1/p) embed compactly into Lp(Ω), summarized graphically in Figure 4.4.The embeddings are governed by inequalities in smoothness and integrability parameters, including the stated BE1–BE3 results.Source: §4.3 · p. 48, Figure 4.4 · p. 48

4.3.1. Atomic decompositions

• Atomic decompositions using tensor-product cardinal B-splines represent functions in Lp, Sobolev, and Besov spaces, and characterize Besov membership through coefficient conditions.The resulting coefficient quasi-norm is equivalent to the usual Besov norm and supports approximation theorems.Source: §4.3.1 · pp. 49–50

4.4. Interpolation of operators

• Interpolation spaces generated by the real method extend known approximation error bounds from classical pairs such as Lp and Besov/Sobolev spaces.The two stated extension principles transfer bounds to interpolated spaces, including linear combinations of approximants in the second principle.Source: §4.4 · pp. 50–51

5.1. Approximation of model classes

• For a model class K, performance is measured by worst-case class error, and faster error decay is meaningful only alongside the computability of the approximant.The class K encodes prior knowledge such as PDE regularity, bandlimits, or sparsity; the section focuses on worst-case rather than averaged error.Source: §5.1 · p. 54

5.2. Widths for measuring approximation error

• Kolmogorov widths quantify optimal approximation by n-dimensional linear spaces, while classical spaces such as polynomials, splines, spectral methods, and FEMs can achieve asymptotic optimality for standard classes.In general Banach spaces, finding stable projections and best approximations can itself be numerically difficult.Source: §5.2 · pp. 55–56

5.3. Nonlinear widths

• Neural-network approximation uses manifolds Σ_n = M_n(R^n), so evaluating it requires comparing both approximation error and the parameter-selection map a_n.The best error from Σ_n does not explain how to find a best or near-best approximant, making parameter selection central to numerical construction.Source: §5 · p. 54, §5.3 · pp. 57–58
• Unrestricted manifold approximation is degenerate because space-filling manifolds can achieve zero error even with one parameter, while small parameter perturbations cause large output changes.Therefore, meaningful nonlinear-width theories must restrict the mappings a and M to reflect reasonable numerical methods.Source: §5.3 · p. 58

5.4. Restrictions on a, M in manifold approximation

• Requiring continuous a and M yields manifold widths that are no better than n-term wavelet or adaptive finite-element approximation on classical smoothness classes.This does not rule out advantages on non-traditional model classes, but continuity alone may still be too weak for numerical implementation.Source: §5.4 · p. 59

5.5. Stable manifold widths

• Stable manifold approximation requires robustness to noisy inputs and parameter errors, ensured by Lipschitz conditions on A = M ◦ a and M.If both maps have Lipschitz constant γ, input-output stability follows with Lipschitz constant γ^2 for A.Source: §5.5 · pp. 59–60

5.6. Bounds for stable manifold widths

• Upper and lower bounds for stable manifold widths are known, and these bounds are tight in Hilbert spaces commonly used in neural-network applications.These results provide quantitative limits for stable manifold approximation rather than unrestricted approximation alone.Source: §5.6 · p. 60
• In Hilbert spaces, stable manifold widths are bounded below and above in relation to entropy numbers, but the constructive approximation achieving the upper bound is generally difficult to obtain.The upper-bound construction uses dimension reduction and Lipschitz extensions, yielding error at most 3ε_n(K)_H, while the resulting maps are not very constructive.Source: §5.6 · pp. 61–62

5.7. Weaker measures of stability

• Weaker assumptions (SP1)–(SP3), including Lip α stability and bounded parameter encodings, still imply Carl-type entropy lower bounds that limit approximation performance.The argument assumes α = 1, Lipschitz constant γ, and a_n(K) contained in the unit ball of R^n, then bounds packings through volumetric estimates.Source: §5.7 · pp. 62–63

5.8. Optimal performance for classical model classes described by smoothness

• For classical Sobolev- and Besov-type smoothness classes, continuous manifold approximation cannot improve on the rate O(n^-s/d), which is attained by methods such as adaptive finite elements and n-term wavelets.The restriction applies in particular to neural-network approximation when the associated parameter and reconstruction maps are continuous.Source: §5.8 · p. 63

5.9. VC dimension also limits approximation rates for model classes

• VC-dimension bounds impose lower rates for ReLU approximation: shallow networks are limited, up to logarithms, by classical rates, whereas deep networks retain the possibility of faster n^-2s/d rates.For deep networks, the text states that these improved rates are attained later; the VC-dimension argument is presented for approximation in the C(Ω) norm.Source: §5.9 · pp. 65–66

5.10. Another measure of optimal performance: approximation classes

• Approximation classes collect functions achieving a prescribed decay rate, but although their structure is characterized for several classical methods, it remains far from understood for neural-network approximation.Some structural results for neural-network approximation classes are available in (Gribonval, Kutyniok, Nielsen & Voigtlaender 2019).Source: §5.10 · p. 66

• The analysis represents network families as Σ_n, including Υ_n,1 and Υ_W0,n, with n proportional to the number of network parameters.The outputs are studied as nonlinear sets of continuous piecewise-linear functions determined by network parameters.Source: §6 · p. 67
• The survey must organize results across varying width, depth, connectivity, weight structure, and activation functions, so it primarily concentrates on two ReLU architectures.These are depth-one networks with W increasing and fixed-width networks with L increasing.Source: §6 · p. 67
• Fixed-width ReLU networks with depth L increasing can yield a dramatic gain in approximation efficiency, known as the power of depth.The survey contrasts this regime with width W increasing at depth one, which more closely matches classical approximation but remains incompletely understood.Source: §6 · p. 67
• Approximation is studied on Ω = [0, 1]^d using Lp(Ω) error for 1 ≤ p ≤ ∞, with uniform error for continuous functions when p = ∞.The main cases are p = ∞, p = 2, and p = 1; the results also extend to general Lipschitz domains.Source: §6 · pp. 67–68
• The approximation framework compares neural networks with classical methods on smoothness classes and seeks new function classes where neural networks outperform classical approximation.The goal is to understand error decay as n increases and identify application-relevant classes explaining when neural-network approximation is beneficial.Source: §6 · p. 68
• Neural-network approximation lacks general inverse theorems and precise characterizations of its approximation classes, leaving a substantial gap relative to classical approximation theory.The subsequent results therefore provide sufficient conditions for membership in approximation classes rather than complete characterizations.Source: §6 · pp. 68–69

7. Approximation using single layer ReLU networks

• Single-layer ReLU outputs form a complex nonlinear family of continuous piecewise-linear functions on convex-polytope partitions, and this complexity limits current understanding of their approximation properties.The family excludes some piecewise-linear functions, including compactly supported ones, despite potentially rich approximation behavior.Source: §7 · pp. 69–70

7.1. Approximation by single layer networks when d = 1

• For d = 1, single-layer ReLU approximation is essentially free-knot linear spline approximation, providing the best-understood setting and guidance for higher dimensions.Most fundamental approximation questions for general d remain unresolved.Source: §7 · pp. 70–71, §7.1 · p. 71

7.1.1. Approximation of classical model classes when d = 1

• For one-dimensional classical model classes, rates O((n + 1)^−s) hold up to smoothness s ≤ 2, with nonlinear breakpoint selection needed when τ < p.For s > 2, locally linear approximation does not improve beyond O((n + 1)^−2).Source: §7.1.1 · pp. 71–72

7.1.2. Approximation classes for d = 1

• When d = 1 and 0 < r < 2, the approximation class A_r is characterized by interpolation-space embeddings between Besov spaces of smoothness r.These embeddings imply approximation error O((n + 1)^−r), while no classical Besov characterization extends to r > 2.Source: §7.1.2 · pp. 72–73

7.2. Results for d ≥2

• For d ≥ 2, the absence of locally supported nodal functions forces approximation methods toward global constructions, with the strongest results presently in L2(Ω).This structural difficulty distinguishes higher-dimensional single-layer networks from standard local approximation methods.Source: §7.2 · p. 73

7.2.1. Approximation in L2(Ω)

• In L2(Ω), Sobolev functions admit known upper bounds for s ≤ 2 + (d − 1)/2 via ridge-function constructions embedded in single-layer ReLU networks.Matching lower bounds for the network family are not known, and the nonlinear gain for Besov classes above the Sobolev embedding line remains unquantified.Source: §7.2.1 · p. 74

7.2.2. Approximation in Lp, p̸ = 2

• For p ≠ 2, established results are largely limited to C(Ω), where the unit Lip 1 ball has approximation rate n^−1/d modulo logarithmic factors.Interpolation yields bounds for Lip α classes, but results for other p and nonlinear gains remain largely absent.Source: §7.2.2 · p. 75

7.2.3. Novel model classes for single layer approximation

• Barron classes provide a novel model class approximated by single-layer networks, with ReLU inheriting the sigmoidal-activation result and nonlinear terms selected adaptively to the target function.Generalizations extend approximation to Lq and improve the rate to O(n^−1/2−1/(q* d)), where q* is the smallest even integer ≥ q.Source: §7.2.3 · pp. 76–77

8. Approximation using deep ReLU networks

• Deep ReLU networks approximate classical model classes at least as well as known nonlinear methods, can achieve dramatically better error rates, and enable rates unavailable to standard methods for self-similar classes.The approximation framework uses function decompositions whose building blocks are captured efficiently, usually with exponential accuracy, by deep networks.Source: §8 · p. 77, §8.1 · p. 77

8.1. Results obtained from basic decompositions

• Basic decompositions combine controlled coefficient expansions with efficient neural approximations of their building blocks, while network depth implements linear combinations through concatenation and collation channels.The construction applies to wavelets, shearlets, ridgelets, power series, and Fourier decompositions; a sharper depth bound is available when blocks are iterated compositions of one univariate function.Source: §8.1 · pp. 77–79

8.2. Approximation of products

• Deep ReLU networks approximate products on [0,1]^k with exponential accuracy, including derivative error bounded by 2 · 2^-n for the bivariate construction.The bivariate product network has width 5 and depth 3n, and higher-order products are built inductively from the bivariate multiplier.Source: §8.2 · p. 82

8.3. Approximation of polynomials

• Monomials of total degree m are approximated by deep ReLU networks with width bounded by 3 + d, and polynomial sums are formed by concatenating the corresponding monomial networks.The construction repeats the input coordinates through source channels, allowing the width bound to depend on dimension rather than monomial degree.Source: §8.3 · p. 84

8.4. Approximation of tensor products

• A rank-one tensor product is well approximated whenever its univariate factors are well approximated, by stacking their networks and feeding their outputs into a deep product network.For factors valued in [0,1], the resulting network combines the factor approximators with the multidimensional product construction.Source: §8.4 · p. 85
• For approximated tensor products, the error satisfies ||g − S||C(Ω) ≤ dε + e^d·4^-n for n ≥ 1 + log2 d.The dε term accumulates the individual factor-approximation errors, while e^d·4^-n comes from product approximation.Source: §8.4 · p. 85

8.5. Approximation of B-splines

• A d-variate B-spline can be approximated by a deep ReLU network with width 6d, depth Cn, and support contained in the original spline’s support.The constant C depends only on the spline order r and dimension d; the construction uses univariate spline approximations, multiplication, and a support-truncation network.Source: §8.5 · pp. 86–87

8.6. Approximation of Besov classes with deep ReLU networks

• Deep ReLU networks approximate Besov unit balls in Lp with the classical O(n^-s/d) rate, using fixed width and at most C(s,d)n parameters.The theorem’s network class has width 6d and depth Cn, with constants depending on s, d, and an arbitrarily small δ; the rate is not quite optimal because of a logarithmic factor.Source: §8.6 · pp. 87, 91

8.7. Super convergence for deep ReLU networks

• For K = U(Lip 1) on [0,1], deep ReLU networks with width 11 achieve the superconvergent O(n^-2) approximation rate.The construction first interpolates f by a piecewise-linear network with error at most 1/n, then corrects the residual using a deeper network of depth 16n + 2.Source: §8.7 · pp. 92–93, §8.7.1 · p. 94

8.7.1. Remarks on Theorem 8.11

• The O(n^-2) rate is twice the rate delivered by standard approximation methods using n parameters, highlighting the unusual approximation power of the nonlinear network manifold.The comparison is stated for Σn = ΥW0,Cn(ReLU; d, 1), with W0 and C fixed depending only on d.Source: §8.7.1 · pp. 93–94
• For Lipschitz spaces, deep-network approximation has matching upper and lower bounds, although attaining the super-convergent parameters is numerically unstable and difficult by parameter search.The parameter-selection map cannot be continuous, while sufficiently long searches with a posteriori error estimation may still find suitable parameters.Source: §8.7.1 · p. 94

8.8. Super convergence for approximation in Lp

• Interpolation theory extends super-convergence to Lp approximation of Besov balls above the Sobolev embedding line, using combinations of approximants for endpoint spaces.The construction combines separate approximants through K-functionals and interpolation, yielding the stated general theorem for 1 ≤ p ≤ ∞.Source: §8.8 · pp. 95–96

8.9. A summary of known approximation rates for classical smoothness spaces

• For p = ∞ and 0 < s ≤ 1, the optimal approximation rate for classical smoothness classes is n^-2s/d, but logarithmic losses prevent a complete description for larger s.For p < ∞, super rates are available, but matching lower bounds and the best rates remain unknown because VC-dimension methods do not apply.Source: §8.9 · p. 97

8.10. Novel model classes

• Classical smoothness classes do not explain deep-learning success in high dimensions, motivating new model classes with manageable entropy that match applications and avoid the curse of dimensionality.The paper emphasizes that rigorous definitions, entropy bounds, and approximation analyses for such classes remain limited.Source: §8.10 · pp. 97–98

8.10.1. Comments on the curse of dimensionality

• Stable manifold approximation makes model-class entropy govern optimal rates, whereas space-filling manifolds may require inordinate computation despite arbitrarily accurate approximation.A practical high-dimensional method must find an approximant within a computational budget that does not grow excessively with dimension.Source: §8.10.1 · pp. 98–99

8.10.2. Model classes in high dimension

• Candidate high-dimensional model classes use sparsity, composition, self-similarity, or dimension reduction; ReLU self-similarity efficiently represents fractal functions and certain fractal-set indicators.Sparsity requires coefficient decay and application fit, while composition suggests mixed norms and low-dimensional manifolds align naturally with parameterized NN outputs.Source: §8.10.2 · pp. 100–101

9. Stable approximation

• Stable approximation is governed by two issues: how stability restrictions affect rates and how stable numerical approximation algorithms can be constructed.For stable methods, approximation decay cannot exceed the entropy numbers ε_n(K)_X, which provide a benchmark for optimal performance.Source: §9 · p. 102
• For embedded Besov balls, entropy yields the stable-method lower bound O(n^-s/d), but stable or merely continuous neural-network mappings attaining this rate are not provided.Matching upper and lower bounds remain a stated problem for classical model classes.Source: §9 · p. 102

9.1. Stability of M_n

• For a fixed ReLU architecture, the natural parameter-to-output map M_n is Lipschitz on every finite ℓ∞ parameter ball, with a constant depending on the ball, width, depth, and dimension.The map is continuous globally, while the theorem establishes local Lipschitz stability into C(Ω).Source: §9.1 · p. 103
• The Lipschitz constant for M_n can be controlled on a small ball around the origin, but the required ball becomes smaller as the network width increases.This control is obtained by keeping the layer matrices’ norms small.Source: §9.1 · p. 104

9.2. Stability of a_n

• Superconvergent parameter-selection procedures for Besov and Sobolev classes cannot be continuous, and continuous selection cannot beat O(n^-s/d) for compactly embedded Besov balls.The rate is attainable in the known cases τ ≥ p through linear spaces such as FEM spaces contained in Σ_n and linear approximation operators.Source: §9.2 · p. 104
• The section leaves matching upper and lower rates unresolved when both mappings are Lipschitz or when parameter selection is merely continuous, while bounded selection is linked to prior encoding-focused work.These questions are posed for approximation of Besov balls using the manifold Σ_n.Source: §9.2 · p. 105

10. Approximation from data

• Data observations alone provide no rigorous guarantee for predicting f at new points, so recovery requires additional model-class information about f.Learning guarantees must be weaker than full-information approximation guarantees, making the loss in performance an important issue.Source: §10 · p. 105

10.1. Deterministic learning; optimal recovery

• Optimal recovery uses the data and compact model class K to approximate f by the Chebyshev center of the feasible set Kw, with error equal to its radius Rw.A near-best feasible solution w + η achieves accuracy at worst 2Rw, but finding the Chebyshev ball numerically is difficult and K may be uncertain.Source: §10.1 · p. 106, §10.2 · p. 107

10.2. Optimal recovery in a Hilbert space

• In a Hilbert space, linear observations λj(f)=⟨f,ωj⟩ determine a projection onto the data subspace, while model-class membership constrains the unseen orthogonal component.The optimal approximation is the Chebyshev center of the feasible set rather than simply the projection onto the observation subspace.Source: §10.2 · pp. 106–107

10.3. Optimal recovery by linear space surrogates

• Surrogate sets built from linear spaces or neural-network manifolds make recovery more tractable, but the data-quality factor can diverge when surrogate dimension n exceeds the number of observations m.For ReLU-network surrogates, least-squares recovery inherits analogous bounds after accounting for nonlinearity; when n ≥ m, interpolation makes µn infinite and the bound useless, requiring an added ingredient.Source: §10.3 · pp. 108, 110

11.1. Deep learning

• Deep learning handles very high-dimensional data through empirical generalization, but lacks a priori guarantees because no specific model class for f is usually assumed.Point observations minimally suggest an RKHS setting, although the relevant RKHS is not specified.Source: §11.1 · p. 112

11.2. Possible model class assumption in high dimension

• A plausible high-dimensional model assumption is that f belongs to an approximation class generated by neural-network outputs, making rate-distortion analysis central.The appropriate error space X and the functions well approximated by the network family remain to be determined.Source: §11.2 · p. 112

11.3. Overparameterization

• Overparameterized networks typically have more trainable parameters than data sites, yielding infinitely many parameter choices that interpolate the observations.The central question is which interpolant the numerical method selects and whether that selection defines a hidden model class assumption.Source: §11.3 · p. 113

11.3.1. Selecting the interpolant by gradient descent

• Gradient descent from random initialization can produce an interpolating surrogate that generalizes well without the explicit regularizers commonly used to prevent overfitting.The section discusses implicit regularization as a possible explanation, but existing a priori guarantees are often vacuous without assumptions on f.Source: §11.3.1 · pp. 113–114

11.3.2. Gradient descent for linear regression

• In overparameterized linear regression, gradient descent preserves the initialization’s null-space component while converging on the data-informed subspace, thereby imposing implicit regularization and a low-complexity assumption.The initialization controls the distance from the minimum-norm solution, while the unchanged null-space component is not seen by the data.Source: §11.3.2 · pp. 115–116

11.3.3. Gradient descent selection for neural networks

• For sufficiently wide fixed-depth networks, gradient descent on mean squared error is approximately equivalent to least-squares regression in an RKHS determined by the neural tangent kernel.As width tends to infinity, the training kernel becomes fixed and depends on the activation function and depth.Source: §11.3.3 · pp. 116–117
• Despite these kernel-limit interpretations, why overparameterized neural-network learning generalizes well remains poorly understood.The section characterizes this as an important open issue rather than a resolved theoretical explanation.Source: §11.3.3 · p. 117

11.3.4. Stability of gradient descent

• Stability analysis asks how the learned surrogate changes with perturbed data, step sizes, stopping rules, initialization, and the parameter region reached by gradient descent.The framework also considers unknown compact model classes and algorithmic widths for comparing stable mappings and expected performance.Source: §11.3.4 · pp. 117–119

• Grohs et al. (2019) is cited for deep neural network approximation theory.Source: Manuscripta Math · p. 121
• Hornik et al. (1989) is cited for multilayer feed-forward networks as universal approximators.Source: Manuscripta Math · p. 122
• Lu et al. (2020) is cited for deep network approximation of smooth functions.Source: Manuscripta Math · p. 122