Source-linked AI summary
Shallow neural network approximation in mixed Sobolev spaces
Yuwen Li, Guozhi Zhang
TL;DR
The paper asks how shallow networks approximate mixed Sobolev spaces when both target smoothness and activation choice matter. It separates multivariate Fourier-block assembly from univariate activation resolution, proving rates governed by min{α,ρ}; for ReLU^k this becomes the optimal algebraic exponent min{α,k+1}, while ELU and cosine attain α up to logarithmic factors.
Problem
Existing shallow-network theory does not make explicit how mixed-smoothness geometry and activation-dependent resolution jointly determine approximation rates.
Method
The paper introduces the Fourier-block property FB(ρ,β) and the univariate condition SUA(ρ), then combines block approximation with hyperbolic-cross truncation and neuron allocation.
Results
For ReLU^k, the optimal algebraic approximation exponent is min{α,k+1} up to logarithmic factors; ELU and cosine achieve exponent α up to logarithmic factors.
Takeaways & Limitations
Activation univariate approximation order determines the algebraic ceiling, while the mixed-smoothness assembly mechanism is activation-independent once the block property holds.
Takeaways & Limitations
The stated framework assumes real-valued target functions, with complex-valued networks used only as auxiliary objects, and its real-valued block formulation is given on a fixed bounded cube after affine rescaling.
Abstract
from arXiv · showhide
We investigate the best $L_2$ approximation of mixed Sobolev spaces by shallow neural networks with $n$ neurons and general activation functions. We first establish an activation-independent Fourier-block principle: if an activation has univariate approximation order $ρ$ in the sense of the Fourier-block property, then the global approximation rate has algebraic order $\min\{α,ρ\}$ for target functions of mixed smoothness $α$, up to explicit logarithmic factors. To verify this property for concrete activations, we introduce a structured univariate approximation condition that implies the Fourier-block property with explicit parameters. For $\mathrm{ReLU}^k$, a matching algebraic lower bound identifies $\min\{α,k+1\}$ as the optimal algebraic approximation exponent in any dimension, up to logarithmic factors in the upper bound. The framework also yields the exponent $\min\{α,k+1\}$ for cardinal B-splines and soft-$\mathrm{ReLU}^k$, and the full mixed-smoothness exponent $α$ for ELU and cosine activations, again up to logarithmic~factors.
1 Introduction
The paper separates mixed-smoothness geometry from activation-dependent resolution by approximating dyadic Fourier blocks with shallow networks. This framework yields algebraic rates governed by min{α,ρ}, with nearly optimal ReLU^k results and activation-specific consequences.
- Motivation: The paper targets best L2 approximation of mixed Sobolev and Korobov spaces by one-hidden-layer networks, focusing on how activation choice affects rates.The framework addresses approximation error in relation to target smoothness, dimension, network width, and activation properties.
- General framework: A uniform relative approximation property on dyadic Fourier blocks separates mixed-smoothness assembly from the activation’s block-resolution efficiency.Hyperbolic-cross truncation and neuron allocation assemble global approximations, while the activation controls resolution of each block.
- General framework: Under FB(ρ,β), the global algebraic decay exponent is determined by the target smoothness α and activation order ρ, with explicit logarithmic contributions.The logarithmic exponent reflects dyadic block counts, neuron allocation, and the logarithmic factor in the Fourier-block property.
- Activation conditions: SUA(ρ) implies FB(ρ,βd,ρ), where βd,ρ = ρ(d−1)+2d−1, providing a one-dimensional route to global approximation bounds.SUA combines a periodic Jackson estimate, translation covariance, and linear-complexity ridge realization.
- Activation-specific results: For ReLU^k, the optimal algebraic exponent is min{α,k+1} up to logarithmic factors, with a matching lower bound in any dimension.The one-dimensional rate is optimal without logarithmic loss; the theorem’s logarithmic exponent is not claimed optimal in every higher-dimensional regime.
- Activation-specific results: The framework also covers cardinal B-splines and soft-ReLU^k, while ELU and cosine attain the full mixed-smoothness exponent α up to logarithmic factors.These examples demonstrate how the global rate follows from the activation’s univariate approximation order.
2 Preliminaries
The preliminaries define mixed Sobolev and Korobov spaces through Fourier and derivative-based descriptions, then decompose functions into dyadic Fourier blocks. They introduce FB(ρ,β) and SUA(ρ), which formalize block approximation and its univariate activation conditions.
- Function spaces: The paper works on the cube Ω=(0,1)^d and relates nonperiodic mixed Sobolev spaces to periodic Korobov spaces through extension and periodization.The preliminaries also record zero-boundary variants and inherited Korobov norms.
- Function spaces: Dominating mixed smoothness is characterized through mixed derivatives and an equivalent periodic Fourier norm.The Fourier representation uses the unitary transform and weak-derivative descriptions for integer smoothness.
- Fourier decomposition: Dyadic Fourier blocks Bℓ generate finite-dimensional trigonometric spaces Tℓ with dimension Dℓ=2^|ℓ|_1 and projections Δℓ.The block norms’ decay with |ℓ|_1 expresses dominating mixed smoothness.
- Fourier-block property: FB(ρ,β) requires approximating any function in a dyadic block with a shallow network once the neuron count is at least proportional to the block dimension.The definition is formulated using complexified networks, while real targets retain the same rate through real parts.
- Structured univariate approximation: SUA(ρ) is a one-dimensional activation condition built from finite-dimensional periodic spaces, operators Qm, band Jackson estimates, translation covariance, and ridge realizations.Its structure is designed to generate the multivariate Fourier-block approximations required by the framework.
3 Applications to selected activation functions
The paper verifies the structured univariate condition for several common activations and converts those orders into global mixed-smoothness approximation bounds. ReLU-derived activations saturate at their univariate order, whereas ELU and cosine retain exponent α.
- Activation examples: The applications section verifies SUA(ρ) for several classes of commonly used activation functions.The verification is activation-specific but follows the common conditions introduced earlier.
- Spline activations: Cardinal B-splines of degree k use the spline spaces V_m^(k), while softplus is the k=1 case.The spline construction is tied to the ReLU-based positive-part function.
- ELU and cosine: For ELU, the construction chooses a spline space V_m^(p) with p+1≥ρ, while cosine uses an orthogonal Fourier projection.Both verifications establish the required structured approximation through activation-specific realizations.
- Global rates: Combining Proposition 3.1 with Corollary 1.3 converts each activation’s univariate order ρ into an explicit global approximation upper bound.The general theory supplies the rate after Proposition 3.1 identifies the relevant activation order.
- Global rates: For ELU and cosine, the global error satisfies n,d(σ) ≲ n^−α polylog(2+n).These activations achieve the full mixed-smoothness algebraic exponent, up to logarithmic factors.
4 Upper bounds for general activation functions
The proof combines Fourier-block approximation with dyadic decomposition to obtain global mixed-Sobolev upper bounds. The resulting algebraic rate is governed by min{α,ρ}, with explicit logarithmic factors determined by the approximation regime and construction.
- General framework: The proof first combines the Fourier-block property with dyadic decomposition, then transfers the periodic estimate to Ω=(0,1)^d by extension and restriction.Affine changes preserve the shallow ridge form and affect only constants.
- Network construction: The network is built by approximating each dyadic Fourier block separately and concatenating the hidden units of all block networks.The block width allocation depends on the dyadic level and the Fourier-block constants.
- Rate regimes: ρ<α yields an algebraic rate n^-ρ(log(2+n))^β, while ρ=α yields n^-α(log(2+n))^(αd+β+p).These are the two regimes in which activation resolution does not exceed or matches target smoothness.
- Rate regimes: ρ>α yields n^-α(log(2+n))^(α(d−1)+α(β+p)/ρ), so the algebraic exponent saturates at the target smoothness.The bound is obtained after selecting the largest admissible dyadic truncation level under the width constraint.
- Interpretation: The activation enters only through the Fourier-block parameters ρ, β, and associated constants; dyadic decomposition and the algebraic rate n^-min{α,ρ} are activation-independent.The framework therefore separates mixed-smoothness geometry from activation-dependent approximation resolution.
5 Optimal algebraic bounds for ReLUk activation functions
For ReLU^k, the paper proves an activation-specific approximation order k + 1 and matching lower bounds, yielding the optimal algebraic exponent min{α, k + 1} up to logarithmic factors in every dimension.
- Saturation lower bound: For α > k + 1, line restrictions reduce shallow ReLU^k networks to piecewise polynomials of degree at most k, producing a matching algebraic lower bound.The lower-bound construction uses a smooth target whose restriction contains t^(k+1), which cannot be approximated arbitrarily well by degree-k pieces.
- Conclusion: The lower bounds hold for all dimensions, so min{α, k + 1} is the optimal algebraic approximation exponent up to the logarithmic factor in the upper bound.The two lower-bound regimes cover α above and below the activation order, while the upper estimate is activation-derived.
- Upper bound: ReLU^k satisfies the structured univariate approximation condition SUA(k + 1), which gives the corresponding shallow-network upper bound.The proof verifies the condition through periodic spline approximation and translation-invariant spline spaces.
- Upper bound: The ReLU^k upper-bound construction represents suitable periodic splines exactly with O(M) neurons, with constants depending on d and k.The realization uses truncated-power spline representations and represents the polynomial part using shifted ReLU^k functions.
- Nonsaturation lower bound: For 0 < α ≤ k + 1, compactly supported oscillatory targets yield lower bounds with exponent α while remaining in the mixed Sobolev unit ball.The construction scales oscillatory functions according to the network width and controls their mixed Sobolev norm through Fourier estimates.
6 Conclusion
The conclusion presents a unified framework for shallow-network approximation based on activation-dependent univariate conditions. It identifies near-optimal ReLU^k rates and corresponding exponents for several other activations.
- Framework: The paper introduces Fourier-block approximation FB(ρ, β) and structured univariate approximation SUA(ρ) as activation-dependent conditions.SUA(ρ) implies FB(ρ, β) with explicit parameters and is formulated entirely in one dimension.
- ReLU^k: For ReLU^k, the upper algebraic exponent is min{α, k + 1}, and a matching lower bound makes the rate nearly optimal.The upper bounds retain logarithmic factors, whose reduction or elimination remains open.
- Other activations: Cardinal B-splines and soft-ReLU^k attain the exponent min{α, k + 1}, while ELU and cosine attain the full mixed-smoothness exponent α.These results follow by applying the general framework after identifying each activation's univariate approximation order.
Proof of Lemma 2.1
The lemma constructs a periodic extension of a function on the domain and proves boundedness of the periodization operator in mixed Sobolev norms for all nonnegative smoothness orders.
- Extension construction: A compactly supported extension is formed by multiplying a mixed Sobolev extension by a smooth cutoff equal to one near the domain.The cutoff preserves the function on the target domain while producing compact support in a larger cube.
- Integer smoothness: For integer smoothness orders, Fourier norm identities and the Leibniz rule establish boundedness of the periodization operator.The argument uses bounded derivatives of the cutoff and sums over the admissible mixed derivatives.
- Arbitrary smoothness: For noninteger smoothness, complex interpolation between adjacent integer orders extends the operator estimate to every α ≥ 0.The mixed-smoothness spaces form an interpolation scale with equivalent norms.