Source-linked AI summary
On the approximation of functions by tanh neural networks
Tim De Ryck, Samuel Lanthaler, Siddhartha Mishra
TL;DR
The paper studies how network size and shallow architecture affect approximation of Sobolev-regular and analytic functions by tanh networks. It constructs explicit approximations using polynomial, multiplication, and partition-of-unity components, and shows that two hidden layers suffice for exponential approximation of analytic functions in Sobolev norms. The results also identify rapidly growing weights and dimensionality as important scope boundaries.
Problem
Universal-approximation and expressivity results often establish existence or asymptotic rates without explicit size bounds, while detailed high-order Sobolev results for tanh networks remain limited.
Method
The paper constructs tanh-network approximations from local polynomial approximations, multiplication operators, and partition-of-unity components, using at most two hidden layers.
Results
Two-hidden-layer tanh networks provide explicit approximation-error bounds for Sobolev and analytic functions, with exponential convergence in width for analytic functions in Sobolev norms.
Takeaways & Limitations
Rather shallow tanh networks can achieve approximation rates comparable to or better than much deeper ReLU networks for the function classes studied.
Takeaways & Limitations
The constructive network’s weights can grow very fast, and the approximation results suffer from the curse of dimensionality, although analytic cases can have dimension-independent rates with exponentially growing constants.
Abstract
from arXiv · showhide
We derive bounds on the error, in high-order Sobolev norms, incurred in the approximation of Sobolev-regular as well as analytic functions by neural networks with the hyperbolic tangent activation function. These bounds provide explicit estimates on the approximation error with respect to the size of the neural networks. We show that tanh neural networks with only two hidden layers suffice to approximate functions at comparable or better rates than much deeper ReLU neural networks.
1. Introduction
The paper addresses gaps in quantitative approximation theory for tanh networks, especially explicit size-dependent error bounds in high-order Sobolev norms. It develops results for networks with at most two hidden layers, including exponential-width convergence for analytic functions.
- Motivation: Most detailed quantitative approximation results concern ReLU networks, while comparable detail for tanh networks remains unavailable.Tanh is widely used in recurrent architectures and physics-informed neural networks.
- Motivation: Higher-order Sobolev norms are important for applications such as PINNs, where neural networks are differentiated to evaluate PDE residuals.Many smooth-activation results instead measure errors in Lp norms.
- Motivation: Existing expressivity results often provide asymptotic rates but not explicit network sizes needed to reach a chosen accuracy.The paper targets explicit bounds on approximation error and network complexity.
- Motivation: Approximation error is only one component of total neural-network error, alongside optimization and generalization errors.Explicit weight estimates are also relevant because standard generalization bounds depend on them.
- Contribution: The paper focuses on expressivity with tanh activation and proves explicit approximation-error bounds for architectures with at most two hidden layers.This directly addresses the question of how large a network must be for a specified accuracy ϵ > 0.
- Contribution: For analytic functions, a two-hidden-layer tanh network achieves exponential convergence in network width, including in Sobolev norms.The paper presents this as an improvement over available results for ReLU and other smooth activations.
2. Preliminaries
The preliminaries define the function spaces, multi-index notation, and feedforward network architecture used throughout the paper. They specialize the activation to tanh and record basic network-composition and parallelization operations.
- Multi-index notation: Multi-indices α and β are compared componentwise, with α ≤ β exactly when α_i ≤ β_i for every coordinate.The notation supports multinomial coefficients and sets of multi-indices with prescribed total degree.
- Sobolev spaces: The paper defines W^k,p(Ω) as the Sobolev space over an open set Ω, equipped with norms based on derivatives through order k.The resulting normed space is a Banach space.
- Neural networks: A neural network realization composes affine layer maps with elementwise activation functions, and its width is the maximum layer width.Weights and biases parameterize the affine maps.
- Neural networks: A shallow network has exactly one hidden layer, whereas a deep network has two or more hidden layers.This terminology follows from counting the first L − 1 layers as hidden layers.
- Activation function: The analysis uses tanh as its activation function and applies directly to sigmoid or logistic activation because it is a shifted and scaled tanh.Restricting attention to tanh enables precise explicit bounds while retaining applicability to the related sigmoid function.
- Network operations: Parallelization and composition propositions allow networks to combine subnetworks and realize function compositions with corresponding widths and layers.These constructions are used as basic neural-network calculus throughout the paper.
1703. Uniform approximation of polynomials
The paper constructs shallow tanh networks that uniformly approximate univariate and multivariate polynomials in Sobolev norms, with explicit error and complexity bounds. These polynomial approximations support later constructions for smooth-function approximation.
- Univariate polynomials: The strategy begins by deriving uniform Sobolev-norm bounds for approximating polynomials with shallow tanh networks.The construction uses the p-th order central finite-difference operator for univariate polynomials.
- Univariate polynomials: The construction gives explicit estimates for simultaneously approximating all polynomials up to a prescribed maximal degree, rather than only asymptotic convergence rates.These estimates are identified as crucial for efficiently approximating analytic functions.
- Univariate polynomials: Shallow tanh networks approximate odd-degree monomials through finite differences and even-degree monomials recursively.The even-degree construction avoids the difficulty of explicitly finding uniform lower bounds for derivatives of tanh at a single point.
- Univariate polynomials: For every ϵ > 0, the polynomial-approximation networks can have width independent of ϵ, after choosing the construction parameter α = 1/s.The network architecture reuses neurons for the approximations of all monomials up to degree s.
- Multivariate polynomials: The univariate construction extends to multivariate monomials by applying the multinomial theorem and representing total-degree monomials through lifted variables.For q = d + 1 and ω = (1, x), the resulting monomials cover all d-variate monomials of total degree at most n.
- Multivariate polynomials: The approach is not restricted to tanh: it can generalize to activation functions possessing a suitable nonvanishing derivative structure.The paper nevertheless focuses on tanh because of its widespread use.
4. Approximation of partition of unity
The paper approximates a partition of unity on [0, 1]^d with tanh networks by constructing localized functions whose sum is approximately one. The construction controls derivative errors and can incorporate neural multiplication approximations.
- Construction: The construction seeks localized functions that mimic a partition of unity, where the component functions are nonzero only on small parts of [0, 1]^d.Unlike ReLU and RePU networks, which can construct such partitions exactly, tanh networks construct an approximate partition.
- Construction: For each index j with ∥j∥∞ ≤ N, the paper defines localized functions ρN_j and combines them into multivariate functions ΦN,d_j.The resulting functions are intended to approximate the partition-of-unity property on the relevant domain.
- Error control: The approximation parameters are selected using tanh’s limiting behavior and derivative decay to control the functions and their derivatives.A sufficiently large α = α(N, ϵ) is chosen, with a monotonicity condition imposed on higher derivatives beyond a threshold R.
- Illustration: Figure 1 illustrates the one-dimensional construction for N = 7, with thin lines representing the component functions Φ7,1_j = ρ7_j.The example is drawn on [0, 1].
- Neural implementation: A tanh neural-network approximation of each ΦN,d_j can be obtained by replacing the multiplication operator with a previously constructed multiplication network.The paper cites two earlier multiplication constructions for this replacement.
2855. Main results
The paper constructs two-hidden-layer tanh networks with explicit width and Sobolev-error bounds for approximating Sobolev-regular and analytic functions. For analytic functions, the resulting approximation converges exponentially with network width, while dimension-dependent constants remain a limitation.
- Sobolev approximation: Theorem 5.1 provides explicit network-size and approximation-error estimates for f ∈ W^{s,∞}([0,1]^d).The theorem applies for sufficiently large N and constructs a tanh network with two hidden layers.
- Sobolev approximation: Two-hidden-layer tanh networks approximate Sobolev-regular functions using localized polynomial approximations and explicit bounds on width and error.The construction partitions the unit cube, approximates the function by local polynomials, and replaces polynomial products and indicators with tanh subnetworks.
- Analytic approximation: Corollary 5.6 gives two-hidden-layer tanh networks of width O(N) for analytic functions, with error bounded by c_{d,k,α,f} N^{α−k/(d+1)}.The paper identifies this as an exponential convergence rate in terms of network width.
- Analytic approximation: The analytic-function rate is independent of dimension, although the hidden constant can still grow superexponentially with dimension.Thus the rate lessens the curse of dimensionality without removing dimension dependence from the bound's constant.
- Analytic approximation: A shallow tanh network can achieve exponential approximation for analytic functions under an additional assumption, without hidden constants growing as O(d^d).This simpler result is restricted to supremum-norm approximation and uses only one hidden layer.
- Examples: For the tested examples, a shallow two-layer tanh network reaches similar error to a three-layer network of the same width, while width increases produce large accuracy gains.The theoretical widths remain overestimates; f_{2π} reaches 1% error with a shallow width-four network.
6. Summary and discussion
The paper develops explicit high-order Sobolev approximation bounds for Sobolev-regular and analytic functions using tanh networks with two hidden layers. Its constructions provide explicit error and weight estimates, while achieving competitive or faster analytic-function convergence than related approaches, subject to stated limitations.
- Contributions: The paper proves explicit approximation-error bounds for Sobolev-regular and analytic functions using tanh networks with only two hidden layers.These results are stated for high-order Sobolev norms and accuracy ϵ > 0.
- Network structure: The constructed networks are generally dense: their non-zero-weight fraction increases with s and decreases with N for d = 1.For analytic functions, increasing s is asymptotically more efficient than increasing N because the convergence rate is O(N^-s).
- Method: The constructive proof combines finite-difference monomial approximation, multiplication-operator approximation, and partition-of-unity approximation.The approach is designed to obtain uniform explicit bounds for multivariate monomial approximation.
- Novelty: The key novelty is a construction that yields explicit bounds on both approximation error and network weights, unlike earlier monomial-based constructions.The paper first approximates odd univariate monomials, then extends the construction to even powers and multivariate monomials.
- Results: For analytic functions, Corollary 5.6 gives asymptotically faster convergence in network width than the compared related works and uses stronger norms.The comparison concerns upper bounds summarized in Table 2, including prior smooth-activation and ReLU results.
- Limitations: The bounds are limited by rapidly growing weight amplitudes, the curse of dimensionality, and continuous dependence of constructed weights on the target function.The authors characterize the bounds primarily as practical upper bounds for setting network size.
Appendix A. Auxiliary results
The appendix collects auxiliary analytical tools used to establish the approximation results, including derivative identities, combinatorial estimates, composition rules, and polynomial approximation lemmas.
- Matrix constructions involving multi-index sets, Stirling numbers, and diagonal scalings establish invertibility properties used in the analysis.
- Derivative formulas for tanh are combined with Stirling-number identities and Bernoulli-number expansions to control higher derivatives.
- The appendix bounds combinatorial complexities arising in multivariate Faà di Bruno expansions and derivative estimates.
- Bramble–Hilbert and Taylor lemmas provide polynomial approximations with Sobolev-norm error bounds under geometric and smoothness assumptions.
- The appendix derives bounds for Taylor remainders by combining multinomial expansions with derivative seminorm estimates.