Source-linked AI summary
Elementary superexpressive activations
Dmitry Yarotsky
TL;DR
Neural-network approximation results had established fixed-architecture approximation, but the known activations were complex. This paper proves that simple elementary families can be superexpressive and shows that most practical non-periodic activations are not; computing suitable weights may nevertheless be impractical.
Problem
Fixed-architecture approximation of arbitrary continuous functions was known, but prior superexpressive activations were complex and not known to have finite elementary-function representations.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
Fixed-size approximation is possible with carefully chosen elementary activations, whereas finitely many piecewise Pfaffian activations cannot be superexpressive.
Source: §2 · p. 2, §3 · p. 6The existence result is theoretical: the required weight choices hide huge approximation complexity, and finding them is practically unfeasible even for relatively small numbers of fitted data points.
Source: §4 · p. 7
Abstract
from arXiv · showhide
We call a finite family of activation functions superexpressive if any multivariate continuous function can be approximated by a neural network that uses these activations and has a fixed architecture only depending on the number of input variables (i.e., to achieve any accuracy we only need to adjust the weights, without increasing the number of neurons). Previously, it was known that superexpressive activations exist, but their form was quite complex. We give examples of very simple superexpressive families: for example, we prove that the family {sin, arcsin} is superexpressive. We also show that most practical activations (not involving periodic functions) are not superexpressive.
1. Introduction
Superexpressiveness allows arbitrary-accuracy approximation of continuous functions with a fixed network architecture, but earlier constructions used complex activations. This paper introduces simple elementary superexpressive families and shows that most practical activations without periodic functions lack this property.
1. Introduction
- Superexpressiveness means approximating every continuous function on a compact domain to arbitrary accuracy with a finite fixed architecture by adjusting weights alone.Its existence follows from a result of Maiorov & Pinkus (1999).Source: §1 · p. 1
- Earlier superexpressive activations were analytic, monotone, and bounded, but were not known to have representations using finitely many elementary functions.The earlier construction joined functions from a dense countable subset for the univariate case, then used the Kolmogorov Superposition Theorem for multivariate functions.Source: §1 · p. 1
- Earlier elementary-activation results were weaker: network size could grow with accuracy, although more slowly than power laws expected under standard regularity assumptions.Examples use sin with ReLU, or floor, exponential, and step activations.Source: §1 · p. 1
- The paper proves that simple elementary activation families, including {sin, arcsin}, are superexpressive, while most practically used activations without periodic functions are not.The networks can approximate any continuous function to arbitrary accuracy without increasing their fixed architecture.Source: §1 · p. 1
2. Elementary superexpressive families
Theorem 3 establishes superexpressiveness for finite families built from elementary functions, including {sin, arcsin} and a family containing an explicit smooth, bounded, increasing activation. The proofs construct fixed-size approximating networks, with the multivariate case reducible to the univariate one.
2. Elementary superexpressive families
- Any family superexpressive for one variable is superexpressive in every dimension, with architecture size scaling as O(d^2).The reduction approximates the univariate functions in the Kolmogorov Superposition Theorem with univariate superexpressive networks.Source: §2 · p. 2
- Theorem 3 establishes that each listed activation-function family is superexpressive; one permits any real-analytic function non-polynomial on some interval, and σ3 is C1, bounded, and strictly monotone increasing.Source: §2 · p. 2
- For A1, the proof approximates f by constants on a fine grid, choosing weights so activation values are rationally independent and using dense torus winding to match the grid values.Uniform continuity controls the grid approximation; the resulting computation has fixed size depending on d.Source: §2 · pp. 2–3
- For A2, a partition of unity splits f into four locally supported functions because the continuous periodic ν cannot assign the whole interval finitely many values as the floor function does.On each support, the proof uses rationally independent sine values and dense torus winding to approximate f at segment centers; the partition then gives uniform approximation.Source: §2 · p. 4
- The {sin, arcsin} construction also requires approximate multiplication, which can be implemented by a fixed-size network when the activation has a nonzero second derivative at some point.The proof applies this multiplier result to sin to complete the A2 case.Source: §2 · pp. 4–5
- The A3 proof reduces to A2 by approximating arcsin and then sin with fixed-size σ3-networks, so an A2 network can be replaced with only a constant-factor size increase.The paper states that σ3 is C1, bounded, and increasing.Source: §2 · p. 5
- Replacing continuous activations by their antiderivatives preserves superexpressiveness.The proof uses uniform approximation of each continuous activation on compact sets.Source: §2 · p. 5
3. Absence of superexpressiveness for standard activations
The section proves that finite families of piecewise Pfaffian activations are not superexpressive, covering most practical activations that do not involve sin or cos. The proof bounds the sign changes possible in a fixed architecture using bounds on zeros of univariate Pfaffian functions.
3. Absence of superexpressiveness for standard activations
- Pfaffian functions include elementary functions on suitable domains, and sums, products, and compositions preserve the Pfaffian property.Sin is Pfaffian on bounded intervals, with complexity depending on interval length, but is not Pfaffian on all of R.Source: §3 · p. 5
- Piecewise Pfaffian activations are defined as Pfaffian on finitely many open intervals, with any remaining domain points listed separately.The cited examples include tanh, sigmoid, ReLU, leaky ReLU, the binary step function, Gaussian, softplus, and ELU.Source: §3 · p. 6
- Any finite family of piecewise Pfaffian activation functions is not superexpressive.The section says this class covers most practically used activations that do not involve sin or cos.Source: §3 · pp. 5–6
- A fixed one-input architecture would need outputs with arbitrarily many sign changes to approximate increasingly oscillatory sine functions, but piecewise Pfaffian networks have bounded complexity.The proof uses the bounded number of zeros for univariate Pfaffian functions to contradict arbitrarily many sign changes.Source: §3 · pp. 6–7
- For a fixed architecture and finite piecewise Pfaffian activation family, the network output is piecewise Pfaffian with bounded interval count and restriction complexity.The bounds depend only on the family and architecture.Source: §3 · p. 6
4. Discussion
The paper gives simple explicit activations for fixed-size approximation and rules out superexpressiveness for common practical activations. Its existence result is theoretical, with substantial complexity hidden in the weights, and it leaves open whether sin alone is superexpressive.
4. Discussion
- The existence result is purely theoretical: a small network hides huge approximation complexity in its specially chosen weights.The paper asks whether good weights can be found algorithmically, even assuming computations at arbitrary precision.Source: §4 · p. 7
- The constructive search is practically infeasible even for relatively small N fitted points because the pigeonhole-based proof is highly vulnerable to the curse of dimensionality.The proof is essentially constructive, but its dimensionality corresponds to the number of fitted data points.Source: §4 · p. 7
- Whether sin alone is superexpressive remains open; Section 3 cannot rule it out because sin has infinite Pfaffian complexity on R.The paper also asks whether an individual activation can be both elementary and real analytic on all of R.Source: §4 · p. 7
- Repeated antiderivative computations yield piecewise elementary superexpressive functions of any finite smoothness, but not functions analytic on R.Source: §4 · p. 7
A. Proof of Lemma 2
The proof equips the unit cube with torus topology and uses induction on dimension to establish density for an irrational winding. It approximates one face first, then extends the approximation to every cross-section.
A. Proof of Lemma 2
- The cube [0, 1)N is treated as a torus by gluing interval endpoints, with distance measured between the corresponding cosets.The resulting metric is shift-invariant.Source: A. Proof of Lemma 2 · p. 7
- The induction step chooses s0 = 1/aN so integer multiples place the last coordinate at zero, reducing the points to the face [0, 1)N−1.The proof proceeds by induction on N, starting from the immediate N = 1 case.Source: A. Proof of Lemma 2 · p. 7
- Rational independence makes the integer-indexed points distinct, so arbitrarily close pairs provide the small shift used in the induction.Source: A. Proof of Lemma 2 · p. 7
- Density in the lower-dimensional face yields points of QN within 2ε, and shifted copies then approximate every cross-section bN = c.The construction uses s values in S for the face and S + cs0 for the cross-section.Source: A. Proof of Lemma 2 · p. 8