Source-linked AI summary
A concentration result for multilayer feedforward neural networks
Vera Koponen
TL;DR
The paper asks how large multilayer feedforward networks behave when their internal weights and input values are only characterized probabilistically. It formalizes fixed-depth networks with continuous aggregation-based activations and layerwise weight distributions, and proves that independently sampled continuous inputs make the output concentrate near a fixed value as n grows.
Problem
The paper studies how output values correlate with large networks’ layers, weight distributions, and input values when exact network details are unavailable.
Method
The paper models fixed-depth networks using continuous structures, continuous aggregation functions, continuous activation components, and continuous approximations to layerwise weight distributions.
Results
For every ε > 0, the probability that the output lies in [ψ − ε, ψ + ε] tends to 1 as n →∞.
Takeaways & Limitations
Large networks satisfying the stated assumptions have output values that concentrate around a single value despite independently distributed continuous inputs.
Takeaways & Limitations
The result assumes that neurons in the same non-input layer have approximately the same incoming-weight distribution, and weaker assumptions may prevent convergence to one value.
Abstract
from arXiv · showhide
We consider for an arbitrary fixed $ρ$ and for each positive integer $n$ a multilayer feedforward artificial neural network with $ρ$ layers, $n$ neurons in the first layer (the input layer) and only one neuron, the output neuron, in the last layer. Very roughly formulated, the main result is that if the distribution of weights of connections from a layer to the next are, for all large $n$, approximated well by a fixed continuous (but otherwise arbitrary) curve which does not depend on $n$, and if the values of the $n$ input neurons are independently and identically distributed with a continuous probability density function, then there is a number $ψ$ such that for all $\varepsilon > 0$ the probability that the value of the output neuron is in $[ψ- \varepsilon, ψ+ \varepsilon]$ tends to 1 as $n$ tends to infinity.
1. Introduction
The paper studies how large multilayer feedforward networks behave when their weights follow stable layerwise distributions and inputs are independently sampled. Under these conditions, the output concentrates near a fixed value, although the theorem relies on strong assumptions.
- Motivation: Neural networks often predict or classify successfully without explaining their outputs, motivating analysis of their internal behavior.The paper links this opacity to ethical and legal questions.
- Research question: The paper asks how output values correlate with network depth, layer sizes, weight distributions, and input values in large networks.Large networks are harder to analyze because many weighted connections contribute to the output.
- Main result: The main result assumes fixed depth, n input neurons, one output neuron, continuous layerwise weight-distribution curves, and continuous activation components.Neuron values and connection weights are restricted to [0, m].
- Main result: The output converges in probability to a value ψ: for every ε > 0, its probability of lying in [ψ − ε, ψ + ε] tends to 1 as n →∞.This is the paper’s concentration result for the output neuron.
- Proof consequence: The proof shows that a two-layer network with n inputs can reproduce the same concentrated output value with high probability.This reduction is identified as Corollary 3.4.
- Limitations: The authors identify strong assumptions, including approximately shared incoming-weight distributions among neurons in the same layer.They suggest weaker assumptions or alternative input distributions could yield interval-probability estimates rather than convergence to one value.
2. Aggregation functions
The paper formalizes aggregation functions as symmetric operations on finite sequences and requires continuity under distributional similarity. Average, maximum, and minimum are treated as continuous examples.
- Definitions: Aggregation functions map finite nonempty sequences from [0, m] to [0, m] and are symmetric under reordering.Symmetry means sequence order does not affect the output.
- Examples: Examples include the average and maximum of a sequence.The average is defined by summing entries and dividing by n, while maximum selects the largest entry.
- Continuity: Continuity means similarly distributed entries produce roughly similar aggregation outputs, even for sequences of different lengths.The definition is designed to make activation functions independent of network size.
- Continuity: A continuous aggregation function satisfies two formal conditions governing perturbation stability and sufficiently large sequences with controlled value proportions.These conditions are stated in Definition 2.2.
- Continuity results: Maximum, minimum, and average are continuous aggregation functions under the paper’s definition.The average result follows from prior work, while maximum and minimum require a modified proof.
- Closure property: Composing a continuous aggregation function with a continuous scalar function yields another continuous aggregation function.This closure property supports constructing activation functions from aggregators.
3. Model theoretic representation
The paper represents network sequences as continuous structures with explicit layers, edges, weights, and neuron values. It then defines input-induced probability measures and shows the output map is continuous and measurable under the model assumptions.
- Network representation: Each network has fixed ρ ≥ 2 layers, exactly n input neurons, and one output neuron in the final layer.The networks form a sequence Bn indexed by the input-layer size n.
- Network representation: Every neuron in one layer connects to every neuron in the next, with weights in [0, m], and later neuron values depend on inputs, weights, and activation functions.Connections are restricted to consecutive layers.
- Model-theoretic encoding: The model uses a finite relational signature to encode layers and edges, then adds a unary predicate P to represent neuron values in network states.A state is an expansion of the base network structure assigning values to neurons.
- Assumptions: Activation functions use continuous aggregation functions together with continuous functions gi, while incoming weight distributions are approximated by continuous functions Wi.The assumptions also bound layer sizes by a polynomial and require sufficiently accurate distributional approximation.
- Assumptions: The weight-distribution assumptions force the number of neurons in every non-output layer to grow without bound as n increases.The proof derives this from positive-mass intervals under the continuous Wi functions.
- Probability model: Independent inputs with density μ induce the product density μn on [0, m]n and a corresponding probability measure on network states.The state-space encoding is bijective with input assignments.
- Output map: The output function Outn maps input assignments to the output-neuron value and is continuous, so inverse images of intervals are measurable.This makes output probabilities well-defined.
P Sn(x1,...,xn)(on) = F2(g2,w1(x1), . . . , g2,wn(xn))
The output mapping is shown to be continuous in the input values, and the paper then establishes that sufficiently large multilayer networks can be approximated by two-layer networks.
- The output function is continuous because it is built by composing continuous aggregation, activation, and updating functions.
- The continuity argument starts with two-layer networks and extends inductively through deeper layers.
- For every ε > 0, the main theorem gives ψ and c such that the output concentrates near ψ for all sufficiently large n.
- Two layers suffice to obtain, up to an arbitrarily small error, the same output distribution as the original large network.
- The equivalent two-layer sequence retains W2 and g2 while replacing F2 with a modified continuous aggregation function.
4. Proofs
The proofs establish concentration first for two-layer networks, then extend it inductively to any fixed number of layers. The argument uses independence of input events, continuity of aggregation, and high-probability control of neuron-value distributions.
- 4.1. The base case: 2 layers: The two-layer base case proves that the output concentrates near some ψ as n grows.Lemma 4.1 supplies an exponentially high-probability concentration statement for networks with two layers.
- 4.1. The base case: 2 layers: Input coordinates are independent under the product density, so interval-membership events for distinct input neurons are independent.The product form of µ_n and Fubini’s theorem yield the required independence.
- 4.1. The base case: 2 layers: For positive interval mass, the number of neurons in each relevant value bin grows linearly with n; for zero mass, the corresponding bin is empty.This separates the concentration proof into bins that occur frequently and bins that never occur.
- 4.2. The induction step: Continuity of F2 transfers similar empirical distributions of connection inputs into similar output values, completing the induction.The proof controls the relevant bin frequencies and invokes the continuity condition for aggregation functions.
- Remark 4.6. Generalization to several features: The same proof strategy extends to neurons with several features, producing a concentration value ψj for every feature.The generalization uses continuous aggregation and update functions for each feature and assumes continuous feature-wise input densities.