Source-linked AI summary
Deep ReLU Networks Have Surprisingly Few Activation Patterns
Boris Hanin, David Rolnick
TL;DR
Deep ReLU networks may express exponentially many activation patterns with depth, yet the functions they learn can occupy a much smaller complexity range. The paper proves a depth-independent average bound at initialization and finds empirically that comparable bounds persist during training, including memorization tasks.
Problem
Prior work leaves uncertain whether the expressive advantage of depth is realized by functions that deep ReLU networks actually learn in practice.
Method
The paper combines theoretical analysis of expected activation-region counts with experiments during training, including memorization-based tasks.
Results
The expected activation-region count at initialization has a depth-independent upper bound based on neuron count and input dimension, and experiments find similar behavior throughout training.
Takeaways & Limitations
Activation patterns are typically far below their theoretical maximum, so practical ReLU-network complexity depends mainly on neuron count rather than depth within the stated conditions.
Takeaways & Limitations
The conclusions assume well-behaved gradients and biases, and the authors have not verified that the relevant constants remain bounded throughout training.
Abstract
from arXiv · showhide
The success of deep networks has been attributed in part to their expressivity: per parameter, deep networks can approximate a richer class of functions than shallow networks. In ReLU networks, the number of activation patterns is one measure of expressivity; and the maximum number of patterns grows exponentially with the depth. However, recent work has showed that the practical expressivity of deep networks - the functions they can learn rather than express - is often far from the theoretical maximum. In this paper, we show that the average number of activation patterns for ReLU networks at initialization is bounded by the total number of neurons raised to the input dimension. We show empirically that this bound, which is independent of the depth, is tight both at initialization and during training, even on memorization tasks that should maximize the number of activation patterns. Our work suggests that realizing the full expressivity of deep networks may not be possible in practice, at least with current methods.
1 Introduction
The paper studies the gap between the maximum complexity deep ReLU networks can express and the complexity they typically learn. It argues that activation-region counts are usually far below depth-based theoretical maxima, both initially and throughout training.
- Deep networks can represent some functions more efficiently than shallow networks, but learned functions may not use their full expressive power.
- The paper analyzes typical activation-region counts rather than worst-case maxima, focusing on average-case upper bounds for ordinary parameter values.Activation regions are polytopes where a ReLU network computes a single linear function associated with a fixed activation pattern.
- Theorem 5 bounds the expected number of activation regions using the input dimension and total neuron count, yielding a depth-independent result under reasonable gradient and bias assumptions.The theorem applies to random weights and biases, including highly correlated distributions and arbitrary connectivity between adjacent layers, provided weights are not tied.
- The bound contrasts with the maximum possible number of activation patterns, which grows exponentially with network depth.
- The paper distinguishes activation regions from linear regions and notes that depth may affect fluctuations such as variance more than the mean region count.
- Empirically, activation-region counts change little during MNIST training and do not increase exponentially, including on memorization-based tasks.Figure 3 reports an initial value of approximately (#neurons)^2/2 in two-dimensional cross-sections; the value decreases slightly and then rebounds over training.
2 How to Think about Activation Regions
The paper distinguishes activation regions, determined by neuron on/off patterns, from linear regions, determined by the functions the network computes. It characterizes activation regions geometrically through hyperplane arrangements that become bent in deeper networks.
- Activation regions: Activation regions are the non-empty input-space sets where every neuron has a fixed on/off sign pattern.Each neuron's pre-activation sign determines whether it is on or off throughout the region.
- Activation regions: Every activation region is convex, although this property is often overlooked when activation regions are conflated with linear regions.The paper notes that convexity extends to other piecewise-linear activations such as leaky ReLU.
- Hyperplane arrangements: Activation regions are connected components formed by removing the neurons’ bent hyperplanes from input space.For deeper layers, each bent hyperplane is locally an ordinary hyperplane within a preceding activation region.
- Linear regions: Linear regions are connected components after removing the set where the network gradient is discontinuous, and they need not match activation regions.Distinct activation regions can coalesce when a layer is zeroed out or neighboring regions compute identical linear functions.
- Depth and complexity: Deep networks can induce more activation regions than shallow networks with the same neuron count, even though average counts have depth-independent bounds at initialization.The paper suggests depth may affect fluctuations such as variance rather than the mean, but does not prove this claim.
3 Main Result
Theorem 5 bounds the average density of activation regions in random ReLU networks independently of depth, under stated distributional and gradient/bias conditions. Intuition and experiments suggest this bound remains comparable during training and is sharp to leading order.
- Formal statement: Theorem 5 bounds the average number of activation regions per unit input volume for random ReLU networks.The theorem applies to feed-forward networks with no tied weights and imposes assumptions on weights, biases, and gradient growth.
- Formal statement: The bound applies at initialization despite the maximum possible number of activation patterns growing exponentially with depth.This establishes a contrast between average-case behavior and worst-case expressivity.
- Sharpness: (#neurons)^nin / nin! is the heuristic upper bound for the average number of activation patterns over all input space at initialization.Experiments confirm this depth-independent heuristic bound.
- Geometric intuition: In one input dimension, the expected number of activation regions over [a, b] is O((b − a) · #neurons).The geometric argument links region endpoints to crossings where a neuron's pre-activation equals its bias.
- Geometric intuition: For higher-dimensional inputs, counting region density suggests an upper bound of #neurons^nin per unit nin-dimensional volume.The argument extends the one-dimensional density bound through unit counting.
- Depth dependence: Per-neuron average contributions to activation-region counts are suggested to be the same across layers, so networks with equal neuron counts should have equal averages.This follows heuristically from layer-independent covariance of the bent-hyperplane normal vectors.
4 Maximizing the Number of Activation Regions
The experiments test whether training tasks and initialization choices can drive activation-region counts toward their depth-dependent theoretical maximum. Across MNIST corruption and memorization tasks, region counts increase only modestly and remain near initialization-scale predictions.
- 4 Maximizing the Number of Activation Regions: 96 regions approximately matched the Theorem 5 prediction throughout MNIST training, with only slight increases as more labels were memorized.Counts averaged over 100 input-space lines and 40 independent runs.
- 4 Maximizing the Number of Activation Regions: Memorization increased activation-region counts with task difficulty, but never by more than a small constant factor from initialization.The experiments interpolate between generalization and memorization by randomizing fractions of MNIST labels.
- 4 Maximizing the Number of Activation Regions: 4608 regions were predicted for the depth-3, width-32 network memorizing random 2D points, and training increased counts more as memorization increased.At maximum memorization, the network essentially failed to learn, possibly because its capacity was insufficient.
- 4.2 The Effect of Initialization: Uniformly rescaling all biases preserves the total number of activation regions, whereas zero biases fall outside Theorem 5 and create fewer regions.The zero-bias behavior is discussed with activation-region visualizations in Appendix D.
- 4.2 The Effect of Initialization: 10^-2 learning rate caused learning to fail, while no tested learning rate produced more than a small constant multiple of the initial region count.Learning rate 10^-3 produced the maximum number of regions and was used in the other experiments.
- 4.2 The Effect of Initialization: Weight scaling did not strongly affect region counts at initialization or during training, although larger scales increased learned regions at a given accuracy and poor initialization impeded high accuracy.Weight scales were measured relative to variance 2/fan-in.
5 Conclusion
The paper finds that activation-region complexity learned in practice depends mainly on neuron count rather than depth and remains far below the theoretical maximum. This holds even for memorization tasks, while the practical utility of depth may lie more in optimization than expressivity.
- 5 Conclusion: Practical activation-region counts depend mainly on total neuron count, not depth, and remain far below the maximum possible.The conclusion concerns ReLU networks under well-behaved gradient and bias conditions.
- 5 Conclusion: The same limited complexity appears after training on memorization tasks, despite more regions being advantageous for fitting randomly labeled inputs.The stated results cover ReLU networks with no tied weights and biases and arbitrary connectivity.
- 5 Conclusion: The practical utility of depth may lie more in its effect on optimization than in expressivity.The paper states that its results do not directly influence architecture selection for deep ReLU networks.
A Experimental Design
The experiments compute activation regions on one- or two-dimensional input subsets by incrementally adding hidden-layer neurons and testing how each neuron cuts existing regions.
- A Experimental Design: Regions are computed on 1D or 2D input-space subsets by adding neurons from the first to last hidden layer.The procedure observes how each newly added neuron cuts existing regions.
- A Experimental Design: A region is tested for cutting by checking whether its corresponding linear function has zeros within the region.The check can be solved by determining whether all region vertices satisfy the relevant condition.
- A Experimental Design: Unless otherwise specified, training uses Adam with learning rate 10^-3 and batch size 128, with normal weights of variance 2/fan-in and biases of variance 10^-6.These are the default training and initialization settings.
B.1 Statement and Proof of Lemma 1 for General Piecewise Linear Activations
The section defines activation patterns and regions for continuous piecewise-linear networks and proves that every activation region is convex.
- B.1 Statement and Proof of Lemma 1 for General Piecewise Linear Activations: A piecewise-linear activation partitions its input into intervals, assigning an affine formula ϕ(t) = p_i t + q_i on each interval.The breakpoints satisfy −∞ = ξ_0 < ξ_1 < ··· < ξ_T.
- B.1 Statement and Proof of Lemma 1 for General Piecewise Linear Activations: An activation pattern assigns each neuron an alphabet element identifying the interval containing its pre-activation.The pattern is an element of {0, . . . , T}^#neurons.
- B.1 Statement and Proof of Lemma 1 for General Piecewise Linear Activations: An activation region is the set of inputs whose neuron pre-activations satisfy all interval constraints for one fixed activation pattern.Only non-empty regions are included in the network’s collection of activation regions.
- B.1 Statement and Proof of Lemma 1 for General Piecewise Linear Activations: Every activation region is convex for any parameter vector of the network.The proof proceeds by induction on depth, using intersections of convex sets.
B.2 Proof of Lemma 2
The proof establishes that connected components of the complement of neuron zero sets are exactly activation regions. It uses sign constancy on connected components and convexity of activation regions.
- B.2 Proof of Lemma 2: Continuous neuron functions have constant signs on each connected component after their zero sets are removed.Connectedness is preserved under continuous maps, so a neuron cannot change sign without vanishing.
- B.2 Proof of Lemma 2: Every such connected component is contained in an activation region because all neurons have definite on-or-off signs there.The activation pattern is therefore fixed throughout the component.
- B.2 Proof of Lemma 2: Activation regions are convex and therefore connected, so each contains exactly the connected component identified by its activation pattern.This proves equality rather than merely containment.
B.3 Proof of Lemma 3
The proof relates activation and linear regions through local sign stability, genericity, and the distinction between activation patterns and the functions they compute. Under generic parameters, distinct activation patterns produce distinct gradients.
- B.3 Proof of Lemma 3: A simple connected-component inclusion fact bounds the components of a larger set by those of a contained set.This reduces the relevant region-count inequality to comparing complements of neuron zero sets.
- B.3 Proof of Lemma 3: Continuity makes the on-off neuron collection locally constant away from zero sets, so the network is linear on a sufficiently small neighborhood.When no neuron is exactly at its bias, the restriction has a continuous gradient.
- B.3 Proof of Lemma 3: For almost every parameter vector, closures of linear regions are closures of unions of activation regions.The argument uses that neuron zero sets are generically piecewise-linear codimension-one manifolds and are open and dense away from nondifferentiability boundaries.
- B.3 Proof of Lemma 3: Distinct nonempty activation patterns generically correspond to different computed linear functions when every layer contains an active neuron.The proof uses path products and shows coincidences occur only on a measure-zero set of parameters.
C Statement and Proof of a Generalization of Theorem 5
The generalized theorem extends region counting to partial activation regions and bounds their expected number under distributional and gradient assumptions. The proof combines local geometry, convexity, volume estimates, and a combinatorial boundary count.
- Definitions: Partial activation patterns assign each neuron a sign −1, 0, or 1, with exactly k neurons assigned zero.The corresponding regions generalize ordinary activation regions by allowing k neurons to lie on their threshold.
- Geometric setup: Every r-partial activation region is convex, providing the geometric structure needed for counting intersections with a cube.Convexity ensures that nonempty bounded intersections have boundary vertices.
- Assumptions: Theorem 10 assumes continuous weight and conditional bias densities, together with a uniform gradient-moment bound.These conditions provide the genericity and volume control used by the expectation bound.
- Local geometry: Near a point where k neurons are exactly at threshold, the threshold set is locally an (nin − k)-dimensional hyperplane with probability one.This follows from the local piecewise-linear structure and general position.
- Expected bound: The proof combines discrete intersection sets, cube-skeleton volume bounds, and Fatou’s Lemma to obtain the expected region-count bound.The final estimate follows by combining the geometric and combinatorial inequalities.
- Combinatorial count: At most 2^k−r r-partial activation patterns can meet a sufficiently small neighborhood of a k-fold threshold point.All other neurons have fixed signs locally, leaving only the k threshold neurons variable.
D Zero Bias
The zero-bias case is handled separately because the general theorem’s bias-density condition fails at initialization. Zero-bias ReLU networks are positively scale equivariant, making their regions convex cones and allowing comparison with networks having arbitrary biases.
- Zero-bias scope: Theorem 5 does not directly apply to zero-bias networks because its bias-density condition fails, but the number of regions remains finite.The paper therefore analyzes zero bias separately rather than treating the theorem’s assumptions as satisfied.
- Comparison with biased networks: The total number of activation regions with all biases set to zero is no more than for the corresponding network with arbitrary biases.The proof constructs an injective mapping from zero-bias regions to regions of the biased network by rescaling inputs and biases.
- Scale equivariance: Zero-bias ReLU networks satisfy N(cx) = cN(x) for every c > 0.Positive homogeneity of each ReLU neuron yields scale equivariance for the whole network.
- Scale equivariance: Every activation region of a zero-bias network is a convex cone, since scaling any point in the region by c > 0 preserves its activation pattern.This geometric structure is central to the zero-bias region comparison.
- Sharper growth estimate: The zero-bias region count is expected to grow no faster than nin (#neurons)^(nin−1), rather than nin (#neurons)^nin.This sharper expectation comes from intersecting conical regions with facets of a unit hypercube.
- Small-bias networks: With very small biases, regions outside a small radius around the origin are infinite and approximately conical, while training enlarges the radius containing finite regions.A plane through the origin reveals this structure, whereas an offset plane does not.