Source-linked AI summary
Randomly initialized autoencoders: fixed points and edge-of-chaos
Leonid Berlyand, Roman Sarapin, Yitzchak Shmalo, Victor Slavin, Sasha Sodin
TL;DR
Bottleneck autoencoders cannot realize the identity on all of R^N, motivating a stability analysis based on contraction and edge-of-chaos behavior. The paper develops local and global edge-of-chaos notions using random-matrix spectral methods and Gaussian-process inequalities, and proves conditions yielding a unique stable zero fixed point with basin R^N.
Problem
Because bottleneck autoencoders cannot implement the identity on all of R^N, their stability and contraction behavior require characterization beyond exact reconstruction.
Method
The paper defines local and global edge-of-chaos criteria for small and arbitrary perturbations, analyzing them with random-matrix spectral techniques and Gaussian-process comparison.
Results
Under the stated width and initialization conditions, the autoencoder has a unique stable fixed point 0 with basin of attraction R^N.
Takeaways & Limitations
Local and global edge-of-chaos distinguish autoencoder stability against small perturbations from stability against arbitrary perturbations.
Takeaways & Limitations
The fixed-point guarantee requires specific layer-width conditions, σ < β_α, and sufficiently large depth L.
Abstract
from arXiv · showhide
In this paper we study autoencoders, a special class of deep neural nets (DNNs) whose performance can be characterized via their fixed points. This perspective naturally raises questions of existence, stability, and basins of attraction of these fixed points. These questions are addressed via the contractive properties of autoencoders, and are closely related to the notion of edge-of-chaos. Edge-of-chaos (EoC) is an important notion in the theory of DNNs. It describes the critical regime separating ordered and chaotic signal propagation through a randomly initialized network. Initialization at or near this critical regime offers several theoretical and practical advantages, including stability of the network w.r.t. perturbations of the input. EoC was previously introduced for broad classes of neural networks using mean-field averaging methods. In this paper we modify the notion of EoC for the study of autoencoders. Specifically, we introduce local and global EoC for autoencoders that control local (small) and global (arbitrary) perturbations of the input respectively. The study of stability of autoencoders falls within the scope of nonlinear problems in Random Matrix Theory (RMT). Our analysis of local EoC is based on spectral techniques of RMT, whereas global EoC is studied by employing Sudakov-Fernique inequality for Gaussian processes.
1 Introduction
The paper studies autoencoders through the existence, stability, and attraction basins of their fixed points under random Gaussian initialization. It introduces local and global edge-of-chaos notions to characterize contraction under small and arbitrary perturbations.
- Autoencoder structure: Autoencoders map inputs through a lower-dimensional latent representation and decode them to reconstruct the original data.The input and output dimensions are equal, while the latent dimension satisfies n ≪ N.
- Fixed points and contraction: Fixed points represent inputs reconstructed exactly, while other inputs in their attraction basins are restored approximately.The paper therefore treats contraction criteria as important for autoencoder design.
- Random initialization: The study examines how centered Gaussian weight initialization and its variance determine fixed-point existence, stability, and attraction basins.Training can create fixed points through iterative weight adjustment, but a given initialization need not have them.
- Local and global edge-of-chaos: Local edge-of-chaos characterizes contraction and stability for small perturbations, whereas global edge-of-chaos addresses contraction over the entire data domain under arbitrary perturbations.The local notion is linked to input-output Jacobians, while the global notion concerns whole-domain stability.
- Global stability results: For ReLU and LeakyReLU autoencoders without biases, sufficiently controlled initialization can yield a unique stable fixed point at 0 with basin of attraction R^N.The stated global-stability results include high-probability guarantees and apply in particular to sufficiently large depth under the corollary conditions.
2 Proof of Theorem 1.2, ReLU case α = 0
The ReLU proof reduces the required supremum bound to a Gaussian-process comparison problem. It constructs a majorizing process, applies Sudakov–Fernique to control the expectation, and uses Borell–TIS to control deviations with high probability.
- Concentration bound: Borell–TIS is then applied to show that the Gaussian-process supremum remains close to its expected value with high probability.The proof estimates the relevant variance term before invoking the concentration inequality.
- Proof setup: The proof considers an L-layer, bias-free ReLU autoencoder with Gaussian initialization and proceeds in four steps.The steps establish a Gaussian-process upper bound, construct a majorizing process, apply Sudakov–Fernique, and then use Borell–TIS.
- Gaussian-process reduction: The norm-gain analysis expresses the relevant quantity through a Gaussian process indexed by recursively defined vectors associated with a single input.Dependencies between the recursively generated vectors are discarded to obtain a computable upper bound, and the one-layer case is exact.
- Gaussian-process comparison: A Gaussian process Q_u is constructed to majorize P_u by comparing the expected squared increments for arbitrary index pairs.The majorization condition is verified using independence of the random matrices and Gaussian-vector identities.
- Expectation bound: Sudakov–Fernique bounds the expected supremum of P_u by that of Q_u, whose supremum can be computed and bounded using Gaussian-vector estimates.The resulting bound uses Cauchy–Schwarz, with the positive-part norm estimate asymptotically sharp as dimension grows.
3 Proof of Theorem 1.1
The proof of Theorem 1.1 analyzes the singular spectrum of the input-output Jacobian through the eigenvalue distribution of its associated nonnegative matrix. It proceeds by deriving a moment-generating-function equation, extracting the spectrum’s right edge, and using its limits to establish local edge-of-chaos behavior.
- Spectral reduction: Because the Jacobian is generally non-symmetric, its norm is determined by the singular values, equivalently the eigenvalues of the associated matrix M(N).The analysis therefore focuses on the maximal eigenvalue and its double-limit behavior.
- Proof strategy: The proof has three steps: derive an infinite-width moment-generating-function equation, obtain the right spectral edge, and calculate its infinite-depth limits.The generalized equation applies to ReLU activation with proportional layer widths.
- Consequences: The proof derives an asymptotic right-edge formula for arbitrary proportional layer widths, an exact same-width expression, and local edge-of-chaos in the limit L →∞.The same-width case permits an explicit solution of the defining equation.
- Right-edge analysis: For L > 2, the moment-generating-function equation cannot be solved explicitly, so the proof obtains the right edge from the function’s radius of convergence and its singularities.The relevant measure is supported on the nonnegative real line because M(N) is nonnegative.
4 Appendix · A: Proof of Theorem 1.2, LeakyReLU case 0 < α < 1
The appendix proves Theorem 1.2 for LeakyReLU with 0 < α < 1 by following the α = 0 proof structure while modifying its key estimates. The argument uses neuron-vector recurrences, Gaussian-process comparison via Sudakov-Fernique inequality, and Gaussian concentration.
- A: Proof of Theorem 1.2, LeakyReLU case 0 < α < 1: The autoencoder is an L-layer, bias-free composition of LeakyReLU-transformed Gaussian weight matrices.Each layer applies Φ^(k)(s) = ϕ_α(W^(k)s), with independently Gaussian-initialized weights.
- A: Proof of Theorem 1.2, LeakyReLU case 0 < α < 1: The LeakyReLU proof for 0 < α < 1 follows the same steps as the α = 0 proof, with specified modifications.The appendix explicitly frames the proof as structurally identical to the earlier case while identifying differences.
- A: Proof of Theorem 1.2, LeakyReLU case 0 < α < 1: The proof defines neuron vectors s^(k), their norms r^(k), and a recurrence for the associated u_k quantities.These objects provide the layerwise variables used in the subsequent bounds.
- A: Proof of Theorem 1.2, LeakyReLU case 0 < α < 1: Step 1 bounds the relevant network quantity over a product of unit spheres using a Gaussian process {P_u} indexed by u ∈ S.The index vector is u = (u_0, …, u_L), and S is the product of the corresponding spheres.
- A: Proof of Theorem 1.2, LeakyReLU case 0 < α < 1: Step 2 constructs an independent Gaussian process Q_u that majorizes P_u according to the required comparison relation.The construction uses independent Gaussian vectors g_0, …, g_{L−1} and h_1, …, h_L with standard normal entries.
- A: Proof of Theorem 1.2, LeakyReLU case 0 < α < 1: Step 3 bounds E{sup Q_u} by estimating its summands and applying Lemma 4.1 for independent Gaussian vectors.Lemma 4.1 assumes ρ ≥ 0, µ ≥ 0, and α ∈ (0, 1); Sudakov-Fernique then transfers the bound from Q_u to P_u.
- A: Proof of Theorem 1.2, LeakyReLU case 0 < α < 1: Step 4 completes the proof through Gaussian concentration, using an argument stated to be analogous to Subsection 2.6.This follows the completion statement at the end of the appendix proof.
B: Singularities of implicit analytic function and radius of convergence
The section identifies the radius of convergence R of mML,σ2(z) and analyzes its singularity using standard singularity analysis for implicitly defined analytic functions. Nonnegative support yields a singularity at z = R, while an implicit-function continuation argument forces a critical solution at the limiting value m∗.
- Radius of convergence: R is the radius of convergence of the moment generating function mML,σ2(z), analyzed through singularity methods for implicitly given analytic functions.L,σ2 is an N × N nonnegative matrix, and νML,σ2 is its limiting NCM.
- Singularity at R: Because supp νML,σ2 ⊂ [0, +∞), the coefficients ak are nonnegative, so Vivanti-Pringsheim yields a singularity at z = R.The function mML,σ2(z) is also monotone increasing on [0, R).
- Limiting solution: As z approaches R from below, mML(z) converges to a finite positive limit m∗, and passing to the limit in the defining equation gives a solution at (m∗, R).The limit satisfies m∗ ∈ (0, +∞).
- Criticality condition: If ∂m(m∗, R) ≠ 0, the implicit function theorem provides a unique analytic continuation near z = R with m(R) = m∗.This continuation contradicts the established singularity at z = R.
- Criticality condition: Therefore, m = m∗ and z = R form a solution of the critical system, with m∗ > 0.The conclusion follows because the nonzero derivative case would contradict singularity of mML,σ2(z) at R.
C: Proof of Lemma 4.1
The section completes the proof of Lemma 4.1.
- C: Proof of Lemma 4.1: The proof of Lemma 4.1 is completed.