Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity

Amit Daniely; Roy Frostig; Yoram Singer

Source-linked AI summary

Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity

Amit Daniely, Roy Frostig, Yoram Singer

arXiv:1602.05897v2Feb 18, 2016cs.LGcs.AIcs.CCcs.DSstat.ML

TL;DR

Neural-network learning remains poorly understood, especially regarding why optimization finds good weights and how initialization and architecture affect learning. The paper develops a duality with compositional kernels and shows that common random initialization yields rich representations, making last-layer optimization a convex route to approximating functions in the associated kernel space.

Problem
Neural-network learning achieves strong empirical results, but the roles of optimization, architecture, activations, and random initialization remain unclear.
Method
The paper defines computation skeletons and analyzes the dual kernel spaces induced by network architectures and random initial representations.
Results
Random initialization produces representations sufficiently rich to approximately express all functions in the skeleton-induced hypothesis class through last-layer weight tuning.
Takeaways & Limitations
The dual view provides design principles for architectures, initialization schemes, activation functions, and convex proxies for architecture selection.
Takeaways & Limitations
The analysis mostly covers fully connected and convolutional layers, leaving components such as max-pooling and recursive neural-network elements for future work.

Abstract

from arXiv · show

We develop a general duality between neural networks and compositional kernels, striving towards a better understanding of deep learning. We show that initial representations generated by common random initializations are sufficiently rich to express all functions in the dual kernel space. Hence, though the training objective is hard to optimize in the worst case, the initial weights form a good starting point for optimization. Our dual view also reveals a pragmatic and aesthetic perspective of neural networks and underscores their expressive power.

1 Introduction

The paper studies why neural networks can learn effectively despite difficult optimization, focusing on random initialization and a duality between architectures and compositional kernels.

Random initialization: Random initialization produces representations rich enough to approximately express every function in the skeleton-induced hypothesis class H.All functions in H can be approximated by tuning only the last-layer weights, a convex optimization task.
Dual perspective: The framework establishes a tight connection between neural network architectures and their dual kernel spaces.This dual perspective generalizes previous constructions and supports design principles for neural networks.
Architectural design: Convolutional-network duals appear better suited to vision and acoustic tasks than fully connected-network duals.
Architectural design: The framework yields an initialization scheme similar to common practice but with a small correction.
Architectural design: Modifying activation functions can replace two consecutive fully connected layers with one while preserving the network’s dual kernel.
Activation functions: The ReLU dual kernel is expressive, and random initialization approximates it even when the initialization scale changes moderately.
Activation functions: For any nonlinear activation, the dual kernel of a fully connected network converges to a degenerate form as depth becomes very large.
Optimization: Optimizing last-layer weights can serve as a convex proxy for choosing among architectures before training.

2 Related work

Prior work frames neural-network understanding around expressivity, generalization, and optimization, while this paper builds on earlier connections between composed kernels and neural networks.

Research questions: Neural-network research commonly asks what functions networks express, when empirical loss predicts population loss, and why efficient algorithms find good weights.
Expressivity: Complexity theory and Barron’s theorem show that broad function classes can be expressed by moderately sized or even two-layer networks.
Compositional kernels: Composed kernels have appeared repeatedly in machine-learning research, including work on deep kernel composition.
Compositional kernels: Prior studies examined the correspondence between random-weight neural networks and kernels, including a formal connection for the RBF kernel.

3 Setting

The paper formalizes neural networks, kernels, learning objectives, and inputs for analyzing representations and their dual kernel spaces.

Notation: The setting also introduces vector and matrix notation, norms, positive-part and indicator notation, and the sphere representation used for inputs.
Input space: Inputs are sequences of n unit vectors, with encodings covering binary features, continuous signals, and categorical variables.
Supervised learning: Supervised learning seeks a predictor with low population loss, using empirical loss on an i.i.d. sample as a proxy.
Neural networks: A neural network is a weighted directed acyclic graph whose internal units use activation functions and whose weights determine initial-weight distributions.
Neural networks: The representation removes the output layer, and learning searches for weights with small empirical loss.
Kernel learning: A kernel induces a reproducing-kernel Hilbert space, and kernel learning minimizes a convex objective that can often be efficiently optimized.

4 Computation skeletons

Computation skeletons compactly represent feed-forward structure, each inducing a family of neural-network architectures and a corresponding compositional kernel space. Their realizations connect replication parameters and network layers to kernel constructions.

A single skeleton represents multiple networks sharing its topology while also defining a corresponding kernel space.
A computation skeleton is a DAG with activation-labeled non-input nodes that compactly describes feed-forward computation.
Skeletons distinguish abstract topology from concrete weighted networks and are naturally more compact, with irreducibility requiring distinct input neighborhoods.
From computation skeletons to neural networks: A realization N(S, r, k) replicates each internal skeleton node r times and adds k identity-activated output neurons.Input neurons use weight 1/d, replicated internal neurons use weight 1/r, and output neurons use weight 1.
From computation skeletons to neural networks: The replication parameter corresponds to channels in convolutional networks and hidden units in fully connected layers.
From computation skeletons to reproducing kernels: A skeleton also induces a normalized compositional kernel, whose function norm is small for functions formed by simple compositions matching the skeleton structure.
From computation skeletons to reproducing kernels: Dual activations define kernel transformations through Gaussian-correlated expectations, and recursively applying them across skeleton nodes yields the final kernel.

5 Main results

The main results show that random initialization produces representations whose empirical kernels and last-layer function classes approximate those induced by the computation skeleton. The guarantees cover bounded smooth activations and ReLU, with explicit replication requirements and acknowledged large polynomial bounds.

Randomly initialized representations approximate the skeleton kernel, while bounded linear functions of those representations approximate bounded-norm functions in the associated RKHS.The analysis restricts attention to bounded activations with bounded first and second derivatives, or ReLU.
Theorems 2 and 3 establish empirical-kernel approximation for C-bounded and ReLU activations, respectively.
For fixed activations and logarithmic skeleton depth, the required replication bounds are polynomial; for ReLU, the bound has quadratic depth dependence under ϵ ≤ 1/depth(S).
The approximation results are formulated probabilistically over random initialization and use a Lipschitz loss with distribution-dependent approximation error.
The polynomial replication bounds in Theorems 4 and 5 are rather large, and improving or proving their optimality is left for future work.

6 Mathematical background

The mathematical background establishes the correspondence among kernels, Hilbert-space embeddings, positive-definite functions, and activation duals. Hermite expansions provide the basis for analyzing activation functions and their dual kernels.

Reproducing kernel Hilbert spaces: Every kernel has a unique reproducing kernel Hilbert space, and every kernel space arises from a kernel.
Kernel embeddings: A function is a kernel exactly when it is an inner product between embeddings of inputs into some Hilbert space.
Positive definite functions: A continuous function on [−1, 1] is positive definite exactly when applying it to inner products on every sphere produces a kernel.
Positive definite functions: Many machine-learning kernels restricted to the unit sphere correspond to positive-definite functions, including Gaussian and polynomial examples.
Hermite polynomials: Normalized Hermite polynomials form an orthonormal basis for square-integrable activations under the Gaussian measure.
Hermite polynomials: Each activation has a unique Hermite expansion, with convergence in ℓ2 under the Gaussian measure.

7 Compositional kernel spaces

Compositional kernel spaces are built by recursively transforming and combining the spaces associated with a network’s input nodes. Direct averaging combines incoming kernels, while activation-specific extension produces the kernel space at each node.

Recursive construction: The recursion generates a kernel space from other kernel spaces by applying a node operation at every vertex.This operation is decomposed into direct averaging followed by extension according to the node’s dual activation.
Recursive construction: A computation skeleton propagates input-node kernel spaces through the network structure to construct the final space H_S.Each input node corresponding to x_i induces linear functions, and the output node’s kernel defines H_S.
Direct average: Direct averaging forms a valid kernel by averaging the kernels of incoming spaces, and preserves normalization when all inputs are normalized.The proof uses that averages of positive semidefinite matrices remain positive semidefinite.
Kernel validity: The construction is valid because the transformed kernel admits an inner-product representation through a feature mapping.This representation allows the kernel properties to follow from the corresponding Hilbert-space theorem.
Activation extension: Activation extension transforms a kernel κ into κ′(x, x′) = µ(κ(x, x′)), where µ is the function associated with the activation.The resulting function is a kernel, and the extended space is normalized exactly when µ is normalized.

8 The dual activation function

The dual activation maps an activation function to the kernel it induces on correlated Gaussian inputs. The section derives this correspondence for several activations and analyzes how repeated composition behaves in very deep fully connected networks.

Duality: Every positive definite function is the dual of some activation, establishing a broad correspondence between activations and kernel functions.The dual activation framework therefore includes polynomial and other positive-definite kernels.
Examples: The exponential activation induces the RBF kernel when restricted to the d-dimensional sphere.The dual is obtained by solving the differential equation for the dual activation and enforcing normalization.
Examples: Hermite activation h_n has dual activation ˆh_n(ρ) = ρ^n, yielding the polynomial kernel.The result follows directly from the Hermite expansion characterization of the dual.
Examples: For ReLU, the coefficients through degree 4 yield an approximation with less than 1% error.The cumulative coefficient sums through degrees 0, 1, and 2, and through degrees 4 or 6, are 0.9774, 0.9907, and 0.9947 respectively.
Deep fully connected networks: For any non-linear activation other than the identity or its negation, repeated dual-activation composition converges pointwise to a degenerate kernel form.The iterates move toward a unique fixed point α_σ for correlations in (−1, 1).

9 Proofs

The proofs establish regularity conditions for dual activations and concentration guarantees for randomly initialized networks. These results support the claim that finite-width initial representations approximate their compositional kernel counterparts under stated assumptions.

Well-behaved activations: C-bounded activations are (C^2, 2C^2, γ)-decent for every γ ≥ 0.Their extended dual activation is Lipschitz, while concentration follows from Hoeffding’s bound.
Well-behaved activations: The normalized ReLU activation is decent under a depth-dependent covariance condition.The proof combines concentration for sub-exponential variables with a Lipschitz bound for the extended dual activation.
Initialization concentration: For decent activations, well-initialized incoming nodes imply that a node is well-initialized with probability at least 1 − δ/|S|.The argument uses empirical covariance concentration and the Lipschitz property of the extended dual activation.
Initialization concentration: Applying the node-wise guarantee inductively yields simultaneous well-initialization across the skeleton with probability at least 1 − δ.Nodes are ordered by depth, and the proof propagates the conditional guarantee from inputs to the output.

10 Discussion

The discussion connects initialization, architecture, and activation choices to what neural networks can learn, while identifying open limits of the framework and its quantitative guarantees.

Role of initialization and training: Random initialization is presented as a potentially important contributor to neural-network learning success, though its full role remains an open question.The paper cites empirical evidence that initialization similar to the analyzed scheme is crucial, but leaves the broader question for future research.
The role of architecture: Skeletons and compositional kernel spaces let the theory reason about functions networks can actually learn, not merely express.This perspective may explain past architectural choices and guide future ones, including localized processing for object recognition.
The role of architecture: Convolutional skeletons encode local-to-global processing, and the result helps explain why convolutional networks outperform fully connected networks in vision applications.The framework makes the intuition behind convolutional architectures compatible with provable learnability.
The role of other architectural choices: Two consecutive fully connected layers can be replaced by one layer with composed dual activation while preserving the corresponding compositional kernel space.The observation may allow saving a whole layer in the corresponding networks.
The role of other architectural choices: ReLU combines an expressive dual kernel, random-initialization approximation in deep networks, and positive homogeneity that makes initialization robust to variance rescaling.Scaling the random-weight variance by a constant leaves the generated representation and space H_w unchanged up to scaling.
Future work: The framework has mainly analyzed fully connected and convolutional layers, while max-pooling, recursive components, dropout, and batch normalization remain for future study.The paper also leaves open whether dependencies on depth, 1/ϵ, R, and L can be quantitatively improved.