Source-linked AI summary
Avoiding pathologies in very deep networks
David Duvenaud, Oren Rippel, Ryan P. Adams, Zoubin Ghahramani
TL;DR
The paper asks how to choose architectures and priors for deep networks without relying on a particular dataset, loss, or training method. It analyzes deep Gaussian processes as compositions of GP-drawn functions, characterizes their depth-dependent behavior, and proposes an input-connected architecture. Standard compositions retain only one effective direction in the deep limit, while input connections alleviate this pathology; the paper also analyzes deep kernels and GP dropout.
Problem
The paper examines how neural-network architecture choices affect the properties of priors over functions, a question important for understanding deep networks independently of datasets, losses, and training methods.
Method
The paper analyzes infinitely wide deep Gaussian processes, their derivatives and Jacobians, deep kernels, and Gaussian processes with dropout.
Results
Standard repeated compositions become invariant to all but one direction of variation, while connecting the input to every layer alleviates this pathology.
Takeaways & Limitations
Input-connected networks provide an alternate architecture whose deep Gaussian processes have well-behaved singular-value spectra even at 50 layers.
Takeaways & Limitations
Deep kernels may lack representation-learning capacity unless richly parameterized, and composing kernels with infinitely many hidden features may not have a closed form.
Abstract
from arXiv · showhide
Choosing appropriate architectures and regularization strategies for deep networks is crucial to good predictive performance. To shed light on this problem, we analyze the analogous problem of constructing useful priors on compositions of functions. Specifically, we study the deep Gaussian process, a type of infinitely-wide, deep neural network. We show that in standard architectures, the representational capacity of the network tends to capture fewer degrees of freedom as the number of layers increases, retaining only a single degree of freedom in the limit. We propose an alternate network architecture which does not suffer from this pathology. We also examine deep covariance functions, obtained by composing infinitely many feature transforms. Lastly, we characterize the class of models obtained by performing dropout on Gaussian processes.
1 Relating deep neural networks to deep GPs
The paper relates multilayer neural networks to Gaussian processes by viewing both as compositions of functions, distinguishing deep kernels from deep GPs and presenting two equivalent deep-GP architectures.
- 1.1 Definition of deep GPs: Deep GPs are distributions over functions formed by composing independently drawn functions from Gaussian-process priors.A deep function is written as f^(1:L)(x) = f^(L)(f^(L−1)(...f^(1)(x)...)).
- 1 Relating deep neural networks to deep GPs: Multilayer perceptrons and deep GPs both implement compositions of vector-valued functions, making composition properties relevant to deep-network analysis.The paper uses this correspondence to study neural-network priors without tying the analysis to a particular dataset, loss, or training method.
- 1.2 Single-hidden-layer models: An infinitely wide, one-hidden-layer MLP with unknown weights corresponds to a Gaussian process as the number of fixed features grows.The central limit theorem makes finite collections of function values approach a joint Gaussian, and a joint Gaussian distribution over function values defines a GP.
- 1.3 Multiple hidden layers: Multiple layers of fixed feature mappings produce a shallow GP with a deep kernel rather than a deep GP.Deep kernels use fixed features, so their representation-learning capacity can be limited unless the kernels are richly parameterized.
- 1.4 Two network architectures equivalent to deep GPs: Deep GPs correspond to two equivalent architectures: alternating weighted sums of fixed basis functions or finite layers with GP-distributed nonparametric activation functions.In the alternating view, each pair of layers forms a random rank-D^(ℓ) information bottleneck; integrating out the weights yields the nonparametric-activation view.
2 Characterizing deep Gaussian process priors
The analysis characterizes how deep GP derivatives and Jacobians behave with increasing depth. These results expose heavy-tailed one-dimensional derivatives and products of independent Gaussian matrices as mechanisms underlying deep-network pathologies.
- One-dimensional asymptotics: Deep GP derivatives become log-normal distributed as the number of layers grows.The analysis uses products of independently drawn layer derivatives and the central limit theorem.
- One-dimensional asymptotics: σ2/w2 = π/2 keeps the expected derivative magnitude constant regardless of depth.This normalization does not prevent the variance of the log-normal distribution from increasing without bound.
- One-dimensional asymptotics: Increasing depth makes derivatives very small almost everywhere, with rare but very large jumps.The heavy-tailed distribution also makes regions with extreme derivatives persist through subsequent layers.
- Distribution of the Jacobian: These Jacobian results enable analysis of deep-GP representations through the properties of products of independent Gaussian matrices.The Jacobian characterization supplies the mathematical basis for studying representational behavior at depth.
- Distribution of the Jacobian: A multivariate deep GP has a Jacobian formed as a product of independent Gaussian matrices with independently drawn entries.The result follows from the chain rule together with independent Gaussian partial derivatives under product kernels.
3 Formalizing a pathology
Deep GP representations lose effective dimensionality as depth increases. Their singular values, warped densities, and local feature maps increasingly reflect only one varying direction, limiting representations for higher-dimensional manifolds and multi-property decisions.
- Manifold representations: Good manifold representations should be invariant orthogonally while varying tangentially to preserve relevant information.Figure 4 visualizes this combination of noise robustness and information preservation.
- Jacobian singular values: As depth increases, the largest Jacobian singular value dominates, leaving usually one effective degree of freedom.The singular-value spectrum is used to count directions in which representations vary significantly.
- Warped densities: Deep-GP density models eventually concentrate observed density onto one-dimensional filaments.This makes sufficiently narrow deep compositions unsuitable for manifolds with underlying dimensionality greater than one.
- Feature maps: After 10 layers, deep-GP representations usually vary locally in only one input direction for changing representation values or crossing decision boundaries.Such representations are therefore likely unsuitable for decisions depending on more than one input property.
- Practical scope: With fixed hidden dimension D > 100, simulations kept singular-value spectra relatively flat for hundreds of layers.The authors therefore state that these pathologies are unlikely to severely affect relatively shallow, wide networks commonly used in practice.
4 Fixing the pathology
Input-connected networks address the loss of representational capacity in standard deep architectures by feeding the original input into every layer. Their deep-GP priors retain richer input dependence and well-behaved variation even at substantial depth.
- Architecture: Input-connected networks make each layer depend on both the previous layer’s output and the original input x.This architecture is proposed to fix the pathologies of standard deep compositions.
- Function behavior: Input-connected deep-GP functions remain relatively smooth in some regions while varying rapidly in others after many layers.The corresponding one-dimensional draws differ from the standard-connectivity behavior shown in Figure 3.
- Analysis: Analyzing these pathologies is easier with a well-defined function prior than with finite Bayesian neural networks, where hidden-unit saturation can create degenerate mappings.The function-prior view supports architecture analysis independently of any dataset, loss function, or training method.
- Representations: Input-connected priors produce more complex densities and representations that can depend on multiple or all aspects of the input.Figures 10 and 12 illustrate richer representational behavior than one-dimensional filamentary mappings.
- Jacobian behavior: Input-connected deep GPs retain sensitivity to several input directions, with singular values more likely to remain similar even at 25 and 50 layers.This indicates better-conditioned local variation than the standard architecture.
5 Deep kernels
Deep kernels arise by composing fixed feature maps, but repeated composition can collapse their covariance structure. Input connections avoid this collapse in one construction, while other connectivity patterns converge to white noise.
- Construction: Deep kernels are formed by composing fixed feature maps, extending kernel constructions beyond local squared-exponential behavior.The paper frames kernel composition as a route to non-local kernels and implicit representation learning.
- Construction: Composing the squared-exponential kernel with another kernel has a closed form expressed exactly through evaluations of the original kernel.This gives an exact representation of the composed kernel without explicitly constructing the other kernel’s feature map.
- Degeneracy: Repeated squared-exponential composition converges to constant covariance, corresponding to a degenerate prior on constant functions f(x) = c.The result holds when the output variance is normalized to k(x, x) = 1.
- Input-connected kernels: Input-connected deep kernels concatenate the composed feature vector at each layer with the original input x to overcome this degeneracy.The resulting recurrence has no closed form and is related to the Lambert W-function.
- Input-connected kernels: In one input dimension, the input-connected limiting kernel resembles an Ornstein-Uhlenbeck covariance with lighter tails, producing locally fractal, nondifferentiable samples.Figure 13 shows the kernel and corresponding GP-prior samples at different depths.
- Alternative limits: Output-connected and all-subsequent-layer connectivity instead converge to the white-noise kernel k(x, x′) = δ(x, x′) at infinite depth.These architectures therefore have a different infinite-depth limit from the input-connected construction.
6 Dropout in Gaussian processes
The paper characterizes Gaussian-process priors induced by dropout on hidden features and inputs. Hidden-layer dropout is ineffective in infinitely wide MLPs after variance rescaling, whereas input dropout yields mixtures and additive-GP interpretations.
- Dropout on infinitely-wide hidden layers has no effect: In infinitely wide MLPs, dropout on hidden features leaves the resulting model unchanged except for output-variance rescaling.Dividing all weights by √p restores the initial variance.
- Dropout on infinitely-wide hidden layers has no effect: The result follows from the central limit theorem and therefore depends on weight mean and variance rather than the exact weight distribution.
- Dropout on infinitely-wide hidden layers has no effect: No individual hidden feature affects the output substantially in the infinite-width limit, explaining why rescaled dropout has no effect.
- Dropout on infinitely-wide hidden layers has no effect: This conclusion does not extend to neural networks with finitely many hidden features and Gaussian-distributed weights.
- Dropout on inputs gives additive covariance: Input dropout produces a mixture of 2^D Gaussian processes, each depending on a different subset of the input variables.
- Dropout on inputs gives additive covariance: For stationary product kernels, input dropout corresponds to independent spike-and-slab priors on lengthscales, with probability mass 1/2 at ℓ_d = ∞ for each dimension.
- Dropout on inputs gives additive covariance: Input dropout yields a non-Gaussian distribution sharing the first two moments of a GP with additive covariance, whose component functions may support non-local extrapolation.
7 Related work
Related work established deep Gaussian processes, deep kernels, and alternative architectures for preserving information or analyzing deep compositions. The paper connects its pathology analysis to prior work on random deep networks, recurrent gradients, and kernel compositions.
- Deep-network pathologies: Neal analyzed arbitrarily deep Bayesian neural networks and proposed input connections to prevent degenerate priors caused by infinitely deep networks without extra connections.
- Deep Gaussian processes: Hierarchical GP-LVMs provided an early practical use of compositions of multiple Gaussian processes to map time into observations.
- Deep Gaussian processes: Damianou and Lawrence introduced the term “deep Gaussian processes” for supervised and unsupervised compositions and developed variational inference and automatic-relevance analyses.
- Related GP and Bayesian-network models: Other work studied latent-input GP regression, unbounded-width Bayesian networks, and GP regression networks based on matrix products rather than function composition.
- Related architectural problems: Deep-density networks, recurrent architectures, and LSTM units addressed information-preservation or gradient-stability problems related to deep-network pathologies.
- Deep kernels: Prior kernel studies derived composition rules, built recurrent deep kernels, and reported competitive performance for deep arc-cosine kernels in SVM vision applications.
- Empirical and dynamical analyses: Earlier analyses found that random-weight MLP performance degrades with depth, while other work examined architecture effects and training dynamics in deep linear or convolutional networks.
- Implementation: The authors released source code for reproducing figures and visualizing analogous mappings using neural networks instead of Gaussian processes.
8 Conclusions
The conclusions use explicit priors to analyze assumptions in multilayer neural networks. They identify loss of representational directions under repeated composition, propose input connections as mitigation, and extend the analysis to deep kernels and GP dropout.
- Conclusions: The paper uses well-defined priors to examine assumptions made by neural-network models without tying the analysis to a particular dataset or training procedure.
- Conclusions: Repeated composition of independent functions makes representations invariant to all but one direction of variation, revealing a deep-network pathology.
- Conclusions: Connecting the input to each layer alleviates the loss of representational directions caused by repeated composition.
- Conclusions: The paper also examines deep kernels formed from arbitrarily many fixed-feature compositions and derives models from dropout applied to Gaussian processes.
- Conclusions: The GP-dropout analysis provides a tractable approximation to exact dropout in Gaussian processes.