PaperlayerView exact revision on arXiv

Source-linked AI summary

How Deep Are Deep Gaussian Processes?

Matthew M. Dunlop, Mark A. Girolami, Andrew M. Stuart, Aretha L. Teckentrup

arXiv:1711.11280v2math.STstat.TH

TL;DR

Direct Gaussian processes have limits in expressing some phenomena, motivating hierarchical Gaussian constructions. This paper develops a common framework for deep Gaussian processes and analyzes their depth through Markov-chain ergodicity. Its results show that effective depth requires care, while a few layers can still be useful for inference.

  • Problem

    Direct Gaussian processes have limits in expressing some phenomena, including in sparse-data settings, motivating hierarchical Gaussian constructions.

    Source: §1.1 · p. 2

  • Method

    The paper develops a common framework for recursively constructed deep Gaussian processes, studies their Markov-chain ergodicity, and presents function-space inference using Metropolis-within-Gibbs sampling.

    Source: §1.2 · p. 2, §1.3 · p. 4, §4.2.1 · p. 28

  • Results

    The analyzed processes can become ergodic or converge to trivial limits, so designing processes with significant depth requires care, although a few layers can be useful for inference.

    Source: §5.2 · p. 39

  • Takeaways & Limitations

    For ergodic constructions, prior expressive power does not increase beyond a certain number of layers, supporting the use of a moderate layer count.

    Source: §5.2 · p. 39

    The analysis concerns the prior process u_n, not the conditioned process u_n|y, whose practical layer needs depend on the observed data's information content.

    Source: §5.2 · p. 39

Abstract

from arXiv · show

Recent research has shown the potential utility of Deep Gaussian Processes. These deep structures are probability distributions, designed through hierarchical construction, which are conditionally Gaussian. In this paper, the current published body of work is placed in a common framework and, through recursion, several classes of deep Gaussian processes are defined. The resulting samples generated from a deep Gaussian process have a Markovian structure with respect to the depth parameter, and the effective depth of the resulting process is interpreted in terms of the ergodicity, or non-ergodicity, of the resulting Markov chain. For the classes of deep Gaussian processes introduced, we provide results concerning their ergodicity and hence their effective depth. We also demonstrate how these processes may be used for inference; in particular we show how a Metropolis-within-Gibbs construction across the levels of the hierarchy can be used to derive sampling tools which are robust to the level of resolution used to represent the functions on a computer. For illustration, we consider the effect of ergodicity in some simple numerical examples.

1. Introduction

The introduction motivates hierarchical Gaussian constructions as a way to represent phenomena not readily captured by direct Gaussian processes, and sets out a common framework for studying deep Gaussian processes. It connects their depth to Markov-chain ergodicity and previews theoretical results and function-space inference methods.

1.2 Our Contribution

  • The paper places hierarchical Gaussian-process constructions in a common framework that supports the construction of a wide class of deep Gaussian processes.Source: §1.2 · p. 2
  • Deep Gaussian processes have a Markovian structure, so ergodicity or non-ergodicity can characterize effective depth; for ergodic constructions, depth corresponds to mixing time.Source: §1.2 · p. 2, §1.4 · p. 4
  • A Metropolis-within-Gibbs construction across hierarchy levels yields inference tools robust to the function representation's computer resolution.Source: §1.2 · p. 3
  • Theorem 4 shows that composition-based deep Gaussian processes can become approximately constant with enough layers, a pathology avoidable by widening hidden layers or connecting each layer to the input.Source: §1.2 · p. 3
  • Theorem 8 establishes ergodicity for a discretized class with non-stationary covariance functions, while Theorem 14 gives a similar result on function space for covariance-operator constructions.Source: §1.2 · p. 3
  • Theorem 16 shows that iterated-convolution processes either converge weakly to zero or diverge as layers increase, limiting their flexibility for inference.Source: §1.2 · p. 3

1.3 Overview

  • The framework defines deep Gaussian processes as conditionally Gaussian function sequences, with zero-mean iterations expressible through a layer-dependent linear operator applied to Gaussian noise.Source: §1.3 · pp. 3–4

1.4 Notation

  • The processes can be viewed as Markov chains on a Hilbert space of functions, a structure used to analyze their ergodicity or lack of ergodicity.Source: §1.4 · p. 4

2. Four Constructions

Four constructions are presented as instances of a shared deep-GP framework, spanning composition, covariance-function and covariance-operator designs, and convolution.

2. Four Constructions

  • The section presents four deep-Gaussian-process constructions within a common framework and notes that readers can design further constructions.The examples include composition, covariance-function, covariance-operator, and convolution constructions.Source: §2 · p. 4

2.1 Composition

  • Composition layers evaluate an independent Gaussian process after mapping the preceding function through F, yielding a linear operator L(u) and covariance L(u)ΣL(u)*.The two-layer case with identity F was introduced by Damianou and Lawrence (2013).Source: §2.1 · p. 5

2.2 Covariance Function

  • Under Proposition 1's conditions, the constructed non-stationary correlation is positive definite; allowing Σ=0 requires finite-dimensional processes because no function-valued Gaussian process results.The hierarchy uses non-stationary correlations parameterized through another Gaussian process; the analysis also confines pointwise variance to a simpler setting for exposition.Source: §2.2 · pp. 6–7

2.3 Covariance Operator

  • A precision-operator construction uses u-dependent multiplication to modify Gaussian covariance; choosing differential P permits SPDE-based sampling and links operator order to sample regularity.The analysis is confined to the described setting for exposition, while broader generalizations are noted.Source: §2.3 · p. 8

2.4 Convolution

  • The convolution construction iterates functions on [0, 1]^d using independent centered Gaussian random functions, with periodic extension assumed to define the convolution.Source: §2.4 · pp. 8–9

3. The Role of Ergodicity

This section examines ergodicity in deep Gaussian process iterations and its implications for effective depth, including settings where iterations become spatially constant or unstable at large depth. It establishes results for several construction classes and uses invariant-distribution, coupling, and Fourier-mode arguments to characterize their behavior.

3. The Role of Ergodicity

  • In many cases, effective depth is limited by mixing time; non-ergodic iterations may instead blow up, so achieving substantial depth requires care.Ergodicity can also be trivial when the process converges to a constant.Source: §3 · p. 9

3.1 Composition

  • For squared exponential composition, σ2 < w2/m yields convergence to a spatial constant, so piecewise-constant initial conditions make ergodicity trivial.Piecewise-constant functions remain piecewise constant at every iteration, and ergodicity requires the discontinuity set to vanish.Source: §3.1 · pp. 9–10, Proof · p. 11

Proof

  • With a general transformation F, convergence to a constant still follows if σ2 < w2/(∥F′∥∞m); increasing layer width makes this triviality less likely.The condition is met when the length scale is large, ∥F′∥∞ is small, or the amplitude is small.Source: Proof · p. 11
  • Connecting every layer to the input preserves the Markovian structure but prevents the same convergence proof, offering intuition for the greater stability of such architectures.The proof’s bound no longer vanishes as iteration depth grows.Source: Proof · p. 11

3.2 Covariance Function

  • Under Assumptions 1 and 2, finite-dimensional covariance-function constructions are ergodic, with a unique invariant measure and geometric convergence in total variation.The result applies to the Markov chain on R^N.Source: §3.2 · pp. 13–14

3.3 Covariance Operator

  • Under Assumptions 3, the covariance-operator chain on H = L2(D; R) has a unique invariant distribution and is ergodic.The theorem also gives a total-variation convergence bound for any initial state in H.Source: §3.3 · pp. 15–16

Proof

  • An invariant distribution plus global one-step minorization lets a coupling argument establish the preceding result with the same ε.The invariant distribution follows from tightness; the minorization applies to the one-step transition probabilities.Source: Proof · p. 18

3.4 Convolution

  • Fourier series and the law of large numbers make the long-time behavior of the convolution iteration analytically characterizable.The iteration is treated as a random map on H = L2(D; C), with D = (0, 1)^d.Source: §3.4 · p. 18
  • When |λ_k|^2 < 2e^γ, every Fourier coefficient tends to zero almost surely (u_n ⇀ 0); when |λ_k|^2 > 2e^γ, |u_n(k)|^2 diverges.The result assumes C is a positive, symmetric, trace-class negative fractional power of a differential operator.Source: §3.4 · pp. 19–20
  • For 1 < |λ_k|^2 < 2e^γ, |û_n(k)|^2 converges to zero almost surely but diverges in mean square.Thus almost-sure decay in this range does not imply mean-square convergence.Source: §3.4 · p. 20
  • With Brownian-bridge noise on (0, 1), u_n ⇀ 0 almost surely; the preceding results also hold for mean-zero Neumann-boundary Gaussian processes with negative-Laplacian precision.For the Brownian bridge, the eigenvalues satisfy |λ_k|^2 < 1 < 2e^γ.Source: §3.4 · pp. 20–21

4. Numerical Illustrations

Numerical illustrations examine covariance-function and covariance-operator deep Gaussian processes, including ergodicity in deeper layers and function-space MCMC for Bayesian inverse problems. In a two-dimensional inverse problem, reconstructions were broadly similar across layer counts and data resolutions, while accuracy varied with the number of layers and inferred length-scales.

4. Numerical Illustrations

  • The numerical section does not add composition-construction experiments and omits convolution experiments because Theorem 16 indicates that interesting behavior is not expected.Earlier numerical examples for the composition construction are cited as Damianou and Lawrence (2013) and Duvenaud et al. (2014).Source: §4 · p. 22

4.1 Covariance Function

  • In one-dimensional covariance-function samples, layers become non-stationary, and their distributions look similar at larger depths, reflecting the chain’s ergodicity.The figure shows four independent realizations of the first seven layers.Source: §4.1 · p. 22, Figure 1 · p. 23

4.2 Covariance Operator

  • Rescaling the covariance operator to keep sample amplitude O(1) yields an ergodic discretized process; sample levels show anisotropy and increasingly similar properties at depth.The norm trace’s running mean converges rapidly, also reflecting ergodicity.Source: §4.2 · pp. 22, 25, Figure 3 · p. 26, Figure 4 · p. 27

4.2.1 Algorithms

  • The non-centred MCMC algorithm is well-defined on function space and avoids the poor mixing and divergent log-determinant issues that can affect centred updates.Its mixing time is designed not to depend on the number of mesh points used to represent the function; the method works with all layers despite interest often centring on the deepest one.Source: §4.2.1 · pp. 28, 30

4.3.1 One-Dimensional Simulations

  • With two or more layers, one-dimensional reconstructions are visually similar; added depth improves accuracy for J = 50 and 100, but errors rise beyond two layers for J = 25.The largest accuracy gain occurs from one to two layers. A single-layer prior fails to capture the discontinuities, and Table 1 reports the L1-errors across setups.Source: §4.3.1 · p. 31, Table 1 · p. 32
  • With 106 randomly placed observations, adding layers improves one-dimensional reconstruction accuracy with diminishing returns, while data influence reaches uN−2 but not uN−3 in the four-layer case.The truth has higher posterior probability with four layers than with one; the authors note that a single-layer result could be improved by adjusting its constant length-scale parameter.Source: §4.3.1 · pp. 31–32

4.3.2 Two-Dimensional Simulations

  • The two-dimensional inverse problem uses J = 210 point observations on a 25 × 25 grid and a true field built from truncated trigonometric functions with different length-scales.The supplied passages describe the setup but do not report reconstruction findings.Source: §4.3.2 · p. 32
  • For J = 210, reconstruction quality generally improves with the number of layers, while too-short or too-long length-scales produce visible errors in some layers.The too-short-scale effect is evident for N = 1; too-long scales are evident for N = 1 and N = 2 in the region where the shortest scale is expected.Source: §4.3.2 · p. 34
  • For J = 210, the N = 2 and N = 3 length-scale fields are similar, but N = 3 captures the true length-scales more accurately.Source: §4.3.2 · p. 34
  • For J = 28, reconstructions remain similar but contour shapes are less accurate, and too-short length-scales are especially evident for N = 1.Source: §4.3.2 · p. 34

5. Conclusions, Discussion, and Actionable Advice

The comparison shows that deep GP constructions trade off interpretability, scalability, and tractable inference, while the numerical results suggest that useful depth depends on data quality and structure. The discussion concludes that effective depth is construction-dependent and that inference-layer needs depend on the observed data, which the prior analysis does not fully resolve.

5.1 Comparison of Deep GP Constructions

  • The composition construction scales with data dimension and has established variational methods, but accurate MCMC becomes intractable for large datasets because each step requires dense covariance construction and factorization.The section also notes that variational inference is tractable but an uncontrolled approximation that may inadequately represent the posterior, particularly its uncertainty.Source: §5.1 · pp. 36, 38
  • The covariance function construction makes each layer interpretable as the next layer’s anisotropic length-scale field, while the covariance operator construction enables fast low-dimensional sampling and tractable MCMC but appears intractable at large data dimension.Variational inference for the covariance function construction had not yet been studied.Source: §5.1 · p. 38
  • The convolution construction is amenable to analysis, but its trivial behavior at large depth makes it a likely poor choice for inference.Source: §5.1 · p. 38
  • The numerical examples suggest few layers suffice with high-quality or low-quality data, while deeper processes may help for intermediate-quality data with sufficiently rich truth structure.The authors caution that the number of layers should be limited.Source: §5.1 · p. 38

5.2 Summary and Future Work

  • Future directions include covariance operators with anisotropy or forcing, ResNet-type iterations, non-ergodic systems, and transitions defined by stochastic integration against Brownian motions.Source: §5.2 · pp. 38–39
  • The paper studies effective depth through ergodicity or convergence to trivial solutions, concluding that construction design requires care even though a few layers can be useful for inference.Source: §5.2 · p. 39
  • The prior-process analysis does not determine inference depth: ergodicity of u_n does not directly transfer to u_n|y, and the practical layer count depends on information in the observations.For ergodic constructions, prior expressive power does not increase beyond a certain layer count, providing some justification for moderate-depth priors.Source: §5.2 · p. 39
  • Approximation theory remains open, significant value from more than two or three layers seems difficult for noisy inverse problems, and efficient sampling under nonlinear observations remains unresolved.The authors note that centred methods or mixtures may be desirable in some applications.Source: §5.2 · p. 39

Appendix

The appendix establishes positive definiteness for the constructed kernel and identifies it as a non-stationary correlation function. Its proof also addresses cases where the spatially varying function G vanishes.

Appendix

  • The proof establishes that ρ is positive definite on R^d × R^d under the stated measure condition that ν is not concentrated at zero.The proof uses positive definiteness of the stationary kernel and linear independence of the relevant kernel functions.Source: Appendix · pp. 42–43
  • The constructed kernel ρ is non-stationary and is a correlation function because ρ(x, x) = 1.Source: Appendix · p. 43
  • When G vanishes at one or more points, the proof takes limits to obtain a correlation matrix with an isolated first row and column, then uses positive definiteness of the remaining submatrix.Source: Appendix · pp. 44–45
Loading 1711.11280v2…