Source-linked AI summary
Continuous-depth Deep Gaussian Processes
Ying Li, Sanyou Wu, Zi Yang, Qiaochu Xu, Yuhao Liu, Zhidi Lin, Michael Minyi Zhang, Petar Djuric, Alexandre H. Thiery
TL;DR
Deep Gaussian processes become difficult to optimize as discrete depth grows, motivating a continuous-depth latent SDE with GP-based priors on its drift family. The paper studies CDGP and FlowDGP and reports that residual DGP improves over plain DGP, while the continuous-depth models are more stable or competitive across studied regimes.
Problem
Standard discrete-depth DGPs become harder to optimize as depth grows, limiting the practical use of deeper hierarchical probabilistic regression models.
Source: §1 · p. 1Method
Results
Across the studied regimes, residual DGP improves over plain DGP, while continuous-depth models are more stable or competitive and FlowDGP performs best at larger scale.
Source: §7 · p. 10Takeaways & Limitations
Continuous depth is a useful redesign for probabilistic regression, with the reported advantage over neural ODE and SDE alternatives tied to a GP-based drift prior rather than continuous depth alone.
Source: §7 · p. 10The evaluation focuses on moderate-dimensional regression, leaving classification and structured prediction to future work; the depth-uniform sensitivity result also requires contractivity not explicitly enforced by either model.
Source: §4.4 · p. 6, §8 · p. 10
Abstract
from arXiv · showhide
Deep Gaussian processes (DGPs) are appealing Bayesian regression models, but their standard discrete-depth parameterization often becomes harder to optimize as depth grows. We argue that depth is better treated as a latent continuous evolution than as a long discrete stack. We therefore propose a continuous-depth formulation in which the latent representation follows a stochastic differential equation with a depth-indexed drift family equipped with a GP-based prior. This reframes the core design choice as the prior placed on that family, rather than adopting a purely neural continuous-depth parameterization as in neural ODE or neural SDE models. We study two instantiations of this idea: CDGP, a direct GP prior over the state--depth drift, and FlowDGP, a flow-evolved prior that captures richer depth dependence at lower training cost. To motivate the redesign, we analyze how the single-sample DGP objective can become increasingly sensitive under repeated layer composition, while residual DGP is a useful first repair that still remains a finite discrete construction. We also give pathwise sensitivity analysis for the continuous-depth formulation. Experiments on synthetic, benchmark, and large-scale regression tasks show that residual DGP improves over plain DGP, while CDGP and FlowDGP provide a stronger overall story across the regimes we study.
1 Introduction
Deep Gaussian processes become difficult to optimize as depth grows, limiting their practical use as hierarchical probabilistic regression models. This paper treats depth as continuous latent evolution and places GP-based priors on the resulting drift family.
1 Introduction
- Repeated discrete layer composition can amplify parameter perturbations, making deep DGPs harder to optimize and sometimes degrading performance.Source: §1 · pp. 1–2
- Residual DGPs retain an identity path to weaken perturbation amplification, but still represent depth as a finite stack of separately parameterized layers.Source: §1 · p. 1
- The proposed continuous-depth system replaces the stack with a latent SDE whose depth-indexed drift family retains a GP-based prior.Source: §1 · p. 2
- CDGP places a direct separable GP prior on the drift family, while FlowDGP places a GP prior on a base drift and evolves it across depth.Source: §1 · p. 2
- The paper analyzes discrete-objective sensitivity and continuous-depth pathwise sensitivity, and reports that CDGP and FlowDGP are more stable or competitive across studied settings.Source: §1 · p. 2
2 Background
The background reviews Gaussian processes as priors over latent functions and deep Gaussian processes as compositions of GP mappings. The latter provide hierarchical representations but become harder to train as depth increases.
2 Background
- A Gaussian process defines a prior over scalar-valued functions through a mean function and covariance kernel; vector outputs are modeled as independent scalar GPs sharing a kernel.Source: §2 · p. 2
- A DGP composes GP mappings into a hierarchy of latent representations, with additive Gaussian layer noise and a noisy observation of the deepest representation.Source: §2 · p. 2
- Although this hierarchy increases expressive power, training becomes significantly harder as depth increases.Source: §2 · p. 2
3 From Discrete-Depth Pathology to the Residual Bridge
The section links discrete-depth optimization difficulty to the sensitivity of a single-sample DGP objective under repeated composition. Residual connections weaken this amplification but leave depth as a finite stack.
3 From Discrete-Depth Pathology to the Residual Bridge
- The analysis studies the Lipschitz constant of the variational DGP ELBO using a single-sample estimator, with data-fit and KL-regularization terms.Source: §3 · p. 2
- Repeated discrete composition can make the single-sample DGP objective increasingly sensitive to parameter perturbations, with data-fit sensitivity potentially growing exponentially with depth.The bound's KL contribution is O(L), while its constant C is independent of L.Source: §3 · pp. 2–3
- Residual DGP layers add a learned correction to the current representation, reducing perturbation amplification through an identity path while retaining repeated Jacobian products and finite-depth accumulation.Source: §3 · p. 3
4 Continuous-Depth Modeling and Inference
The paper replaces a finite stack with a latent SDE evolving over continuous depth, then compares GP-based priors for its drift family. Shared variational inference and pathwise sensitivity analysis complete the formulation.
4 Continuous-Depth Modeling and Inference
- Continuous-depth modeling replaces separately parameterized residual layers with gradual latent evolution over a depth variable t in [0, T], and makes the drift-family prior the central modeling choice.Source: §4 · p. 3
4.1 CDGP: Separable Kernel Instantiation
- CDGP places a direct GP prior on the joint state–depth drift with a separable kernel, enabling Kronecker covariance factorization but restricting state–depth interactions.Source: §4.1 · p. 4
4.2 FlowDGP: CNF-Transformed GP
- FlowDGP evolves a GP-sampled base drift across depth through a continuous normalizing flow, avoiding a large joint kernel and extra diffusion-path inference at the cost of ODE solver computation.The authors report substantially faster training than CDGP.Source: §4.2 · p. 4
4.3 Variational Inference
- Both models use inducing-point variational inference and the same simulator-level training logic, but CDGP places inducing variables on the joint state–depth drift while FlowDGP places them on the base GP.The simulator propagates paths on a discretized depth grid using Euler–Maruyama.Source: §4.3 · pp. 4–5
4.4 Optimization Stability
- The generic pathwise sensitivity bound avoids repeated layerwise Jacobian products but still depends exponentially on T; a depth-uniform bound requires contractivity, which neither model explicitly enforces.Under the contractive regime, pathwise sensitivity is bounded by Lθ/µ.Source: §4.4 · pp. 5–6
5 Experiments
The experiments progress from diagnosing discrete-depth behavior on a toy task, to realistic benchmarks, and then to full-size datasets where training cost narrows the comparison to FlowDGP.
5.1 Toy Example
- As discrete depth L increases from 1 to 10, standard DGP deteriorates across all three metrics, while residual DGP is more robust but still degrades.The residual model’s identity shortcut helps materially but does not remove the limitations of discrete composition.Source: §5.1 · p. 6, Figure 1 · p. 7
- With T = 1, CDGP remains stable across metrics as J varies from 1 to 10, matching or outperforming the two-layer Residual DGP reference.FlowDGP achieves lower NLL than CDGP at J ≤5.Source: §5.1 · p. 6, Figure 1 · p. 7
- With J = 3, CDGP maintains stable performance as T increases from 1 to 10; FlowDGP shows better RMSE and MAE at tested T values.The passage describes the performance as well behaved across the tested terminal-depth range.Source: §5.1 · p. 7, Figure 1 · p. 7
5.2 Benchmark Datasets
- CDGP or FlowDGP achieves lower RMSE than TGP on all six benchmarks, while FlowDGP is roughly 5–14× faster to train than CDGP.FlowDGP has the best NLL on heteroscedastic and nonstationary; CDGP has the best NLL on sarcos_small, while TGP leads on heavy_tailed and uci_bike_small.Source: §5.2 · pp. 7–8, Table 1 · p. 8
- On the six benchmarks, NODE often has substantially worse NLL, while NSDE can achieve strong RMSE but has markedly weaker NLL; the authors report stronger probabilistic performance from GP-based drift priors.The comparison is against continuous-depth alternatives that parameterize the vector field with a neural network rather than a GP-based prior.Source: §5.2 · p. 8
5.3 Large-Scale Datasets
- On uci_bike, FlowDGP achieves the best RMSE, NLL, and CRPS; on uci_energy it leads in CRPS and ties for best RMSE, while NSDE leads in NLL.On sarcos, FlowDGP performs comparably to or slightly better than NSDE across all three metrics.Source: §5.3 · p. 9, Figure 2 · p. 9
6 Related Work
Related work spans discrete GP hierarchies, neural continuous-depth dynamics, and GP-driven flows, while this paper focuses on how latent depth is parameterized.
6 Related Work
- DGPs compose GP mappings; residual and thin DGPs modify that hierarchy, while deep kernel learning learns neural features and TGPs transform a GP through an invertible map.The paper’s focus is the parameterization of latent depth.Source: §6 · p. 9
- GP networks model conditional relations in discrete graphical models, and nonlinear multifidelity GPs relate observed fidelity levels; this paper’s depth is latent and continuous.Intermediate depths need not be observed.Source: §6 · p. 9
- Neural ODEs and SDEs use neural vector fields for continuous-depth dynamics; deep state-space GPs use conditional SDEs over input time within a finite hierarchy.Differential GP flows use a GP posterior for an input-warping SDE’s drift and diffusion, followed by a separate output GP.Source: §6 · p. 9
- In FlowDGP, a sampled base drift evolves across latent depth, state diffusion is specified separately, and the terminal state enters the likelihood directly.Additional YearPredictionMSD results are reported in Appendix F.1.Source: §6 · pp. 9–10
7 Conclusion
The conclusion argues that continuous latent evolution is a better depth representation for probabilistic regression than a long discrete stack. It presents GP-based drift priors and reports stronger empirical performance across the studied regimes, while qualifying the strongest depth-uniform theory result.
7 Conclusion
- For probabilistic regression, the conclusion favors treating depth as latent continuous evolution rather than as a long discrete stack.It characterizes residual DGP as a useful first repair, but not the final answer.Source: §7 · p. 10
- The proposed models replace a deep stack with a latent SDE and study CDGP’s direct separable prior and FlowDGP’s structured flow-evolved prior under shared sparse variational inference.The theory analyzes sensitivity relative to repeated discrete composition, but its strongest depth-uniform statement is conditional.Source: §7 · p. 10
- Residual DGP improves over plain DGP, while continuous-depth models are more stable or competitive across studied regimes and FlowDGP performs best at larger scale.Against neural ODEs and SDEs, the conclusion attributes the advantage not to continuous depth alone but to a GP-based prior over the drift family.Source: §7 · p. 10
8 Limitations
The paper identifies modeling and task-scope limits for its continuous-depth GP approaches.
8 Limitations
- CDGP’s separable kernel limits joint state–depth drift dependence, while FlowDGP’s deterministic CNF makes depthwise drift uncertainty depend solely on the base GP f0.The evaluation covers moderate-dimensional regression; classification and structured prediction are left to future work.Source: §8 · p. 10
A Proof of Proposition 3.1
The proof decomposes the discrete DGP objective and bounds its terms under layerwise assumptions, using smoothness and chain-rule arguments to control depth dependence.
A Proof of Proposition 3.1
- The proof bounds the objective’s two terms separately and assumes layerwise conditions, including uniformly bounded gradients for each layerwise KL term.The supplied proof passages also state that sampled layer functions are uniformly Lipschitz with respect to their inputs.Source: A Proof of Proposition 3.1 · pp. 12–13
- The stated constant C is independent of depth L.The proof concludes by taking the supremum over θ.Source: A Proof of Proposition 3.1 · pp. 12–13
- For layer-specific parameter blocks, the proof applies the chain rule and bounds products using sub-multiplicativity of operator norms.It uses standard smoothness arguments and a decomposition of the objective with the triangle inequality.Source: A Proof of Proposition 3.1 · pp. 12–13
- The KL regularizer decomposes by layer, and the proof combines the resulting lemmas to obtain the proposition’s bound.The KL contribution is identified separately in the final expression.Source: A Proof of Proposition 3.1 · pp. 13–14
B Proof of Theorem 4.1
The proof couples two latent SDE paths with the same Brownian motion so their stochastic terms cancel, then bounds their pathwise difference using the drift’s Lipschitz assumptions. The corollary proof adds a one-sided contractivity condition to bound sensitivity.
B Proof of Theorem 4.1
- Using the same Brownian motion makes the stochastic terms cancel, yielding a pathwise equation for the difference between the two SDE solutions.The proof then applies Lipschitz assumptions and integrates the resulting bound over time.Source: B Proof of Theorem 4.1 · pp. 14–15
C Proof of Corollary 4.2
- The corollary proof combines one-sided contractivity with parameter-Lipschitz control to bound the pathwise difference.It applies Young’s inequality and combines the resulting bounds for almost every sample path.Source: C Proof of Corollary 4.2 · pp. 15–16
D Complexity Analysis
FlowDGP performs inference using a single base GP, reducing inducing-point count relative to CDGP and more than offsetting its added ODE solver cost. Benchmark timings show why CDGP and residual DGP are omitted from the full-size study.
D Complexity Analysis
- FlowDGP’s reduction to M inducing points more than offsets its added ODE solver cost, yielding substantially faster training in practice.Inference is over a base GP with M inducing points and has per-iteration complexity O(M^3 + NM^2 + NJCg), where Cg is the cost of evaluating gϕ.Source: D Complexity Analysis · p. 16
- Benchmark timings show FlowDGP is substantially faster than CDGP, explaining why CDGP and residual DGP are omitted from the full-size study.The authors report that CDGP and residual DGP have prohibitive training costs at those scales.Source: D Complexity Analysis · p. 16
E Simulation Details
- Test-input predictions use 300 Monte Carlo samples to estimate moments for a Gaussian predictive distribution before NLL evaluation.The supplied simulation passages describe the sampling and predictive-moment steps, but do not provide the complete NLL expression.Source: E Simulation Details · p. 16
E.1 Datasets Description
The evaluation spans toy, synthetic, subsampled real-world, and full-size regression datasets, with settings and supplementary experiments covering hyperparameter selection, large-scale comparison, contraction, and learned drift evolution.
E.1 Datasets Description
- The one-dimensional toy task combines multiple signal structures and a near-step with increasing output variance, challenging models with a single stationary lengthscale.Its components include multiscale sinusoids, a localized burst, a cusp, and a regime change.Source: §E.1 · p. 17
- The synthetic benchmarks target changing lengthscales, gross outliers, and input-dependent noise across nonstationary, heavy-tailed, and heteroscedastic tasks.The heavy-tailed task uses Student-t noise, while the heteroscedastic task tests fitting input-dependent aleatoric uncertainty.Source: §E.1 · pp. 17–18
- The real-world tasks cover robot-arm torque, hourly bike rentals, appliance energy use, and song-release-year prediction.The SARCOS and UCI tasks are evaluated in both subsampled benchmark and full-size settings; YearPredictionMSD contains 515,345 observations.Source: §E.1 · p. 18
E.2 Hyperparameter Settings
- Benchmark hyperparameters are selected using validation NLL, with near-best-NLL configurations chosen by validation RMSE.The hyperparameter description distinguishes the depth-inducing-point count from terminal horizon T, which is 1.0 unless varied explicitly.Source: §E.2 · p. 18, Table 3 · p. 19
E.1 Datasets Description
- The dataset suite includes a toy example, three synthetic benchmarks, subsampled real-world tasks, and full-size counterparts; all tasks are scalar regression.The benchmark setting uses 300 training and 500 test samples, while the large-scale study uses full-size real-world datasets.Source: §E.1 · pp. 17–18, Table 2 · p. 19
F.1 Large-scale comparison on YearPredictionMSD
- On YearPredictionMSD, FlowDGP has the lowest mean RMSE and NLL, while NSDE has the lowest mean CRPS.Table 5 reports means and standard deviations over random seeds, with lower values better.Source: §F.1 · p. 19, Table 5 · p. 20
F.2 A controlled contraction illustration
- In the controlled SDE illustration, sensitivity levels off below the 1/µ bound for contractive drifts, while the control’s sensitivity increases with horizon.The results are based on 5,000 paired paths with shared Brownian motion; this is an auxiliary SDE, separate from the fitted CDGP and FlowDGP models.Source: §F.2 · p. 20, Table 6 · p. 20
F.3 Learned drift evolution
- In the toy example, FlowDGP’s drift means and latent trajectories vary more substantially across depth than CDGP’s.Figure 3 shows drift posterior means at fixed states for CDGP and fixed inputs for FlowDGP, alongside sampled trajectories and their means.Source: §F.3 · p. 21, Figure 3 · p. 21