PaperlayerView exact revision on arXiv

Source-linked AI summary

Random Feature Expansions for Deep Gaussian Processes

Kurt Cutajar, Edwin V. Bonilla, Pietro Michiardi, Maurizio Filippone

arXiv:1610.04386v2stat.MLstat.CO

TL;DR

DGP inference is difficult to scale to large datasets and deeper architectures despite its probabilistic treatment and uncertainty benefits. The paper uses random feature expansions with stochastic variational inference to build a practical framework that is faster than competing methods, handles million-scale datasets and moderately deep models, and improves uncertainty quantification over dropout-trained DNNs.

  • Problem

    Existing DGP inference approaches have limited scalability and are not practical beyond only a few hidden layers, restricting application to large-scale problems.

  • Method

    The paper approximates GPs at every DGP layer with random feature expansions and trains the resulting probabilistic model using stochastic variational inference.

  • Results

    The framework is faster than competing methods while achieving lower errors, scales to datasets with over 8 million observations, and outperforms dropout-trained DNNs on uncertainty quantification metrics.

  • Takeaways & Limitations

    Random-feature DGPs provide a practical route to scaling probabilistic deep models to large datasets and moderately deep architectures while retaining uncertainty quantification.

  • Takeaways & Limitations

    The method requires selecting the number of random features, so its approximation level must be tuned to running-time or hardware constraints; covariance parameters are not treated variationally in the described variant.

Abstract

from arXiv · show

The composition of multiple Gaussian Processes as a Deep Gaussian Process (DGP) enables a deep probabilistic nonparametric approach to flexibly tackle complex machine learning problems with sound quantification of uncertainty. Existing inference approaches for DGP models have limited scalability and are notoriously cumbersome to construct. In this work, we introduce a novel formulation of DGPs based on random feature expansions that we train using stochastic variational inference. This yields a practical learning framework which significantly advances the state-of-the-art in inference for DGPs, and enables accurate quantification of uncertainty. We extensively showcase the scalability and performance of our proposal on several datasets with up to 8 million observations, and various DGP architectures with up to 30 hidden layers.

1. Introduction

The paper addresses limited scalability and depth in DGP inference with a random-feature and stochastic-variational framework. It reports faster, competitive inference on million-scale datasets, moderate-depth architectures, and uncertainty metrics.

  • 1. Introduction: Random feature expansions combined with stochastic variational inference make DGP learning scalable to large problems and moderately deep architectures while retaining probabilistic uncertainty quantification.The framework approximates GPs at all layers with random features and uses stochastic variational inference for scalable probabilistic learning.
  • 1. Introduction: The approximate DGP connects to Bayesian DNNs with low-rank weight matrices, while covariance choices yield trigonometric or ReLU activation functions.RBF covariance produces trigonometric functions, whereas ARC-COSINE covariance produces ReLU functions.
  • 1. Introduction: The proposal is consistently faster at achieving lower errors than competing methods for a given DGP architecture and outperforms dropout-trained DNNs on uncertainty quantification metrics.These findings are reported across regression and classification problems.
  • 1. Introduction: Over 8 million MNIST8M observations and over 5 million AIRLINE observations demonstrate the framework's large-scale applicability, with experiments run on a single non-GPU machine.The proposal is also designed to exploit GPU and distributed computing for further acceleration.
  • 1. Introduction: The work positions random feature expansions as a novel DGP approximation that systematically outperforms state-of-the-art inference methods, especially on large-scale and moderately deep problems.The contribution also validates superior uncertainty quantification for DGPs compared with DNNs.

2. Preliminaries

DGPs compose Gaussian processes across layers to provide probabilistic, nonparametric mappings, but inference requires generally intractable integrals. Random-feature approximations make these models explicitly representable and support scalable inference using covariance-specific features.

  • DGPs compose possibly transformed multivariate Gaussian processes across layers, placing nonparametric GP priors over hidden mappings rather than fixed parametric functions.
  • Inference in DGPs is challenging because learning and prediction require generally intractable integrals, including marginal-likelihood computations over covariance parameters at all layers.
  • The framework considers RBF and ARC-COSINE covariances, with spectral-frequency and integral representations supplying their respective random-feature constructions.
  • Random-feature expansions approximate covariance functions with explicit finite-dimensional mappings estimated by Monte Carlo, enabling scalable inference in DGPs.
  • The proposed DGP approximation replaces each hidden-layer GP with a two-layer weight-space construction using random features and covariance-controlled weights.

3. Random Feature Expansions for DGPs

The paper approximates each DGP layer with random features, yielding a probabilistic DNN with covariance-dependent activations, low-rank weights, and scalable stochastic variational learning. It compares alternative treatments of spectral frequencies and finds that fixed-randomness variational treatment is most consistent, while avoiding inducing-point matrix decompositions.

  • 3.1. Low-rank weights in the resulting DNN: Random feature expansions approximate each GP layer as a DNN, with sine/cosine activations for RBF covariances and ReLU activations for ARC-COSINE covariances.The resulting model retains Gaussian priors over feature weights controlled by covariance parameters and standard Gaussian priors over the remaining weights.
  • 3.1. Low-rank weights in the resulting DNN: The approximation imposes a low-rank structure on DNN weights by representing each layer with two separate weight transformations instead of their dense product.Collapsing the transformations would require learning considerably more weights at each layer.
  • 3.2. Variational inference: Stochastic variational inference provides a tractable learning objective for the approximate DGP, using a variational posterior over weights and doubly stochastic estimates from mini-batches and Monte Carlo samples.The reparameterization trick supports unbiased stochastic gradients for variational and covariance parameters.
  • 3.3. Treatment of the spectral frequencies Ω: VAR-FIXED treats spectral frequencies variationally while fixing the randomness used to compute them, and performs more consistently than the alternative strategies across the reported comparisons.PRIOR-FIXED improves with more random features, whereas VAR-RESAMPLED can struggle with optimization as feature count increases.
  • 3.2. Variational inference: The framework avoids Cholesky decompositions involving inducing-point counts, enabling scalable optimization compared with inducing-point stochastic variational inference.This supports the paper’s emphasis on large-scale DGP learning without matrix operations that grow with the number of inducing points.

4. Experiments

The experiments compare the proposed DGP variants with competing inference methods and DNNs across standard and large-scale datasets, including models with up to 30 hidden layers. The proposed models generally converge faster, achieve competitive or superior predictive performance, and scale to datasets with more than 8 million observations.

  • Regression and classification: DGP-RBF and DGP-ARC consistently outperform competing techniques in convergence speed and predictive accuracy.The comparison covers error rate or RMSE and MNLL over time for one-hidden-layer configurations.
  • Regression and classification: DGP-RBF and DGP-ARC remain competitive after adding a second hidden layer, whereas the regularized DNN becomes notably inferior and more prone to overfitting.The two-hidden-layer results are reported in the supplementary material.
  • Large-scale datasets: The framework scales to MNIST8M and AIRLINE, containing over 8 million and 5 million observations, respectively.The experiments assess scalability on datasets beyond the scale typically used for GP and DGP models.
  • Large-scale datasets: 99.14% accuracy on MNIST8M with one hidden layer is comparable to AutoGP's 99.11% on the same large-scale task.MNIST8M contains more than 8 million observations.
  • Model depth: With increasing depth on AIRLINE, training takes longer, but every model converges in under a couple of hours and 10- and 20-layer models improve over the shallower two-layer model.The depth study evaluates DGP-RBF models with up to 30 layers.

5. Conclusions

The paper formulates DGP inference with random feature covariance approximations and stochastic variational inference, preserving a probabilistic treatment while improving scalability. Experiments indicate faster and often superior inference, including on AIRLINE and MNIST8M, while higher-dimensional settings motivate more efficient random-feature approximations.

  • 5. Conclusions: The proposed DGP formulation combines random feature covariance approximations with stochastic variational inference to preserve probabilistic inference.The approach is presented as a scalable reformulation of DGP inference.
  • 5. Conclusions: Inference is faster and frequently superior to other state-of-the-art methods, particularly for competing DGP models.The reported results emphasize AIRLINE and MNIST8M as especially large datasets for DGP applications.
  • 5. Conclusions: Higher-dimensional datasets motivate approximations such as Fastfood to support using more random features.The paper also reports ongoing work to mitigate performance decline when variational parameters are optimized with resampling.

A. Additional Experiments

With two hidden layers, DGP-ARC and DGP-RBF continued to perform well, while the regularized DNN performed worse and overfit more. DGP-EP did not produce sensible MNIST results in this configuration.

  • DGP-ARC and DGP-RBF continued to perform well after adding a second hidden layer.
  • The regularized DNN was notably inferior and exhibited substantially greater overfitting.
  • DGP-EP produced no sensible MNIST results with two hidden layers, indicating low scalability in this configuration.

REGRESSION CLASSIFICATION

Across regression and classification datasets, VAR-FIXED produced more consistent results than alternative strategies for treating Ω. Its classification error rate was markedly superior, although ARC-COSINE models showed some overfitting on high-dimensional SPAM and MNIST.

  • VAR-FIXED yielded more consistent results than alternative strategies across the reported datasets.
  • VAR-FIXED produced a markedly superior error rate on the classification datasets.
  • ARC-COSINE models optimized with VAR-FIXED showed some overfitting on the higher-dimensional SPAM and MNIST datasets.

B. Comparison with MCMC

The paper compares its variational approximation with MCMC on a two-layer DGP regression model using different numbers of fixed spectral frequencies. MCMC explores two sign-symmetric posterior modes, whereas the variational approximation identifies one mode and becomes accurate with more features.

  • The overall variational approximation is accurate when using more random Fourier features.
  • MCMC explores two opposite-sign posterior modes because the two-layer output is invariant to flipping weight signs.

REGRESSION CLASSIFICATION

The optimisation-strategy experiments compare RMSE and MNLL progression for DGP-ARC and DGP-RBF models with one and two hidden layers. Across these settings, VAR-FIXED is reported as the most consistent strategy, while higher-dimensional ARC-COSINE cases can overfit.

  • REGRESSION CLASSIFICATION: VAR-FIXED was more consistent than alternative optimisation strategies for both one- and two-hidden-layer models.
  • REGRESSION CLASSIFICATION: The figures track RMSE and MNLL progression over time for DGP-ARC and DGP-RBF under one- and two-hidden-layer configurations.

B.1. Details of MCMC sampler for a two-layer DGP with a Gaussian likelihood

The sampler targets posterior inference in a two-layer DGP with a Gaussian likelihood by alternating conditional updates for latent functions. It treats the hidden-layer variables as covariance-controlling quantities and uses this construction as a reference for evaluating the proposed approximation.

  • B.1. Details of MCMC sampler for a two-layer DGP with a Gaussian likelihood: The MCMC procedure provides a gold-standard reference for evaluating the proposed DGP approximation and inference.For tractability, it assumes a two-layer DGP with a Gaussian likelihood and fixed GP hyperparameters.
  • B.1. Details of MCMC sampler for a two-layer DGP with a Gaussian likelihood: Figure 8 compares MCMC and variational inference for the two-layer DGP using 100 samples from each posterior.The comparison uses a single GP in the hidden layer and a Gaussian likelihood.
  • B.1. Details of MCMC sampler for a two-layer DGP with a Gaussian likelihood: Gaussian likelihood integration eliminates the second-layer latent function analytically, yielding a compact model whose covariance depends on first-layer GP variables.These first-layer variables can be interpreted as hyper-parameters governing the covariance structure.
  • B.1. Details of MCMC sampler for a two-layer DGP with a Gaussian likelihood: The latent-variable posterior is factorized to support a Gibbs sampling strategy that iteratively updates its conditional components.The procedure uses separate sampling steps for the latent variables, while the second-layer posterior is available in closed-form Gaussian form.

C. Derivation of the lower bound

The lower-bound derivation formulates variational learning for the approximate DGP by optimizing distributions over weights and, optionally, spectral frequencies. The resulting parameters are optimized with mini-batch stochastic inference and a reparameterization-based procedure.

  • C. Derivation of the lower bound: The variational objective is derived for learning the posterior over weights while optimizing the model parameters under fixed spectral frequencies.The derivation is presented as a lower bound used to optimize Θ.
  • C. Derivation of the lower bound: The formulation can also treat spectral frequencies variationally alongside the weights by defining Ψ = {W, Ω}.This extends the variational treatment beyond the weight variables.
  • C. Derivation of the lower bound: The derivation assumes a factorized prior over weights across layers.This assumption structures the variational objective for the multilayer model.
  • C. Derivation of the lower bound: The lower bound is optimized with mini-batches and the reparameterization trick, using means and variances as variational parameters for the approximating factors.The optimization updates the variational parameters of each factor.
  • C. Derivation of the lower bound: For Gaussian factors, the derivation includes the KL divergence between the corresponding normal distributions.The KL term is specified for distributions with means and variances.

F. Distributed Implementation

The distributed implementation uses asynchronous stochastic gradient updates coordinated by parameter servers and workers. On MNIST, adding workers reduces training time sub-linearly but increases error rate because asynchronous updates become less precise.

  • F. Distributed Implementation: Workers asynchronously fetch parameters, compute mini-batch lower-bound gradients, and push updates to parameter servers.The system supports one or more parameter servers and multiple workers operating on randomly selected data batches.
  • F. Distributed Implementation: The MNIST study evaluates how training time and error rates change with worker count while keeping the model parameters fixed across experiments.The experiment varies the number of workers rather than the model configuration.
  • F. Distributed Implementation: Training time decreases in proportion to the number of workers, albeit sub-linearly, while error rate increases in the asynchronous MNIST experiment.The reported trend appears in Figure 9 and reflects the trade-off between speed and optimization accuracy.
  • F. Distributed Implementation: Imprecise asynchronous gradient updates negatively impact optimization, motivating methods to alleviate this issue.The paper connects this observation with corroborating work on asynchronous optimization.
Loading 1610.04386v2…