PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Scalable Deep Kernels with Recurrent Structure

Maruan Al-Shedivat, Andrew Gordon Wilson, Yunus Saatchi, Zhiting Hu, Eric P. Xing

arXiv:1610.08936v3cs.LGcs.AIstat.ML

TL;DR

Standard kernels do not readily capture sequential structure, and sequence models have rarely addressed predictive uncertainty across time. This paper proposes GP-LSTM, which builds recurrent structure into Gaussian-process kernels and uses convergent semi-stochastic optimization and kernel algebra for scalable learning. The model outperforms alternatives on several sequence-to-real regression tasks while providing a probabilistic representation of uncertainty.

  • Problem

    Standard Gaussian processes are limited to pairwise input correlations and may miss long-term dependencies, while predictive uncertainty across time has rarely been addressed by neural sequence models.

    Source: §1 · p. 1, §2 · p. 5

  • Method

    GP-LSTM builds recurrent LSTM structure into Gaussian-process kernels and uses semi-stochastic optimization and structured covariance algebra for scalable learning and prediction.

    Source: §1 · p. 2, §4.2 · p. 11

  • Results

    GP-LSTM outperforms a range of alternatives on several sequence-to-real regression tasks and works across low and high signal-to-noise ratios.

    Source: §6 · p. 20

  • Takeaways & Limitations

    GP-LSTM provides a practical Bayesian-LSTM representation that combines recurrent learning biases with probabilistic predictions and uncertainty quantification.

    Source: §1 · p. 2, §6 · p. 20

    GP-LSTM quantifies predictive uncertainty but does not model propagation of input uncertainty through the recurrent structure.

    Source: §6 · p. 20

Abstract

from arXiv · show

Many applications in speech, robotics, finance, and biology deal with sequential data, where ordering matters and recurrent structures are common. However, this structure cannot be easily captured by standard kernel functions. To model such structure, we propose expressive closed-form kernel functions for Gaussian processes. The resulting model, GP-LSTM, fully encapsulates the inductive biases of long short-term memory (LSTM) recurrent networks, while retaining the non-parametric probabilistic advantages of Gaussian processes. We learn the properties of the proposed kernels by optimizing the Gaussian process marginal likelihood using a new provably convergent semi-stochastic gradient procedure and exploit the structure of these kernels for scalable training and prediction. This approach provides a practical representation for Bayesian LSTMs. We demonstrate state-of-the-art performance on several benchmarks, and thoroughly investigate a consequential autonomous driving application, where the predictive uncertainties provided by GP-LSTM are uniquely valuable.

1. Introduction

Sequential applications need models that capture temporal structure and represent predictive uncertainty, particularly in settings such as self-driving. The paper proposes GP-LSTM, combining recurrently structured kernels with Gaussian processes, and develops optimization and scaling techniques for them.

1. Introduction

  • GP-LSTM uses kernels that encapsulate LSTM structure, giving Gaussian processes state-of-the-art performance on sequential regression while representing predictive uncertainty.The model retains non-parametric flexibility.Source: §1 · p. 2
  • A provably convergent semi-stochastic procedure enables mini-batch updates and improves runtime and converged-solution quality over standard non-stochastic first-order methods.Source: §1 · p. 2
  • Covariance structure yields O(n) training time and O(1) test predictions, supporting scalability to datasets approaching a million examples.The evaluation spans system identification, energy forecasting, and self-driving applications.Source: §1 · p. 2
  • The evaluation reports state-of-the-art performance across tasks, data sizes from hundreds to almost a million points, and varied signal-to-noise ratios.Source: §1 · p. 2

2. Background

Learning from sequences requires exploiting temporal structure without turning a finite history into an unwieldy feature space. This background introduces recurrent mappings, LSTMs, and Gaussian processes, while identifying limits of ordinary GP kernels for temporal dependencies.

2. Background

  • Flattening L previous p-dimensional observations creates an (L×p)-dimensional regression problem, making sequential structure important for learning from finite data.Source: §2 · p. 3
  • A recurrent mapping updates a latent representation from each observation and maps the final representation to a target, typically using time-invariant transition and emission functions.Source: §2 · p. 3
  • LSTMs use memory cells and differentiable gates to stabilize error flow and manage memorized information through soft gating.Source: §2 · p. 4
  • A Gaussian process defines a distribution over functions through a mean and covariance kernel, and its kernel hyperparameters are learned by optimizing the data evidence.The evidence objective combines model fit with a complexity penalty.Source: §2 · pp. 4–5
  • Ordinary GPs learn pairwise input correlations and may miss long-term dependencies and complex temporal structure, motivating their combination with recurrent models.Source: §2 · p. 5

3. Related work

Prior sequence models include autoregressive and state-space approaches, as well as Gaussian-process extensions with differing flexibility and inference costs. The paper instead directly learns kernels with recurrent structure, preserving an analytical objective without chaining Gaussian processes recurrently.

3. Related work

  • Autoregressive and state-space models describe stochastic temporal processes, with state-space approaches explicitly representing latent states and propagating uncertainty through time.Source: §3 · pp. 5–6
  • GP state-space models offer flexible state-dynamics uncertainty, but their approximate training and inference are difficult to scale.Source: §3 · p. 6
  • Recurrent Gaussian processes replace recurrent-network layers with GP activations and propagate uncertainty, but require intractable inference and sophisticated variational approximations.Source: §3 · p. 6
  • Rather than chain GPs recurrently, the proposed model jointly optimizes a standard GP composed with a recurrent model, retaining an analytical objective.Source: §3 · p. 6
  • Figure 1 contrasts a one-step recurrent model, GP-LSTM with a Gaussian field over latent representations, and a GP kernel using a parametric map.Source: Figure 1 · p. 7

4. Learning recurrent kernels

The paper constructs recurrent GP kernels by embedding input sequences with an LSTM and applying a kernel in the transformed space. It pairs joint marginal-likelihood optimization with structure-exploiting computations to scale training and prediction.

4. Learning recurrent kernels

  • An LSTM embeds the input sequence into a hidden-space vector, and a base kernel on that representation defines the recurrent kernel.The paper uses the recurrent network’s last hidden vector; the composed function is a valid kernel.Source: §4 · pp. 7–8

4.1 Optimization

  • Because the negative log marginal likelihood does not factorize over data, the method alternates full-data kernel-hyperparameter updates with mini-batch recurrent-weight updates.With a fixed kernel matrix, the weight objective becomes a weighted sum of independent per-point functions.Source: §4 · p. 8, §4.1 · p. 9
  • Delayed kernel-matrix updates permit multiple mini-batch steps between recomputations; convergence requires balancing the recurrent-weight learning rate with update frequency.The stated convergence condition uses a learning rate decaying as Θ(1/τt^(1+δ/2)) for δ in (0, 1].Source: §4.1 · p. 10
  • Semi-stochastic training converges faster and performs better than full-batch techniques, requiring fewer passes through the data.The comparison is reported for GPs with recurrent kernels trained with Algorithm 2.Source: §4.1 · p. 10
  • The mini-batch approach avoids variational approximation to the Gaussian-process marginal likelihood.This distinguishes the proposed training approach from stochastic variational methods for Gaussian processes.Source: §4.1 · p. 11

4.2 Scalability

  • Sparse interpolation and Kronecker products of circulant inducing-point covariances yield O(n) learning and inference and O(1) test predictions.The structure also supports fast matrix-vector products and linear-system solves.Source: §4.2 · p. 11

5. Experiments

Experiments span system identification, smart-grid forecasting, and autonomous-driving tasks, comparing recurrent Gaussian-process models with neural and other baselines. The results report state-of-the-art performance for recurrent kernels, benefits from semi-stochastic optimization, and scalable learning and prediction.

5.1 Data and the setup

  • Experiments cover Actuator and Drives system identification, smart-grid power and wind forecasting, and four self-driving prediction tasks, using data from hundreds to almost a million time points.The car data were recorded on two trips at 10 ms intervals, with the first used for training and the second for validation and testing.Source: §5 · p. 11, §5.1 · pp. 12–13

5.2.1 Convergence of the optimization

  • For GP-LSTM models with one recurrent hidden layer and 16 or 64 hidden units, semi-stochastic optimization with delayed kernel updates converged faster and achieved better final held-out-set error and training-set NLML than full-batch iterative training.Source: §5.2.1 · p. 15
  • Semi-stochastic optimization requires balancing mini-batch frequency and learning rate: long delays between kernel updates combined with a sufficiently high learning rate can prevent convergence.An appropriate combination can generalize better than the default batch approach of Wilson et al. (2016a).Source: §5.2.1 · p. 15

5.2.2 Regression, Auto-regression, and Free Simulation

  • Across Actuator and Drives, GP-based architectures improve predictive performance over vanilla deep-learning counterparts in regression, autoregression, and free simulation, while RGP performs best in free simulation.The reported RGP comparison is available only for free simulation; the authors attribute its result to RGP's design for representing and propagating uncertainty through recurrence.Source: §5.2.2 · p. 16, Table 3 · p. 16

5.2.3 Prediction for smart grid and self-driving car applications

  • Recurrent-kernel Gaussian processes attain state-of-the-art performance on the GEF and self-driving car tasks.The models were initialized with pretrained recurrent weights and fine-tuned with Algorithm 2.Source: §5.2.3 · p. 17
  • On GEF-power, GP-based models are less prone to overfitting with limited data, and their negative log likelihood can guide architecture selection.The experiments also find a hidden-unit count with the best power-prediction performance; GP-LSTM estimates uncertainty alongside its predictions for lane estimation and front-vehicle tracking.Source: §5.2.3 · pp. 17–18, Figure 5 · p. 18

5.2.4 Scalability of the model

  • With a fixed number of inducing points per dimension, MSGP-LSTM training time grows linearly with training examples, while prediction time per test point is virtually constant.Training time also depends linearly on the number of inducing points; prediction time does not depend on training-set size or inducing-point count.Source: §5.2.4 · p. 19

6. Discussion

The discussion presents GP-LSTM as a probabilistic model that combines recurrent LSTM structure with Gaussian processes. The appendix describes structured interpolation and algebraic methods for efficient inference and prediction.

6. Discussion

  • GP-LSTM combines LSTM structure and learning biases with a Bayesian nonparametric representation, outperforming alternatives on several sequence-to-real regression tasks.The discussion also reports performance across low and high signal-to-noise ratios and scalability to very large datasets.Source: §6 · p. 20

A.1 Structured kernel interpolation

  • Structured kernel interpolation approximates cross-covariances using inducing points and sparse interpolation weights, with local cubic interpolation used in practice.The approach uses regularly spaced inducing points and covariance-matrix structure to improve scalability.Source: §A.1 · p. 22

A.2 Kernel approximations

  • For regularly spaced grid points and a stationary product kernel, the inducing-point covariance decomposes into a Kronecker product of Toeplitz matrices.Circulant approximations allow fast Fourier transforms to compute approximate eigendecompositions in O(m log m) time.Source: §A.2 · pp. 22–23

A.3 Structure exploiting inference

  • Inference solves the structured-kernel linear system with conjugate gradients and computes the log determinant using an approximate eigendecomposition.The eigendecomposition exploits the Kronecker and Toeplitz structure of the inducing-point covariance.Source: §A.3 · p. 23

A.4 Fast Test Predictions

  • Fast test prediction applies structured kernel interpolation to approximate the latent mean and variance for new inputs.Training-time precomputation leaves O(1) operations per test point, and the prediction method applies to trained Gaussian processes regardless of how they were learned.Source: §A.4 · p. 23

Appendix B. Gradients for GPs with recurrent kernels

The appendix derives gradients for Gaussian processes with recurrent kernels by applying the chain rule through the transformed inputs. It also allows finite-difference derivatives when analytical kernel derivatives are impractical.

Appendix B. Gradients for GPs with recurrent kernels

  • The chain rule expresses each kernel-entry derivative through its dependence on the transformed inputs and the recurrent-map parameters.Because a kernel entry depends only on its two corresponding hidden representations, the derivative has nonzero terms only for those inputs.Source: Appendix B · pp. 23–24
  • The log marginal likelihood derivatives with respect to kernel hyperparameters and recurrent-map parameters follow a generic form, with the kernel-specific derivative depending on the chosen kernel.The appendix develops the recurrent-map derivative by differentiating kernel entries through the transformed data vectors.Source: Appendix B · pp. 23–24
  • Finite differences can approximate derivatives with respect to hidden representations when analytical forms are impractical, without affecting the stated convergence result.The appendix gives spectral mixture kernels with grid-based approximations as an example.Source: Appendix B · p. 25

Appendix C. Convergence results

The convergence appendix adapts prior results to the paper’s semi-stochastic optimization setting. Its objective is the negative log marginal likelihood for a particular dataset, with deterministic and stochastic updates across two parameter blocks.

Appendix C. Convergence results

  • The optimization minimizes negative log marginal likelihood on a fixed dataset, equivalently using that dataset’s empirical distribution in the adapted formulation.This differs from the original stochastic-program setting, which takes expectation under an underlying data distribution.Source: Appendix C · p. 25
  • The algorithm computes the kernel matrix using the entire training dataset at each step, rather than accessing only one random data point.This is one of the stated differences from the original optimization setting.Source: Appendix C · p. 25
  • The method has two parameter blocks: θ is updated deterministically on the full dataset, while W uses stochastic gradients from empirical-distribution samples.The appendix adapts smooth non-convex convergence results to this scheme and then considers delayed kernel-matrix updates.Source: Appendix C · p. 25

C.1 Semi-stochastic alternating gradient

The semi-stochastic procedure alternates updates to θ and W, using mini-batches for stochastic gradients. Under the stated assumptions, the nonconvex objective permits a stationary-point guarantee rather than a global-optimum guarantee.

C.1 Semi-stochastic alternating gradient

  • Each iteration alternates between θ and W updates, with a mini-batch sampled from the full training set for the stochastic step.The stated update order is θ first, then W.Source: §C.1 · pp. 25–26
  • With the kernel matrix fixed, the alternating scheme computes unbiased estimates of the gradients at each step.The appendix notes that the W-gradient factorizes over the full data and the mini-batch, respectively.Source: §C.1 · p. 26
  • The analysis assumes a lower-bounded objective with uniformly Lipschitz partial derivatives and bounded parameter iterates.The appendix states that its assumptions are stronger than the original condition because of the semi-stochastic scheme, simplifying the analysis but not being critical.Source: §C.1 · p. 27
  • Because the structured-kernel negative log marginal likelihood is nonconvex, the guarantee is convergence to a stationary point, not a global optimum.A stationary point is where the objective gradient is zero.Source: §C.1 · p. 27

C.2 Semi-stochastic gradient with delayed kernel matrix updates

The delayed-kernel-update algorithm remains convergent under bounded delay and stated assumptions. Its analysis bounds perturbations to the kernel matrix, while noting a trade-off between delay and learning rate.

C.2 Semi-stochastic gradient with delayed kernel matrix updates

  • The semi-stochastic gradient algorithm remains convergent when kernel-matrix updates have bounded delay.The proof extends the argument for Theorem 1 to Algorithm 2 while accounting for perturbations from delayed updates.Source: §C.2 · pp. 30–31
  • The analysis bounds delayed kernel perturbations using Lipschitz conditions on the recurrent transformation and kernel, plus a positive lower bound on kernel-matrix eigenvalues.The recurrent transformation is assumed L-Lipschitz in W; the kernel is uniformly G-Lipschitz with uniformly J-Lipschitz first partial derivatives, and its smallest eigenvalue is lower bounded by γ > 0.Source: §C.2 · p. 30
  • The proof estimates entrywise differences between updated and delayed kernel matrices, then bounds the perturbed matrix’s spectral norm and uses Lemma 3 to extend Theorem 1.The derivation also bounds the spectral norm of the target expression using the largest eigenvalues of matrix inverses, which the authors call pessimistic.Source: §C.2 · p. 31
  • The convergence bounds are crude, and their constant reflects a balance between update delay τ and learning rate λ.The authors note that tighter bounds could use kernel-specific perturbation analysis or more specific assumptions about the data distribution.Source: §C.2 · p. 32

Appendix D. Details on the datasets

The evaluated datasets span a wide range of sequence lengths, dimensionalities, and estimation tasks.

Appendix D. Details on the datasets

  • Sequence lengths across the datasets range from hundreds to a million time steps.Source: Appendix D · p. 32
  • The datasets vary in input and output dimensionality and in the nature of their estimation tasks.Source: Appendix D · p. 32
  • The dataset collection covers estimation tasks with differing sequence lengths and dimensionalities.Source: Appendix D · p. 32

D.1 Self-driving car

The self-driving evaluation covers several estimation tasks, including vehicle state estimation and lane sequence prediction. Lane curves are discretized into points rather than represented by polynomial coefficients.

D.1 Self-driving car

  • Speed estimation uses GPS velocity and fiber-gyroscope features to predict speedometer measurements, while yaw estimation uses acceleration and compass measurements to predict car-centric yaw.Source: §D.1 · p. 32
  • Lane sequence prediction uses seven discretization points per lane instead of polynomial coefficients, which produced less stable overall results.Each lane is represented by eight cubic-polynomial coefficients, while the discretized representation uses initial, final, and five equidistant intermediate points.Source: §D.1 · p. 32
  • The lane targets comprise coordinates for seven points per lane, yielding 28 total output dimensions.Source: §D.1 · p. 32
  • The self-driving tasks include nearest-front-vehicle position, car speed, car yaw, and lane sequence prediction.The described tasks use different vehicle sensor features and targets.Source: §D.1 · pp. 32–33
  • For nearest-vehicle position estimation, inputs include prior position, GPS velocity, fiber gyroscope, compass, and steering angle; targets are x and y coordinates.Source: §D.1 · p. 33

Appendix E. Neural architectures

The appendix directs readers to Table 5 for the best neural architectures used on each dataset.

Appendix E. Neural architectures

Related papers

Loading 1610.08936v3…