PaperlayerView exact revision on arXiv

Source-linked AI summary

Linear Latent Force Models using Gaussian Processes

Mauricio A. Álvarez, David Luengo, Neil D. Lawrence

arXiv:1107.2699v2stat.MLcs.AI

TL;DR

Data-driven models face difficulties with scarce data and extrapolation, while mechanistic models can be too complex to specify. This paper combines Gaussian processes with linear differential equations to build flexible, physically inspired models. In motion-capture tests, its latent force model outperforms the comparison methods on RMSE and explained variance; in the Swiss Jura case, heat-kernel gains over multi-task GPs are within standard deviations when using two latent forces.

  • Problem

    Data-driven models can struggle with scarce data and extrapolation, while mechanistic models may require specifying interactions and parameters that are infeasible to identify.

    Source: §1 · p. 1

  • Method

    The paper combines Gaussian-process priors over latent functions with linear differential equations to form probabilistic latent force models.

    Source: §1 · p. 1, §2 · p. 3, §3.1 · p. 3

  • Results

    The latent force model outperforms MTGP, SLFM, and independent Gaussian processes on motion-capture RMSE and R2; for Jura with Q = 2, heat-kernel results are within standard deviations of alternatives.

    Source: Motion Capture data · p. 8, Prediction of Metal Concentrations · p. 12

  • Takeaways & Limitations

    The paper formulates a probabilistic kernel model encoding coupled dynamical-system behavior and reports more accurate predictions.

    Source: §7 · p. 16

    The first-order latent force model is a simplification that includes only output first derivatives, with masses set to zero and damper constants to one.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Purely data driven approaches for machine learning present difficulties when data is scarce relative to the complexity of the model or when the model is forced to extrapolate. On the other hand, purely mechanistic approaches need to identify and specify all the interactions in the problem at hand (which may not be feasible) and still leave the issue of how to parameterize the system. In this paper, we present a hybrid approach using Gaussian processes and differential equations to combine data driven modelling with a physical model of the system. We show how different, physically-inspired, kernel functions can be developed through sensible, simple, mechanistic assumptions about the underlying system. The versatility of our approach is illustrated with three case studies from motion capture, computational biology and geostatistics.

1 Introduction

Data-driven models can struggle with scarce data and extrapolation, while mechanistic models can be too complex to specify and parameterize. The paper proposes a flexible hybrid that uses simple mechanistic assumptions to shape Gaussian-process models.

1 Introduction

  • Data-driven models may predict poorly with scarce data relative to system complexity or when extrapolating beyond observed regimes.Their regularization can also encode physical intuition, such as smoothness.Source: §1 · p. 1
  • Mechanistic models can be difficult to construct for complex systems, and identifying all interactions and parameters may be infeasible.Even when physical processes can be described, the resulting model may become extremely complex.Source: §1 · p. 1
  • Mechanistic structure can support accurate predictions in regions without training data, an advantage motivating its combination with data-driven flexibility.The paper illustrates this extrapolation advantage with spacecraft entering orbits without available data.Source: §1 · p. 1
  • The paper combines a typically simplified mechanistic model with enough flexibility to fit systems when its assumptions are not fully met.It uses mechanistic structure to encode salient characteristics, without requiring model components to correspond to actual physical components.Source: §1 · p. 1
  • The paper develops linear dynamical and partial differential equation models and illustrates them in motion capture, Drosophila development, and Swiss Jura pollutant data.The motion-capture model is intended to capture salient movement features without exactly replicating human physics.Source: §1 · pp. 1–2

2 From latent variables to latent functions

The paper turns latent variables into time-varying latent functions with Gaussian-process priors, then couples them to a physical system. This yields latent force models that retain probabilistic flexibility while incorporating mechanistic structure.

2 From latent variables to latent functions

  • A reduced-dimensional latent-variable model relates high-dimensional observations Y to latent representations U, with Q < D.The familiar probabilistic formulation can be extended to temporal data by placing a Gaussian-process prior on latent values across time.Source: §2 · p. 2
  • A Gaussian process makes latent variables functions of time, with a covariance function describing their values at the input times.The paper assumes zero mean and independence across latent dimensions.Source: §2 · p. 2
  • The paper interprets latent functions as forces acting on outputs and extends the spring model to include dampers and mass.The resulting second-order system can represent inertia and resonance, characteristics absent from the simpler latent-variable model.Source: §2 · pp. 2–3
  • The latent force model combines a physical system with a probabilistic prior over latent variables, using unobserved functions as inputs.The paper uses a marionette analogy: a small number of latent inputs control a high-dimensional representation of movement.Source: §2 · p. 3

3 Latent Force Models in Practise

The paper derives Gaussian-process models from linear latent-force systems, estimates their hyperparameters through marginal likelihood, and extends the framework to higher-order systems and multidimensional inputs. It also reviews sparse approximations for reducing computational cost.

3 Latent Force Models in Practise

  • The section lays out a practical procedure: derive output covariances, estimate hyperparameters by marginal likelihood, and extend the model to higher-order and multidimensional settings.It also reviews efficient approximations to reduce computational complexity.Source: §3 · p. 3

3.1 First-order Latent Force Model

  • The first-order model sets masses to zero and damper constants to one, and its latent forces receive independent Gaussian-process priors.Linearity makes the outputs Gaussian processes; their covariance depends on the latent-function covariance, for which the paper assumes a squared-exponential form.Source: §3.1 · pp. 3–4

3.2 Hyperparameter learning

  • After marginalizing latent forces, the model estimates unknown covariance parameters by maximizing the marginal likelihood and uses Gaussian-process regression for prediction.The posterior distribution of latent forces can also be computed.Source: §3.2 · p. 5

3.3 Higher-order Latent Force Models

  • Higher-order models use a Green’s function to solve the differential equation and derive output covariances from the latent-force priors.The same marginal-likelihood learning and prediction procedure applies, with parameters describing the Green’s function and latent-process length-scales.Source: §3.3 · pp. 5–6

3.4 Multidimensional inputs

  • For multidimensional inputs, linear partial differential equations relate latent forces and outputs, with boundary conditions replacing initial conditions.Covariances and hyperparameters are handled in a manner similar to the one-dimensional-input model.Source: §3.4 · p. 7

3.5 Efficient approximations

  • The paper describes multi-output PITC and DTCVAR approximations for reducing computation in Gaussian-process models.DTCVAR establishes a lower bound on marginal likelihood and reduces computational complexity to O(DNK^2).Source: §3.5 · p. 7

4 Second Order Dynamical System

The paper linearizes a second-order dynamical system to model motion-capture angles, retaining richer dynamics such as inertia and resonance. In cross-motion tests, the latent force model outperforms the comparison methods on RMSE and explained variance.

4 Second Order Dynamical System

  • The motion-capture model linearizes a second-order system whose true dynamics are nonlinear, while retaining the capacity to represent inertia and resonance.The authors state that this is not the correct physical model for the system, but expect it to help extrapolate across motions.Source: §4 · p. 7

Motion Capture data

  • The latent force model outperforms MTGP, SLFM, and independent Gaussian processes on motion-capture RMSE and R2 in the reported tests.The authors particularly note its R2 performance as indicating an ability to generate more realistic motions.Source: Motion Capture data · p. 8

5 Partial Differential Equations and Latent Forces

The section extends latent-force Gaussian processes to partial differential equations, deriving multi-input covariance functions and applying them to gene expression and pollutant prediction.

5 Partial Differential Equations and Latent Forces

  • The section extends time-only ODE latent-force models to PDEs, yielding multioutput Gaussian processes over multiple inputs.It presents a spatio-temporal covariance example and a simplified diffusion-based covariance for higher-dimensional inputs.Source: §5 · pp. 8–9

5.1 Gap-gene network of Drosophila melanogaster

  • The gap-gene model replaces nonlinear production with a linear combination of latent forces, retaining decay and spatial diffusion.Its exact solution is specified for a zero initial condition and zero boundary values at both ends of the spatial domain.Source: §5.1 · pp. 9–10

Prediction of gene expression data

5.2 Diffusion in the Swiss Jura

  • The Jura pollutant model treats concentrations as a snapshot of diffusion from latent functions, with diffusion scale estimated from the data.The proposed covariance assumes zero external force and uses a Gaussian smoothing Green’s function.Source: §5.2 · pp. 11–12

Prediction of Metal Concentrations

  • All methods beat independent GPs on RMSE and explained variance; with Q=1, the Heat kernel beats MTGP, while Q=2 differences fall within standard deviations.With Q=1, the Heat kernel also performs similarly to its Q=2 version, using one latent force as a simplified explanation for the outputs.Source: Prediction of Metal Concentrations · p. 12, Table 3 · p. 13, Table 4 · p. 13

6 Related work

Related work covers differential-equation parameter estimation, Gaussian-process models using derivative information, and multi-output covariance constructions. The latent force model differs by using an equation as prior knowledge about unknown dynamics, rather than assuming it precisely describes the system.

6 Related work

  • Latent force models use differential equations to encode prior knowledge about important features of unknown dynamics, unlike parameter-estimation methods that begin with an accurate interaction description.The equation is part of the modeling problem, not assumed to represent the system's real dynamics exactly.Source: §6 · p. 13
  • Classical parameter-fitting methods solve an ODE numerically or approximate its solution with basis functions, then optimize equation parameters against observations or residuals.Numerical methods may use sensitivity equations for gradients; collocation requires smooth basis functions to obtain derivatives.Source: §6 · p. 14
  • In Gaussian-process regression for differential equations, the equation's output is assigned a GP prior whose covariance is obtained by differentiating the underlying kernel.The cited formulation treats observations as noisy measurements of the differential-equation output.Source: §6 · p. 14
  • A related GP approach places priors on derivatives around an equilibrium, and including derivative information reduces posterior predictive uncertainty relative to using function observations alone.The derivatives are estimated a posteriori through Bayesian linear regression.Source: §6 · p. 15
  • Multi-output GP work includes linear model of coregionalization and process-convolution constructions for valid cross-output covariances.The LMC represents each output as a linear combination of basic processes, some sharing covariance functions.Source: §6 · p. 15

7 Conclusion

The paper presents a hybrid of Gaussian-process priors and linear differential equations to model interactions among variables. Its kernel-based probabilistic model encodes coupled dynamical behavior and is reported to allow more accurate predictions.

7 Conclusion

  • The paper combines Gaussian-process priors with linear differential equations in a hybrid model of interactions between variables.Source: §7 · p. 16
  • The resulting kernel-based probabilistic model encodes coupled behavior among several dynamical systems and allows more accurate predictions.Source: §7 · p. 16
  • Extensions described include non-linear, cascaded, stochastic, and switching latent force models, with different inference methods or covariance and parameter assumptions.Non-Gaussian likelihoods call for Laplace approximation or sampling; stochastic models treat latent forces as white noise, while switching models allow parameters to change over time.Source: §7 · p. 16

A Preprocessing for the mocap data

The appendix describes signal-to-noise ratio as the performance measure for selecting motion angles, using Gaussian-process regressors for the outputs.

A Preprocessing for the mocap data

B Expression for C(n, m, ℓx)

The appendix gives cases for the expression C(n, m, ℓx), distinguishing equal indices, unequal indices, and parity combinations.

B Expression for C(n, m, ℓx)

C Expression for C(x′, n, ℓx)

The supplied C-expression fragment contains two plus-linked terms involving γn and ℓx. The accompanying notation defines γn and the imaginary-part operator I[·].

C Expression for C(x′, n, ℓx)

Loading 1107.2699v2…