Gaussian Processes for Data-Efficient Learning in Robotics and Control

Marc Peter Deisenroth; Dieter Fox; Carl Edward Rasmussen

Source-linked AI summary

Gaussian Processes for Data-Efficient Learning in Robotics and Control

Marc Peter Deisenroth, Dieter Fox, Carl Edward Rasmussen

arXiv:1502.02860v2Feb 10, 2015stat.MLcs.LGcs.ROeess.SY

TL;DR

Reinforcement learning can require impractically many real-system interactions and often relies on task-specific knowledge. PILCO addresses this by learning probabilistic GP dynamics, propagating uncertainty for planning and evaluation, and optimizing policies with analytic gradients. It advances state-of-the-art RL learning speed by at least an order of magnitude and has been applied to robotics without demonstrations or informative initialization.

Problem
Many RL methods require impractically many environment interactions, while data-efficient alternatives often depend on task-specific knowledge.
Method
PILCO uses a probabilistic nonparametric GP dynamics model, deterministic approximate inference, and analytic policy gradients for model-based policy search.
Results
PILCO advances state-of-the-art RL learning speed by at least an order of magnitude in continuous state and control spaces.
Takeaways & Limitations
PILCO was directly applied to robotics without human demonstrations or other informative initializations or prior knowledge.
Takeaways & Limitations
PILCO does not model temporal correlation in model uncertainty, which can underestimate uncertainty, although moment matching is typically conservative.

Abstract

from arXiv · show

Autonomous learning has been a promising direction in control and robotics for more than a decade since data-driven learning allows to reduce the amount of engineering knowledge, which is otherwise required. However, autonomous reinforcement learning (RL) approaches typically require many interactions with the system to learn controllers, which is a practical limitation in real systems, such as robots, where many interactions can be impractical and time consuming. To address this problem, current learning approaches typically require task-specific knowledge in form of expert demonstrations, realistic simulators, pre-shaped policies, or specific knowledge about the underlying dynamics. In this article, we follow a different approach and speed up learning by extracting more information from data. In particular, we learn a probabilistic, non-parametric Gaussian process transition model of the system. By explicitly incorporating model uncertainty into long-term planning and controller learning our approach reduces the effects of model errors, a key problem in model-based learning. Compared to state-of-the art RL our model-based policy search method achieves an unprecedented speed of learning. We demonstrate its applicability to autonomous learning in real robot and control tasks.

1 INTRODUCTION

PILCO addresses the interaction burden of reinforcement learning by learning a probabilistic GP dynamics model and incorporating uncertainty into planning and policy learning. The framework is presented as data-efficient and applicable to realistic robotic control without task-specific expert knowledge.

Many RL algorithms require thousands of trials even for low-dimensional systems, making real-world robotic learning impractical.
Data-efficient RL typically depends on expert demonstrations, realistic simulators, pre-shaped policies, or explicit dynamics knowledge.
PILCO learns nonparametric GP dynamics, propagates uncertainty through long-term predictions, and improves policies using analytic gradients.
The probabilistic model represents uncertainty over plausible transition functions rather than assigning full confidence to one deterministic model.
PILCO’s learning speed is reported as directly enabling application to realistic control and robotic hardware platforms.

2 RELATED WORK

Prior controller-learning approaches often rely on manually specified parametric models or value functions, limiting their generality and scalability. PILCO instead combines nonparametric GP dynamics with Bayesian averaging and policy search without state-space discretization.

Parametric models require problem-specific manual specification and are not suited to learning models across a broad range of tasks.
Nonparametric regression can extract latent dynamics features from data while representing model uncertainty through Bayesian methods or noise assumptions.
PILCO enables closed-form Bayesian averaging over infinitely many plausible dynamics models using nonparametric GPs.
Earlier GP-based RL approaches modeled global value functions, which could require many data points because value functions are often discontinuous.

3 MODEL-BASED POLICY SEARCH

PILCO learns a probabilistic GP forward model and uses approximate distribution propagation, analytic policy gradients, and iterative policy updates to search for controllers. Its dynamics model trains on state-action pairs and state differences.

Problem formulation: The system evolves according to unknown transition dynamics with continuous states and controls, additive Gaussian system noise, and a policy parameterized by θ.
Learning algorithm: Learning alternates between fitting the GP model, evaluating policies through approximate inference, computing gradients, and updating controller parameters.
Core framework: PILCO combines a probabilistic GP dynamics model, deterministic approximate inference for long-term predictions, and analytic policy gradients for policy improvement.
Model learning: The GP uses state-action pairs as inputs and state differences ∆t = xt+1 − xt as training targets.
Model learning: Using state differences gives the dynamics model an implicit prior mean m(x) = x, so predictions outside the training data remain constant rather than reverting to zero.
Policy evaluation: Long-term policy evaluation cascades Gaussian one-step predictions while approximating uncertain state-action inputs as Gaussian distributions.
Policy improvement: Analytic gradients are obtained by repeated chain-rule application through predicted state-distribution moments and policy-dependent control moments.

4 LONG-TERM PREDICTIONS

PILCO propagates uncertain state-action distributions through GP dynamics over multiple steps, approximating each successor distribution as Gaussian. It compares moment matching with linearization, trading exact first two moments against computational efficiency.

Long-term prediction: Long-term predictions are computed iteratively by approximating each predictive state distribution as Gaussian and propagating it through the GP model.
Inference methods: Moment matching computes the first two predictive moments exactly, whereas linearization of the posterior GP mean is computationally advantageous.
Moment propagation: Predictive covariance includes expected function variance and system noise, while conditionally independent target GPs omit conditional cross-target covariance terms.
Moment propagation: The covariance of the next state combines current-state covariance, predictive difference covariance, and state-difference cross-covariances.
Linearization: Linearization evaluates the posterior GP mean at the input mean and maps the Gaussian input through the resulting linear approximation.
Approximation caveat: Linearization can produce predictive distributions that are too tight, while the exact predictive distribution is analytically intractable.

5 POLICY

PILCO designs policies that support probabilistic long-term prediction while respecting bounded control amplitudes. It provides linear and deterministic-GP representations with analytically tractable predictive moments and control distributions.

Policy requirements: PILCO maps Gaussian state distributions to predictive control distributions, using moment matching to approximate p(u) = p(π(x)).This supports long-term prediction even for deterministic policies.
Constrained control signals: Control limits u ∈ [−u_max, u_max] are enforced by applying a bounded, differentiable squashing function to an unconstrained preliminary policy.The resulting policy is π(x) = σ(˜π(x)) scaled by u_max.
Constrained control signals: The squashing function is a third-order Fourier expansion of a trapezoidal wave, normalized to [−1, 1], enabling analytical predictive moments for Gaussian states.Its periodicity is typically irrelevant because the preliminary policy operates within a single half wave.
Policy representations: PILCO supports linear and nonlinear preliminary policies whose moments can be computed in closed form for Gaussian-distributed states.The linear policy is exact under Gaussian inputs, while the nonlinear representation uses a deterministic GP with fixed basis functions.
Policy representations: The deterministic GP policy is functionally equivalent to a regularized RBF network and contains no uncertainty about its underlying function.Its parameters include basis-function centers, length-scales, and targets.
Successor-state prediction: Successor-state computation analytically propagates state distributions through the policy, joint state-control distribution, and probabilistic GP dynamics model.Cross-covariance information between state and control is used when forming the joint distribution.

6 COST FUNCTION

PILCO uses a saturating distance-based cost that supports both exploration and exploitation under predictive uncertainty. Far from the target it favors uncertain states, while near the target it favors concentrated, confident trajectories.

Cost design: The cost penalizes Euclidean distance to the target and is locally quadratic but saturates at unity for large deviations.The width is controlled by σ_c.
Cost design: The saturating cost is preferred to a quadratic cost because large early predictive uncertainty makes quadratic-cost policy gradients less useful.PILCO therefore uses the saturating cost as its default.
Exploration and exploitation: Far from the target, uncertainty can place probability mass in low-cost tails, producing automatic exploration in regions with poor dynamics models.Interaction in those regions reduces local model uncertainty and tightens later predictions.
Exploration and exploitation: Close to the target, peaked state distributions concentrate mass in low-cost regions, so the policy favors exploitation rather than broad uncertainty.Wide distributions are more likely to include high-cost tails near the target.
Exploration and exploitation: Combining probabilistic dynamics, Bayesian inference, and the saturating cost yields exploration far from the target and confident trajectories near it.This behavior occurs even when the policy greedily minimizes expected cost.

7 EXPERIMENTAL RESULTS

Experiments show that PILCO learns controllers efficiently across simulated and robotic tasks, with moment matching generally outperforming linearization in reliability and success. The results also demonstrate scaling to higher-dimensional control and highlight dependence on approximate inference quality and Bayesian uncertainty handling.

Approximate Inference Assessment: Moment matching has computational complexity O(n^2E^2D), requiring more resources than linearization but generally learning faster and more reliably.Linearization often became stuck in local minima because it underestimated predictive variances.
Cart-Pole Swing-Up: PILCO learned a cart-pole controller reliably after 15–20 s of experience, solving approximately 95% of test runs with moment matching.Linearization achieved about 83% success in the same task.
Double-Pendulum Swing-Up: Moment matching learned the double-pendulum task faster and more successfully than linearization, reaching approximately 95% success after about 50 s of data.Linearization achieved about 80% success on average.
Importance of Bayesian Averaging: Bayesian model uncertainty was necessary for successful cart-pole learning, because early models extrapolate into regions with large posterior uncertainty.Deterministic point predictions can produce arbitrary trajectories and vanishing policy gradients.
Scaling to Higher Dimensions: Unicycling: PILCO learned a unicycle controller in about 20 trials and 30 s of experience, achieving approximately 93% success across 1,000 test runs.Extreme initial configurations were sometimes impossible to recover because of torque constraints.
Hardware Applicability: PILCO also learned a real cart-pole controller fully automatically in ten trials and 17.5 s of total experience.The learned controller and dynamics model were obtained without relying on the simulated system’s exact parameters.

8 DISCUSSION

The discussion highlights PILCO’s model-based policy search, Bayesian treatment of uncertainty, and practical applicability, while noting approximation and observation-noise limitations.

PILCO framework: PILCO uses a GP forward model, deterministic approximate inference, and analytic gradients to optimize expected long-term cost.This avoids trajectory-sampling gradients and supports learning thousands of policy parameters.
Exploration: PILCO’s Bayesian averaging provides natural exploration, while an explicit UCB utility can encourage additional exploration.The UCB formulation adds cost standard deviations weighted by κ to expected immediate costs.
Uncertainty modeling: PILCO reduces model-error effects by incorporating model uncertainty into planning and control, but treats temporal uncertainty correlation as noise.This treatment can underestimate model uncertainty, although moment matching is typically conservative.
Scope: The article focuses on MDPs with transition noise; measurement noise presents a more challenging setting for learning GP models.The stated scope excludes the more difficult partially observed measurement-noise case.
Applications: The framework has been extended to reference tracking, constrained environments, and obstacle avoidance without collisions during training.Initially uncertain policies stayed away from obstacles.

9 CONCLUSION

The conclusion presents PILCO as a practical model-based policy search method that improves learning speed and demonstrates the practicality of Bayesian nonparametric control models.

Conclusion: PILCO advances state-of-the-art RL methods for continuous state and control spaces by at least an order of magnitude in learning speed.The method uses analytic gradients and reduces model-error effects during model learning, planning, and policy learning.
Conclusion: PILCO has been applied directly to robotics without human demonstrations, informative initializations, or other prior knowledge.This supports autonomous learning from scratch on robotic hardware.
Conclusion: Bayesian inference and nonparametric models can provide practicable approaches for learning controllers while avoiding excessive reliance on explicit models.The conclusion identifies these approaches as relevant to classical control settings.

APPENDIX A TRIGONOMETRIC INTEGRATION

Appendix A provides exact integral equations for trigonometric functions required to implement the algorithms discussed in the paper.

APPENDIX A TRIGONOMETRIC INTEGRATION: The appendix gives exact trigonometric integral equations used in implementing the discussed algorithms.It assumes x is Gaussian distributed with mean µ and variance σ^2.

APPENDIX B GRADIENTS

Appendix B introduces derivative identities and develops gradient calculations for the moment-matching approximation.

APPENDIX B GRADIENTS: The appendix begins with basic derivative identities used in subsequent gradient computations.These identities support the derivative calculations for moment matching.
APPENDIX B GRADIENTS: The appendix distinguishes B(:, i) as the ith column of B and B(i, :) as the ith row.These notation conventions clarify the matrix derivatives.
APPENDIX B GRADIENTS: The derivatives cover predictive mean, predictive covariance, and cross-covariance with respect to the input distribution’s mean and covariance.The input is the control-augmented state.

B.2.1 Derivatives of the Predictive Mean with Respect to the Input Distribution

This section derives derivatives of the predictive Gaussian-process mean with respect to the input distribution's mean and covariance. It additionally accounts for covariance-matrix symmetry and tensor contraction in the covariance derivative.

The predictive GP mean is differentiated with respect to the mean and covariance of the input distribution.
Derivative with respect to the Input Mean: The derivative with respect to the input mean is obtained from the predictive-mean function value in Eq. (51).
Derivative with Respect to the Input Covariance Matrix: The derivative with respect to the input covariance matrix Σt−1 is computed explicitly.
Derivative with Respect to the Input Covariance Matrix: The covariance derivative accounts explicitly for the symmetry of Λa + ˜Σt−1.
Derivative with Respect to the Input Covariance Matrix: A tensor contraction is used when difference vectors are multiplied onto the matrix derivative.

B.2.2 Derivatives of the Predictive Covariance with Respect to the Input Distribution

This section derives derivatives of the predictive covariance matrix with respect to the input distribution's mean and covariance. The covariance derivative is assembled from derivatives of q_a, q_b, and Q.

The predictive covariance matrix entries are defined for target dimensions a, b = 1, . . . , E.
Derivative with Respect to the Input Mean: Derivatives of predictive covariance entries are taken with respect to the predictive mean.
Derivative with Respect to the Input Covariance Matrix: Derivatives of predictive covariance entries are also taken with respect to the input distribution's covariance matrix.
Derivative with Respect to the Input Covariance Matrix: Because ∂q_a/∂˜Σt−1 and ∂q_b/∂˜Σt−1 are known from Eq. (56), the remaining term to compute is ∂Q/∂˜Σt−1.
Derivative with Respect to the Input Covariance Matrix: The desired derivative is obtained by defining intermediate quantities and evaluating partial derivatives of Q_ij with respect to the covariance matrix.
Derivative with Respect to the Input Covariance Matrix: The derivation concludes the partial derivative after specifying the relevant index ranges and covariance-matrix derivative.