PaperlayerView exact revision on arXiv

Source-linked AI summary

Virtual vs. Real: Trading Off Simulations and Physical Experiments in Reinforcement Learning with Bayesian Optimization

Alonso Marco, Felix Berkenkamp, Philipp Hennig, Angela P. Schoellig, Andreas Krause, Stefan Schaal, Sebastian Trimpe

arXiv:1703.01250v1cs.ROcs.LGcs.SYeess.SY

TL;DR

Policy tuning through physical experiments can be time-consuming, while reinforcement-learning search may require many experiments. This paper extends Entropy Search to choose between simulations and physical experiments by trading information gain against evaluation cost. On cart-pole control, it reports better controllers with fewer physical experiments than standard Entropy Search.

  • Problem

    Policy parameters are often tuned manually, and reinforcement-learning methods can require substantial experimental time to find good settings.

    Source: §I · p. 1

  • Method

    The method extends Entropy Search to compare information gains from simulations and physical experiments relative to their costs.

    Source: §I · pp. 1–2

  • Results

    MF-ES finds controllers that are 33.23% better, on average, and uses 2.7 physical experiments versus ES’s 3.5, 22.86% less, plus 11.9 simulations.

    Source: D. Results · p. 7

  • Takeaways & Limitations

    On cart-pole policy optimization, simulator information can reduce the data needed to find good control policies globally.

    Source: §VI · p. 7

Abstract

from arXiv · show

In practice, the parameters of control policies are often tuned manually. This is time-consuming and frustrating. Reinforcement learning is a promising alternative that aims to automate this process, yet often requires too many experiments to be practical. In this paper, we propose a solution to this problem by exploiting prior knowledge from simulations, which are readily available for most robotic platforms. Specifically, we extend Entropy Search, a Bayesian optimization algorithm that maximizes information gain from each experiment, to the case of multiple information sources. The result is a principled way to automatically combine cheap, but inaccurate information from simulations with expensive and accurate physical experiments in a cost-effective manner. We apply the resulting method to a cart-pole system, which confirms that the algorithm can find good control policies with fewer experiments than standard Bayesian optimization on the physical system only.

I. INTRODUCTION

The introduction frames policy tuning as costly physical experimentation and identifies the lack of a principled way to trade off simulator and experiment costs against accuracy. It proposes extending Entropy Search to combine both information sources and reports faster cart-pole policy optimization than standard Bayesian optimization.

I. INTRODUCTION

  • Manual tuning and reinforcement-learning policy search can require substantial physical experimental time before finding optimal or reasonable parameters.Robotics simulators are commonly available, but are approximations of the real system.Source: §I · p. 1
  • Earlier simulator-transfer approaches did not explicitly trade off the costs and information gains of simulation and real experiments.The introduction also notes that prior Bayesian optimization methods operated directly on the real system.Source: §I · pp. 1–2
  • The proposed multi-source Bayesian optimization extends Entropy Search, using entropy to compare information from cheap, inaccurate simulations and costly, precise experiments against their costs.The method automatically selects which source to evaluate.Source: §I · pp. 1–2
  • On cart-pole policy optimization, the approach is reported to speed optimization significantly relative to standard Bayesian optimization.The paper presents this as its first application of a multiple-information-source framework to reinforcement learning and controller-parameter optimization.Source: §I · p. 2

II. PROBLEM STATEMENT

The problem is to find policy parameters that globally minimize task cost on the real system while limiting the experiments needed to do so. The method uses an approximate simulator as an information source and chooses both the parameters and source for each evaluation.

II. PROBLEM STATEMENT

  • The objective is to find policy parameters θ that globally minimize task cost on the real system.The policy parameters lie within a domain D.Source: §II · p. 2
  • Physical evaluations take time and cause robot wear, so the goal is to minimize iterations before determining the real-system optimum.An evaluation measures policy performance on the real system, for example through an error signal over a fixed time horizon.Source: §II · p. 2
  • An approximate simulator estimates real-system cost and informs the search, but cannot directly determine the real-system optimum.At each iteration, the method chooses both the parameter vector and whether to simulate or experiment; the overall objective is to minimize total evaluation effort.Source: §II · p. 2

III. PRELIMINARIES

The preliminaries introduce Gaussian processes as models for unknown cost functions and explain how they represent prior knowledge and uncertainty. GP predictions are conditioned on noisy observations.

A. Gaussian Processes (GPs)

  • Gaussian processes provide a nonparametric model for the unknown relationship between policy parameters and task cost.They model function values as random variables whose finite collections have a joint Gaussian distribution.Source: A. Gaussian Processes (GPs) · p. 2
  • A GP prior uses a mean function to encode prior knowledge and a covariance function, or kernel, to model relationships and uncertainty between function values.Kernel choice encodes assumptions about smoothness and rate of change.Source: A. Gaussian Processes (GPs) · pp. 2–3
  • Predictions at new inputs are conditioned on past observations, which the paper assumes are noisy measurements of the true function.The noise variance may depend on the input.Source: A. Gaussian Processes (GPs) · p. 3

B. Bayesian Optimization

Bayesian optimization uses a statistical model of an expensive objective to search for its global optimum. Entropy Search selects evaluations for their expected reduction in uncertainty about the minimum’s location.

B. Bayesian Optimization

  • Bayesian optimization uses statistical models of expensive-to-evaluate objective functions to seek their global optimum.For GP models, the posterior mean and variance help identify promising parameters.Source: B. Bayesian Optimization · p. 3
  • Entropy Search selects parameters expected to maximally reduce uncertainty about the location of the cost function’s minimum.It measures uncertainty with the entropy of the distribution over the minimum’s location and uses a non-uniform grid approximation.Source: B. Bayesian Optimization · p. 3
  • The next evaluation is chosen for its expected information gain, while the current best parameter estimate minimizes the GP posterior.Computing the expected entropy change requires several approximations.Source: B. Bayesian Optimization · p. 3

IV. REINFORCEMENT LEARNING WITH SIMULATIONS

The method extends Entropy Search to combine simulations and physical experiments, addressing simulator error and the different costs of each source. A multi-source Gaussian process models how simulation data informs—but does not fully determine—the physical-system cost.

IV. REINFORCEMENT LEARNING WITH SIMULATIONS

  • The approach extends Entropy Search to multiple information sources, with simulator error and evaluation effort as its two main modeling challenges.The exposition focuses on one simulator but states that the approach can extend to an arbitrary number of sources.Source: §IV · p. 3

A. GP Model for Multiple Information Sources

  • The physical-system cost is modeled as the simulator cost plus a systematic error term, with a binary source indicator distinguishing simulation from physical evaluation.The extended parameter is a = (θ, δ), where δ = 0 denotes simulation and δ = 1 denotes a physical experiment.Source: A. GP Model for Multiple Information… · p. 3
  • The kernel combines simulator covariance with an error covariance that is switched off for simulations, so simulator observations explain only part of the physical-system cost.The simulator and error kernels represent the simulator cost and its difference from the physical-system cost, respectively.Source: A. GP Model for Multiple Information… · pp. 3–4
  • Different measurement-noise variances for simulation and physical experiments let one Gaussian process model both information sources.The text characterizes simulation as typically noise-free and states that predictions are made using the resulting kernel and noise structure.Source: A. GP Model for Multiple Information… · p. 4

B. Optimization

MF-ES selects between simulations and physical experiments by comparing expected information gain about the physical-system minimum against evaluation effort. In a typical run, it begins with cheaper simulations, switches to physical experiments, and can return to simulation when useful.

B. Optimization

  • MF-ES targets low entropy in the distribution over the physical-system minimum, but information gain alone would always select physical experiments.Physical evaluations inform the cost directly, whereas simulations inform only part of it.Source: B. Optimization · p. 4
  • The algorithm compares expected entropy reduction per unit of effort for simulations and physical experiments, whose effort measures satisfy tsim < texp.Entropy provides a consistent information measure for both sources, including when their noise variances differ.Source: B. Optimization · p. 4
  • The typical run converges to a distribution concentrated around the cost minimum while using fewer physical experiments than optimization using only physical experiments.Simulations initially reduce uncertainty about the simulator and provide partial information about the true system cost.Source: B. Optimization · p. 5
  • The acquisition rule adapts source selection to simulator quality, allowing MF-ES to switch back to simulations instead of following a fixed two-stage schedule.In the illustrated run, simulations are initially advantageous because their effort is set 40% below that of the real system.Source: B. Optimization · pp. 4–5

V. EXPERIMENTAL RESULTS

The paper evaluates MF-ES on feedback-controller optimization for an unstable cart-pole system.

V. EXPERIMENTAL RESULTS

  • The evaluation applies MF-ES to an unstable cart-pole system.The system is identified as the setting illustrated in Fig. 1.Source: §V · p. 5
  • The optimization task is to tune the system's feedback controller.Source: §V · p. 5
  • The cart-pole application is presented as an evaluation of MF-ES.Source: §V · p. 5

A. Experimental Setup

The experimental setting uses a Quanser linear inverted pendulum with a manufacturer-provided Simulink dynamics model. The controller-tuning problem uses LQR-parameterized feedback gains and evaluates cost in 30-second balancing runs or simulations.

A. Experimental Setup

  • The testbed is a Quanser Linear Inverted Pendulum, with a manufacturer-provided nonlinear Simulink model used as the simulator.The setup is controlled through dedicated hardware from a standard laptop using Matlab/Simulink.Source: A. Experimental Setup · p. 5

B. Controller Tuning Problem

  • The task tunes static state-feedback gains to minimize a cost penalizing deviations from upright equilibrium and control effort.The gains are parameterized through an LQR design using weights based on a nominal linearized model; this parameterization reduces dimensionality and offers stability and robustness properties with respect to that nominal system.Source: B. Controller Tuning Problem · p. 5
  • Each physical cost evaluation uses a 30-second balancing experiment, while each simulation evaluation runs for 30 seconds.The cost is computed from the recorded states and inputs.Source: B. Controller Tuning Problem · p. 5
  • Controllers violating state or input safety limits are classified as unstable and assigned fixed penalties of Jexp = 0.06 and Jsim = 0.04.The penalties were chosen conservatively above the cost of the worse stabilizing controller observed in initial evaluations.Source: B. Controller Tuning Problem · p. 5
  • The final MF-ES posterior predicts the real-system cost despite simulator evaluations being systematically lower, a bias represented by a lower simulator prior mean.The figure marks the best-guess global minimum and shows posterior mean and ±2 standard deviations conditioned on simulation and experiment data.Source: Figure 3 · p. 6

C. Bayesian Optimization Settings

The settings assign simulations lower effort than physical experiments, use Gaussian-process priors and noise models for both sources, and stop exploration when the best-guess estimate is sufficiently stable and certain.

C. Bayesian Optimization Settings

  • Simulations are assigned 1 s and physical experiments 30 s, making simulations 30 times cheaper in effort.Source: C. Bayesian Optimization Settings · p. 6
  • Both simulation and error GPs use rational-quadratic kernels, with hyperparameters fixed after initial experiments and pessimistic prior means corresponding to unstable-controller penalties.The prior means are 0.04 for simulation and 0.02 for the error GP.Source: C. Bayesian Optimization Settings · p. 6
  • Real-system evaluation noise is ηexp = 2.08 × 10^-4, versus simulator noise ηsim = 10^-5.The simulator noise is described as roughly twenty times lower.Source: C. Bayesian Optimization Settings · p. 6
  • Exploration stops when the best-guess posterior mean is steady within σerr/4 over three iterations and its posterior standard deviation is below σerr/2; the final guess is then tested physically.The final controller's cost on the physical experiment is the learning procedure's outcome.Source: C. Bayesian Optimization Settings · p. 6

D. Results

MF-ES achieved better final controller costs than physical-only ES while using fewer physical experiments on average; a single run illustrates its nontrivial alternation between sources.

D. Results

  • In one run, MF-ES first used cheap simulations to identify regions of predicted instability, then requested physical experiments at iterations 10 and 14.The final selected controller had cost J(θbg) = 0.0194 on the physical system.Source: D. Results · p. 6
  • The source switching was nontrivial, and the pessimistic prior predicted instability in regions without evaluations.The paper notes that this switching cannot be reproduced by simple two-stage learning.Source: D. Results · p. 6
  • Across ten runs, MF-ES final-controller costs were 33.23% better on average than ES, while ES failed to find a stabilizing controller in 4 of 10 cases (cost 0.06).The final controller costs are compared in Fig. 5.Source: D. Results · pp. 6–7, Figure 5 · p. 7
  • MF-ES used 2.7 physical experiments on average versus ES's 3.5 (22.86% less), plus 11.9 simulations.Fig. 6 compares physical-experiment counts and shows MF-ES simulations separately.Source: Figure 6 · p. 7, D. Results · p. 7

VI. CONCLUSION

The method adaptively selects among information sources with different accuracies and evaluation costs, and its cart-pole application found that simulator information can reduce data needed to find good control policies globally.

  • Simulator model information reduced the data required to globally find good control policies in the cart-pole task.Source: §VI · p. 7
  • The method adaptively selects among information sources with different accuracies and evaluation efforts, including real-robot experiments and simulations.Source: §VI · p. 7
  • The method was applied to policy optimization on a cart-pole system.Source: §VI · p. 7
Loading 1703.01250v1…