Automatic LQR Tuning Based on Gaussian Process Global Optimization

Alonso Marco; Philipp Hennig; Jeannette Bohg; Stefan Schaal; Sebastian Trimpe

Source-linked AI summary

Automatic LQR Tuning Based on Gaussian Process Global Optimization

Alonso Marco, Philipp Hennig, Jeannette Bohg, Stefan Schaal, Sebastian Trimpe

arXiv:1605.01950v1May 6, 2016cs.ROcs.LGeess.SY

TL;DR

The paper addresses automatic controller tuning when nominal designs are affected by modeling and measurement imperfections and experiments are limited. It combines LQR tuning with Entropy Search, modeling experimental cost with a Gaussian process to select informative evaluations. On a robot balancing task, the approach improved controllers in both 2D and 4D tuning experiments, including poor initialization.

Problem
Nominal controllers can be suboptimal under nonlinear dynamics, model uncertainty, and imperfect measurements, while controller tuning on physical plants should use few experiments.
Method
The framework combines LQR tuning with Entropy Search, using a Gaussian process to represent the unknown cost and select evaluations with high information gain.
Results
31.9% improvement was obtained over the initial controller after 20 iterations in the accurate-nominal-model experiment.
Takeaways & Limitations
The experiments demonstrate potential for automatic controller tuning on robotic platforms across 2D and 4D parameter spaces.

Abstract

from arXiv · show

This paper proposes an automatic controller tuning framework based on linear optimal control combined with Bayesian optimization. With this framework, an initial set of controller gains is automatically improved according to a pre-defined performance objective evaluated from experimental data. The underlying Bayesian optimization algorithm is Entropy Search, which represents the latent objective as a Gaussian process and constructs an explicit belief over the location of the objective minimum. This is used to maximize the information gain from each experimental evaluation. Thus, this framework shall yield improved controllers with fewer evaluations compared to alternative approaches. A seven-degree-of-freedom robot arm balancing an inverted pole is used as the experimental demonstrator. Results of a two- and four-dimensional tuning problems highlight the method's potential for automatic controller tuning on robotic platforms.

I. INTRODUCTION

The paper targets information-efficient automatic controller tuning on physical robots, replacing time-consuming manual or grid-search protocols with Entropy Search combined with LQR tuning. Experiments on a humanoid robot demonstrate tuning across controller dimensions and initialization qualities.

Manual tuning and grid search can be highly time-consuming, motivating automated fine-tuning from physical-plant performance with few experiments.
The framework assumes a limited budget of experimental evaluations because plant time may be limited or experiments costly.
The method globally explores a bounded controller range and returns the best known controller after a fixed number of experiments, while allowing controller failures under available safety provisions.
The proposed framework combines Entropy Search with LQR tuning and selects evaluations expected to provide information about the global minimum.
Experiments demonstrate successful automatic tuning in 2D and 4D parameter spaces, including relatively good and poor initial controllers.
The work reports direct tuning of continuous state-feedback controllers on a real robotic platform using Bayesian optimization.

II. LQR TUNING PROBLEM

The LQR tuning problem starts from a nominal linear approximation of unknown nonlinear dynamics and improves its controller using experimental performance data. This addresses suboptimality caused by modeling and measurement imperfections.

The system is modeled by unknown discrete-time nonlinear dynamics, with measurable states and an equilibrium at zero.
A nominal linear model approximates the dynamics around the zero equilibrium for controller design.
The quadratic performance cost balances state regulation against control effort through positive-definite performance weights Q and R.
The LQR computes a static gain for the nominal model and performance weights, providing a locally optimal controller for that approximation.
Experimental fine-tuning remains desirable because nonlinear dynamics, model uncertainty, and imperfect state measurement can make the nominal LQR suboptimal.

B. LQR tuning problem

The method varies LQR design weights rather than arbitrary gains, producing a structured controller search that is tuned against experimental cost. This restriction helps focus the search and avoid undesirable nominal-controller behavior.

Controller gains are parameterized through design weights Wx(θ) and Wu(θ), whose parameters θ are varied during automatic tuning.
Weight-based parameterization restricts the search space, helping avoid controllers that destabilize the nominal plant or have poor robustness properties.
Varying θ changes the controller gain and therefore the experimentally observed cost.
The tuning objective is to vary θ so that the performance cost is minimized.
The performance weights Q and R remain fixed while the design weights change during tuning.

C. Optimization problem

The optimization searches a bounded parameter domain using noisy, finite-horizon experimental cost evaluations. Because the objective shape is unknown and evaluations are expensive, the formulation is suited to limited experimental budgets.

The LQR tuning problem restricts θ to a bounded domain D representing a region where improvement or safe exploration is expected.
The cost-function shape is unknown, with neither gradient information nor convexity guarantees available.
The practical objective uses a finite but sufficiently long horizon, making each experimental cost a noisy evaluation of the infinite-horizon cost.
Each cost evaluation is expensive because it requires a physical balancing experiment lasting several minutes.

III. LQR TUNING WITH ENTROPY SEARCH

The method models the unknown, noisy controller cost with a Gaussian process and updates that model as experiments provide new evaluations. Its covariance encodes assumptions about cost-function structure, while posterior uncertainty guides subsequent optimization.

Bayesian optimization represents uncertainty over the unknown controller cost function with a Gaussian process conditioned on noisy experimental evaluations.
The GP prior is specified by a mean function and covariance function, with the squared exponential covariance used here.
The covariance function encodes assumptions about smoothness, characteristic length-scales, and signal variance of the cost function.
Conditioning on observed data yields a posterior mean and posterior variance for the cost function.
Three successive evaluations illustrate mean adjustment near observations and reduced uncertainty there, while unevaluated regions remain uncertain.
After each new cost evaluation, GP hyperparameters are improved by maximizing marginal likelihood, while automatic relevance determination removes low-influence parameter dimensions.

B. Probability measure over the location of the minimum

Entropy Search explicitly represents uncertainty about where the cost function reaches its minimum. It chooses experiments for information gain about that location rather than simply sampling where the minimum currently appears most likely.

Entropy Search explicitly models pmin(θ), the probability that each θ is the minimum of the cost function over domain D.
The minimum-location distribution is induced by the Gaussian-process distribution over possible cost functions.
Entropy Search approximates pmin(θ) on a finite non-uniform grid with higher resolution in influential regions.
The next evaluation is selected to maximize expected information gain about the minimum location, formalized through relative entropy against a uniform distribution.
Rather than evaluating where the minimum is currently believed to be, the algorithm evaluates where it expects to learn most about the minimum.
After proposing evaluations, Entropy Search returns the current best minimum-location estimate as the point with maximum approximated pmin(θ).

D. Automatic LQR tuning

Automatic LQR tuning combines the LQR design framework with Entropy Search in an iterative experimental loop. Each proposed controller is tested, its cost is added to the data, and the process seeks informative exploration under a fixed evaluation budget.

The framework combines LQR tuning with Entropy Search, using ES to solve the LQR parameter-optimization problem.
Each iteration proposes a controller through θ, tests it experimentally, records a cost evaluation, and updates the dataset.
The procedure is designed to explore relevant cost regions, infer the cost-function shape, and seek the global minimum within D.
After a fixed number N of evaluations, ES returns an improved controller based on experiments expected to provide maximal information about better parameter settings.
The initial controller typically uses LQR design weights equal to the desired performance weights Q and R.
Compared with methods that directly seek low function values, ES is intended for prototyping settings where experiment information matters more than each individual outcome.
ES requires several seconds to select the next experiment, but this computation is reported as minor when physical experiments take substantially longer.

IV. EXPERIMENTAL RESULTS

The experimental platform is a seven-degree-of-freedom robot arm balancing a pole through end-effector acceleration control. The setup combines model-based state feedback with measured pole and arm states, while integral action compensates steady-state position offsets.

The experiment balances a one-degree-of-freedom pole using end-effector acceleration control on a seven-degree-of-freedom Kuka robot arm.
The pole angle is measured by external motion capture, while end-effector position and velocity are estimated from robot joint encoders.
The balancing dynamics describe pole angle, end-effector displacement, and acceleration, with the pole angle and displacement defined relative to gravity and the zero position.
Experiments use short and long poles with different lengths, masses, and friction-related parameters.
A nominal discrete-time model is obtained by linearizing the dynamics around the upright, zero-displacement equilibrium and sampling at 1 ms.
The commanded acceleration is realized through an end-effector tracking controller rather than directly applied as a physical input.
Integral action is added to compensate steady-state end-effector position offsets caused by angle-measurement bias.
The integral gain is fixed at Fz = −0.3 and excluded from the cost because the integrator is implemented in the controller and affects long-term behavior.

B. Automatic LQR tuning: Implementation choices

The implementation defines LQR tuning experiments over 2D and 4D controller-parameter spaces, with experiments lasting 2 minutes and safety-aware cost handling. Entropy Search is initialized and updated through Gaussian-process hyperparameter estimation, while posterior figures track controller evaluations and the inferred minimum.

Tuning dimensions: The experiments tune either two parameters θ = [θ1, θ2] or four parameters θ = [θ1, θ2, θ3, θ4], with each parameter constrained to [0.01, 10].The initial values are θ0 = [2, 4] for 2D and θ0 = [1, 4, 1, 8] for 4D, corresponding to the nominal LQR design.
Experiment duration: Each balancing experiment runs for 2 minutes, corresponding to a discrete horizon of K = 1.2 · 10^5 steps.
Safety and cost handling: When a controller destabilizes the system, the experiment is stopped after safety violations and assigned a fixed heuristic cost Ju.The reported heuristic values include Ju = 3.0 and Ju = 5.0, chosen slightly above the performance of a stable but poor controller.
Gaussian-process hyperparameters: Entropy Search updates the Gaussian-process hyperparameters H after each iteration by maximizing the GP marginal likelihood.Initial hyperparameter values are taken from the expected values of Gamma priors acquired through preliminary experiments.

C. Results from 2D experiments

The 2D experiments test automatic LQR tuning with an accurate model and with a 50% underestimated model. After 20 iterations, Entropy Search identifies candidate controllers, and the accurate-model case improves performance by 31.9%.

Experimental conditions: The 2D experiments use a short pole with the best available linear model, then a double-length pole while retaining the same nominal model.The second case tests tuning under a 50% underestimated model.
Accurate nominal model: Five initial evaluations consisting of θ0 and the four domain corners initialize Entropy Search for the first 2D experiment.This initialization provides Gaussian-process prestructuring and tends to speed learning.
Accurate nominal model: After 20 balancing experiments, Entropy Search selected θBG = [0.01, 2.80] as the estimated location of the cost minimum.
Accurate nominal model: 31.9% improvement was achieved over the initial controller in five separate evaluation experiments.The comparison uses the average cost of the tuned controller and the initial controller, each computed across five balancing experiments.
Poor nominal model: With the longer pole and mismatched nominal model, the initial controller destabilized the system, while 20 iterations produced θBG = [3.25, 0.01].The resulting controller was evaluated five times against the destabilizing initial controller; the supplied passages do not report the table values.

D. Results from 4D experiment

The 4D experiment improved a controller in a higher-dimensional tuning space, including from an initially destabilizing controller, while global exploration sampled unstable regions. The experiment also exposed practical dependence on ES parameter choices and prior assumptions.

D. Results from 4D experiment: The framework improved the controller found in the long-pole 2D experiments while tuning in a higher-dimensional 4D space.The 4D experiment used the same nominal model as earlier experiments, despite that model being poor for the real plant.
D. Results from 4D experiment: 31.7% better performance was obtained after 46 iterations with θBG = [4.21, 7.47, 0.43, 0.01], compared with the 2D long-pole experiments.The initial controller θ0 destabilized the system, and running the experiment to iteration 50 produced no further improvements.
D. Results from 4D experiment: Unstable controllers appeared throughout tuning, reflecting global search across the controller space.Figure 5 records cost evaluations by experiment and distinguishes stable from unstable controllers.
D. Results from 4D experiment: The 4D search required two restarts to slightly adjust H and Ju, whereas the 2D experiments received prior tuning of hyperparameters and parameter ranges.The 4D experiments began without the same degree of prior parameter selection used for the 2D experiments.
D. Results from 4D experiment: ES-based tuning relies on prior assumptions over continuous functions, and general rules for choosing GP-optimizer parameters remain an open development issue.The authors were encouraged that little changed when moving from 2D to 4D, but identify parameter selection as future work.
D. Results from 4D experiment: The paper introduces Bayesian optimization for automatic controller tuning and demonstrates an LQR-plus-ES framework on a robotic platform.The framework was demonstrated in 2D and 4D experiments, including initialization with unstable and stable controllers.