PaperlayerView exact revision on arXiv

Source-linked AI summary

Safe Controller Optimization for Quadrotors with Gaussian Processes

Felix Berkenkamp, Angela P. Schoellig, Andreas Krause

arXiv:1509.01066v4cs.RO

TL;DR

Controller-parameter tuning is difficult, time consuming, and potentially unsafe when optimization evaluates unstable controllers on a real system. The paper applies a modified SAFEOPT algorithm using Gaussian-process estimates and a safe performance threshold to tune controllers automatically. Quadrotor experiments identify the performance function after 30 experiments, evaluate no parameters below J_min = 0, and safely explore controller parameters without failures.

  • Problem

    Controller tuning requires domain knowledge and time, while ordinary Bayesian optimization can evaluate unstable controllers and cause real-system failures.

  • Method

    A modified SAFEOPT algorithm uses Gaussian-process confidence bounds to evaluate only controller parameters likely to exceed a safe performance threshold.

  • Results

    After 30 experiments, the unknown performance function was reliably identified, no parameters below J_min = 0 were evaluated, and the method safely explored the parameter space.

  • Takeaways & Limitations

    The approach enables efficient, automatic, and global controller-parameter optimization on a quadrotor without dangerous or expensive system failures.

Abstract

from arXiv · show

One of the most fundamental problems when designing controllers for dynamic systems is the tuning of the controller parameters. Typically, a model of the system is used to obtain an initial controller, but ultimately the controller parameters must be tuned manually on the real system to achieve the best performance. To avoid this manual tuning step, methods from machine learning, such as Bayesian optimization, have been used. However, as these methods evaluate different controller parameters on the real system, safety-critical system failures may happen. In this paper, we overcome this problem by applying, for the first time, a recently developed safe optimization algorithm, SafeOpt, to the problem of automatic controller parameter tuning. Given an initial, low-performance controller, SafeOpt automatically optimizes the parameters of a control law while guaranteeing safety. It models the underlying performance measure as a Gaussian process and only explores new controller parameters whose performance lies above a safe performance threshold with high probability. Experimental results on a quadrotor vehicle indicate that the proposed method enables fast, automatic, and safe optimization of controller parameters without human intervention.

SUPPLEMENTARY MATERIAL

The supplementary material provides a video demonstration and a Python implementation of the proposed safe, automatic controller optimization algorithm.

  • A video demonstrates safe, automatic controller optimization on a quadrotor vehicle.
  • A Python implementation of the algorithm is available.
  • The supplementary material directs readers to both the demonstration video and implementation.

I. INTRODUCTION

The paper addresses the difficulty and safety risks of tuning controller parameters automatically. It proposes modified SAFEOPT Bayesian optimization to explore controller parameters safely without a model or gradients.

  • Controller tuning requires significant domain knowledge and can be time consuming, while existing methods may rely on restrictive assumptions, many evaluations, or converge locally.
  • The method automatically tunes controller parameters without a dynamic-system model or gradient computation, while guaranteeing safety during convergence to the global optimum.
  • Bayesian optimization models unknown, noisy performance functions with Gaussian processes and can find global optima using few parameter evaluations.
  • Standard Bayesian optimization can evaluate informative but unstable controllers because exploratory samples may lie far from the original control law.
  • The proposed method modifies SAFEOPT to evaluate only controller parameters whose performance exceeds a safe threshold with high probability.
  • The paper demonstrates safety and performance experimentally on a quadrotor, learning optimal controller parameters without failures during the experiments.

II. PROBLEM STATEMENT

The paper formulates controller tuning as maximizing an unknown, noisy performance function over controller parameters while satisfying a safety threshold during real-system evaluations.

  • The controller maps noisy system measurements and reference signals to control actions using parameters a_n in a domain A.
  • The performance measure J(a) is evaluated on the real system and assigns higher values to better-performing controllers over a finite time horizon.
  • The optimization seeks controller parameters that maximize J(a) from noisy observations, with Gaussian measurement noise.
  • The setting assumes a safety-critical system, no prior dynamic model, and a sample-efficient procedure requiring few performance evaluations.
  • Safety is encoded as the high-probability constraint J(a_n) ≥ J_min for every evaluated parameter, with low performance indicating likely instability over a sufficiently long horizon.

III. METHODOLOGY

The methodology section reviews Gaussian processes and Bayesian optimization, then presents the theory behind SAFEOPT.

  • The section reviews Gaussian processes.
  • The section reviews Bayesian optimization.
  • The section illustrates the theory behind SAFEOPT.

A. Gaussian Process (GP)

The paper models the unknown performance function J(a) with a Gaussian process, using prior mean and covariance assumptions to predict performance at unobserved controller parameters from noisy observations.

  • GP model: A Gaussian process approximates the unknown nonlinear map from controller parameters a to performance values J(a).It treats function values at different parameters as jointly Gaussian random variables.
  • GP model: The covariance function k(a_i, a_j), or kernel, encodes assumptions about the unknown function’s smoothness and rate of change.The prior mean is assumed to be zero without loss of generality.
  • Prediction: Given past observations D_n, the GP predicts J(a*) at an arbitrary parameter a* using the observed noisy performance data.The observations satisfy ˆJ(a)=J(a)+ω, with Gaussian noise ω.
  • Prediction: The GP prediction uses covariances between the new parameter and observed points together with the covariance matrix of the observations.The supplied passages identify k_n(a*) and K_n as these covariance quantities.

B. Bayesian Optimization

Bayesian optimization addresses expensive black-box optimization by modeling an unknown objective and selecting informative evaluation locations, making it suitable when real-system experiments consume time and cause wear.

  • Purpose: Bayesian optimization seeks the global maximum of an unknown function when evaluations are expensive but computation is comparatively cheap.Here, each performance evaluation is an experiment on the real system.
  • Sampling: GP-based Bayesian optimization uses predicted means and variances to determine informative controller-parameter samples.The GP model supplies the quantities used to choose the next location.
  • Sampling: The upper-confidence-bound rule selects parameters where the GP estimate’s upper confidence bound is maximal.Repeated evaluations improve the mean estimate and reduce uncertainty near candidate maxima.
  • Sampling: Choosing the next sample uses only the GP model, so solving the selection problem does not require another real-system evaluation.This reflects the assumption that computational resources are cheap relative to experiments.

C. Safe Bayesian Optimization

Safe Bayesian optimization restricts exploration to controller parameters whose predicted performance clears a safety threshold, while balancing improvement of the known safe region with optimization within it.

  • Safe set: SAFEOPT models the unknown performance function with a GP over a finite parameter set and restricts optimization to the safe set.The safe set contains parameters whose performance is at least J_min.
  • Safe set: The initial safe set consists of the initial safe controller parameters, and the estimated safe set is updated after each function evaluation.Safety is therefore established progressively from experimental data.
  • Exploration and exploitation: SAFEOPT selects uncertain parameters from potential maximizers and expanders to optimize within the known safe set while enlarging it.Potential maximizers support exploitation, whereas expanders support exploration.
  • Modification: The modified method estimates the safe, maximizer, and expander sets directly from GP predictions rather than using a Lipschitz constant.This removes the need to specify an additional Lipschitz tuning parameter.

IV. MODIFIED SAFEOPT ALGORITHM

The modified SAFEOPT algorithm uses GP confidence bounds to identify safe parameters, potential maximizers, and expanders, then evaluates the most uncertain candidate to enlarge the safe region while seeking the optimum.

  • GP-based set construction: The modified algorithm estimates S_n, G_n, and M_n directly from GP confidence bounds, avoiding specification of a Lipschitz constant.It also includes a faster Python implementation than a naive implementation.
  • Safe set: Using β_n to define confidence intervals, the safe set contains parameters whose lower confidence bound is at least J_min.The experiments use β_n = 2.
  • Candidate sets: Potential maximizers are safe parameters whose upper confidence bound exceeds the best safe lower bound.They represent parameters that could still achieve the maximum under the confidence bounds.
  • Candidate sets: An expander is a parameter whose optimistic measurement could classify previously unsafe points as safe, thereby enlarging the safe set.The indicator is positive when the new measurement has a non-negligible chance of expanding safety.
  • Iteration: The algorithm selects the most uncertain parameter from G_n ∪ M_n, evaluates the real system, and updates the GP with the noisy measurement.The process repeats until user termination or the desired confidence criterion is reached.
  • Implementation: The implementation reduces computation by checking candidate expanders in descending uncertainty and stopping after finding a suitable expander.Typically, few or no parameters need to be checked.
  • Extensions: The method can be extended to input or state constraints that do not depend on the performance function.The passage directs readers to [22] for details.

V. QUADROTOR EXPERIMENTS

The quadrotor experiments use a two-parameter linear position controller and Gaussian-process SafeOpt to optimize performance over real-system trials while avoiding unsafe parameters. After 30 experiments, the method identifies high-performing gains through staged safe exploration and refinement.

  • Experimental setup: The performance function is evaluated over a 5 s horizon using weighted state and input costs, with safety threshold Jmin = 0.Performance is defined as cost improvement relative to 95% of the initial controller cost.
  • Gaussian-process model: The experiment uses a Matérn kernel and conservative Gaussian-process hyperparameters to represent uncertainty in the unknown performance function.The selected length-scales are 0.05 for both controller parameters, while prior and noise standard deviations are scaled to the initial controller performance.
  • Results: The optimized controller produced a smooth, fast response, while the initial controller was stable but performed poorly and other evaluated controllers produced intermediate trajectories.The trajectories were measured during a 5 s interval containing a 1 m reference position change.
  • Results: After 30 experiments, the unknown performance function was reliably identified and no parameters below the safe threshold Jmin = 0 were evaluated.Samples covered the safe set, with denser sampling near the maximum and initial controller parameters.
  • Optimization behavior: The algorithm first samples near the initial gains to identify a safe region, then expands that region coarsely before finely evaluating promising maximizers.This staged behavior concentrates samples near the initial controller and near the estimated maximum while sampling other areas more sparsely.
  • Results: The identified performance landscape resembles but does not equal the quadratic function expected from linear-quadratic control theory.The difference is attributed to omitted on-board controller states and the nonlinear quadrotor dynamics.

VI. CONCLUSION

The paper applies Safe Bayesian Optimization to position control of a real quadrotor. The method achieves automatic global controller-parameter optimization without dangerous or expensive system failures.

  • The study presents the first application of Safe Bayesian Optimization on a real robotic system.
  • SafeOpt is modified to work directly with Gaussian-process estimates and applied to quadrotor position control.
  • The algorithm enables efficient, automatic, and global optimization of controller parameters without risking dangerous and expensive system failures.
Loading 1509.01066v4…