PaperlayerView exact revision on arXiv

Source-linked AI summary

A Decade of Bayesian Optimization for Controller Tuning and Robot Learning: Tutorial, Review, and Future Prospects

David Stenger, Paul Brunzema, Johanna Menn, Alexander von Rohr, Angela P. Schoellig, Sebastian Trimpe

arXiv:2609.09403v1cs.ROeess.SY

TL;DR

Controller tuning and robot learning need data-efficient methods for costly, noisy experiments, while control and robotics lack standardized benchmarks. This paper surveys and tutorials BO methods, reviews applications, and proposes benchmarking guidance and a lightweight suite; its cart-pole benchmark finds BO outperforming Sobol sampling under a small budget, without significant acquisition-function differences in that case.

  • Problem

    Control and robotics lack standardized benchmark problems and systematic evaluations for comparing BO methods rigorously.

  • Method

    The paper combines a practitioner tutorial, a comprehensive review of BO methodologies and applications, and benchmarking guidelines with a lightweight controller-tuning benchmark.

  • Results

    BO outperformed Sobol sampling on the cart-pole tuning task under a small evaluation budget, while acquisition-function choice did not significantly influence performance in that case.

  • Takeaways & Limitations

    The paper provides control- and robotics-specific guidance for applying BO and for comparing algorithms across controller-tuning and robotic tasks.

  • Takeaways & Limitations

    Safe BO hardware applications commonly use fixed GP hyperparameters and β values, and practical procedures for determining them before optimization remain underdeveloped.

Abstract

from arXiv · show

In the past decade, Bayesian optimization (BO) has emerged as a powerful and adaptable framework for automatic controller tuning and robot learning. This article offers a comprehensive overview of the state-of-the-art in BO, designed to support both researchers and practitioners in understanding recent advancements, practical applications, and future research directions. We begin by adopting a practitioner's perspective, illustrating how to effectively set up BO through a representative controller tuning example. We position BO within the broader context of learning paradigms, ranging from deep reinforcement learning to data-driven control, and highlight scenarios where BO is most advantageous. Next, we discuss the diverse range of BO methods that have been developed to tackle complex problems and specific applications. This article provides a unified perspective on the current landscape of BO, emphasizing its relevance to control systems and robotics, and it highlights future prospects by identifying key research challenges and promising avenues for advancing BO in the field. This includes addressing a significant gap in the BO landscape: the lack of standardized benchmark problems specifically for control-related applications. To foster future research and ensure rigorous evaluation, we start an effort towards a lightweight benchmark suite for control engineering and robotics. We also present metrics and best practices to facilitate direct comparisons between new BO algorithms and established state-of-the-art methods.

1 Introduction

The paper presents BO as an efficient framework for controller tuning and robot learning, combining a practitioner tutorial, methodological review, and benchmarking guidance. It addresses limited standardized evaluation through a lightweight benchmark effort for control and robotics.

  • 1 Introduction: BO automates controller tuning with limited system assumptions and fewer manual experiments, especially when evaluations are costly and noisy.It uses measurable performance outputs, probabilistic models, and exploration–exploitation balancing.
  • 1 Introduction: BO variants address real-world requirements including safety, constraints, multiple objectives, and changing environments.
  • 1 Introduction: The paper guides practitioners from a practical BO tutorial to advanced methods, applications, and future research directions in control and robotics.
  • 1 Introduction: The paper reviews 110 empirical hardware studies and develops benchmarking guidance covering baselines, metrics, statistical analysis, and an extensible lightweight suite.
  • 1 Introduction: The review identifies scarce comprehensive overviews and positions its contribution as broader than earlier robotics and control tutorials.

PART A: TUTORIAL

Part A introduces BO for practitioners by formulating controller tuning as black-box optimization and explaining when BO is advantageous relative to other learning paradigms.

  • PART A: TUTORIAL: The tutorial uses controller-tuning examples to explain vanilla single-objective BO with a known feasible domain before introducing more advanced problem settings.

2 Controller Tuning as Black-Box Optimization

Controller tuning is formulated as noisy black-box optimization of a trajectory-based objective over adjustable controller parameters. BO then searches this objective using costly closed-loop experiments while allowing prior knowledge and advanced constraints.

  • 2.1 Single-Objective Unconstrained Problem Formulation: Controller tuning requires only measurable closed-loop performance influenced by adjustable parameters, without assuming Markovian, stateless, differentiable, or known control laws.
  • 2.1 Single-Objective Unconstrained Problem Formulation: The objective maps a closed-loop trajectory to a cost, so unknown dynamics and state trajectories need not be explicitly known.
  • 2.1 Single-Objective Unconstrained Problem Formulation: Each controller parameterization is evaluated through a noisy, potentially costly closed-loop experiment, producing data for sequential black-box optimization.
  • 2.1 Single-Objective Unconstrained Problem Formulation: The cart-pole example defines a controller, episode, task-specific weighted mean absolute error cost, and bounded search domain before optimization.
  • 2.1 Single-Objective Unconstrained Problem Formulation: The basic formulation extends to multiple objectives, constraints, adaptive or safe optimization, and incorporation of models, stability margins, starting points, or other prior information.

3 Introduction to Bayesian Optimization and Gaussian Process Regression

The paper introduces GP-based BO as a sequential loop that models an unknown tuning objective, selects promising parameters through an acquisition function, evaluates them, and updates the model. It presents the basic algorithm and its computational and modeling choices for noisy controller-tuning experiments.

  • 3 Introduction to Bayesian Optimization and Gaussian Process Regression: GP regression models the unknown tuning objective probabilistically, while an acquisition function selects candidate parameters for evaluation and model updating.
  • 3.1 Gaussian Process Regression: The GP observation model treats measurements as noisy samples of a latent objective, commonly using independent Gaussian noise.
  • 3.1 Gaussian Process Regression: The GP prior encodes expected objective values and covariance structure through its mean and kernel, and conditioning yields predictive distributions for latent objectives and future observations.
  • 3.1 Gaussian Process Regression: Direct GP computations scale cubically with the number of observations, and hyperparameter estimates can be poor when data are scarce.
  • 3 Introduction to Bayesian Optimization and Gaussian Process Regression: BO starts with evaluated parameterizations, fits a GP to the resulting data, selects the next parameterization, evaluates it, and repeats until stopping.
  • 3.2 Bayesian Optimization: The LCB acquisition function favors parameters with low predicted cost and/or high uncertainty, with β controlling the exploration–exploitation balance.

4 Distinction from Other Controller Learning Paradigms: When to use BO?

BO is most useful when evaluations are expensive or noisy and system interactions are limited, because structured controllers reduce tuning to a small parameter space. Compared with other learning and black-box optimization approaches, BO offers sample efficiency and flexibility for non-Markovian or sparse-reward settings, but incurs computational overhead and sacrifices asymptotic flexibility.

  • When to use BO?: BO is preferable when objective evaluations are expensive and noisy, while its computational overhead makes it unsuitable for inexpensive evaluations.Each iteration requires fitting a GP model and optimizing an acquisition function.
  • Compared with black-box optimizers: BO can outperform other black-box optimizers in sample efficiency, although GLIS and SMGO have achieved competitive or similar sample efficiency.BO additionally accommodates noisy evaluations and has extensions for more complex problem formulations.
  • Compared with reinforcement learning: Relative to deep reinforcement learning, BO trades asymptotic flexibility for data efficiency because performance is bounded by the expressiveness of the chosen controller structure.Designing suitable controller structures can also require substantial domain expertise.
  • Compared with reinforcement learning: BO-based controller tuning exploits structured PID, LQR, or MPC policies, typically leaving fewer than ten interpretable parameters instead of thousands or millions of neural-network weights.This prior control knowledge places controller tuning in the micro-data regime, where tens of experiments may suffice instead of millions.
  • Compared with reinforcement learning: BO handles non-Markovian objectives, policies, and dynamics, sparse rewards, and episode-level objectives without requiring changes to the underlying algorithm.BO can also optimize reinforcement-learning hyperparameters, making it complementary to RL rather than mutually exclusive.
  • Compared with learning-based control: Within learning-based control, BO treats the closed-loop system as a black box, optimizes tens of parameters, and targets expensive hardware or high-fidelity simulation evaluations.BO can also tune parameters and hyperparameters of other learning-based control methods, including learning-based MPC and DRL.

PART B: REVIEW

Part B targets advanced BO users and algorithm developers by reviewing core design choices, advanced controller-tuning challenges, and hardware applications.

  • PART B: REVIEW: Part B reviews vanilla BO design choices, advanced optimization formulations for controller and robotics tuning, and a systematic survey of 110 hardware papers.

5 Design Choices in Vanilla Bayesian Optimization

The review presents practical starting points for vanilla BO while emphasizing that acquisition-function performance depends on the task and that common modeling assumptions can fail in practice.

  • 5. Design Recommendations: BO design recommendations are informed starting points rather than definitive conclusions because representative control- and robotics-specific benchmark evaluations remain lacking.
  • 5. Design Recommendations: The review recommends GPR with homoscedastic Gaussian noise, Matérn or SE kernels with ARD, standardized observations, MAP hyperparameter optimization, and task-dependent acquisition functions.
  • 5.1 Probabilistic Surrogate Model: Gaussian-process inference scales cubically with dataset size, while random forests and TPEs better support categorical or integer variables and larger dimensions.
  • 5.1 Probabilistic Surrogate Model: Homoscedastic Gaussian noise is analytically tractable with one noise-variance hyperparameter, but the assumption may fail and heteroscedastic models generally require more data.
  • 5.1 Probabilistic Surrogate Model: Gaussian likelihoods are vulnerable to outliers, motivating heavier-tailed Student’s-t alternatives that add a hyperparameter and tend to require more data.
  • 5.2 Acquisition Function: Acquisition-function performance is task-dependent: EI and MES performed similarly on average across ten deterministic controllers, while UCB was slightly worse.
  • 5.2 Acquisition Function: Standard EI should not be used in noisy settings, while logEI has shown strong performance in high-dimensional problems.

6 Advanced Variants

Advanced BO variants formalize practical controller-tuning requirements including feasibility, safety, crashes, and competing objectives. These formulations use specialized constraint models, safe-set exploration, failure indicators, and Pareto-optimal solutions.

  • 6.1 Constrained BO: Constrained BO requires only the final solution to be feasible, while probabilistic models estimate unknown constraint violations during acquisition optimization.Expected improvement can be multiplied by the probability of feasibility.
  • Overview: BO extensions cover practical requirements such as choosing settings, constraints, crash handling, and adaptive operation in controller-tuning and robot-learning tasks.Figure 5 summarizes these extensions.
  • 6.2 Safe BO: Safe BO optimizes the objective without violating safety constraints in any iteration, requiring safe-set identification and an initial safe parameter set.The main exploration challenge is balancing expansion of the safe set with optimization within it.
  • 6.2 Safe BO: Safe BO methods commonly rely on confidence bounds and GP assumptions, but heuristic parameters and kernel misspecification can invalidate guarantees or cause safety violations.Practical applications often use β = 2 or β = 3 and Matérn kernels with ν = 3/2.
  • 6.3 Crash Constraints: Crash-constrained BO steers optimization away from unstable regions using a success indicator when objective values are unavailable after failed experiments.The objective is observed only when the experiment succeeds.
  • 6.4 Multi-Objective BO: Multi-objective BO returns a Pareto-optimal set, allowing practitioners to select a parameterization that balances competing performance metrics.The acquisition strategy must improve the Pareto set rather than a single metric.

6.6 Contextual BO

Contextual BO optimizes controller parameters as a function of operating conditions, while time-varying BO tracks changing optima as data become stale. Related variants address non-real decision spaces, local search, multiple fidelities, and early stopping.

  • 6.6 Contextual BO: Contextual BO searches for an optimal parameter function θ∗(s), adapting controller settings to known operating conditions such as ambient temperature.The context is provided by the environment before BO selects and evaluates parameters.
  • 6.6 Contextual BO: Contextual BO can combine with safe exploration by calculating the safe set for the current context.This combination has been demonstrated in controller and robot applications.
  • 6.7 Time-Varying BO: Time-varying BO tracks an optimum through time because changing objectives make previous data potentially stale and alter the exploration–exploitation trade-off.Methods either model temporal dynamics explicitly or adapt online to detected changes.
  • 6.8 Decision Spaces: Non-real decision spaces require BO formulations for integer variables and discrete choices, such as MPC horizon length or optional controller components.These variables make the parameter domain a set outside R^d.
  • 6.9 Local BO: Local optimization restricts search to a neighborhood around an initial guess, enabling continuous local improvement but remaining sensitive to initialization and potentially suboptimal.Local updates may also avoid evaluating unstable parameters.
  • 6.10 Further Variants: Multi-fidelity BO combines inexpensive, lower-accuracy simulations with costly, accurate hardware experiments, while early stopping terminates evidently suboptimal episodes sooner.Early stopping has been transferred to both simulated and hardware control experiments.

7 Review of Practical Applications of BO

The review finds BO applications spanning diverse control policies, domains, and practical problem formulations, while noting that hardware use remains concentrated in low-dimensional settings and often uses vanilla BO. It also surveys extensions and related applications beyond direct online controller tuning.

  • 7.1 Review Method: The review covers online hardware tuning selected from papers published by 2025, organized by application domain, optimized policy, and problem formulation.The review criteria require each BO query to evaluate a controller or policy on hardware.
  • 7.2 Application Domains: BO is applied across lab-scale demonstrators, industrial processes, human-in-the-loop systems, wind farms, CNC machines, and particle accelerators.The reviewed applications include 26 lab-scale demonstrators and practical cases involving vehicle calibration, industrial efficiency, sustainable energy, and scientific discovery.
  • 7.3 Optimized Controller Type: Most hardware applications optimize fewer than 10 parameters, despite BO also supporting hierarchical controllers and latent spaces of high-dimensional policies.Examples span MPC, LQR, PID, impedance control, neural controllers, grasping, gait, and hierarchical structures.
  • 7.4 Problem Formulations: Vanilla BO achieves reported real-world efficiency gains, while unknown constraints, safe exploration, and contextual BO are the most popular extensions.The review also concludes that advanced BO methods are more developed than the methods used in real-world applications beyond lab-scale demonstrators.
  • 7.5 Other Use Cases in Control and Robotics: BO concepts also support path planning, environment exploration, run-to-run control, controller and hardware co-design, and joint optimization of guidance, navigation, and control stacks.For co-design, most studies lack experimental validation, with a quadruped study identified as an exception.

PART C: BENCHMARKING

The paper identifies a lack of realistic, public controller-tuning benchmarks, making it difficult to compare BO methods and assess generalization beyond individual tasks. It responds with benchmarking guidance and TUNECONTROL, a reproducible lightweight suite.

  • Motivation: BO evaluations are difficult to generalize when demonstrations depend on favorable task-specific design choices.Performance depends on modeling assumptions, acquisition functions, initialization, hyperparameter fitting, and implementation details.
  • Benchmarking: TUNECONTROL provides a reproducible format for 34 variants of cart-pole and cascaded-tank tasks and is designed for community extension.The paper also demonstrates a minimal comparison of common acquisition functions on one task.

8 Best Practices in Benchmarking Black-Box Optimization Algorithms

The paper recommends benchmarking BO across diverse problem classes with explicit baselines, metrics, statistical procedures, and behavioral diagnostics. It introduces TUNECONTROL as a lightweight controller-tuning benchmark while emphasizing that synthetic results alone do not establish relevance to control and robotics.

  • 8.3 Metrics and 8.4 Statistical Analysis: Reporting should include task and budget details, seeds, initialization, surrogate and acquisition settings, baselines, metrics, runtime, statistical tests, and state trajectories.Behavioral diagnostics are recommended because scalar performance metrics may not verify intended controller behavior.
  • 8.1 Benchmark Problems: Reliable BO evidence requires multiple benchmark structures, noise levels, dimensionalities, and model-mismatch conditions rather than a single objective function.A single objective can accidentally favor a method’s kernel, smoothness assumptions, or exploration strategy.
  • 8.1 Benchmark Problems: Synthetic functions enable controlled regret-based evaluation but should not be the only evidence for claims about BO in control and robotics.They may fail to capture controller-tuning and robotics structure despite offering known optima and targeted difficulty patterns.
  • 8.1 Benchmark Problems: TUNECONTROL offers a consistent API with 34 cart-pole and cascaded-tank task variants, and is intended to be extended with additional tuning tasks.It is presented as a lightweight benchmark suite for black-box controller tuning.
  • 8.2 Baselines: Benchmarks should include Sobol sampling and, for larger budgets, low-overhead optimizers such as CMA-ES, pattern search, or particle swarm optimization.These baselines calibrate task difficulty and test whether BO’s modeling overhead is justified.
  • 8.4 Statistical Analysis: Statistical comparisons should match the experimental design, using Wilcoxon signed-rank tests for paired samples and Friedman tests with Holm correction for multiple algorithms.Directly averaging costs or regrets across differently scaled objectives can be misleading; ranking or normalized alternatives are suggested.

9 Example Benchmark

This minimal cart-pole benchmark compares three acquisition functions under a shared GP model and low evaluation budget. BO improves faster than Sobol sampling, while acquisition-function differences are not statistically significant among BO methods.

  • Benchmark Setup: The benchmark compares EI, UCB, and MES against Sobol sampling using the same GP surrogate, differing only in acquisition function.The setup is designed to isolate acquisition-function effects.
  • Results: The example is intended as a minimal, reproducible benchmark template rather than evidence that one acquisition function is generally superior.The authors motivate broader benchmarking across controller-tuning and robotics tasks.
  • Benchmark Setup: The four-dimensional task uses 40 evaluations after five initial Sobol samples, representing a low-budget controller-tuning regime.The budget is defined as 10d with d = 4, and no acquisition-function hyperparameters are tuned for the task.
  • Results: BO variants improve faster than Sobol sampling, and LogEI achieves the lowest final median cost in the cart-pole benchmark.The comparison uses Figure 6's best-observed cost over the optimization budget.
  • Results: LogEI performs significantly better than Sobol sampling, but its differences from UCB and MES are not statistically significant.The post-hoc comparison follows a significant Friedman test across methods.
  • Results: Figure 7 illustrates state trajectories from one LogEI optimization run at median performance.It provides a trajectory-level view alongside aggregate optimization results.

10 Future Research Directions

Future work centers on stronger empirical comparisons, broader benchmarking, safer and more adaptable BO methods, and integration with other learning paradigms. The paper contributes TUNECONTROL as a lightweight framework for more rigorous evaluation.

  • Empirical Comparisons: Rigorous comparisons of BO with model-based reinforcement learning and other surrogate-based optimizers remain scarce.The paper identifies systematic sample-efficiency and method comparisons as open needs.
  • Surrogate Models: Broader benchmarking is needed to determine which surrogate models and GP modeling choices are most appropriate across controller-tuning and robot-learning tasks.The paper notes alternatives including random forests, tree-structured Parzen estimators, and neural-network models such as VBLLs.
  • Benchmarking: TUNECONTROL provides a lightweight framework for community-contributed controller-tuning benchmarks and more rigorous evaluation of BO design choices.The framework is intended to increase the visibility and impact of BO research in control and robotics.
  • Safe BO: Safe BO still needs methods for selecting parameters before optimization in use cases beyond lab-scale demonstrations and clearer categories of safety guarantees.The discussion distinguishes deterministic or stochastic guarantees from cautious exploration.
  • Dynamic Settings: BO remains largely restricted to repetitive episodic tasks, motivating contextual, time-varying, and early-stopping approaches for more dynamic settings.Further research is needed to transfer BO's sample efficiency beyond highly repetitive tasks.
  • Hybrid Methods: Combining BO with other learning-based control methods, black-box optimization paradigms, and foundation-model priors is identified as a promising direction.Foundation-model applications in control and robotics remain limited, but may combine large-scale priors with sparse task-specific data.

Declaration of generative AI in the manuscript preparation process.

The manuscript reports that large language models assisted with code generation and grammar and spelling correction during preparation.

  • Declaration: The authors used various LLMs for code generation and grammar and spelling correction, then reviewed and edited the resulting content.They state responsibility for the published article's content.
Loading 2609.09403v1…