Goal-Driven Dynamics Learning via Bayesian Optimization

I. INTRODUCTION

• For complex systems, a globally accurate dynamics model may be unnecessary when a task-specific model can achieve the desired control performance.The introduction notes that local linearization can suffice for trajectory tracking.Source: §I · p. 1
• aDOBO uses Bayesian optimization in active learning to update a dynamics model from physical-system performance and select informative experiments.The framework uses past experiments to guide subsequent queries, aiming to improve performance with few physical evaluations.Source: §I · p. 1
• Unlike prediction-error system identification, aDOBO seeks the model whose controller performs best on the actual system.Traditional system identification separates model fitting from controller design; prediction-error minimization can yield suboptimal control performance.Source: §I · pp. 1–2

II. PROBLEM FORMULATION

• The problem formulation represents the unknown dynamics with a parameterized LTI model and searches for the model whose controller minimizes cost on the actual plant.The selected model need not correspond to an actual linearization; it is judged by closed-loop performance.Source: §II · p. 2
• Because each cost evaluation requires an experiment, the optimization seeks to minimize the number of physical-system evaluations.The cost function is unknown in advance and is evaluated empirically.Source: §II · p. 2

III. BACKGROUND

• The paper introduces Gaussian processes and Bayesian optimization as background for optimizing the parameterized dynamics model.Source: §III · p. 2

A. Gaussian Process (GP)

• A Gaussian process models the unknown performance function J(θ), with a prior mean and covariance kernel specifying relationships between function values.The paper uses a 5/2 Matérn kernel, whose hyperparameters are selected by maximizing marginal likelihood.Source: A. Gaussian Process (GP) · p. 3

B. Bayesian Optimization (BO)

• The GP predicts both expected performance and uncertainty at candidate dynamics parameters, which Bayesian optimization uses to select informative samples.The BO acquisition function is optimized using the GP's mean and variance predictions.Source: A. Gaussian Process (GP) · p. 3, B. Bayesian Optimization (BO) · p. 3
• Expected improvement selects the next parameter point by maximizing expected improvement over the best explored value, without requiring a physical-system evaluation to optimize the acquisition function.The selected parameters are subsequently evaluated on the real system.Source: B. Bayesian Optimization (BO) · p. 3

• The procedure measures z_{i+1} on the real system.Source: §3 · p. 3
• After θ* is determined, it is finally evaluated on the real objective J, identified as the system.Source: §3 · p. 3

IV. DYNAMICS OPTIMIZATION VIA BO (aDOBO)

• The current model designs an optimal control sequence, whose first action is applied to the real system before the next state is measured and the process continues.After the resulting trajectory, the cost is computed on the actual system and used to update the Gaussian process.Source: §IV · p. 3
• aDOBO directly searches linearizations for the one whose controller minimizes cost on the actual system, updating a Gaussian process with experiment results to choose informative next models.The method learns the relationship between linearization parameters and observed performance rather than fitting the true dynamics.Source: §IV · p. 3
• On the Dubins car, aDOBO reaches within 10% of optimal cost in 100 iterations from a random model, with log cost warping accelerating learning.The figure reports mean and standard deviation across 10 trials.Source: Figure 2 · p. 4
• The cost function determines the control scheme: quadratic costs yield linear feedback via a Riccati equation, while general convex costs can use MPC and produce nonlinear controllers.The framework is modular and can use other control schemes suited to the cost function.Source: §IV · p. 4
• The Gaussian process can start from known high-performing models or random models, and aDOBO minimizes expected cost when the real system is stochastic.A good known linearization can speed convergence.Source: §IV · p. 4

A. Dubins Car System

• The simulated Dubins car is a three-dimensional nonlinear system with position and heading states and speed and turn-rate inputs.Its continuous dynamics are discretized at 10 Hz.Source: A. Dubins Car System · p. 4
• The Dubins-car simulation starts from z0 = (1.5, 1, π/2) and aims for the equilibrium z* = 0 using a 3-state, 2-input linear model with 15 learned parameters.The quadratic-cost control problem is solved as LQR; the comparison uses percentage error against the true optimal cost.Source: A. Dubins Car System · p. 4
• Within 10% of optimal cost, aDOBO learns the Dubins-car controller as learning progresses, and its learned state and control trajectories closely match the true system.The trajectories indicate accurate behavior around the desired state.Source: A. Dubins Car System · pp. 4–5, Figure 3 · p. 4

B. A Simple 1D Linear System

• After 45 iterations, aDOBO reaches the true optimal performance, J* = 1.61, with parameters θ1 = 1.69 and θ2 = 2.45 rather than the true values θ1 = 1 and θ2 = 1.The optimum is computed using LQR on the real system.Source: B. A Simple 1D Linear System · p. 5
• A wide range of parameters can produce similar actual-system performance, so performance-based learning can recover optimal control and trajectories without recovering the true dynamics.The simulation suggests that true dynamics may not be required for control performance.Source: B. A Simple 1D Linear System · p. 5
• The aDOBO parameters yield performance very close to the true-system parameters, although they do not converge to those true parameters.Fig. 4 plots actual-system cost over the linearization parameters.Source: Figure 4 · p. 5, B. A Simple 1D Linear System · p. 5

C. Cart-pole System

• The cart-pole experiment uses unknown dynamics, 10 Hz discretization, and MPC to control the pendulum while keeping cart position within [-2, 2].The setup also limits pendulum overshoot to 0.1.Source: C. Cart-pole System · p. 5
• Within 20% of optimal performance in 250 iterations, aDOBO continues improving on the nonlinear cart-pole task.The simulation uses convex MPC and a cost that penalizes state-bound violations.Source: C. Cart-pole System · p. 5, Figure 5 · p. 6

A. Tuning (Q, R) vs aDOBO

• For the tuning comparison, penalty-matrix optimization uses the known linearization (A*, B*), whereas aDOBO optimizes the dynamics parameters.Both methods are initialized using the noisy linearization in the Dubins car comparison.Source: A. Tuning (Q, R) vs aDOBO · p. 6
• With no noise in (A*, B*), (Q, R) tuning outperforms aDOBO, but its performance deteriorates as noise increases while aDOBO remains fairly indifferent.Tuning learns fewer parameters but relies on an accurate dynamics linearization; aDOBO assumes the penalty matrices (Q, R) are known.Source: A. Tuning (Q, R) vs aDOBO · p. 6, Figure 6 · p. 6

B. Learning K vs aDOBO

• Within 20% of optimal cost in a few iterations, aDOBO outperforms directly learning K, which has error as high as 80% on the non-quadratic cost.The non-quadratic setting can have a nonlinear optimal controller, while directly learning K is restricted to linear controllers.Source: B. Learning K vs aDOBO · p. 6, Figure 7 · p. 7
• For quadratic cost, directly learning K converges faster than aDOBO because it learns fewer parameters.The figure compares mean and standard deviation of η for both methods across quadratic and non-quadratic costs.Source: B. Learning K vs aDOBO · p. 6, Figure 7 · p. 7
• More than 250% performance error remains after 600 iterations of direct control-sequence learning, while aDOBO learns dynamics parameters instead.The table compares mean and standard deviation of η for aDOBO and direct control-sequence learning on system (15).Source: B. Learning K vs aDOBO · p. 7, Table I · p. 7
• With linear dynamics and convex MPC, learning A and B can implicitly represent a richer controller space than a linear feedback matrix K.The underlying linear system's true optimal controller is also in the search space.Source: B. Learning K vs aDOBO · pp. 6–7

C. Adaptive Control vs aDOBO

• Least-squares fitting converges to true dynamics in 5 iterations on a linear system but performs poorly on the nonlinear Dubins car.The authors note that task-relevant trajectories near the region of interest may be needed for a useful nonlinear-system linearization.Source: C. Adaptive Control vs aDOBO · p. 7, Table II · p. 7

VII. QUADROTOR POSITION TRACKING EXPERIMENTS

• For Crazyflie tracking, the authors treat horizontal velocity dynamics as unknown and optimize their parameterized model with aDOBO.The goal is to track a desired position from p0 = [0, 0, 1], using LQR for each learned linearization.Source: §VII · p. 8
• 12% better than the nominal controller after 45 iterations, aDOBO improves Crazyflie position tracking on the physical system.The nominal controller performs better initially; the authors attribute the later advantage to aDOBO accounting for unmodeled effects through optimization on actual-system performance.Source: §VII · p. 8

VIII. CONCLUSIONS AND FUTURE WORK

• The work introduces aDOBO, an active learning framework that optimizes system dynamics to maximize controller performance.Source: §VIII · p. 8
• As learning progresses, aDOBO outperforms the nominal controller by 12% on the actual system.The nominal controller uses the quadrotor's full 12D nonlinear dynamics model.Source: Figure 9 · p. 8
• aDOBO achieves optimal control performance even when no prior information about system dynamics is known.Source: §VIII · p. 8
• A model optimized for one task may not perform well on a similar, different task; exploration may also query unstable parameters, and BO is not scalable beyond roughly 30–40 parameters.Future directions include optimizing across task classes, using state and input trajectories to improve data efficiency, safety-aware exploration, and scaling to richer nonlinear models.Source: §VIII · p. 8