PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning-based Model Predictive Control for Safe Exploration

Torsten Koller, Felix Berkenkamp, Matteo Turchetta, Andreas Krause

arXiv:1803.08287v3eess.SYcs.AIcs.LGcs.RO

TL;DR

Learning-based control lacks reliable safety guarantees for safety-critical exploration. The paper introduces SAFEMPC, combining GP confidence bounds, multi-step uncertainty propagation, constrained MPC, and a terminal-set constraint; it proves high-probability safety and demonstrates safe learning and exploration.

  • Problem

    Model-based reinforcement learning can use exploratory actions that lead to unpredictable and possibly unsafe system behavior, limiting its use in safety-critical systems.

  • Method

    SAFEMPC combines GP-based confidence intervals and multi-step ellipsoidal uncertainty propagation with constrained MPC and a terminal-set constraint.

  • Results

    Theorem 2 establishes δ-safety for the system under the SAFEMPC controller when the initial state lies in the safe region.

  • Takeaways & Limitations

    The algorithm supports safe exploration of partially unknown systems while model updates improve accuracy and control performance during learning.

Abstract

from arXiv · show

Learning-based methods have been successful in solving complex control tasks without significant prior knowledge about the system. However, these methods typically do not provide any safety guarantees, which prevents their use in safety-critical, real-world applications. In this paper, we present a learning-based model predictive control scheme that can provide provable high-probability safety guarantees. To this end, we exploit regularity assumptions on the dynamics in terms of a Gaussian process prior to construct provably accurate confidence intervals on predicted trajectories. Unlike previous approaches, we do not assume that model uncertainties are independent. Based on these predictions, we guarantee that trajectories satisfy safety constraints. Moreover, we use a terminal set constraint to recursively guarantee the existence of safe control actions at every iteration. In our experiments, we show that the resulting algorithm can be used to safely and efficiently explore and learn about dynamic systems.

I. INTRODUCTION

The paper addresses unsafe exploration in model-based reinforcement learning by introducing SAFEMPC, a learning-based MPC scheme with high-probability safety guarantees. It combines GP-based uncertainty propagation, constrained MPC, and a terminal-set condition to preserve feasible return trajectories during learning.

  • Motivation: Model-based RL learns unknown system dynamics from data to optimize long-term behavior, but exploratory actions can produce unpredictable and unsafe behavior.The passages identify this limitation as preventing application to real-world safety-critical systems.
  • Contribution: SAFEMPC recursively guarantees feasible return trajectories to a safe region with high probability by combining uncertainty propagation, constrained MPC, and a terminal-set constraint.
  • Related work: Existing robust MPC methods provide safety through recursive feasibility and robust constraint satisfaction but can be conservative because they do not update the model.
  • Related work: Learning-based MPC approaches adapt models online, but prior guarantees require a known nominal linear model, bounded deviations, or sampling-based mechanisms.
  • Related work: GP-based MPC can handle learned dynamics, yet complex trajectory dependencies and unbounded stochastic uncertainties make approximate propagation insufficient for theoretical safety guarantees.
  • Problem formulation: The problem formulation uses a prior model plus an unknown error, polytopic state and control constraints, and a backup controller whose safe region is robustly control positive invariant.
  • Problem formulation: Because enforcing safety for arbitrary unknown dynamics is generally impossible, the paper relaxes the requirement to high-probability safety over the controller’s operation time.

III. BACKGROUND

The background section introduces Gaussian processes and ellipsoidal set operations as the tools for modeling system uncertainty and computing multi-step predictions.

  • The paper uses Gaussian processes and ellipsoidal set-theoretic properties to model the system and perform multi-step-ahead predictions.

A. Gaussian Processes (GPs)

Gaussian processes provide high-probability confidence intervals for the unknown dynamics, while ellipsoids support tractable outer approximations of uncertainty during multi-step prediction.

  • Gaussian Processes: The unknown model error is learned with a GP posterior whose mean and variance are computed from noisy observations of the system response relative to the prior model.
  • Gaussian Processes: Under a bounded RKHS norm and sub-Gaussian measurement noise, the GP error interval holds uniformly over inputs and output dimensions with probability at least 1 −δ.
  • Gaussian Processes: The confidence-width factor β_n depends on information capacity, which can be greedily approximated and is sublinear in n for many commonly used kernels.
  • Gaussian Processes: The regularity assumptions also imply that the unknown model error is Lipschitz with respect to the relevant inputs.
  • Ellipsoids: Ellipsoids provide outer bounds on reachable states under uncertain set-valued inputs, making them useful for robust multi-step prediction.
  • Ellipsoids: Affine transformations preserve ellipsoidal form, whereas Minkowski sums generally require ellipsoidal over-approximations whose shape can be optimized by trace.
  • Ellipsoids: The maximum transformed distance from an ellipsoid center is obtained from the largest generalized eigenvalue of the pair (Q, S^T S).

IV. SAFE MODEL PREDICTIVE CONTROL

The method constructs high-probability ellipsoidal confidence regions for one- and multi-step system predictions, then uses them in constrained MPC. A terminal-set constraint and recursive propagation support high-probability safety throughout operation.

  • Method: The scheme combines GP-based model-error confidence intervals with MPC to optimize predicted trajectories under safety constraints.The controller relies on regularity assumptions and multi-step uncertainty propagation.
  • One-step predictions: The one-step prediction linearizes the prior model, approximates model error with a GP confidence interval, and bounds approximation errors using Lipschitz arguments.The resulting approximation accounts for both confidence and linearization errors.
  • One-step predictions: The next-state ellipsoid combines the linearized-model approximation and remainder ellipsoid through a Minkowski sum.The remainder captures confidence-interval and linearization errors.
  • Multi-step predictions: Lemma 2 provides high-probability one-step confidence regions, and these regions extend to sequences of multi-step ellipsoids containing the true trajectory.The construction iteratively applies the one-step prediction update.
  • Multi-step predictions: Corollary 1 states that the true state remains in the propagated ellipsoids jointly for all time with probability at least 1 −δ.This uniform containment lets safety of the ellipsoids imply high-probability safety of the system.
  • State-feedback predictions: Affine state-feedback controllers are used because open-loop input sequences can produce excessively large reachability sets when they cannot correct prediction deviations.The feedback law can contract ellipsoids toward their centers.

B. Safety constraints

Safety is enforced by requiring every propagated state ellipsoid and its corresponding affine control image to remain inside the state and input constraint sets.

  • Constraint formulation: The predicted ellipsoid sequence must satisfy state and control safety constraints for every prediction step.The state constraints are checked on Rt, while control constraints are checked on πt(Rt).
  • State constraints: State safety is expressed as Rt = E(pt, Qt) ⊂X, decomposed into individual constraints Rt ⊂Xi.An analytical formulation is available for these ellipsoid-to-polytope constraints.
  • Control constraints: Because the controller is affine, its image of an ellipsoid is also represented as an ellipsoid, enabling direct enforcement of πt(Rt) ⊂U.The resulting control-set constraint is imposed for each input constraint.

C. The SafeMPC algorithm

SAFEMPC solves a constrained finite-horizon MPC problem and uses a terminal safe-set condition plus a backup policy to preserve safety and recursive feasibility.

  • MPC formulation: The MPC problem propagates ellipsoids with Rt+1 = ˜m(Rt, πt) while imposing intermediate constraints and the terminal condition RT ⊂Xsafe.The objective can be selected for the control task.
  • Terminal set: The terminal constraint produces feedback controllers that steer the predicted system back to the safe region Xsafe.This return trajectory is central to maintaining safe operation.
  • Recursive feasibility: When MPC is infeasible, SAFEMPC shifts the previous controller sequence and appends πsafe until a new feasible solution is found.The procedure is implemented in receding-horizon fashion.
  • Safety guarantee: Theorem 2 states that the algorithm-defined controller makes the system δ-safe when initialized in Xsafe.The proof uses the high-probability propagated-ellipsoid guarantee and the terminal constraint.
  • Safety guarantee: The safety proof proceeds by induction: feasible MPC uses the return path, while infeasibility invokes the backup controller from the safe set.Uniform high-probability bounds apply over the relevant closed-loop horizons.

D. Optimizing long-term behavior

The extended MPC jointly plans a safety-preserving return trajectory and a longer-horizon performance trajectory. Sharing the first r controls preserves the safety guarantees while enabling performance optimization.

  • D. Optimizing long-term behavior: The controller simultaneously plans a return strategy and a performance trajectory over a possibly longer horizon.The performance trajectory uses a performance model, while the return strategy comes from the safety-constrained MPC problem.
  • D. Optimizing long-term behavior: The first r controls are constrained to be identical for the safety and performance trajectories.Here r is bounded by the return horizon and performance horizon.
  • D. Optimizing long-term behavior: The safety guarantees transfer directly because the controller can always fall back to the feasible return strategy.This preserves recursive safety while optimizing performance through the extended MPC problem.

E. Discussion

The implementation combines nonlinear programming with uncertainty propagation that is not analytic but still provides exact function values and derivative information. Linearizing the GP mean is identified as a way to reduce conservatism.

  • E. Discussion: The performance-optimizing MPC problem can be solved with commonly used nonlinear programming solvers such as Ipopt.The uncertainty propagation requires an eigenvalue problem, making the scheme non-analytic.
  • E. Discussion: Linearizing the GP mean prediction can further reduce conservatism in multi-step-ahead predictions.This linearization was omitted from the presentation for clarity.

V. EXPERIMENTS

The experiments evaluate SAFEMPC on an inverted pendulum, using safe-set initialization and mutual information to assess exploration. The static setup compares exploration paths and collected information across horizon choices.

  • V. EXPERIMENTS: The experiments evaluate SAFEMPC for safely exploring the dynamics of an inverted pendulum system.The pendulum state comprises angle and angular velocity, with torque as the control input.
  • V. EXPERIMENTS: The pendulum uses bounded torque inputs and a terminal-set constraint that prevents falling by acting as a stability constraint.The control constraint is U = {u ∈R| −1 ≤u ≤1}, and the safety region is based on a conservative polytopic inner approximation.
  • V. EXPERIMENTS: The initial model is trained from n0 = 25 samples collected inside the safe set using the backup controller.A fixed βn = 2 is used for confidence intervals instead of the theoretically conservative scaling choice.
  • V. EXPERIMENTS: Exploration performance is evaluated using mutual information between collected samples and the GP prior on the unknown model error.The mutual information is computed in closed form.
  • V. EXPERIMENTS: For T ∈{1, 4, 5}, Fig. 3 visualizes informative paths to the safe set, with T = 4 balancing cautiousness and return-trajectory length.T = 1 concentrates samples near the origin, while T = 5 is too cautious because final-state uncertainty is too large.

A. Static Exploration

Static exploration resets the system to arbitrary states and selects samples through safety-constrained MPC. Horizon length determines the trade-off between early informativeness, long-run learning, and propagated uncertainty.

  • A. Static Exploration: The static experiment resets the system to an arbitrary state at every iteration.Without terminal constraints, selecting the largest predictive standard deviation is close to optimal, but SAFEMPC restricts samples to feasible return trajectories.
  • A. Static Exploration: The MPC exploration objective minimizes the negative sum of predictive standard deviations while optimizing the initial state.This targets high-uncertainty states while requiring the resulting state-action pairs to lie on feasible return trajectories to XSafe.
  • A. Static Exploration: The procedure updates the sample set, collects an observation, and updates the GP models for varying horizon lengths.Because initial-state optimization is highly non-convex, 25 random initializations are used at each iteration.
  • A. Static Exploration: For T = 1, exploration is slow and mutual information quickly levels off because the algorithm can move only one step outside the safe set.For T = 4, the algorithm explores more of the state space and gains more information.
  • A. Static Exploration: Short horizons gather more informative samples initially, whereas longer horizons become superior after knowledge accumulates; T = 4 gives the best trade-off.Long horizons initially suffer from uncertainty propagated over many steps.
  • A. Static Exploration: The results suggest adaptively changing the horizon, including increasing it when mutual information saturates for the current horizon.A variable-horizon MPC approach is proposed as one possible implementation.

B. Dynamic Exploration

Dynamic exploration collects informative samples online while preserving a safety trajectory. Planning an additional performance trajectory improves exploration, with the best performance at T = 4.

  • Experimental setup: SAFEMPC collects informative samples during operation over 200 iterations without resetting the system at every iteration.The experiment starts from x0 ∈ Xsafe and applies SAFEMPC throughout.
  • Performance trajectory: The performance trajectory represents states as Gaussians and propagates them using the predictive mean and variance of the current state and applied action.The propagation uses mt+1 = µn(mt, ut) and St+1 = Σn(mt, ut).
  • Performance trajectory: Its cost maximizes predictive confidence intervals along the trajectory while penalizing deviation from the safety trajectory.The first actions of the safety and performance trajectories are constrained to be identical.
  • Results: Except for T = 1, performance-trajectory decomposition consistently outperforms the standard setting in mutual information.The comparison uses varying T ∈ {1, .., 5}, fixed H = 5, and information gathered after 200 iterations.
  • Results: The best exploration performance occurs at T = 4, with a slight degradation at T = 5.Planning the additional performance trajectory provides an additional degree of freedom for exploration.
  • Implications: As data accumulate and the statistical model is updated, SAFEMPC becomes more accurate and improves control performance while maintaining safety guarantees.This conclusion applies to safely exploring partially unknown systems.
Loading 1803.08287v3…