Source-linked AI summary
Probabilistic model predictive safety certification for learning-based control
Kim P. Wabersich, Lukas Hewing, Andrea Carron, Melanie N. Zeilinger
TL;DR
Safety-critical uses of reinforcement learning need closed-loop constraint guarantees, but RL methods often do not explicitly address physical limits at each step. PMPSC filters learning inputs through a safe backup policy and provides probabilistic constraint guarantees. Its feasible scheme satisfies the stated chance constraints for all times, and a car simulation reports safe learning in every episode.
Problem
RL methods often do not explicitly address physical system limitations and safety requirements at each time step, limiting their applicability in safety-critical settings.
Source: §I · p. 1Method
PMPSC checks proposed learning inputs using a safe backup policy, applying the proposal when safe and otherwise overriding it to maintain probabilistic constraint guarantees.
Source: §I · p. 1Results
The feasible PMPSC scheme satisfies the chance constraints for all times, and car simulation shows safe learning in every episode versus frequent road exits without PMPSC.
Source: B. Probabilistic model predictive… · p. 6, §VI · p. 10Takeaways & Limitations
The framework can enhance arbitrary learning-based control methods with safety guarantees during learning.
Source: §VII · pp. 10–11
Abstract
from arXiv · showhide
Reinforcement learning (RL) methods have demonstrated their efficiency in simulation environments. However, many applications for which RL offers great potential, such as autonomous driving, are also safety critical and require a certified closed-loop behavior in order to meet safety specifications in the presence of physical constraints. This paper introduces a concept, called probabilistic model predictive safety certification (PMPSC), which can be combined with any RL algorithm and provides provable safety certificates in terms of state and input chance constraints for potentially large-scale systems. The certificate is realized through a stochastic tube that safely connects the current system state with a terminal set of states, that is known to be safe. A novel formulation in terms of a convex receding horizon problem allows a recursively feasible real-time computation of such probabilistic tubes, despite the presence of possibly unbounded disturbances. A design procedure for PMPSC relying on bayesian inference and recent advances in probabilistic set invariance is presented. Using a numerical car simulation, the method and its design procedure are illustrated by enhancing a simple RL algorithm with safety certificates.
I. INTRODUCTION
The introduction motivates safety certification for learning-based control and presents PMPSC as a policy-independent way to filter unsafe learning inputs. It contrasts this approach with prior safe-RL methods and introduces the paper’s notation.
I. INTRODUCTION
- RL methods often do not explicitly enforce physical constraints at each time step, limiting their use in safety-critical industrial applications.Conventional controllers may require time-consuming manual tuning to meet safety specifications.Source: §I · p. 1
- PMPSC certifies proposed learning inputs using a safe backup policy, applying the proposal when safe and otherwise overriding it.The safety policy is recomputed using the predicted next state, and the framework can be combined with arbitrary learning-based controllers.Source: §I · p. 1
- The framework targets continuous, chance-constrained state and input spaces, with an online, data-driven synthesis that avoids explicitly computing the safe state-action set offline.The paper presents this implicit representation as a way to avoid the curse of dimensionality associated with explicit computation.Source: §I · pp. 1–2
II. RELATED WORK
- Unlike task-specific or policy-tied safe-RL approaches, the proposed MPC-based safety mechanism is independent of the learning policy.This allows RL to address complex tasks and measured black-box rewards that MPC’s explicit online cost formulation may not accommodate.Source: §II · p. 2
- The method formulates safety probabilistically for all future times, allowing a chosen conservatism level and accommodating disturbances with unbounded support.It also relies on an implicit safe-set description to support scalability with state dimension.Source: §II · pp. 2–3
A. Notation
- The notation section defines set operations including Minkowski sum and Pontryagin difference, and uses Pr(E), E(x), and var(x) for probability, expectation, and variance.It also defines N(μ, Σ) as a multivariate Gaussian distribution.Source: A. Notation · p. 3
B. Problem statement
The problem statement models unknown nonlinear dynamics through an uncertain linear approximation and defines safety as probabilistic constraint satisfaction. A safety policy uses proposed inputs when certifiable and otherwise supplies a safe modification and backup trajectory.
B. Problem statement
- The unknown system is nonlinear and time-invariant, while controller design uses a linear model with model error and stochastic disturbance terms.The model-error set is chosen from available data; the paper notes that nonlinear-system error-set computation can require non-convex optimization and that Gaussian-process regression is another option.Source: B. Problem statement · p. 3
- Safety is defined by state and input chance constraints that must hold at every time step with specified probabilities.The stated probability accounts for uncertain parameters and noise realizations.Source: B. Problem statement · p. 3
- A safety policy maps the current state and proposed learning input to a safe input and a backup trajectory that maintains the constraints in future steps.The proposal is applied when certified; otherwise the policy supplies a safe modification.Source: B. Problem statement · p. 3
- The safety policy is defined implicitly by an optimization that certifies the current input while seeking a safe backup for the predicted next state with minimal modification to the learning input.This optimization uses MPC techniques intended to provide real-time feasibility and scalability while defining a large safe set implicitly.Source: B. Problem statement · p. 3
IV. PROBABILISTIC MODEL PREDICTIVE SAFETY CERTIFICATION
The scheme computes an MPC safety policy that accepts a learning input when a safe backup trajectory exists, extending deterministic certification with probabilistic tubes and chance constraints.
A. Nominal model predictive safety certification scheme
- The nominal scheme computes a constraint-satisfying input sequence to an invariant terminal set, applying the learning input when feasible or the closest safe alternative otherwise.The invariant terminal set makes feasibility recursive, so a feasible solution continues to exist at subsequent time steps.Source: A. Nominal model predictive safety… · p. 4
B. Probabilistic model predictive safety certification scheme
- The probabilistic scheme tracks a nominal trajectory and bounds deviations with probabilistic reachable sets, tightening state and input constraints to meet chance constraints.The auxiliary feedback keeps the true state within a probabilistic tube around the nominal trajectory.Source: B. Probabilistic model predictive… · pp. 4–5
- Under Assumptions IV.3 and IV.4, feasibility of (6) guarantees state and input chance constraints for all times and learning inputs.The error remains within the probabilistic reachable sets with probabilities at least px and pu, while recursive feasibility preserves tightened nominal constraints.Source: B. Probabilistic model predictive… · p. 6
- Unlike formulations that may require recovery, the formulation retains recursive feasibility even for unbounded disturbance realizations.The nominal system is simulated rather than optimized over its initial state, preserving error dynamics across online replanning.Source: B. Probabilistic model predictive… · pp. 6–7
C. Discussion: Safe RL with predictive safety certification
Safety filtering can introduce nonlinearities that slow learning, but the PMPSC policy is continuous in the state and proposed input; safety restrictions can still reduce performance.
C. Discussion: Safe RL with predictive safety certification
- The safety mechanism can introduce additional nonlinearities into the system being optimized, potentially slowing learning convergence.The paper notes that this can occur when combining PMPSC with RL.Source: C. Discussion: Safe RL with predictive… · p. 7
- Adding a cost for deviations between proposed and applied inputs can encourage convergence, while several RL algorithms require continuity of the applied policy.The paper identifies TRPO, deterministic actor-critic policy search, and Bayesian Optimization as examples with continuity-related theoretical properties.Source: C. Discussion: Safe RL with predictive… · p. 7
- The PMPSC policy is continuous in the state and learning input, and the paper states that RL convergence properties can typically be preserved with the convex formulation.It can also compute partial derivatives of the safety policy with respect to the proposed learning input for policy updates.Source: C. Discussion: Safe RL with predictive… · p. 7
- Safety filtering can reduce performance compared with plain unsafe RL because it restricts the space of possible inputs.Source: C. Discussion: Safe RL with predictive… · p. 7
V. DATA BASED DESIGN
The data-based design procedure computes the model, probabilistic error tubes, and nominal terminal set from measurements collected from the system.
- PMPSC requires a model description, probabilistic state and input error tubes, and a nominal terminal set to support recursive feasibility.Source: §V · p. 7
- The paper presents efficient techniques tailored to learning-based control for computing these components.Source: §V · p. 7
- The design techniques rely solely on data collected from the system.Source: §V · p. 7
A. Model design
The model design estimates a linear Bayesian model from noisy measurements and uses posterior confidence sets to bound model uncertainty. Because model error can be unbounded over an unbounded domain, the procedure assumes it is bounded outside a sufficiently large outer state and input space.
A. Model design
- Linear Bayesian regression estimates system parameters from noisy measurements using a prior distribution on the parameters.The measurements pair state and input values with noisy outputs.Source: A. Model design · p. 7
- A posterior confidence region at probability level pθ represents parameter realizations conditioned on the collected data.This region is used to derive a set of possible system dynamics and model errors.Source: A. Model design · p. 7
- Model error can grow unbounded on an unbounded domain, so the design assumes it is bounded outside a sufficiently large outer state and input space.The outer space contains the state and input spaces of interest.Source: A. Model design · p. 7
- A singular value decomposition of the posterior covariance yields major axes for a polytopic outer approximation of parameter uncertainty and, in turn, the model-error set.The outer approximation of the parameter confidence set is used to construct an outer approximation of Wθ.Source: A. Model design · p. 8
B. Calculation of R for uncertain linear dynamics and unbounded disturbances
The error set combines a robustly invariant bound for model mismatch with a probabilistic reachable set for stochastic disturbances. Their separate construction supports a probabilistic bound for cumulative error, including when disturbances may be unbounded.
B. Calculation of R for uncertain linear dynamics and unbounded disturbances
- The error is decomposed into model mismatch and stochastic disturbance; the disturbance PRS can be computed independently of the RIS used for model uncertainty.The model-mismatch component is bounded robustly in probability, while the disturbance component uses its known distribution.Source: B. Calculation of R for uncertain… · pp. 8–9
- If E is a robustly invariant set for model error at probability pθ and Rs is a probabilistic reachable set for disturbance error at probability ps, then E ⊕ Rs bounds cumulative error with probability pθps.The construction separates the two error sources before combining their sets.Source: B. Calculation of R for uncertain… · p. 9
- The disturbance PRS can be computed from the noise variance using a Chebyshev bound, or from a chi-squared quantile when the noise is normally distributed.The selected probability level depends on the chosen bound.Source: B. Calculation of R for uncertain… · p. 9
- For ellipsoidal model-error bounds, robust-control methods construct an ellipsoidal RIS by solving an optimization problem.The ellipsoid’s weighting matrix is pre-selected, and the stated optimization can be solved efficiently by bisection on α^-1.Source: B. Calculation of R for uncertain… · p. 9
- The combined PRS is R = Rs ⊕ E, with probability pspθ; the allocation between ps and pθ can be chosen to meet the desired state and input tube probability levels.The component sets are computed for stochastic disturbance and model uncertainty, respectively.Source: B. Calculation of R for uncertain… · p. 9
C. Iterative construction of the terminal safe set Zf
The terminal safe set can be enlarged using nominal trajectories from successfully solved instances of the safety optimization. Under the stated convexity and terminal-set assumptions, the enlarged set retains the required terminal-set property.
C. Iterative construction of the terminal safe set Zf
- The terminal set is iteratively enlarged by adjoining the convex hull of nominal states from successful optimization solutions, reducing potentially conservative modifications to the learning-based input.The procedure can start from a conservative terminal set.Source: C. Iterative construction of the… · p. 9
- If the original terminal set satisfies Assumption IV.4 and the optimization problem is convex, the enlarged set also satisfies Assumption IV.4.The proposition applies to the convex hull of the successful predicted states together with the existing terminal set.Source: C. Iterative construction of the… · p. 9
- Convex combinations of feasible predicted states preserve state and input constraints, allowing a nominal terminal control law to be defined for the enlarged set.The construction relies on convexity of the feasible set and of the constraints.Source: C. Iterative construction of the… · p. 9
D. Overall MPSC design procedure
The overall design procedure derives a model and uncertainty sets from collected data, tightens the constraints, and initializes a terminal set. It then safeguards a learning-based control law online, with closed-loop data and trajectories available to refine the design.
D. Overall MPSC design procedure
- Estimate a linear dynamical model from measurements, then use singular value decomposition to form a polytopic confidence set and model-uncertainty set.These are the first two synthesis steps.Source: D. Overall MPSC design procedure · p. 10
- Compute PRS sets for stochastic uncertainty and RIS sets for model uncertainty, then tighten state and input constraints using their Minkowski-sum bounds.The tightened constraints use Rx = Rs,x ⊕ Ex and KRRu = KR{Rs,u ⊕ Eu}.Source: D. Overall MPSC design procedure · p. 10
- Initialize Zf = {0}, then safeguard a potentially unsafe learning-based control law by solving the safety optimization and applying its control at each time step.The procedure also notes that less restrictive initial terminal sets can be computed using standard invariant-set methods.Source: D. Overall MPSC design procedure · p. 10
- During closed-loop operation, collected measurements can reduce model uncertainty and permit recomputation of tighter error bounds, while nominal trajectories can enlarge Zf.The text states that tighter bounds enable greater exploration in the future.Source: D. Overall MPSC design procedure · p. 10
VI. NUMERICAL EXAMPLE: SAFELY LEARNING TO CONTROL A CAR
The car simulation tests whether PMPSC can keep learning within road constraints while tracking a periodic reference. It enables safe learning in every episode, with slightly slower convergence and slightly worse post-convergence performance on average than direct application of the learning controller.
VI. NUMERICAL EXAMPLE: SAFELY LEARNING TO CONTROL A CAR
- Bayesian Optimization learned a linear control law for tracking a periodic reference on a narrow road with the additional constraint ||y|| ≤1.The controller used a 30-step planning horizon and model and probabilistic reachable sets designed at a 98% probability level.Source: §VI · p. 10
- PMPSC kept learning within the safety constraints in every episode, while many episodes without it left the road constraints.The task was to track a periodic reference on a narrow road.Source: §VI · p. 10, Figure 4 · p. 10
- PMPSC interventions decreased as learning converged after 30 episodes, but convergence was slightly slower and average post-convergence performance slightly worse.The comparison was with direct application of the learning-based controller.Source: §VI · p. 10
VII. CONCLUSION
The conclusion presents PMPSC as a way to add safety guarantees to arbitrary RL methods using a data-driven belief approximation and on-the-fly safety policies. Its design uses Bayesian inference and convex optimization to establish closed-loop safety at all time steps with little expert system knowledge.
VII. CONCLUSION
- A data-driven, linear belief approximation of system dynamics is used to compute safety policies for learning-based controllers.The methodology is intended to enhance arbitrary RL algorithms with safety guarantees during learning.Source: §VII · p. 10
- The framework establishes closed-loop safety by proving that a safety policy exists at all time steps.The policies are computed on the fly.Source: §VII · p. 11
- The design procedure relies on Bayesian inference and convex optimization and requires little expert system knowledge.Source: §VII · p. 11
APPENDIX
The appendix describes data-based model and constraint design for the numerical example. It also shows how nominal trajectories can iteratively enlarge the terminal set to reduce conservatism.
A. Details of numerical example
- System measurements and a linear-quadratic regulator-based mean dynamics model are used to compute the probabilistic invariance set.The model calculation uses sensor noise σs = 0.01 and a prior distribution.Source: A. Details of numerical example · p. 11, Figure 7 · p. 11
- The final design tightens the constraints to ||uδ|| ≤1.39 rad, −5.4 ≤ua ≤1.3 m s−2, ||y|| ≤0.87 m, ||δ|| ≤0.64 rad, ||v|| ≤19.97 m s−1, and −4.97 ≤a ≤0.44 m s−2.Different measurement counts produce different state and input constraint tightenings.Source: A. Details of numerical example · p. 11, Figure 8 · p. 11
- Starting from Zf = {0}, nominal state trajectories can iteratively enlarge the terminal set to reduce conservatism of its constraint.The procedure follows Proposition V.4.Source: A. Details of numerical example · p. 11