PaperlayerView exact revision on arXiv

Source-linked AI summary

A predictive safety filter for learning-based control of constrained nonlinear dynamical systems

Kim P. Wabersich, Melanie N. Zeilinger

arXiv:1812.05506v4eess.SYcs.LG

TL;DR

Real-world RL must handle physical and safety constraints that popular algorithms do not explicitly support. The paper introduces a predictive safety filter using MPC, a data-driven probabilistic model, and state- and input-dependent uncertainties to certify or modify RL inputs. The approach provides probabilistic safety for nonlinear systems, supports modular RL integration, and is illustrated in pendulum and quadrotor simulations.

  • Problem

    RL methods generally do not explicitly account for state and input constraints, creating a safety gap for real-world control of physically limited systems.

  • Method

    A predictive safety filter uses online MPC backup trajectories and a data-driven probabilistic nonlinear system model to certify proposed RL inputs or modify them when safety cannot be guaranteed.

  • Results

    The formulation provides rigorous probabilistic safety guarantees for nonlinear systems and is demonstrated in pendulum swing-up and 12-state, 4-input quadrotor simulations.

  • Takeaways & Limitations

    The filter makes safety modular with respect to the learning controller and can support exploration beyond available data while limiting intervention to unsafe proposed inputs.

  • Takeaways & Limitations

    Large model uncertainties can make the safety-filter optimization infeasible initially, requiring a longer prediction horizon or a lower safety probability level.

Abstract

from arXiv · show

The transfer of reinforcement learning (RL) techniques into real-world applications is challenged by safety requirements in the presence of physical limitations. Most RL methods, in particular the most popular algorithms, do not support explicit consideration of state and input constraints. In this paper, we address this problem for nonlinear systems with continuous state and input spaces by introducing a predictive safety filter, which is able to turn a constrained dynamical system into an unconstrained safe system and to which any RL algorithm can be applied `out-of-the-box'. The predictive safety filter receives the proposed control input and decides, based on the current system state, if it can be safely applied to the real system, or if it has to be modified otherwise. Safety is thereby established by a continuously updated safety policy, which is based on a model predictive control formulation using a data-driven system model and considering state and input dependent uncertainties.

1 Introduction

The paper proposes a predictive safety filter that lets unconstrained RL controllers operate safely on constrained nonlinear systems. It combines MPC-based backup trajectories with probabilistic, data-driven uncertainty handling and demonstrates the approach in pendulum and quadrotor simulations.

  • Motivation: Physical limitations and safety specifications create state and input constraints that popular RL methods generally do not explicitly handle.Examples include limited robot-arm torque, building power supply, and crash prevention for autonomous vehicles or aircraft.
  • Concept: The predictive safety filter verifies each RL-proposed input and modifies it only when necessary to maintain safe operation at future times.It searches for safe backup trajectories and aims to make the smallest possible intervention while preserving safety.
  • Concept: The filter provides modular safety certification, allowing any RL algorithm without safety certificates to control the resulting safe system.The safety architecture separates the learning controller from the safety policy that processes its input.
  • Contributions: The implicit safe-set representation avoids offline computations that scale exponentially with state dimension, but requires solving an optimization problem online.The authors note that efficient optimization tools are available for this online computation.
  • Contributions: State- and input-dependent uncertainty estimates support probabilistic robustness for nonlinear learned models and can enable exploration beyond available data.The formulation relates filter parameters and model accuracy to safety at a desired probability level and can be less conservative than uniform uncertainty bounds.
  • Experiments: Simulations cover a data-scarce pendulum swing-up task with prohibited overshoots and a 12-state, 4-input quadrotor landing task.The examples address challenging safety constraints, scalability, and practical implementation.

2 Related Work

Related work spans safe model-free RL, learning-based MPC, and barrier-function safety frameworks. The paper targets their reported gaps in rigor, conservatism, scalability, system-class coverage, and task dependence.

  • Safe model-free reinforcement learning: Safe model-free RL methods often provide weaker guarantees, such as one-step constraint satisfaction in expectation, than probabilistic safety at all future time steps.Policy-based safety can also couple certificates to a specific controller or task, limiting generalization.
  • Learning-based model predictive control: Learning-based MPC methods may lack rigorous safety guarantees, become overly conservative through Lipschitz-based uncertainty estimates, or apply only to specific system classes.These limitations are reported despite practical demonstrations of some methods.
  • Model-based policy certification: Barrier-function and safety-framework approaches can have limited scalability or rely on principled computations for restricted constrained model classes.The cited frameworks include methods based on linear or polynomial models and task-specific constructions.
  • Position of this paper: Unlike similar MPC-inspired mechanisms, the proposed approach handles nonlinear models with stochastic parameter uncertainty and uses state- and input-dependent uncertainty estimates.This design is intended to reduce conservatism relative to approaches using uniform uncertainty bounds.

3 Problem Statement

The problem is to learn a performance-oriented policy for a constrained stochastic nonlinear system while enforcing safety at a desired probability level. The safety filter separates learning from real-system actuation by certifying or minimally modifying proposed inputs.

  • System and constraints: The system has nonlinear discrete-time dynamics with unknown real parameters represented by a distribution inferred from available information or data.The initial condition is stochastic, and the model is parameterized by θR.
  • System and constraints: Physical limitations and safety requirements are represented by polyhedral state and input constraint sets X and U.The constraints restrict both x and u during operation.
  • Learning objective: The learning policy πL seeks to minimize an episodic, finite-horizon, or infinite-horizon objective while satisfying safety at a desired probability level pS.The stage cost may include deterministic and zero-mean stochastic components.
  • Safety-filter objective: The safety filter maps the learning input and current state to the real-system input, enabling any RL algorithm to act through a virtual input while closed-loop safety is enforced.This creates an unconstrained learning interface relative to the safe system.
  • Safety certification: An input is certified when the filter leaves it unchanged and the resulting closed loop remains safe for all future times.Inputs that cannot be certified are replaced with alternative safe inputs, with the filter seeking the smallest modification.

4 Predictive safety filter

The predictive safety filter certifies or minimally modifies learning inputs by constructing safe backup trajectories with MPC, extending the approach to uncertain nonlinear models through probabilistic constraint handling. Feasibility and recursive backup mechanisms maintain safety, while confident-subspace planning reduces conservatism.

  • Predictive safety filter: The method computes safe backup plans online and modifies the learning input only as much as needed to maintain future constraint satisfaction.The optimization minimizes deviation between the planned first input and the requested learning input.
  • Nominal predictive safety filter: In the nominal setting, backup trajectories remain in a confident state-input subset and terminate in a safe set with a locally valid terminal safety filter.The terminal set supports safety beyond the finite planning horizon.
  • Predictive safety filter under uncertainty: Under model uncertainty, predictions use nominal expected states and tightened constraints so the true trajectory remains safe with probability pS.The probabilistic formulation compensates prediction errors induced by the learned model.
  • Nominal predictive safety filter: If the full-horizon problem becomes infeasible, the filter reuses the tail of the previously feasible trajectory with a reduced horizon, preserving admissible states and inputs.A terminal safety filter can be applied when the shrinking-horizon mechanism reaches the relevant condition.
  • Predictive safety filter under uncertainty: Planning is restricted to regions with sufficiently confident dynamics, using an allowable error set scaled by γ to reduce conservatism while supporting cautious exploration.The approach links the tolerated model error to constraint tightening through γ and ϵ.
  • Predictive safety filter under uncertainty: Safety in probability follows when the model-confidence map varies sufficiently slowly, the tightening factor is selected appropriately, and the initial planning problem is feasible.Large uncertainties can instead make the initial predictive safety filter problem infeasible, requiring a longer horizon or lower safety probability.

5 Application to numerical examples

The numerical examples apply the predictive safety filter to pendulum swing-up and quadrotor learning, using data-driven models and uncertainty-aware safety constraints. The filter enables safe exploration and constraint satisfaction while learning performance-oriented policies.

  • 5.1 Swing-up: Safe exploration beyond initial data: The pendulum task swings from 0° to 180° under limited torque and safety constraints −90° ≤ α ≤ 190°.The torque is restricted to |u| ≤ 0.7 [Nms/rad].
  • 5.1 Swing-up: Safe exploration beyond initial data: Bayesian regression models the pendulum dynamics from data, with a set-valued confidence map representing parametric uncertainty.The model starts with 10 data points, updates after each episode, and uses ρ = 0.999, ϵ = 0.02, and γ = 0.02.
  • 5.1 Swing-up: Safe exploration beyond initial data: After 120 learning episodes, additional data from 18,000 samples enables less conservative behavior and supports a complete safe swing-up beyond the initial data.The initial 10-point model produces cautious trajectories; the refined model supports complete swing-up.
  • 5.2 Safe data-driven quadrotor learning control: The quadrotor experiment models a 10-state, 3-input system with minimum height z ≥ 0.175 [m] and uses a 20-step predictive horizon.The terminal safe set is selected at z ≥ 1.5 [m], with desired chance constraint satisfaction pS = 0.9.
  • 5.2 Safe data-driven quadrotor learning control: The predictive safety filter enables constraint satisfaction during all 240 quadrotor learning episodes and remains inactive on the favorable optimal trajectory.Unsafe policy search produces ground contacts, defined as states less than 0.01 [m] from the ground; safety-augmented search avoids these contacts.

6 Conclusion

The paper introduces a predictive safety filter for modular safe reinforcement learning with nonlinear, potentially data-driven system models. It reports rigorous safety guarantees while supporting complex descriptions without being overly conservative.

  • 6 Conclusion: The predictive safety filter separates safety from the employed RL algorithm, enabling modular safe RL.The formulation can use a possibly data-driven approximate system model.
  • 6 Conclusion: The optimization-based formulation provides rigorous safety guarantees for nonlinear and complex system descriptions.The authors characterize the approach as an important step toward safe RL for realistic applications.

A.1 Lipschitz continuity w.r.t. Hausdorff metric

This appendix defines the Hausdorff metric for comparing sets and Lipschitz continuity for set-valued maps under that metric.

  • A.1 Lipschitz continuity w.r.t. Hausdorff metric: The Hausdorff metric compares two sets within a metric space using the underlying distance function.It is introduced as the basis for measuring set-valued-map variation.
  • A.1 Lipschitz continuity w.r.t. Hausdorff metric: A set-valued map is Lipschitz continuous when Hausdorff distance between its outputs is bounded by a constant times the input 2-norm distance.The bound is dH(E(a), E(b)) ≤ LE ∥a − b∥2.

A.2 Proof of Theorem 4.6

The proof establishes recursive feasibility of the predictive safety filter and uses this property to prove probabilistic satisfaction of state and input constraints for all future times.

  • A.2 Proof of Theorem 4.6: Constraint satisfaction follows by bounding trajectory changes through Lipschitz continuity, constraint tightening, and a sufficiently small uncertainty-set scaling parameter γ.The proof separately verifies state, terminal, input, and uncertainty constraints.
  • A.2 Proof of Theorem 4.6: The admissible error magnitude depends on the constraint-tightening fraction ϵ and the region where local incremental stabilizability holds.Increasing tightening beyond the stated bound does not necessarily increase the uncertainty magnitude that can be tolerated.
  • A.2 Proof of Theorem 4.6: Conditioning on the model error remaining in the probabilistic confidence set reduces the safety proof to a deterministic feasibility argument.The confidence-set assumption then yields chance-constraint satisfaction with probability at least pS.

A.3 Offline design verification

Offline design verification uses sampled system trajectories and concentration bounds to certify the safety probability and select a sufficiently large simulation budget.

  • A.3 Offline design verification: The verification procedure samples model parameters and initial states to estimate the probability of satisfying state and input constraints over all time steps.The sampled executions define an indicator for safe execution and an empirical safety probability.
  • A.3 Offline design verification: Hoeffding’s inequality provides a finite-sample confidence bound for the estimated infinite-horizon safety probability.The bound relates the error margin and confidence level to the required number of offline simulations.
  • A.3 Offline design verification: If verification is infeasible, the design parameters must be adjusted or additional data must be collected.The proposition formalizes a lower bound on the total number of offline simulations when the estimated probability exceeds the target.
  • A.3 Offline design verification: 1656 simulations are required when pS = 0.95 and the estimated safety probability is 0.99, to obtain 99% confidence for the parametrization.The required sample count is independent of system complexity, while tighter probability gaps can increase it rapidly.

A.4 Sufficient condition for Assumption 4.3

The paper gives a sufficient, verifiable condition for the incremental-stabilizability assumption using stabilizable linearizations and a quadratic Lyapunov-like function.

  • A.4 Sufficient condition for Assumption 4.3: For every state-input reference, stabilizability of the linearized pair permits continuous positive-definite matrices and feedback gains to be selected.The matrices and gains vary continuously with the reference.
  • A.4 Sufficient condition for Assumption 4.3: The condition also bounds how the matrix associated with the quadratic function changes along successive references.This accounts for linearization errors as the nonlinear system evolves.
  • A.4 Sufficient condition for Assumption 4.3: Choosing V(x, µ, v) = (x−µ)⊤Pr(x−µ) provides a bound on its one-step change under the local feedback law.The resulting condition is sufficient for Assumption 4.3, rather than necessary.
  • A.4 Sufficient condition for Assumption 4.3: The analytical condition can be conservative for nonlinear systems whose linearizations change rapidly along references.Its conservatism is tied to large changes between consecutive linearized system matrices.
Loading 1812.05506v4…