Source-linked AI summary
Safe Model-based Reinforcement Learning with Stability Guarantees
Felix Berkenkamp, Matteo Turchetta, Angela P. Schoellig, Andreas Krause
TL;DR
Reinforcement learning can apply unsafe exploratory actions, limiting its use on real-world safety-critical systems. The paper develops stability-based safe learning with statistical models and Lyapunov verification, showing high-probability policy safety while expanding the estimated safe region and improving control performance. Its practical method is demonstrated on a simulated inverted pendulum without leaving the region of attraction.
Problem
Reinforcement learning may use unsafe exploratory actions, making it difficult to apply on real-world physical and safety-critical systems.
Method
The paper combines Lyapunov stability verification with statistical models of dynamics to safely collect informative data, optimize policies, and expand the estimated region of attraction.
Results
The algorithm provides high-probability stability guarantees while safely improving policies and increasing the estimated region of attraction.
Takeaways & Limitations
Safe learning can optimize policies and gather dynamics data without leaving the safe region, supporting stability-certified reinforcement learning for continuous state-action spaces.
Takeaways & Limitations
Region-of-attraction verification suffers from the curse of dimensionality, requiring adaptive discretization to scale to higher-dimensional systems.
Abstract
from arXiv · showhide
Reinforcement learning is a powerful paradigm for learning optimal policies from experimental data. However, to find optimal policies, most reinforcement learning algorithms explore all possible actions, which may be harmful for real-world systems. As a consequence, learning algorithms are rarely applied on safety-critical systems in the real world. In this paper, we present a learning algorithm that explicitly considers safety, defined in terms of stability guarantees. Specifically, we extend control-theoretic results on Lyapunov stability verification and show how to use statistical models of the dynamics to obtain high-performance control policies with provable stability certificates. Moreover, under additional regularity assumptions in terms of a Gaussian process prior, we prove that one can effectively and safely collect data in order to learn about the dynamics and thus both improve control performance and expand the safe region of the state space. In our experiments, we show how the resulting algorithm can safely optimize a neural network policy on a simulated inverted pendulum, without the pendulum ever falling down.
1 Introduction
Reinforcement learning rarely reaches real-world physical systems because intermediate policies can be unsafe, while safety remains an open problem. This paper addresses that gap with stability-based, high-probability safety guarantees that support safe exploration and policy improvement.
- Motivation: Intermediate reinforcement-learning policies may break physical systems or harm their environments before the long-term optimal policy is reached.This concern is especially acute for safety-critical systems affecting human lives.
- Stability-based safety: Asymptotic stability requires exploratory behavior to recover to a safe state, with the region of attraction containing trajectories that remain bounded and converge to a goal.The region of attraction is forward invariant under the closed-loop policy.
- Contribution: The proposed algorithm provides high-probability safety guarantees for policies in continuous state-action spaces.Safety is defined in terms of stability guarantees.
- Contribution: Starting from an initially safe policy, the algorithm collects data inside the safe region, expands the estimated region of attraction, and improves control performance.The policy is adapted without leaving the region used for safe data collection.
- Related work: Existing safe-reinforcement-learning approaches cover risk-sensitive rewards, robust MDPs, safe exploration, constraints, and Bayesian optimization, but often require accurate probabilistic models or focus on different safety notions.Prior stability results primarily address verification or safe learning for fixed policies.
2 Background and Assumptions
The paper models unknown dynamics as errors around a known prior model and defines safety through a policy’s region of attraction. Safe learning relies on continuity, calibrated uncertainty, Lyapunov verification, and an initially stabilizing policy.
- System model: The system combines a known prior model with unknown model errors, while noisy measurements of the dynamics are obtained by applying actions at selected states.The policy maps states to control actions and the goal state is the origin.
- Safety objective: Safe learning must preserve the region of attraction and prevent exploratory actions from driving the system outside it.The region of attraction depends implicitly on both the system dynamics and the policy.
- Assumptions: Lipschitz-continuous dynamics and policies are assumed so learned dynamics can generalize to unvisited states and policy changes do not produce discontinuous behavior.The policy class is restricted to Lπ-Lipschitz functions.
- Assumptions: The statistical model must be well calibrated so confidence intervals contain the true dynamics with probability at least 1 − δ.The bound is expressed as ∥f(x, u) − µn(x, u)∥1 ≤ βnσn(x, u) for all n, x, and u.
- Lyapunov safety: Lyapunov functions certify a region of attraction by replacing convergence analysis with a one-step decrease condition on the Lyapunov function.For a fixed policy, a suitable level set remains invariant and converges to the goal when the Lyapunov value decreases.
- Initialization: Value functions with strictly positive costs away from the origin can provide Lyapunov candidates, and learning assumes an initial policy that stabilizes the origin locally.The initial policy is not assumed safe throughout the entire state space.
3 Theory
The theory combines Lyapunov-based stability verification with statistical dynamics models to optimize policies and expand a certified safe region without leaving it. Under regularity assumptions, safe exploration reduces model uncertainty while preserving high-probability stability guarantees.
- Region of attraction: The algorithm computes a region of attraction for a fixed policy using high-probability confidence intervals on Lyapunov-function predictions.It discretizes the state space and uses Lipschitz continuity to extend the decrease condition from grid points to the continuous domain.
- Policy optimization: Policy optimization selects the largest Lyapunov level set whose policy-induced state-action pairs satisfy the certified decrease constraints.The resulting set V(cn) is a region of attraction for the true system with probability at least 1−δ.
- Safe exploration: The exploration strategy samples the most uncertain state-action pairs within the currently safe set to shrink confidence intervals and expand the estimated safe region.The approach relies on model regularity, including Lipschitz continuity and bounded RKHS norm assumptions, to generalize from collected measurements.
- Safety guarantees: With probability at least 1−δ, the estimated region remains inside the true region of attraction, and selected exploratory transitions do not leave it.These guarantees hold jointly over iterations under the stated assumptions.
- Safety guarantees: After finitely many data points, the explored safe set contains the oracle-certified set Rϵ(S0) while remaining contained in the zero-residual safe set R0(S0).The theorem therefore guarantees at least the oracle baseline’s exploration performance without classifying unsafe state-action pairs as safe.
- Illustrative behavior: In the illustrative run, repeatedly selecting safe data points reduces model uncertainty, improves the policy, and expands the safe region over time.The policy is updated only after collecting data within the current region of attraction, avoiding unsafe evaluations and system failure.
4 Practical Implementation and Experiments
The practical algorithm preserves stability guarantees while replacing intractable exact policy optimization with approximate updates and approximate dynamic programming. On an inverted pendulum, safely collected data improved performance and expanded the estimated region of attraction without leaving the safe region.
- Practical algorithm: The practical variant retains safety guarantees but sacrifices exploration guarantees to obtain a more tractable algorithm.It is summarized in Algorithm 1.
- Practical algorithm: Approximate policy updates optimize performance first, then verify the new policy's region of attraction before collecting further data.A failed update can be replaced by the previous policy, which is guaranteed safe.
- Practical algorithm: The GP confidence intervals guide safety verification and active learning, while exploratory actions are restricted to remain near the current policy for better data efficiency.Actions far from the current policy are unlikely to expand the estimated region of attraction.
- Limitations: Discretized region-of-attraction verification suffers from the curse of dimensionality, motivating adaptive discretization for higher-dimensional systems.Returned policies remain safe for arbitrary numbers of time steps within their verified level sets.
- Experiments: The inverted-pendulum experiment used a GP model whose prior mean had incorrect mass and omitted friction, producing an initially underactuated policy with a small attraction region.Linear and Matérn kernels modeled errors from parameter and integration inaccuracies.
- Practical algorithm: The implementation uses approximate dynamic programming with a neural network policy and a quadratic normalized cost to estimate cost-to-go.The network has two hidden layers with 32 ReLU neurons each.
- Experiments: After 50 data points, the estimated safe region expanded and control performance improved drastically while all learning data were collected safely.The estimated set remained contained within the optimized policy's true safe region.
5 Conclusion
The paper combines reinforcement learning with stability-based safety constraints and statistical dynamics models. It reports safe policy optimization, stability certificates, and theoretical guarantees for safe exploration toward desired state-action pairs.
- The approach combines classical reinforcement learning with safety constraints defined in terms of stability.
- The method safely optimizes policies and provides stability certificates based on statistical models of the dynamics.
- The algorithm has theoretical safety and exploration guarantees while driving the system toward desired state-action pairs during learning.
A.1 Stability verification
Stability verification transfers Lyapunov decrease conditions from a discretized state grid to the continuous domain with high probability. This establishes verified level sets as regions of attraction for learned policies.
- The discretization lemma bounds Lyapunov-function decrease on the continuous domain using a grid whose points lie within τ in 1-norm.
- The confidence-set lemma states that v(f(x,u)) belongs to Q_n for all states and actions with probability at least 1−δ.
- The continuous-domain conclusion follows by combining the grid approximation with the Lyapunov stability theorem.
- Theorem 2 certifies V(c) as a region of attraction when the upper confidence bound satisfies u_n(x,π(x)) < v(x) − L_Δvτ on the discretized level set.
- Theorem 3 guarantees V(c_n) ⊆ R_πn with probability at least 1−δ for every n>0.
- The initial confidence-interval construction ensures that the Lyapunov decrease condition holds for the initial policy.
A.2 Gaussian process model
The dynamics model combines a known prior with GP-modeled errors under Lipschitz and RKHS regularity assumptions. Its posterior confidence intervals cover the true dynamics with high probability, enabling stability verification and exploration guarantees.
- The model-error function is assumed to lie in an RKHS associated with a differentiable kernel and have norm bounded by B_g.This assumption supports the exploration guarantees.
- A GP represents smooth dynamics functions using a prior mean h and a kernel encoding assumptions about function behavior.Noisy dynamics measurements update the posterior mean and covariance.
- For multiple outputs, the GP represents each output dimension through an extended input space and aggregates their posterior standard deviations.
- Kernel smoothness makes the dynamics Lipschitz continuous with constant L_f = L_h + L_g.
- The GP supplies high-probability confidence intervals for the dynamics under the stated regularity and noise assumptions.
- With probability at least 1−δ, the true dynamics lie within the GP posterior confidence intervals for all n≥1, states, and actions.The bound depends on the information capacity γ_n and the confidence scaling β_n.
A.3 Safe exploration
The algorithm expands the safely learnable state-action set by certifying Lyapunov decrease, generalizing through Lipschitz dynamics, and selecting informative points within certified regions. Its guarantees characterize both the maximal safely reachable set and eventual expansion or convergence of exploration.
- Safe exploration: Safe exploration expands knowledge by learning at certified state-action pairs and transferring guarantees to nearby unseen states through Lipschitz continuity.The initial safe set is retained even when it does not satisfy the strict decrease condition, allowing safe initial policies to seed exploration.
- Safe exploration: The algorithm optimizes over Lipschitz policies to find the largest Lyapunov level set whose state-action pairs satisfy the decrease condition.This level set estimates the largest region of attraction supported by the currently learned safe set.
- Safe exploration: Rϵ(S) contains state-action pairs that can be visited safely, and repeated application reaches the largest set classifiable by an algorithm with the same knowledge and policy restrictions.The corresponding region of attraction is Rlev(Rϵ(S)).
- Safe exploration: The safe sets are nested across iterations, so previously certified pairs remain available while confidence estimates become more informative.The induction argument establishes Sn ⊇ Sn−1 and propagates membership to later sets.
- Safe exploration: With probability at least 1 −δ, if the maximal safely classifiable set has not been reached, the safe set expands after a bounded number of iterations.If expansion stops, the analysis instead shows convergence to the fixed point Rϵ(Sn) = Sn.
A.4 Safety and policy adaptation
The learned policies retain stability guarantees because their certified Lyapunov level sets lie within the true regions of attraction. Under Gaussian-process regularity and sub-Gaussian noise assumptions, these safety statements hold jointly with high probability.
- Safety and policy adaptation: For every iteration, the certified level set V(cn) is contained in the true region of attraction Rπn of the selected policy.The result follows by applying the Lyapunov decrease theorem to the policy's certified safe set.
- Safety and policy adaptation: The guarantees assume σ-sub-Gaussian measurement noise and a dynamics-model error with RKHS norm smaller than Bg.The confidence parameter βn incorporates Bg, σ, γn, and δ.
- Safety and policy adaptation: Every successor state f(x, u) generated by a state-action pair in Sn lies within the true region of attraction Rπn.This extends safety from certified state-action pairs to the states reached after applying them.
- Safety and policy adaptation: With probability at least 1 −δ, the learned safe sets remain bounded between the initial safely reachable set Rϵ(S0) and the larger set R0(S0).The guarantee is stated jointly for all n > 0 under the theorem's assumptions.