Source-linked AI summary
End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks
Richard Cheng, Gabor Orosz, Richard M. Murray, Joel W. Burdick
TL;DR
RL often explores unsafe behaviors during learning, limiting deployment on physical systems. The paper combines model-free RL, CBF controllers, and online dynamics learning, using GP uncertainty estimates to support safety. RL-CBF is reported to improve exploration efficiency while maintaining safety during learning across the tested control tasks.
Problem
RL lacks safety guarantees during learning, allowing unsafe policies that can damage hardware or harm humans and contributing to limited physical-system deployment.
Method
The framework integrates model-free RL with CBF controllers and online GP-based dynamics learning to guide exploration within safe policies under uncertain model information.
Results
The paper reports greater policy exploration efficiency and safety throughout learning, with the framework applicable regardless of the RL algorithm used.
Takeaways & Limitations
Combining crude model information and CBFs with model-free RL supports efficient learning with end-to-end safety for physical-system deployment.
Takeaways & Limitations
The computationally efficient approximation does not necessarily retain the performance guarantee in Theorem 2.
Abstract
from arXiv · showhide
Reinforcement Learning (RL) algorithms have found limited success beyond simulated applications, and one main reason is the absence of safety guarantees during the learning process. Real world systems would realistically fail or break before an optimal controller can be learned. To address this issue, we propose a controller architecture that combines (1) a model-free RL-based controller with (2) model-based controllers utilizing control barrier functions (CBFs) and (3) on-line learning of the unknown system dynamics, in order to ensure safety during learning. Our general framework leverages the success of RL algorithms to learn high-performance controllers, while the CBF-based controllers both guarantee safety and guide the learning process by constraining the set of explorable polices. We utilize Gaussian Processes (GPs) to model the system dynamics and its uncertainties. Our novel controller synthesis algorithm, RL-CBF, guarantees safety with high probability during the learning process, regardless of the RL algorithm used, and demonstrates greater policy exploration efficiency. We test our algorithm on (1) control of an inverted pendulum and (2) autonomous car-following with wireless vehicle-to-vehicle communication, and show that our algorithm attains much greater sample efficiency in learning than other state-of-the-art algorithms and maintains safety during the entire learning process.
Introduction
RL can learn effective continuous-control policies, but unsafe exploration limits deployment on physical systems. The paper combines model-free RL with CBF controllers to preserve safety while improving exploration efficiency.
- RL learns policies by repeatedly observing states, acting, receiving rewards, and updating the controller to maximize long-term reward.
- Unsafe learning policies can damage hardware or harm humans, so most successful physical-system RL applications have remained in simulation.
- Existing model-free safe-RL methods approximately enforce safety only after sufficient learning, while model-based methods can neglect exploration efficiency or overconstrain policy exploration.
- The proposed framework integrates model-free RL with CBF controllers and online dynamics learning to ensure safety and guide policy exploration despite uncertain model information.
- The RL-CBF algorithm combines TRPO or DDPG with CBF controllers and dynamical-model learning for inverted-pendulum balancing and autonomous car following.
- The paper claims end-to-end safety for nonlinear control systems and improved learning efficiency by guiding policy exploration with barrier functions.
Preliminaries
The framework models unknown dynamics with Gaussian processes and uses control barrier functions to constrain exploration to safe states. Its CBF optimization can provide high-probability safety guarantees, including under dynamics uncertainty.
- Preliminaries: The setting is an infinite-horizon discounted MDP with deterministic control-affine dynamics comprising nominal drift, nominal actuation, and unknown system dynamics.
- Policy Gradient-Based RL: Model-free RL algorithms maximize expected long-term reward through policy iteration, derivative-free optimization, or policy gradients; DDPG and TRPO are benchmarked examples.
- Policy Gradient-Based RL: Because standard model-free RL lacks safety guarantees, the framework complements RL controllers with model-based CBF controllers to improve search efficiency and ensure safety.
- Gaussian Processes: Gaussian processes estimate unknown dynamics from data using a mean and uncertainty estimate that yield high-probability confidence intervals.
- Gaussian Processes: As data accumulate, the GP mean becomes a better dynamics estimate while uncertainty decreases, but GP training scales poorly with dataset size and is batch-limited here to about 1000 recent points.
- Control Barrier Functions: CBFs define safe sets whose forward invariance can be enforced through an optimization that accounts for actuator limits and uncertain dynamics.
- Control Barrier Functions: The resulting deterministic CBF controller applies minimal intervention to maintain safety or gracefully degrades when safety cannot be enforced.
Reinforcement Learning
The framework combines a model-free RL controller with CBF-based filtering and updates policy learning around safer deployed controllers. RL-CBF preserves high-probability safety while improving exploration toward relevant regions of the state space.
- Barrier-compensating control: The initial architecture combines a model-free RL controller with a CBF controller that compensates for unsafe actions but does not guide exploration.The CBF-filtered controller is updated around the original RL policy.
- Barrier-compensating control: The RL controller proposes reward-seeking actions, while the CBF controller filters them with minimum intervention to keep the system within the safe set.The CBF controller is formulated through a per-timestep quadratic program that can include actuator limits.
- Barrier-guided exploration: RL-CBF updates around previously deployed controllers by incorporating prior CBF controllers, encouraging policy operation in desired state-space regions.This replaces updates around potentially unsafe RL proposals with updates informed by safe deployed behavior.
- Safety and performance guarantees: The guided controller satisfies the barrier condition and renders the safe set C forward invariant with probability (1 −δ) when ϵmax = 0.When ϵmax > 0, the theorem instead guarantees forward invariance of a larger set Cϵ under its stated condition.
- Safety and performance guarantees: RL-CBF provides high-probability safety during learning and can maintain TRPO performance guarantees, although those guarantees apply to a non-deployed control law.With no uncertainty in the dynamics, safety is guaranteed with probability 1; the CBF intervention is reported to decay as learning proceeds.
Computationally Efficient Algorithm
The framework replaces an impractical history-dependent controller implementation with a neural-network approximation, reducing computation while preserving safety guarantees. This approximation may sacrifice the original performance guarantee.
- The ideal controller is computationally expensive because it requires storing previous neural networks and solving multiple sequential quadratic programs.The summation over prior guided controllers creates k−1 stored networks and QP solves.
- The method approximates the accumulated guided-control term with a feedforward MLP fitted to trajectories from the previous policy iteration.The MLP is parameterized by φ and approximates the history-dependent term.
- Safety with probability (1−δ) remains guaranteed because the approximation affects only the guided RL term, while the safety-constrained optimization still solves the required problem.
- The approximation reduces implementation to storing two neural networks and solving one quadratic program for the controller.
- The approximation preserves safety but does not necessarily preserve the performance guarantee established for the unapproximated controller.
Experiments
Experiments evaluate TRPO-CBF and DDPG-CBF on inverted-pendulum and car-following tasks under model uncertainty. The RL-CBF variants maintain safety during learning and generally improve learning efficiency over their model-free baselines.
- Inverted Pendulum: The inverted-pendulum experiment uses TRPO-CBF and DDPG-CBF with a safe angle region of [−1,1] radians and a nominal model containing 40% parameter error.The true pendulum uses m=1 and l=1, while the nominal model assumes m=1.4 and l=1.4.
- Inverted Pendulum: The two RL-CBF algorithms converge near the optimal pendulum reward rapidly and significantly outperform their corresponding baselines without CBFs.
- Inverted Pendulum: The RL-CBF controllers never leave the pendulum safe region during learning, whereas TRPO and DDPG severely violate safety while learning.
- Inverted Pendulum: Early pendulum policies remain near the safe-region boundary under CBF intervention, while final policies quickly converge upright and make the CBF controller inactive.
- Simulated Car Following: In car-following experiments, both RL-CBF variants avoid safety violations, while TRPO and DDPG alone produce collisions in almost all trials.
- Simulated Car Following: TRPO-CBF learns faster and outperforms TRPO; DDPG-CBF also outperforms DDPG, although neither DDPG variant converges to a high-performance controller.
Conclusion
The RL-CBF framework combines model-free reinforcement learning, CBF control, and online model learning to improve exploration while maintaining end-to-end safety. Its deployment depends on a valid forward-invariant safe set, whose computation is nontrivial and computationally intensive.
- Conclusion: The framework integrates model-free RL, model-based CBF control, and online model learning to improve exploration and ensure end-to-end safety.It is designed for physical systems requiring online computation, efficient learning, and safety guarantees.
- Conclusion: The architecture can integrate new RL algorithms and improve its CBF controller using better model information from measurements.These properties support replacing TRPO or DDPG as new algorithms become available.
- Conclusion: The framework assumes a valid safe set h(s) that can be rendered forward invariant.Computing such safe sets is nontrivial and computationally intensive, and future work should address learning the safe set.
- Conclusion: No safety violations occurred with either RL-CBF controller in the simulated car-following experiments, whereas TRPO and DDPG alone experienced collisions in almost all trials.TRPO-CBF learned faster and outperformed TRPO; DDPG-CBF outperformed DDPG, although neither DDPG-based method converged to a high-performance controller.
Appendix A: Proof of Theorem 2
Theorem 2 establishes high-probability forward invariance for the safe set, or for a relaxed set when the safety slack remains within its maximum allowable value. The appendix derives related performance bounds for TRPO-based policy updates and barrier-controller constructions.
- Appendix A: Proof of Theorem 2: If ϵmax = 0 and problem (16) has a solution, the control law renders the safe set C forward invariant with probability (1 − δ).This is the exact safety guarantee stated in Theorem 2.
- Appendix A: Proof of Theorem 2: For TRPO, the proposed controller uses the advantage function and policy-distance constraints to obtain a performance bound.The derivation uses the total variational distance between successive policies and the TRPO optimization bound.
- Appendix A: Proof of Theorem 2: The proof derives the bound by applying the advantage-function property and substituting the proposed policy into the resulting inequality.The total variational distance is simplified using its shift invariance.
Driver Behavior and System Dynamics
The car-following experiment models driver acceleration using the specified dynamics and adds Gaussian acceleration noise. Driver four’s nominal model uses kp = 3.5, kb = 18, and kd = 0 for the other drivers’ behavior.
- Driver Behavior and System Dynamics: The car-following drivers’ dynamics follow equation (20), with acceleration a(i) representing driver i’s acceleration.Gaussian noise is added to each driver’s acceleration.
- Driver Behavior and System Dynamics: Driver four’s nominal model of the other drivers uses kp = 3.5, kb = 18, and kd = 0.These parameters specify the nominal behavior model used in the experiment.
Explanation for High Reward of DDPG in Initial Trials
DDPG-CBF’s reward is initially high in some trials because stochastic driver behavior makes a low-acceleration strategy appear favorable. In most trials, avoiding collision requires acceleration, while the DDPG policy learns to avoid initially doing nothing.
- Explanation for High Reward of DDPG in Initial Trials: DDPG-CBF’s reward starts high in early trials and then drops because stochastic driver behavior makes poor strategies perform well in rare cases.The initial high rewards correspond to trials where the rear driver slows significantly.
- Explanation for High Reward of DDPG in Initial Trials: In most trials, the controlled car must accelerate to avoid collision with the driver behind, reducing reward.Cruising with little acceleration is favorable only when the rear driver slows sufficiently.
- Explanation for High Reward of DDPG in Initial Trials: The initially favorable cruising strategy is generally poor because a non-slowing rear driver forces heavy last-second acceleration and penalties.DDPG learns to avoid this initially inactive strategy.