PaperlayerView exact revision on arXiv

Source-linked AI summary

A Review of Safe Reinforcement Learning: Methods, Theory and Applications

Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Alois Knoll

arXiv:2205.10330v5cs.AIcs.LG

TL;DR

Safe RL deployment must balance safety and reward during learning, while providing rigorous guarantees for exploration, stability, and convergence. This paper reviews safe RL methods, theory, applications, and benchmarks through five critical problems, synthesizing progress and identifying challenges for future research.

  • Problem

    Safe RL must guarantee safety while improving reward during learning, yet rigorous guarantees for exploration, monotonic reward improvement, stability, and convergence remain open problems.

  • Method

    The paper reviews safe RL methods across a unified safety framework, analyzes theory and sample complexity, examines applications and benchmarks, and organizes the discussion around five critical problems.

  • Results

    The review synthesizes 20 years of safe RL research, covering model-based and model-free methods, convergence, sample complexity, applications, benchmarks, and unresolved challenges.

  • Takeaways & Limitations

    The review provides a foundation and research direction for safe RL, while its benchmark analysis may help advance RL toward real-world applications.

Abstract

from arXiv · show

Reinforcement Learning (RL) has achieved tremendous success in many complex decision-making tasks. However, safety concerns are raised during deploying RL in real-world applications, leading to a growing demand for safe RL algorithms, such as in autonomous driving and robotics scenarios. While safe control has a long history, the study of safe RL algorithms is still in the early stages. To establish a good foundation for future safe RL research, in this paper, we provide a review of safe RL from the perspectives of methods, theories, and applications. Firstly, we review the progress of safe RL from five dimensions and come up with five crucial problems for safe RL being deployed in real-world applications, coined as "2H3W". Secondly, we analyze the algorithm and theory progress from the perspectives of answering the "2H3W" problems. Particularly, the sample complexity of safe RL algorithms is reviewed and discussed, followed by an introduction to the applications and benchmarks of safe RL algorithms. Finally, we open the discussion of the challenging problems in safe RL, hoping to inspire future research on this thread. To advance the study of safe RL algorithms, we release an open-sourced repository containing the implementations of major safe RL algorithms at the link: https://github.com/chauncygu/Safe-Reinforcement-Learning-Baselines.git.

A Review of Safe Reinforcement Learning: Methods, Theories and Applications

The paper involves authors from institutions in Germany, China, and the United Kingdom, and focuses on safe reinforcement learning and related safety topics.

  • The listed affiliations span the Technical University of Munich in Germany, Peking University and BIGAI in China, and King’s College London and University College London in the UK.
  • The paper’s keywords include safe reinforcement learning, safety optimisation, constrained Markov decision processes, and safety problems.

1. Introduction

The introduction frames safe RL as an important, unresolved practical and AI-safety problem, then organizes the field around five deployment questions called “2H3W.” The review surveys methods and applications while identifying challenges for real-world use.

  • RL is widely used across domains, but safety becomes especially important when deploying agents in real-world applications.
  • Safe RL is framed through five “2H3W” problems: safety policy, safety complexity, safety applications, safety benchmarks, and safety challenges.
  • Safety Policy: Safety policy asks how to optimize policies so agents reach safe states or perform safe actions, using approaches including primal-dual, trust-region, formal, control-theoretic, and Gaussian-process methods.
  • Safety Complexity and Safety Applications: Safety complexity concerns the training data required for convergence and safety bounds, while safety applications remain limited because many studies are simulations that ignore safety.
  • Safety Benchmarks: Safety benchmarks must balance reward performance against safety because loose cost constraints may fail to ensure safety, whereas conservative constraints may degrade rewards.
  • Safety Challenges: Future challenges include incorporating human, environmental, and robot safety, resolving social dilemmas, and establishing standards for safe multi-agent RL.
  • The review analyzes safe RL algorithms, theory, applications, benchmarks, and remaining challenges from a unified perspective organized around the five problems.

2. Background

The background models safe RL primarily as a constrained Markov decision process that maximizes reward while satisfying safety constraints. It also distinguishes constraint types and positions this review relative to earlier surveys.

  • Problem Formulation: Safe RL is commonly formulated as a constrained Markov decision process, maximizing reward while requiring agents to satisfy safety constraints.
  • Problem Formulation: A CMDP extends an MDP with a constraint set specifying cost functions and safety constraint bounds.
  • Problem Formulation: The safe RL objective maximizes reward and minimizes costs, yielding a feasible policy set whose expected costs satisfy the constraint bounds.
  • Related Surveys: Compared with earlier surveys, this review emphasizes the “2H3W” problems alongside algorithm, sample-complexity, and convergence analyses for model-based and model-free safe RL.
  • Constraint Types: The review distinguishes discounted cumulative, mean-valued, probabilistic, instantaneous explicit, and instantaneous implicit safety constraints.
  • Constraint Types: Most CMDP methods optimize cumulative costs, although some focus on immediate costs; whole-trajectory costs are natural in applications such as robot-motion planning and resource allocation.

3. Methods of Safe Reinforcement Learning

The methods overview classifies safe RL by optimization criteria, knowledge use, and policy or value perspectives. It provides a holistic model-based and model-free framework covering single-agent and multi-agent approaches.

  • Safe RL methods differ by objective criteria, with some treating cost as an optimization objective to achieve safety.
  • Other methods incorporate external knowledge into the exploration process to improve safety.
  • The review organizes methods into model-based and model-free approaches, analyzing safe single-agent and multi-agent RL in detail.
  • The methods overview includes policy optimization, formal methods, control theory, and Gaussian-process approaches, with model-based and model-free analyses summarized in two tables.

3.1. Methods of Model-Based Safe Reinforcement Learning

Model-based safe RL uses environment models and control or formal methods to improve safety, but model requirements, tuning, computation, and guarantees remain important constraints.

  • Model-based methods: Model-based safe RL can improve learning efficiency, with applications including robotics, transportation planning, and logistics.The review notes that accurate models are difficult to build and many applications lack them.
  • Policy optimization-based approaches: Chernoff-function policy optimization balances reward and safety through a parameter δ, but recovering policies from risk areas takes ten times value iteration.Setting δ to 1 ignores safety, whereas δ equal to 0 fully considers risk and cost.
  • Policy optimization-based approaches: Actor-critic approaches analyze primal-dual optimization with three timescales for critic, actor, and dual-ascent updates.The critic, actor, and dual ascent operate on fast, middle, and slow timescales, respectively.
  • Policy optimization-based approaches: Several policy-optimization methods lack complete safety guarantees during training or require stringent learning-rate conditions for real-world use.The review specifically reports that some methods cannot guarantee total safety and that others may fail to guarantee safety during training.
  • Control and formal methods: Control-theoretic methods provide more rigorous safety but often require dynamics models, while Lyapunov functions can be difficult to construct and CBF deployment is not straightforward.Formal methods may also depend heavily on model knowledge, incur expensive verification, and face the curse of dimensionality.
  • Formal and Gaussian-process methods: Neurosymbolic formal verification can guarantee safety in a mirror-descent framework, whereas GP-based reachability methods estimate disturbances that may lead to unsafe states.The neurosymbolic method uses symbolic policies in an iterative gradient-approximation and verification loop.

3.2. Methods of Model-Free Safe Reinforcement Learning

Model-free safe RL spans policy optimization, control-theoretic safety layers, Gaussian processes, and formal methods, trading safety guarantees against computation, assumptions, and theoretical completeness.

  • Overview: Most model-free safe RL studies avoid requiring environment dynamics and are organized around policy optimization, control theory, Gaussian processes, and formal methods.The review emphasizes deployment across domains without known model dynamics.
  • Policy optimization: CPO seeks monotonic reward improvement while satisfying safety constraints, but its Fisher-matrix and second-order computations are more expensive than PPO-Lagrangian.CPO can approach the safety bound and achieve comparable performance to primal-dual methods on some tasks.
  • Policy optimization: PCPO maximizes reward with TRPO before projecting the policy into a feasible safety region, but its two second-order steps increase computation relative to FOCOPS.The review reports better performance than CPO on some tasks.
  • Policy optimization: SAILR uses an advantage-function surrogate to minimize cost and achieve safe policies during training and deployment, while SPRL constructs separate safe and reward policies off-policy.SPRL is applied to shortest-path problems in the traveling-salesperson setting.
  • Policy optimization: IPO uses logarithmic barrier functions to satisfy safety constraints and is easy to implement, but its performance lacks theoretical guarantees.Its empirical effectiveness is reported on MuJoCo and grid-world environments.
  • Value-based methods: Value-based methods use optimistic transition and cost estimates to support safety in reward-free settings while analyzing safety violations and regret bounds.The review also describes least-squares value iteration with upper-confidence bounds for large-scale primal-dual optimization.
  • Safety layers: Safety-layer methods filter actions more directly than cost-based policy optimization, but OptLayer, A-CRL, and DCRL lack convergence-rate analysis.OptLayer integrates a neural-network safety layer, while A-CRL uses dual gradient descent to find feasible trajectories.
  • Gaussian processes: GP methods model safe, unsafe, and uncertain states to balance exploration and exploitation, yet rigorous safety guarantees during exploration remain open.Finite-time convergence and optimization for multiple heterogeneous objectives are also identified as gaps.

3.3. Safe Multi-Agent Reinforcement Learning

Safe MARL extends constrained decision-making to interacting agents, with methods addressing joint rewards and individual costs while facing computational, theoretical, and specification challenges.

  • Motivation: Safe MARL remains relatively new because it must account for the ego agent’s safety, other agents’ safety, and their interactions.Most existing RL studies focus on single-agent settings.
  • Problem formulation: The fully cooperative formulation models agents, joint actions, transitions, rewards, individual costs, and cost limits as a multi-agent constrained Markov game.Each agent acts according to its policy, producing a joint action and constrained joint return.
  • Policy optimization methods: MACPO is the first safe model-free MARL algorithm reported to guarantee monotonic reward improvement and theoretical safety constraints at every iteration.It is built from CPO and HA-TRPO and is evaluated on safe multi-agent Ant and Lift tasks.
  • Policy optimization methods: MACPO requires expensive Fisher-information computations, whereas MAPPO-Lagrangian is first-order and easier to implement but does not guarantee hard safety constraints.Adaptive multiplier updates reduce sensitivity to Lagrange-multiplier initialization.
  • Decentralized methods: Dec-PG searches reward-cost saddle points with decentralized descent-ascent, but parameter sharing among neighboring agents can produce suboptimal solutions and high variance.MACPO avoids the parameter-sharing assumption.
  • Robust methods: Robust MARL models reward and transition uncertainty, but its payoff and equilibrium assumptions may be too strong for real-world applications.Reward uncertainty is represented by randomizing a nature policy with Gaussian noise.
  • Formal and safety-layer methods: Shielding methods require prior LTL safety specifications and problem-specific knowledge, while safety-layer methods do not guarantee zero violation across all evaluated tasks.The review reports that shielding can guarantee safety to some extent.

3.4. Summary of Safe Reinforcement Learning Methods

Safe RL methods show promising reward-safety performance, but real-world deployment remains limited, especially because safety constraints and theoretical guarantees are incomplete.

  • Summary: The review surveys safe RL across value-based, policy-based, model-based, model-free, single-agent, and multi-agent settings.It reports impressive reward scores for some algorithms but a long path toward real-world applications.
  • Summary: Only a few algorithms consider safety constraints in MARL, making safe multi-agent RL an area requiring substantially more research.The review characterizes safe MARL as relatively new.

4. Theory of Safe Reinforcement Learning

The review distinguishes safe RL theory from standard RL through constrained-policy structure and analyzes primal-dual, constrained-policy, sampling-complexity, and safety-violation methods. It also identifies theoretical gaps, including costly feasibility checking, approximation errors, and missing convergence analyses for several algorithms.

  • Theoretical foundations: CMDPs may lack uniformly optimal stationary policies, unlike standard RL, and an optimal CMDP policy can be deterministic or randomized stationary.The randomized case places randomness at a unique state and between two actions.
  • Theoretical foundations: Checking feasibility among deterministic CMDP policies is NP-hard, making safe-policy verification computationally expensive.The review explicitly characterizes feasibility checking as time-expensive.
  • Primal-dual approaches: Primal-dual methods use Lagrange multipliers for CMDPs, with analyses covering local or asymptotic convergence, PAC-style guarantees, and zero duality gaps.The reviewed work also extends constrained CMDPs toward robust formulations and state-based constraints.
  • Constrained policy optimization: Constrained policy optimization replaces non-convex terms with first- and second-order approximations to obtain convex subproblems, but approximation errors and expensive updates remain concerns.CPO and related methods use surrogate costs or convex approximations; conjugate-gradient updates can require an inverse Fisher information matrix.
  • Constrained policy optimization: FOCOPS and CUP formulate constrained policy optimization as policy improvement followed by projection, with analyses of worst-case safety bounds and generalized advantage estimation.The review summarizes asymptotic results for safe RL in Table 3.
  • Sampling complexity and safety analysis: UCBVI-γ achieves sampling complexity matching the minimax lower bound up to logarithmic factors, while several model-free methods still lack convergence-rate analyses.The review separately surveys model-based and model-free O(ε)-optimality and lists algorithms without convergence-rate analysis.

5. Applications of Safe Reinforcement Learning

Safe RL applications span autonomous driving and robotics, with methods adapted to domain-specific deployment requirements. The review also identifies unresolved challenges in guaranteeing safety during exploration while preserving reward, stability, and convergence.

  • Applications of Safe Reinforcement Learning: Application-focused safe RL addresses autonomous driving, robotics, and recommendation systems, emphasizing deployment considerations beyond algorithmic foundations.The application section focuses on how methods are deployed in specific domains and their domain-specific requirements.
  • Safe Reinforcement Learning for Autonomous Driving: Autonomous-driving methods combine constrained RL with motion planning, parallel risk-aware policies, safety layers, rule-based policies, and formal verification.These methods target lane keeping, vehicle decision-making, lane changes, and action safety across different driving settings.
  • Safe Reinforcement Learning for Autonomous Driving: Constrained Q-learning for autonomous driving restricts the action space during Q-value updates and searches for a safe policy in SUMO simulations validated with HighD data.The framework targets high-level autonomous-driving decisions in simulation environments.
  • Safe Reinforcement Learning for Robotics: Safe RL robotics research includes simulation and real-world systems, including a differentiable OptLayer implemented on a 6-DoF industrial manipulator.The review notes that OptLayer may be inefficient for complex, high-dimensional robot-manipulation tasks.
  • Safe Reinforcement Learning for Robotics: SAFER uses safe trajectories collected by a scripted policy to improve data efficiency and safety rate, but it lacks theoretical safety guarantees.In PyBullet experiments, SAFER outperformed demonstration-based safe RL baselines on safety and success rate.
  • Summary of Applications: A central open challenge is rigorously guaranteeing safety during exploration while retaining monotonic reward improvement, stability, and convergence in real-world applications.The review presents this as a major unresolved issue across autonomous-driving and robotics applications.

6. Benchmarks of Safe Reinforcement Learning

Safe RL benchmarks provide environments and baseline comparisons for evaluating safety-oriented algorithms. They range from discrete gridworlds to continuous-control settings that support sim-to-real research.

  • Benchmarks of Safe Reinforcement Learning: Several single-agent and multi-agent safe RL benchmarks have been developed, with baseline comparisons intended to support fair evaluation and progress toward real-world applications.The review frames these benchmarks as an answer to the safety-benchmarking problem.
  • AI Safety Gridworlds: AI Safety Gridworlds use 10×10 two-dimensional grids with discrete actions, obstacles, and task-specific safety challenges.The action space consists of right, left, up, and down movements.
  • Safety Gym: Safety Gym provides continuous-action two-dimensional tasks in which robots navigate toward goals while avoiding dangerous areas, obstacles, and moving objects that incur costs.It is built on OpenAI Gym and MuJoCo environments.
  • Safe Control Gym: Safe Control Gym integrates traditional control, learning-based control, and reinforcement learning with both model-based and data-based approaches.Its environments include cart-pole, quadrotor, stabilization, and trajectory-tracking tasks.
  • Sim-to-Real Relevance: Safety Gym, AI Safety Gridworlds, and Safe Control Gym may support sim-to-real research through randomization, dynamics disturbances, and symbolic dynamics and constraints.These features introduce non-idealities resembling real-world robotics.

6.2. Benchmarks of Safe Multi-Agent Reinforcement Learning

Safe multi-agent benchmarks extend cooperative robot-control environments with explicit safety constraints. They differ in task structure, decentralization, scalability, and computational performance.

  • Safe Multi-Agent Benchmarks: Three safe multi-agent benchmarks—Safe MAMuJoCo, Safe MARobosuite, and Safe MAIG—were developed to promote safe multi-agent RL research.They provide distinct environments for studying safety-aware multi-agent robot control.
  • Safe Multi-Agent MuJoCo: Safe MAMuJoCo requires agents to learn robot manipulation while avoiding unsafe obstacles and positions that emit costs.It extends MAMuJoCo while preserving its background environment, agents, physics simulator, and reward function.
  • Safe Multi-Agent Robosuite: Safe MARobosuite is fully cooperative, continuous, decentralized, and safety-constrained, with agents controlling individual or groups of robot joints.Its tasks include Lift, Stack, and TwoArmLift configurations with varying numbers of agents.
  • Safe Multi-Agent Isaac Gym: Almost ten times faster trajectory sampling and logical computation distinguishes Safe MAIG from Safe MAMuJoCo and Safe MARobosuite on the same server, task, and algorithm.The review also reports better communication speed, while noting that CPU–GPU memory may require optimization.

7. Challenges and Outlook of Safe Reinforcement Learning

The review identifies unresolved challenges for deploying safe RL, spanning human compatibility, ethical decisions, industrial standards, large-scale multi-agent safety, and competing objectives. It also outlines research directions including game theory, information theory, biologically inspired methods, human feedback, and human-robot interaction.

  • 7.1. Human-Compatible Safe Reinforcement Learning: Human preference statements can be misstated maliciously or unintentionally, while multiple users may impose conflicting preferences on one safe RL agent.The review highlights robust agents, preference personalization, and multi-user preference balancing as open questions.
  • 7.1. Human-Compatible Safe Reinforcement Learning: Ethical decisions remain open when safe agents must trade off harms to different people or act according to differing value systems.The review frames these cases as social dilemmas involving human-machine interactions and moral norms.
  • 7.2. Industrial Deployment Standards for Safe Reinforcement Learning: Safe RL lacks industrial deployment standards covering technical efficiency, reward-safety trade-offs, morality, and legal responsibility.The review calls for alignment between academic research and industrial requirements.
  • 7.3. Safety Guarantee Efficiently in Large Number of Agents’ Environments: Efficient safety guarantees remain difficult in large-agent environments because the multi-agent decision space becomes extremely large.The review identifies safety in environments with up to 1 million agents as a continuing challenge, with convergence, sample complexity, and stability also unresolved.
  • 7.4. Trade-off Balances: Safe MARL must balance exploration-exploitation and reward-cost trade-offs across competitive, cooperative, and mixed games.The review also notes unresolved handling of multiple heterogeneous constraints across agents.
  • Research Directions: Proposed directions include game-theoretic, information-theoretic, biologically inspired, human-feedback, and human-robot-interaction approaches.These directions target game structure, uncertainty, scalable representations, diverse safe behaviors, and understanding human preferences and intentions.

8. Conclusion

The conclusion presents the paper as a comprehensive review of safe RL methods, theory, applications, benchmarks, and deployment challenges. It organizes the field around five critical problems and a unified safety framework while identifying directions for future research.

  • Conclusion: The review examines safe RL research from methods, theory, applications, benchmarks, and future challenges.It covers both model-based and model-free approaches and discusses autonomous driving and robotics applications.
  • Conclusion: The paper identifies challenging problems confronting safe RL applications to guide future research.The conclusion positions these challenges as directions for continued study.
  • Conclusion: Five critical safe RL problems are analyzed within a unified framework covering model-based and model-free methods.The paper also briefly investigates sample complexity and convergence for the reviewed methods.
  • Conclusion: The review introduces safe RL benchmarks intended to support progress toward real-world applications.It presents benchmark analysis alongside applications and algorithmic research.
Loading 2205.10330v5…