A Review of Safe Reinforcement Learning: Methods, Theory and Applications

• The authors are affiliated with the Technical University of Munich, Germany.Source: A Review of Safe Reinforcement… · p. 2
• Additional affiliations include Peking University and BIGAI, King’s College London, and Tongji University in China.Source: A Review of Safe Reinforcement… · p. 2
• The listed keywords are safe reinforcement learning, safety optimisation, constrained Markov decision processes, and safety problems.Source: A Review of Safe Reinforcement… · p. 3

Motivation

• RL is widely used across domains, but real-world deployment can cause unacceptable harm unless safety is explicitly guaranteed.Examples include preventing injuries in human-robot interaction, discriminatory recommendations, and unsafe behavior by self-driving cars.Source: §1 · p. 3

Safety definitions

• Safe RL encompasses definitions based on avoiding unsafe states, obeying human desires, controlling recognized dangers, and satisfying broader risk or reversibility requirements.The review consequently studies methods that optimize costs, avoid adversarial attacks, reduce risk, improve undesirable situations, and control agents safely [144] [180] Moldovan and Abbeel.Source: §1 · pp. 3–4

2H3W framework

• The review frames safe RL through five “2H3W” problems: safety policy, safety complexity, safety applications, safety benchmarks, and safety challenges.These address policy optimization, required training data, current application progress, fair performance evaluation, and future obstacles.Source: §1 · p. 4

Methods and theory

• Safety policy research develops primal-dual, trust-region, formal, control-theoretic, and Gaussian-process approaches to keep agents within safe states and actions [55] [186] [270] [342] [6] [35] [127] [15] [26] [28] [59] [62] [63] [142] [147] [51] [94].Safety complexity concerns how many sampled trajectories are needed for convergence under constraints, including value-based [81] [221] and policy-based [357] methods, while balancing exploration, exploitation, reward, and cost.Source: §1 · pp. 4–6

Applications

• Safe RL applications must account for ego-agent, environmental, human, and control safety, yet many existing applications remain simulations rather than real-world deployments.The review covers applications including autonomous driving [154], robotics [292], video compression [205], and vehicle scheduling.Source: §1 · p. 6

Benchmarks and challenges

• Benchmarks must balance reward performance against safety through appropriately designed cost and reward functions, while future work must address human, environmental, robot, social, standardization, and safe MARL challenges.Loose constraints may fail to guarantee safety, whereas overly conservative constraints can degrade reward; examples include AI Safety Gridworlds [181], Safety Gym [259], Safe MAMuJoCo, Safe MARobosuite, and the trolley dilemma.Source: §1 · pp. 6–7

2. Background

• Safe RL commonly models decision-making as a CMDP, maximizing reward while satisfying safety constraints; deep safe RL extends this setting to high-dimensional continuous problems using neural networks.CMDP research includes tabular and linear cases, whereas deep safe RL for continuous optimization is comparatively recent.Source: §2 · p. 8

2.1. Problem Formulation of Safe Reinforcement Learning

• A CMDP augments an MDP with cost functions and bounds, seeking a policy that maximizes reward while keeping each expected cost within its safety constraint.The formulation defines states, actions, transitions, rewards, initial-state distributions, discounting, cost value functions, expected costs, and the feasible policy set.Source: §2.1 · pp. 8–9

2.2. A Survey of Safe Reinforcement Learning related surveys

• Prior surveys classify safe RL by control-theoretic guarantees, optimization criteria or external risk knowledge, learning techniques, and cumulative versus instantaneous constraints [49].Cumulative constraints include discounted, mean-valued, and probabilistic forms; instantaneous constraints may be explicit or implicit.Source: §2.2 · pp. 9–10
• This review differs from earlier surveys by targeting the “2H3W” problems and analyzing safe RL algorithms, sample complexity, and convergence for model-based and model-free methods.Earlier surveys generally lacked comprehensive theoretical analysis of sample complexity and did not focus on the review’s identified critical deployment problems.Source: §2.2 · pp. 9, 11

Method Taxonomy

• Safe RL methods are categorized by objective optimization, external knowledge for exploration, policy versus value learning, and single-agent versus multi-agent settings.Objective-based methods incorporate costs as optimization objectives, while knowledge-based methods use external safety knowledge [62]. Policy-based and value-based methods are also separately surveyed [6] [357] [23] [81] [89] [221] [333].Source: §3 · p. 11

Agent Settings

• Single-agent methods balance reward improvement against cost constraints, whereas multi-agent methods must jointly account for ego and other agents’ rewards and safety in unstable systems.The multi-agent setting extends single-agent safety considerations by including other agents’ safety alongside the ego agent’s safety.Source: §3 · p. 11

Model-Based and Model-Free Methods

• The review organizes safe RL primarily into model-based and model-free methods, analyzing safe single-agent and multi-agent settings in detail.The overview covers policy optimization-, formal methods-, control theory-, and Gaussian processes-based approaches across both model-based and model-free settings.Source: §3 · p. 11

Summary Tables

• Tables 1 and 2 summarize the review’s model-based and model-free safe RL analyses, respectively.Table 1 covers model-based safe RL, while Table 2 covers model-free safe RL.Source: §3 · p. 11

3.1. Methods of Model-Based Safe Reinforcement Learning

• Across these approaches, model-based safe RL can be more learning-efficient than model-free RL and applies to domains including robotics, transportation planning, and logistics.Its practical scope remains constrained because accurate models are difficult to build and many applications lack models.Source: §3.1 · p. 12

3.1.1. Policy optimization-based approaches

• Policy optimization methods tune reward–cost tradeoffs through cumulative costs, primal-dual optimization, risk constraints, or actor-critic schemes, but may be slow to recover and unsafe during training.The Chernoff-based method adjusts safety via δ and requires ten times value iteration to recover policies from risk areas; other methods lack total safety or require difficult learning-rate tuning.Source: §3.1.1 · pp. 12–13

3.1.2. Control theory-based approaches

• Control-theoretic methods use Lyapunov functions, MPC, or control barrier functions to provide more rigorous safety, but require dynamics models or hand-designed safety certificates.CBF-based methods have improved sample efficiency and safety in simulation and autonomous-driving experiments, while model and certificate requirements hinder deployment in complex environments.Source: §3.1.2 · pp. 13–14

3.1.3. Formal methods-based approaches

• Formal methods seek safety without unsafe probabilities, yet model dependence, verification cost, and dimensionality limit reward performance and scalability in continuous high-dimensional RL.Neurosymbolic and external-knowledge approaches show promising safety, but fixed worst-case models and limited continuous-system scalability constrain practical use.Source: §3.1.3 · pp. 14–15

3.1.4. Gaussian processes-based approaches

• Gaussian-process approaches estimate uncertainty, disturbances, and unsafe regions to guide reachability, stability, or PILCO-based policy search while balancing reward and safety.These methods improve safety probabilities or compute maximal safe areas, but training safety and stability still require more rigorous evaluation and a unified framework.Source: §3.1.4 · p. 15

3.2. Methods of Model-Free Safe Reinforcement Learning

• Model-free safe RL is organized around policy optimization, control theory, Gaussian processes, and formal methods for deployment without known model dynamics.The review emphasizes these four perspectives as the main methodological categories.Source: §3.2 · p. 17

3.2.1. Policy optimization-based approaches

• Policy optimization methods enforce safety through constrained updates, projections, barriers, primal-dual schemes, or primal methods, but face computational, stability, sampling, and theoretical limitations.CPO [6] approaches safety bounds but is more expensive than PPO-Lagrangian; PCPO can outperform CPO on some tasks but uses costly second-order optimization, while FOCOPS improves sample efficiency yet may encounter unstable saddle points and unsafe training actions. PCRPO addresses CRPO oscillations through soft policy optimization and gradient manipulation, outperforming CRPO and several primal-dual baselines.Source: §3.2.1 · pp. 17–19

3.2.2. Control theory-based approaches

• Control-theoretic safe-RL methods use Lyapunov functions or control barrier functions to guarantee safety during training, reduce reliance on environment models, improve learning efficiency, and sometimes lack convergence-rate analysis.OptLayer, A-CRL [52], and DCRL lack convergence rate analysis.Source: §3.2.2 · pp. 20–21

3.2.3. Gaussian processes-based approaches

• Gaussian-process methods model unknown rewards, costs, or safe states to support uncertainty-aware exploration and near-optimal safe behavior, but rigorous safety guarantees during exploration remain open.Sui et al. report near-optimal safe performance in movie recommendation and medical applications, while Wachi et al. distinguish safe, unsafe, and uncertain states to balance exploration and exploitation.Source: §3.2.3 · p. 21

3.2.4. Formal methods-based approaches

• Formal methods encode logical constraints or verification layers with LTL, satisfiability modulo theories, and reward engineering to improve safety, though real-world deployment remains insufficiently studied.Hasanbeig et al. use LTL during policy synthesis, Murugesan et al. apply satisfiability modulo theories as a verification layer, and Hasanbeig et al. encode LTL properties through reward engineering.Source: §3.2.4 · pp. 21–22

3.3. Safe Multi-Agent Reinforcement Learning

• Safe MARL extends constrained RL to multiple agents by considering each agent’s safety and reward alongside those of the other agents.The section focuses mainly on fully cooperative settings and introduces safe MARL formulation because relatively few methods exist.Source: §3.3 · p. 22, §3.3.1 · p. 22

3.3.1. Problem Formulation of Safe Multi-Agent Reinforcement Learning

• The formulation models a fully cooperative multi-agent constrained Markov game with joint actions, probabilistic transitions, joint rewards, individual-agent costs, and cost constraints.Agents select actions according to individual policies, form a joint action, maximize joint return, and must satisfy safety constraints.Source: §3.3.1 · pp. 22–24

3.3.2. Methods of Safe Multi-Agent Reinforcement Learning

• MACPO guarantees monotonic reward improvement and theoretically satisfies safety constraints per iteration, whereas MAPPO-Lagrangian is cheaper and easier to implement but lacks hard safety guarantees.MACPO uses Fisher Information Matrix-based conjugate gradients; MAPPO-Lagrangian adaptively updates multipliers to reduce sensitivity to their initialization, while robust theoretical guarantees remain an open need.Source: §3.3.2 · p. 25
• Other methods address decentralized optimization, uncertainty, cost constraints, and formal safety, but parameter sharing, strong equilibrium assumptions, absent safety theory, or required prior specifications limit their applicability.Safe Dec-PG imposes neighboring-agent parameter sharing; Robust MARL models reward and transition uncertainty; CMIX handles peak and average constraints without theoretical safety guarantees; shielding uses LTL with central or factored shielding.Source: §3.3.2 · pp. 25–26
• Safety-layer methods improve safety on some MARL tasks but do not guarantee zero violation across all tasks, while shielding requires prior LTL safety specifications and problem-specific knowledge.The safety layer is built on safe DDPG [74] and MADDPG; shielding is evaluated with MADDPG and Coordinating Q-learning [75].Source: §3.3.2 · p. 26

3.4. Summary of Safe Reinforcement Learning Methods

• Safe RL methods are categorized by value- versus policy-based, single- versus multi-agent, and model-based versus model-free approaches.Source: §3.4 · p. 26
• Despite impressive reward scores from some algorithms, safe RL still faces a substantial gap before real-world application.Source: §3.4 · p. 26
• [176] is identified as one of the earliest safe MARL methods, presenting a convergence guarantee for multi-agent systems in 2000.Source: §3.4 · p. 26
• Only a few algorithms address safety constraints, making safe MARL a relatively new area requiring substantially more attention.Source: §3.4 · p. 27

4. Theory of Safe Reinforcement Learning

• Overall, safe RL theory connects policy structure, constrained optimization, sampling efficiency, and safety-violation control across model-based, model-free, and specialized learning frameworks.The section organizes these results around the theoretical differences and analytical techniques needed for safe RL.Source: §4 · p. 27

4.1. Difference between RL and Safe RL

• Safe RL differs fundamentally from standard RL because CMDPs may lack uniformly optimal stationary policies, and checking deterministic-policy feasibility is NP-hard.CMDP optima may require randomized stationary policies whose randomness occurs at one state.Source: §4.1 · p. 27

4.2. Theory for Primal-Dual Approaches

• Primal-dual methods support safe RL through Lagrangian optimization, with results covering asymptotic and non-asymptotic convergence, zero duality gaps, offline PAC guarantees, robustness, and safety-preserving policy improvement.These analyses use stochastic approximation, duality theory, robust CMDPs, and backward value functions.Source: §4.2 · p. 28

4.3. Theory for Constrained Policy Optimization

• Constrained policy optimization replaces difficult non-convex terms with convex approximations, but approximation error and expensive second-order computations limit the resulting theory and practicality.CPO [6] provides near-constraint-satisfaction guarantees, while FOCOPS and CUP [344] analyze policy-improvement and projection procedures with safety bounds.Source: §4.3 · pp. 29–30

4.4. Sampling Complexity and Safety Violation Analysis

• Sampling-complexity analyses characterize O(𝜖)-optimal safe RL under feasibility and structural assumptions, while model-based and model-free results quantify convergence, constraint violations, and required samples.The reviewed bounds include lower bounds, model-specific guarantees, and safety-violation analyses, although several displayed expressions are incomplete in the supplied passage.Source: §4.4 · p. 32

4.4.1. Model-Based Safe Reinforcement Learning

• Model-based safe RL uses linear programming and Lagrangian methods, with guarantees spanning PAC analysis, zero constraint violation, feature-based bounds, and rates matching minimax lower bounds up to logarithmic factors.OptPess-PrimalDual uses a known strictly safe policy, while UCBVI-𝛾 attains a bound matching the minimax lower bound up to logarithmic factors.Source: §4.4.1 · pp. 32–33

4.4.2. Model-Free Safe Reinforcement Learning

• Many model-free methods lack convergence-rate analyses, whereas NPG-PD establishes sublinear convergence for reward optimality and safety constraints under non-concave and non-convex objectives.Its online extension requires additional O(𝜖−4) trajectories, and other methods incur higher complexity or require generative models.Source: §4.4.2 · pp. 34–35

4.5. Other Theory Techniques

• Other theory techniques address distinct safety settings: SAILR analyzes development and training safety, SNO-MDP unknown constraints, A-CRL [52] augmented-state dynamics, and DCRL imperfect learning.These methods provide guarantees for intervention-free learning, policy improvement, bounded errors, or near-optimality under their respective assumptions.Source: §4.5 · p. 35

5. Applications of Safe Reinforcement Learning

• Safe RL applications adapt learning-safety techniques to domain-specific environments, deployment constraints, and safety-critical requirements across driving, robotics, and other applications.The application perspective complements the methods section by focusing on practical implementation and real-world environmental factors.Source: §5 · p. 36

5.1. Safe Reinforcement Learning for Autonomous Driving

• Autonomous-driving methods combine constrained RL, risk layers, safety verification, prediction, logical rules, and action-space constraints for planning, lane changes, and vehicle control.Approaches range from hybrid motion planning in uncertain corridor scenarios to formal verification for arbitrary-lane highway environments, while explainable safe RL remains immature.Source: §5.1 · pp. 37–38

5.2. Safe Reinforcement Learning for Robotics

• Robotics applications use offline demonstrations, differentiable safety layers, continuous risk functions, Lyapunov constraints, model-based planning, and constrained manifolds, but each has deployment limitations.SAFER improves simulation safety and success without theoretical guarantees; OptLayer reaches a real 6-DoF manipulator but may scale poorly, while model-based and manifold methods require suitable horizons, models, or tracking controllers.Source: §5.2 · pp. 39–41

5.3. Safe Reinforcement Learning for Other Applications

• Beyond driving and robotics, safe RL addresses recommender health risk, video-compression bitrate, wireless-security exploration, satellite docking, edge computing, chemical processes, and scheduling.Examples optimize healthy recommendations with CVaR constraints, video quality under bitrate limits, and UAV communication security against jamming, although video methods may not scale to large datasets.Source: §5.3 · pp. 41–42

5.4. Summary of Applications

• Despite successful application methods, safe RL still lacks rigorous guarantees for safe exploration with monotonic reward improvement and stable, convergent real-world deployment.The section summarizes methods including model-based planning, Lyapunov safety, baseline-policy exploration, formal verification, and constrained Q learning [154].Source: §5.4 · p. 42

6. Benchmarks of Safe Reinforcement Learning

• Safe RL benchmarks compare diverse baselines and have substantially contributed to advancing the field toward real-world applications [259].The review frames these benchmarks as addressing the safety-benchmarking problem across single-agent and multi-agent settings.Source: §6 · p. 43

6.1.1. AI Safety Gridworlds

• AI Safety Gridworlds [181] evaluate safe RL algorithms in 10×10 grids with discrete right, left, up, and down actions.Agents navigate obstacle-filled cells in environments designed for safety evaluation.Source: §6.1.1 · p. 44
• AI Safety Gridworld tasks test interruption avoidance, water-cell navigation, and behavior under probabilistically appearing supervisors [181].Off-switch, navigation, and absent-supervisor tasks impose different interruption, cost, and path-length conditions.Source: Figure 5 · p. 44

6.1.2. Safety Gym

• Safety Gym [259] uses OpenAI Gym [47] and MuJoCo to provide continuous-action 2-D robot navigation tasks.Point, Car, and Doggo robots navigate toward goals while avoiding dangerous areas, obstacles, objects, and collisions that incur costs.Source: §6.1.2 · p. 45

6.1.3. Safe Control Gym

• Safe Control Gym [353] integrates traditional control [50], learning-based control, and reinforcement learning methods [281] in a shared benchmark.It supports model-based and data-based control for cart-pole, quadrotor, stabilization, and trajectory-tracking tasks.Source: §6.1.3 · p. 45
• Safe Control Gym [353] may be more suitable for sim-to-real research than Safety Gym [259] and AI Safety Gridworlds [181].It provides randomization, dynamics disturbances, and symbolic representations of system dynamics and constraints that resemble real-world robotics.Source: §6.1.3 · pp. 45–46, Figure 7 · p. 46

6.2. Benchmarks of Safe Multi-Agent Reinforcement Learning

• The three benchmarks—Safe MAMuJoCo, Safe MARobosuite, and Safe MAIG—were developed by to promote safe multi-agent reinforcement learning research.They provide benchmark environments spanning robotic manipulation, obstacle avoidance, and high-performance GPU-based simulation.Source: §6.2 · p. 46

6.2.1. Safe Multi-Agent MuJoCo

• Safe MAMuJoCo extends MAMuJoCo [244] by adding obstacles and cost emissions, requiring agents to manipulate robots while avoiding unsafe areas.Its environments preserve the background, agents, physics simulator, and reward function of MAMuJoCo, while introducing walls, bombs, and other hazards.Source: §6.2.1 · pp. 46–47, Figure 8 · p. 47

6.2.2. Safe Multi-Agent Robosuite

• Safe MARobosuite is a cooperative, continuous, decentralized benchmark that assigns robot joints across agents and evaluates safety-constrained manipulation under partial observations.It supports varied multi-agent decompositions for Lift, Stack, and TwoArmLift tasks, while enabling modular-robot robustness and scalability.Source: §6.2.2 · pp. 47, 49

6.2.3. Safe Multi-Agent Isaac Gym

• Safe MAIG achieves nearly 10× the computation speed of Safe MAMuJoCo and Safe MARobosuite and better communication speed, but requires CPU–GPU memory optimization.It uses GPUs for trajectory sampling and logical computation on Isaac Gym.Source: §6.2.3 · p. 49

7.1. Human-Compatible Safe Reinforcement Learning

• Human-compatible safe RL must address misstated or conflicting human preferences and unresolved ethical decisions shaped by differing value systems.Malicious or unintentional preference errors can produce unsafe behavior, while multi-agent deployment raises social-dilemma and trolley-problem questions [97].Source: §7.1 · pp. 52–53

7.2. Industrial Deployment Standards for Safe Reinforcement Learning

• Safe RL lacks industrial deployment standards covering technical performance, application-specific safety, legal responsibility, and morality.Standards must address efficiency, cost and reward design, environmental scope, robot-caused harm, and responsibility allocation between programmers and users.Source: §7.2 · p. 53

7.3. Safety Guarantee Efficiently in Large Number of Agents’ Environments

• Large-scale safe MARL remains difficult because safety, convergence, stability, credit assignment, scalability, heterogeneous constraints, robustness, and reward–cost trade-offs grow with agent count.Open problems include complex task execution, simulation-to-reality mismatch under sensor noise or attacks, and balancing exploration–exploitation with local and global objectives.Source: §7.3 · pp. 53–55

7.4. Possible Directions for Future Safe Reinforcement Learning Research •

• Future safe RL research may combine game theory, information theory, biological inspiration, human feedback, and human–robot interaction.These directions target game-specific safety, uncertain rewards and costs, biologically inspired coordination, diverse safe behaviors, and robots’ learning of preferences and intentions from multiple users.Source: §7.4 · pp. 55–56

• The review poses five critical safe RL problems and analyzes model-based and model-free methods within a unified safety framework.Source: §8 · pp. 56–57
• It briefly investigates the sample complexity and convergence of safe RL methods while reviewing progress over the past 20 years.Source: §8 · pp. 56–57
• The review examines applications such as autonomous driving and robotics, reveals community benchmarks, and identifies challenges for future safe RL research.These contributions are intended to help RL progress toward real-world applications and provide clues for further research.Source: §8 · pp. 56–57