PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to be Safe: Deep RL with a Safety Critic

Krishnan Srinivasan, Benjamin Eysenbach, Sehoon Ha, Jie Tan, Chelsea Finn

arXiv:2010.14603v1cs.LGcs.RO

TL;DR

Safe reinforcement learning needs alternatives to manually specified constraints, especially when failures are easier to identify than complete safety rules. The paper learns a safety critic in safer settings, transfers it to new tasks, and reports fewer failures with faster, more stable learning across three domains. Its scope remains bounded by simulation results, assumptions behind safety guarantees, and unresolved out-of-distribution safety-critic queries.

  • Problem

    Manual safety constraints are difficult to specify for real-world reinforcement learning, while catastrophic failures can be easier to identify than complete formal constraints.

  • Method

    SQRL pre-trains a safety critic on prior experience and uses it to constrain policy actions and updates while fine-tuning new target tasks.

  • Results

    SQRL achieves substantially fewer learning failures and consistent gains in learning efficiency across navigation, quadruped locomotion, and dexterous manipulation.

  • Takeaways & Limitations

    Learning safety precautions before target-task training can support safer learning and improved task performance within the evaluated control problems.

  • Takeaways & Limitations

    The evaluation is in simulation, and the method leaves open out-of-distribution safety-critic queries and assumptions underlying safety guarantees.

Abstract

from arXiv · show

Safety is an essential component for deploying reinforcement learning (RL) algorithms in real-world scenarios, and is critical during the learning process itself. A natural first approach toward safe RL is to manually specify constraints on the policy's behavior. However, just as learning has enabled progress in large-scale development of AI systems, learning safety specifications may also be necessary to ensure safety in messy open-world environments where manual safety specifications cannot scale. Akin to how humans learn incrementally starting in child-safe environments, we propose to learn how to be safe in one set of tasks and environments, and then use that learned intuition to constrain future behaviors when learning new, modified tasks. We empirically study this form of safety-constrained transfer learning in three challenging domains: simulated navigation, quadruped locomotion, and dexterous in-hand manipulation. In comparison to standard deep RL techniques and prior approaches to safe RL, we find that our method enables the learning of new tasks and in new environments with both substantially fewer safety incidents, such as falling or dropping an object, and faster, more stable learning. This suggests a path forward not only for safer RL systems, but also for more effective RL systems.

1 Introduction

The paper proposes learning task-agnostic safety precautions from prior experience and using them to constrain reinforcement learning on new tasks. SQRL combines a pre-trained safety critic with policy fine-tuning, yielding safer and more efficient learning across three control domains.

  • Manual safety constraints are difficult to specify for messy real-world reinforcement learning environments.The paper motivates learning safety specifications instead of relying exclusively on hand-engineered reward shaping or policy constraints.
  • Prior experience from safer settings can teach agents general precautions for avoiding failures during new behavior learning.The proposed transfer can use experiences from simulation, slow robot motion, human-supervised settings, or offline safety incidents.
  • Safe policy iterates avoid unsafe states throughout fine-tuning, whereas standard reinforcement learning visits unsafe states and produces more failed episodes.
  • SQRL pre-trains a safety critic and then constrains policy updates and selected actions while fine-tuning a new target task.The safety critic evaluates whether state-action pairs lead to unsafe behavior, with separate pre-training and target-task fine-tuning phases.
  • SQRL provides consistent and substantial gains in learning safety and efficiency across navigation, quadrupedal locomotion, and dexterous manipulation.The comparison includes prior state-of-the-art reinforcement learning and safety approaches.

2 Related Work

Prior safe-RL research uses varied cost, risk, constraint, and safety-function formulations, but many approaches focus on final-policy safety rather than safety during learning. This paper positions SQRL as a framework for transferring learned safety across tasks and safety modes.

  • Safe-RL methods define safety through expected costs, risk measures, catastrophic-state regions, or constrained probabilities of violation.
  • Several prior methods do not guarantee or reason about policy safety during the learning process itself.The paper contrasts this limitation with its emphasis on constraining learning-time behavior.
  • Lyapunov-based approaches use feasible baseline policies to map learned policies into spaces satisfying Lyapunov safety conditions.
  • Risk-value and safety-function approaches estimate unsafe states or provide backup control, but do not transfer learned safety to different tasks or environments.
  • The paper aims for a general safety framework spanning different safety modes rather than only predicting mobile-robot collision probabilities.

3 Preliminaries

The paper formulates its approach within entropy-regularized reinforcement learning on a Markov Decision Process. Policies maximize reward while incorporating action entropy through a tunable regularization parameter.

  • The reinforcement-learning problem is modeled as an MDP with state and action spaces, discounting, rewards, transition dynamics, and initial-state distributions.
  • Entropy-regularized reinforcement learning learns a policy πθ over actions conditioned on states while maximizing the stated objective.
  • The policy action entropy H(πθ(·|st)) is weighted by a positive tuning parameter α.

4 Problem Statement

The problem is to learn a target-task policy while limiting catastrophic failures that are easier to identify than complete formal safety constraints. The proposed two-phase formulation transfers safety knowledge from a safer pre-training task and constrains target-task learning by a failure-probability threshold.

  • Robot falls and dropped objects are identifiable failures, but formally specifying constraints that capture them can be difficult, biased, or hinder learning.
  • The framework combines a failure-tolerant pre-training environment with a safety-critical target-task environment.
  • Learning proceeds by training an exploratory policy on a simpler safer task and transferring it to the target task with safety guarantees.
  • The target policy optimizes expected return while minimizing visits to unsafe states after pre-training.
  • The threshold ϵsafe upper-bounds expected policy risk, and single-step action constraints can guarantee safety up to that probability under stated assumptions.The constraint is intended to apply throughout training, not only after convergence.

5 Safety Q-functions for RL

SQRL learns a safety critic during pre-training and uses it to constrain policy behavior while fine-tuning on a new task. The method combines future-failure estimation, safety-constrained exploration, and theoretical safety guarantees under stated assumptions.

  • 5 Safety Q-functions for RL: SQRL jointly learns a policy and safety critic during pre-training, then fine-tunes the policy on a target task using the learned safety precautions.The safety critic estimates future failure probability for state-action pairs and constrains later policy behavior.
  • 5 Safety Q-functions for RL: The safety critic estimates cumulative future failure probability with dynamic programming and a discount factor γsafe.Its Bellman-style objective propagates failure information across future timesteps.
  • 5 Safety Q-functions for RL: Pre-training uses maximum-entropy reinforcement learning to explore diverse state-action pairs while updating the safety critic under safety-constrained policies.Using constrained policies for critic training mitigates excessive pessimism caused by unsafe downstream behavior in replay data.
  • 5 Safety Q-functions for RL: SQRL constrains actions by projecting or masking the policy so sampled actions satisfy the target safety threshold ϵsafe.The constrained policy assigns zero probability to actions whose estimated failure probability reaches the threshold.
  • 5 Safety Q-functions for RL: During fine-tuning, SQRL adds a safety constraint to the target-task objective and collects data with actions constrained by the pre-trained safety critic.The policy is optimized with dual gradient updates involving entropy and safety constraint multipliers.
  • 5 Safety Q-functions for RL: Under Assumptions 1–4, every new policy encountered during target-task training belongs to the safety-feasible policy set Πϵsafe.The analysis also states that projected policies have discounted future failure probability at most ϵsafe.

6 Practical Implementation of SQRL

The implementation adds SQRL as a layer over off-policy training with replay buffers and safety-constrained action sampling. Practical behavior depends on tuning task- and environment-specific safety hyperparameters, while experiments compare performance and cumulative failures across environments.

  • 6 Practical Implementation of SQRL: SQRL is implemented as a layer above existing off-policy algorithms that use replay buffers during high-dimensional control training.The implementation stores trajectories in an offline replay buffer while collecting samples for training.
  • 6 Practical Implementation of SQRL: Figure 2 compares final task performance with cumulative failure rate during fine-tuning, while Figure 3 shows the environments used in the paper.The Figure 2 caption states that SQRL achieves good performance while being significantly safer during learning.
  • 6 Practical Implementation of SQRL: Safety-critic-guided importance sampling emphasizes actions near the ϵsafe threshold, where distinguishing safe from unsafe actions is most informative.The safety parameters ϵsafe and γsafe are task- and environment-specific and require tuning.
  • 6 Practical Implementation of SQRL: During fine-tuning, SQRL samples candidate actions, masks unsafe ones, and importance-samples the remaining actions using their original policy probabilities.If no safe candidate is found, it selects the action with the lowest estimated failure probability.

7 Experiments

The experiments evaluate safety-constrained transfer across navigation, quadruped locomotion, and manipulation, comparing SQRL with standard and prior safe RL methods. They also test whether the safety threshold controls risk-taking and task efficiency.

  • Environments: Experiments span drunk-spider navigation, Minitaur locomotion, and cube rotation, including a Minitaur variant with changed foot friction.The Minitaur target increases speed from 0.3 m/s during pre-training to 0.4 m/s during fine-tuning.
  • Experimental comparisons: The evaluation compares SQRL against SAC and prior safe RL methods, while separately ablating safety-critic training.The ablation fine-tunes a standard SAC policy pretrained on the source task.
  • Safety and performance: Figure 4 fine-tuning curves suggest that SQRL’s safety critic improves learning stability and efficiency beyond reducing safety incidents.The comparison is shown for the Minitaur and Cube target tasks.
  • Safety and performance: SQRL records only a 1% fall rate in drunk spider and about 5% falling during Minitaur learning, while unconstrained SAC falls about 3x as often.On cube rotation, SQRL drops the cube least, although SAC achieves better performance with more safety incidents.
  • Risk-reward trade-off: With ϵsafe = 0.05, the agent takes a longer route around the lava pits; with ϵsafe = 0.2, it navigates directly through them despite action noise εaction = 0.2.The trajectories are consistent across multiple trials, linking the threshold to risk-taking and task efficiency.

8 Discussion

The paper presents SQRL as a way to learn safety precautions and transfer them to new tasks. Across challenging control problems, it reports fewer learning failures than prior methods, while identifying assumptions and future work needed for broader deployment.

  • Contribution: SQRL learns a safety critic that estimates future failure probability and constrains later policies to limit failures.The approach uses dynamic programming and can provide safety guarantees under standard assumptions.
  • Results: Across challenging control problems, SQRL encounters substantially fewer failures during learning than prior methods and achieves superior task performance on one task.The discussion summarizes this as progress relative to multiple prior methods.
  • Limitations and future work: The authors identify pre-training and curriculum optimization, relaxed safety-guarantee assumptions, out-of-distribution safety-critic queries, and real-world manipulation as future work.The reported results are strong in simulation, while application to real-world manipulation remains planned.

A Guaranteeing Safety with the Safety Critic

The safety critic masks actions whose estimated failure probability exceeds a threshold, and under stated assumptions this keeps every policy iterate safe during target-task fine-tuning.

  • Safety-critic masking: SQRL defines a safety-constrained policy by masking actions with Qsafe(s, a) ≥ ϵ and renormalizing the remaining action probabilities.The masked policy is ¯πI,ϵ(a | s) = 1/Z(s) · 1(Qsafe(s, a) < ϵ) · π(a | s).
  • Theoretical guarantee: Under assumptions covering transition probabilities, available safe alternatives, pre-training support, and critic optimality, all policies encountered during training remain ϵ-safe on Ttarget.The guarantee assumes the initial policy is ϵ-safe under Tpre and the safety critic is an optimal learner.
  • Theoretical guarantee: The proof argues that any masked action with nonzero probability has Qsafe(s, a) < ϵ, contradicting a failure probability at least ϵ.Unsafe transitions would otherwise produce a safety-critic value above the masking threshold.
  • Theoretical guarantee: A safe action is required at every safe state, allowing the constrained policy to replace unsafe actions without entering failure states.This follows from the assumption that each safe state has an action with nonzero probability of transitioning to another safe state.
  • Implications: The resulting safety guarantee applies even when the task policy is trained separately from the safety critic, provided safe actions remain in its action-distribution support.The masked policy is at least as safe as the safety critic’s actor under these conditions.
  • Implications: Figure 6 depicts the mechanism: disallowed actions are removed so failure probability stays below the dashed threshold throughout an episode.Black arrows are actions taken by the policy, gray arrows are actions the safety critic would have taken, and red arrows are disallowed.
  • Empirical evidence: During fine-tuning, SQRL produces significantly fewer cumulative failures than prior safe RL approaches.The comparison is shown over the course of fine-tuning in Figure 7.

C Environment Details

The environments vary task conditions during fine-tuning, while implementation choices address safety-critic estimation and policy performance. Minitaur tasks share pre-training but diverge through friction or velocity changes.

  • Minitaur Tasks: MinitaurFriction and MinitaurVelocity share pre-training at a target velocity of 0.3 m/s under normal foot-friction.
  • Minitaur Tasks: Fine-tuning increases foot-friction by 25% in MinitaurFriction and raises goal velocity by 33% to 0.4 m/s in MinitaurVelocity.
  • CubeRotate: CubeRotate fine-tuning offsets each pre-training goal by an additional eighth turn from four 90-degree target orientations.
  • Safety-Critic Training: Bellman targets use future actions sampled from the unconstrained policy because the initially pessimistic safety-critic can falsely reject safe, reward-increasing actions.
  • Safety-Critic Training: A smaller online replay buffer makes safety-critic training more on-policy, improving failure-probability estimates and the safety-constrained policy.

E Training Plots

The training plots examine failures, pre-training performance and safety, and ablations of online safety-critic training. Together, the reported plots indicate more stable pre-training and improved fine-tuning outcomes for the method.

  • Training Curves: Figure 7 tracks failures accumulated during fine-tuning for different methods.
  • Ablations: Online safety-critic training makes Minitaur pre-training failure rates qualitatively more stable and significantly increases fine-tuning task reward by mitigating critic over-pessimism.
  • Pre-training: Pre-training results across Point-Mass, Minitaur and Cube suggest that jointly learning the policy and safety-critic can improve performance and safety.
  • Pre-training: In Point-Mass, learning to walk safely across the bridge without falling yields significantly more reward at the end of the trajectory.
  • Ablations: The SQRL ablation examines the effects of different design choices.
Loading 2010.14603v1…