PaperlayerView exact revision on arXiv

Source-linked AI summary

Conservative Safety Critics for Exploration

Homanga Bharadhwaj, Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, Animesh Garg

arXiv:2010.14497v2cs.LGcs.AIcs.ROstat.ML

TL;DR

Safe exploration in RL must limit catastrophic failures while retaining trial-and-error learning. CSC addresses this by learning a conservative safety critic and constraining exploration and policy updates, achieving competitive task performance with substantially fewer training failures. The paper also provides high-probability failure bounds and convergence results, while acknowledging residual failures and difficulty on tasks requiring additional assumptions.

  • Problem

    Safe RL needs to support trial-and-error learning while limiting catastrophic failures from partially trained policies.

  • Method

    CSC learns a conservative safety critic that overestimates failure probability and uses it to constrain policy evaluation and policy improvement.

  • Results

    Across five simulated robotic domains, CSC achieves competitive task performance while reducing catastrophic failures during training by up to 50% over prior safe-exploration methods.

  • Takeaways & Limitations

    The method provides high-probability failure bounds throughout training while preserving task performance comparable to or better than prior methods.

  • Takeaways & Limitations

    CSC still exhibits non-zero failures because early safety-critic approximation errors are not precisely captured theoretically, and difficult tasks may require additional assumptions.

Abstract

from arXiv · show

Safe exploration presents a major challenge in reinforcement learning (RL): when active data collection requires deploying partially trained policies, we must ensure that these policies avoid catastrophically unsafe regions, while still enabling trial and error learning. In this paper, we target the problem of safe exploration in RL by learning a conservative safety estimate of environment states through a critic, and provably upper bound the likelihood of catastrophic failures at every training iteration. We theoretically characterize the tradeoff between safety and policy improvement, show that the safety constraints are likely to be satisfied with high probability during training, derive provable convergence guarantees for our approach, which is no worse asymptotically than standard RL, and demonstrate the efficacy of the proposed approach on a suite of challenging navigation, manipulation, and locomotion tasks. Empirically, we show that the proposed approach can achieve competitive task performance while incurring significantly lower catastrophic failure rates during training than prior methods. Videos are at this url https://sites.google.com/view/conservative-safety-critics/home

1 INTRODUCTION

The paper addresses safe exploration in RL by seeking conservative behavior that limits catastrophic failures without eliminating trial-and-error learning. CSC learns a conservative safety estimate, constrains policy updates, and achieves competitive performance with fewer failures during training.

  • Safe exploration must balance trial-and-error learning against the potentially high cost of failures in real-world control.The paper motivates conservative behavior that limits catastrophic failures rather than requiring failure-free learning.
  • Existing safe-exploration methods often require additional assumptions, including safety queries, default safe controllers, or knowledge of unsafe states.
  • CSC learns a conservative safety critic that overestimates failure probability and uses it to constrain exploration and policy updates.The method imposes a KL-divergence constraint on successive policy updates and optimizes a chance constraint with primal-dual gradient descent.
  • 50% lower catastrophic failure rates are achieved by CSC over prior safe-exploration methods while learning effective policies across five simulated robotic domains.The domains span manipulation, navigation, and locomotion.

2 PRELIMINARIES

The paper formulates safety as a constrained Markov decision process, where policies maximize discounted task reward subject to an upper bound on expected failure probability. It builds on conservative Q-learning, which controls value overestimation through action-distribution objectives.

  • A constrained MDP represents states, actions, transitions, rewards, discounting, initial states, and safety constraints with allowable failure limits.The safety constraint is binary: 0 denotes an alive state and 1 denotes failure.
  • Safe exploration requires satisfying CMDP constraints while collecting data from the environment.
  • The policy objective maximizes expected discounted task reward over policies whose expected failure probability does not exceed the safety limit.
  • The paper defines discounted state, value, action-value, and advantage quantities for both task rewards and safety constraints.The constraint value at the starting distribution represents average episodic failures and can be interpreted as expected failure probability.
  • Conservative Q-learning lowers overestimation by minimizing Q-values under one action distribution while maximizing them under the on-policy distribution.CQL is used as the framework underlying the paper’s safety-critic construction.

3 THE CONSERVATIVE SAFE-EXPLORATION FRAMEWORK

CSC constrains safe exploration and policy improvement using a conservative safety critic that overestimates failure probability. It filters actions during rollouts, trains the critic from collected transitions, and optimizes reward subject to a failure-probability constraint.

  • Without structural assumptions such as a known dynamics model, CSC cannot guarantee zero failures and instead reduces failure probability per episode.
  • Policy learning: CSC maximizes task rewards while respecting an expected failure-probability constraint through primal-dual optimization.The policy uses the learned overestimated safety advantage, while the dual variable enforces the constrained objective.
  • Learning the safety critic: The safety critic QC overestimates failure probability so the agent can reject actions whose predicted risk exceeds a threshold.
  • Executing rollouts: During rollouts, CSC samples actions from the current policy and uses QC(s, a) ≤ ϵ to select actions for execution.Actions failing the threshold are resampled, with the lowest-risk available action selected in the practical implementation.
  • Overall algorithm: The algorithm stores state, action, next-state, reward, and binary constraint outcomes in a replay buffer for safety-critic and policy updates.The critic, policy, and Lagrange multiplier are updated using the collected environment data.
  • Policy learning: A KL-divergence constraint keeps successive policies close, helping bound the new policy’s expected failures using the old policy’s failures.

4 THEORETICAL ANALYSIS

The analysis shows that CSC can maintain high-probability safety bounds throughout policy updates while preserving convergence guarantees, with stricter safety thresholds creating a performance tradeoff.

  • Safety guarantees: Each intermediate policy satisfies the CMDP safety constraint with the allowed maximum violation χ.This guarantee applies throughout the sequence of policy updates, not only after training converges.
  • Safety guarantees: The safety-critic constraint is satisfied with high probability, with estimation error decreasing as the sample count N increases.The bound includes sampling and critic overestimation terms, plus a concentration term involving log(1/ω)/N.
  • Safety guarantees: Theorem 1 bounds each updated policy’s expected failure probability with high probability during training.The bound depends on the specified safety threshold χ.
  • Policy improvement: Theorem 2 shows that safety-constrained policy updates retain a convergence rate whose additional safety term is upper bounded and can be made small.The safety threshold χ controls the tradeoff: lower χ imposes stricter safety and increases the bound’s additional term.

5 EXPERIMENTS

Experiments evaluate CSC across navigation, manipulation, and locomotion tasks using binary catastrophic-failure signals. CSC achieves comparable or better task performance with substantially fewer failures during training, while stricter thresholds improve safety at a convergence cost.

  • Experimental setup: The experiments cover five simulated robotic environments spanning navigation, manipulation, and locomotion, each with a task objective and catastrophic-failure criterion.Tasks include trap avoidance, block pushing without toppling or boundary violations, and quadruped walking without falling.
  • Experimental setup: CSC is compared with CPO, Base, BaseShaped, and Q ensembles, alongside ablations using different safety thresholds.The baselines include constrained, unconstrained, reward-shaped, and ensemble-based safety-critic methods.
  • Empirical results: CSC achieves comparable or better task-reward performance while producing significantly fewer average and cumulative failures during training than prior methods.CPO and Q ensembles eventually approach zero average failures, whereas CSC reaches that regime much earlier.
  • Empirical results: Stricter safety thresholds reduce average failures across environments but slow task-reward convergence.This empirical tradeoff complements the theoretical safety and convergence results.
  • Empirical results: Average episodic failures fall below the specified threshold after about 500 training episodes.The result is reported for the threshold ablations, including the practical effects of critic approximation error early in training.

6 RELATED WORK

Prior safe RL and control methods often rely on structural knowledge, continuous safety feedback, recovery mechanisms, or dynamics assumptions. CSC is presented as not requiring system dynamics or state-space regularity conditions needed by HJI reachability methods.

  • Prior safe-exploration methods assume safe-state queries, default safe controllers, known unsafe states, or other domain knowledge.
  • CPO and related methods require a continuous safety cost at every timestep, limiting use when only binary feedback is available.
  • Recovery-based methods such as LNT use reset policies and ensembles of critics, whereas the paper emphasizes a simpler learned safety critic.
  • HJI reachability methods require system dynamics or state-space regularity conditions for computing unsafe states, while CSC does not require them.

7 DISCUSSION, LIMITATIONS, AND CONCLUSION

The paper concludes that CSC bounds failure probability during training while preserving high task performance, but its guarantees do not eliminate empirical failures or cover difficult tasks without additional assumptions.

  • CSC estimates failure probability with a conservative safety critic and uses it to constrain policy evaluation and improvement.
  • Theoretical guarantees bound failure probability throughout training, while empirical results show high task performance with low accident rates.
  • Non-zero failures remain when χ=0 because early safety-critic approximation errors are not precisely captured and bounded failure probability can still permit failures.
  • For difficult tasks, such as those involving stability concerns, CSC can fail without additional assumptions.

A.1 PROOFS OF ALL THEOREMS AND LEMMAS

The proofs establish high-probability safety bounds for policy updates and show that safety-aware optimization preserves convergence guarantees with additional bounded terms. With an appropriately decreasing threshold, cumulative violations grow sub-linearly.

  • The proof uses conservative overestimation of the safety advantage and a KL-divergence constraint on policy updates.
  • With probability at least 1−ω, Algorithm 1 satisfies the safety constraint during policy updates.
  • Theorem 1 provides a high-probability upper bound on the expected failure probability of the updated policy.
  • Because ε_C depends on the updated policy, it cannot be calculated exactly beforehand, so the analysis uses the trivial bound ε_C≤2.
  • The safety-aware convergence rate differs from unconstrained policy optimization by an additional bounded term.
  • If χ_t=O(1/t), cumulative safety violations scale sub-linearly with the number of collected transition samples.

A.2 DERIVATION OF THE POLICY UPDATE EQUATIONS

CSC learns task and safety value functions from reward and binary failure signals, then uses a conservative safety critic within constrained primal-dual policy optimization. The resulting updates combine task advantage with safety penalties and enforce KL limits through backtracking.

  • The policy objective maximizes cumulative task returns while constraining expected failure probability to a tolerance χ.
  • The algorithm learns a task value function and a safety action-value function from rewards and binary dead-or-alive failure feedback.
  • The safety critic conservatively overestimates failure probability to guide exploration toward caution.
  • CQL-style updates provide an overestimated safety advantage that replaces the true safety advantage in the policy constraint.
  • Primal-dual gradient descent updates the policy using a modified advantage combining task reward and safety cost terms.
  • Backtracking line search rejects or shrinks updates until the average KL divergence satisfies the prescribed constraint.

A.3 RELATION TO CPO

CSC differs from CPO by avoiding the first-order Taylor simplification of the safety constraint and optimizing an upper bound through primal-dual updates. This preserves support for sparse binary failure indicators without requiring a continuous safety cost.

  • Unlike CPO, CSC does not linearize the safety constraint with a first-order Taylor expansion.
  • CSC optimizes an upper bound on the safety value through a CQL-inspired objective and primal-dual optimization.CPO instead enforces a first-order Taylor approximation of the constraint in the dual space.
  • CSC handles sparse binary failure indicators without assuming a continuous safety cost function.

A.4 PRACTICAL CONSIDERATIONS

The practical results examine safety–performance tradeoffs, theoretical sensitivity to the policy-update constraint, implementation choices, and empirical behavior across simulated control tasks. Stricter safety settings reduce failures but slow reward convergence, while replay-buffer seeding and continuous safety signals provide additional evaluations of CSC.

  • Theoretical and practical considerations: With γ = 0.99 and χ = 0.05, any δ > 1e-8 makes the third term in Theorem 3 negative.The analysis also notes that increasing α can overestimate QC and reduce the practicality of policy execution.
  • Theoretical and practical considerations: Overestimating QC too much can prevent the policy-execution condition from being satisfied, causing execution to stop and wall-clock time to become infinite.
  • Theoretical and practical considerations: The implementation caps the relevant loop at 100 iterations, so anytime safety guarantees can be violated early when QC approximation is inaccurate.The paper reports that the guarantee holds during most of training empirically.
  • Safety–performance tradeoff: Stricter safety thresholds χ reduce average failures but slow task-reward convergence.The plots compare average reward, average episodic failures, and cumulative episodic failures across five environments.
  • Baseline comparisons: The safety threshold is χ = 0.05 for all baselines and environments in the baseline comparison.For Laikago, baselines start from a controller that keeps the robot standing before learning to walk.
  • Baseline comparisons: TRPO converges slightly faster than SAC and has slightly lower average and cumulative failures in Point-agent navigation.The paper therefore uses TRPO as the Base RL baseline in Figures 3 and 4.
  • Replay-buffer seeding: 1000 replay-buffer tuples particularly help CSC reduce average and cumulative failures during Car navigation training.All baselines improve after seeding, but CSC is reported to leverage the offline data especially effectively.
  • Continuous safety signal: CSC is also evaluated with a continuous safety signal in Car navigation, where trap penalties accumulate until catastrophic failure.
Loading 2010.14497v2…