PaperlayerView exact revision on arXiv

Source-linked AI summary

Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones

Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E. Gonzalez, Julian Ibarz, Chelsea Finn, Ken Goldberg

arXiv:2010.15920v2cs.LGcs.AIcs.RO

TL;DR

Recovery RL targets the conflict between exploratory task learning and safe interaction in uncertain environments. It learns likely constraint-violating regions from offline data and separates task and recovery policies. Across simulation and physical-robot domains, it outperforms prior safe-RL methods and improves the trade-off between task successes and violations.

  • Problem

    Safe RL must support exploration for learning while limiting constraint violations that can damage robots or environments.

  • Method

    Recovery RL uses offline violation data to learn recovery regions and separates task-reward optimization into a task policy from safety behavior into a recovery policy.

  • Results

    Recovery RL outperforms the next best prior method across the evaluated domains, trading off violations and task successes 2 - 20 times more efficiently in simulation and 3 times more efficiently physically.

  • Takeaways & Limitations

    Separating task and recovery policies enables Recovery RL to balance task performance and constraint satisfaction across six simulation domains and a physical-robot task.

Abstract

from arXiv · show

Safety remains a central obstacle preventing widespread use of RL in the real world: learning new tasks in uncertain environments requires extensive exploration, but safety requires limiting exploration. We propose Recovery RL, an algorithm which navigates this tradeoff by (1) leveraging offline data to learn about constraint violating zones before policy learning and (2) separating the goals of improving task performance and constraint satisfaction across two policies: a task policy that only optimizes the task reward and a recovery policy that guides the agent to safety when constraint violation is likely. We evaluate Recovery RL on 6 simulation domains, including two contact-rich manipulation tasks and an image-based navigation task, and an image-based obstacle avoidance task on a physical robot. We compare Recovery RL to 5 prior safe RL methods which jointly optimize for task performance and safety via constrained optimization or reward shaping and find that Recovery RL outperforms the next best prior method across all domains. Results suggest that Recovery RL trades off constraint violations and task successes 2 - 20 times more efficiently in simulation domains and 3 times more efficiently in physical experiments. See https://tinyurl.com/rl-recovery for videos and supplementary material.

I. INTRODUCTION

Recovery RL addresses unsafe exploration by learning constraint-violating regions from offline data and separating task optimization from recovery behavior. Across simulation and physical-robot evaluations, it improves the trade-off between task success and constraint satisfaction.

  • Unconstrained exploration can damage robots or surroundings and interrupt learning when constraint violations occur.
  • Jointly optimizing task performance and safety can create objective conflict and require constraint violations to learn about unsafe regions.
  • Recovery RL uses a task policy for unconstrained task reward and a recovery policy that takes control when future constraint violations are likely.
  • Recovery RL uses offline violation data to learn a recovery set identifying regions where future constraint violations are likely.
  • 2 - 20 times more efficiently, Recovery RL trades off constraint violations and task successes than the next best prior method in simulation.
  • 3 times more efficiently, Recovery RL trades off constraint violations and task successes than the next best prior algorithm on a physical robot.

II. RELATED WORK

Recovery RL belongs to safe-RL approaches that restrict exploration with an auxiliary policy, while learning constraint and dynamics information from prior experience rather than assuming them in advance.

  • Prior safe-RL methods impose constraints on expected return, risk measures, or regions where constraint violations are likely.
  • Recovery RL uses a learned recovery policy to keep the agent within a learned safe region of the MDP.
  • Jointly Optimizing for Task Performance and Safety: Unlike joint task-safety optimization, Recovery RL separates task and recovery objectives across two sub-policies.
  • Restricting Exploration with an Auxiliary Policy: Unlike reachability and shielding methods, Recovery RL learns MDP information from experience instead of requiring approximate dynamics or precise constraints beforehand.
  • Leveraging Demonstrations for Safe RL and Control: Recovery RL uses controlled demonstrations of constraint-violating behavior rather than requiring demonstrations of task successes.
  • The safety critic evaluates task-policy actions, and the recovery policy is queried when predicted risk exceeds εrisk.

III. PROBLEM STATEMENT

The paper formulates safe RL as maximizing expected discounted reward subject to a bound on expected discounted future constraint violations. Recovery RL addresses this problem with task and recovery policies initialized using offline violation data.

  • The constrained MDP augments state, action, dynamics, reward, discount, and initial-state components with binary constraint costs and a risk discount factor.
  • Constraint costs are binary indicators of violating states, and episodes terminate upon violations under the stated formulation.
  • The safety objective is to keep the expected discounted probability of future constraint violation below εrisk.
  • The feasible-policy set consists of policies whose discounted violation probability satisfies the risk threshold; γrisk = 1 captures future violation probability under termination.
  • Recovery RL trains a task policy to maximize reward and a recovery policy to guide the agent toward state-action pairs with risk at most εrisk.
  • Offline transitions need only contain constraint violations, and they constrain task-policy exploration during environment interaction.

IV. RECOVERY RL

Recovery RL learns a safety critic, uses it to define a recovery set and policy, and initializes these components from offline data before applying implementation-specific training details.

  • The method first learns a safety critic estimating the probability of future constraint violations for the agent’s policy.
  • The safety critic defines the recovery policy and the recovery set in which that policy is activated.
  • Offline data initializes the safety critic and recovery policy before subsequent implementation and interaction details.

A. Preliminaries: Training a Safety Critic

Recovery RL trains a safety critic to estimate the current policy’s discounted future probability of constraint violation. The critic is learned from sampled transitions using a target-based mean-squared-error objective.

  • The safety critic Qπrisk estimates the discounted future probability of constraint violation under the current policy π.
  • The binary indicator ct equals 1 for a constraint-violating state and 0 otherwise.
  • The formulation does not assume access to the true constraint cost function C and instead treats episodes as terminating when ct = 1.
  • A sample-based approximation of the safety critic is trained from transitions (st,at,st+1,ct) that approximate the critic equations.
  • The critic parameters are optimized by minimizing an MSE loss against the right-hand side of the safety-critic equation.
  • A target network supplies target values for training the safety critic.

B. Defining a Recovery Set and Policy

Recovery RL combines a task policy with a recovery policy that filters actions likely to cause constraint violations. It defines safety using the critic’s estimated risk and relabels task-policy actions during training when recovery actions are executed.

  • Recovery RL executes a composite policy that selects between πtask and πrec according to the predicted near-future risk of constraint violation.
  • The safe set contains state-action pairs whose discounted probability of constraint violation under π is below εrisk.
  • When πtask proposes an action outside the safe set, Recovery RL executes an action sampled from πrec instead.
  • The recovery policy minimizes the learned safety critic to reduce constraint-violation risk and projects task actions into a region where violations are unlikely.
  • Recovery RL filters unsafe proposed actions, thereby changing the dynamics of the environment in which πtask operates.
  • Task-policy training relabels each transition with the action proposed by πtask, while recovery-policy training uses the action actually executed.
  • The algorithm pretrains πrec and the safety critic, initializes Drec with offline data, and then alternates interaction, transition collection, and policy updates.

C. Offline Pretraining

Recovery RL uses offline transitions containing constraint violations to inform safety before online interaction. The recovery policy and safety critic are pretrained offline, then all components are updated online from composite-policy experience.

  • Offline data Doffline conveys constraint information before environment interaction and can be collected under human-defined policies or supervision.
  • The safety critic is pretrained by minimizing its loss on offline batches sampled from Doffline.
  • The recovery policy is also pretrained using offline data containing constraint violations.
  • After pretraining, πtask, πrec, and the safety critic are updated online using experience generated by the composite policy.
  • Any RL algorithm can represent πtask, any off-policy RL algorithm can learn πrec, and separate task demonstrations may initialize πtask in exploration-challenging environments.

D. Practical Implementation

Recovery RL supports both model-free and model-based recovery policies. Model-free recovery uses safety-critic gradients, while model-based recovery uses MPC over learned dynamics with the critic as a cost.

  • Model-free recovery performs gradient descent on the safety critic evaluated at the recovery policy’s action.
  • Model-based recovery performs model predictive control over a learned dynamics model while using the safety critic as the planning cost.
  • PETS is used for lower-dimensional tasks, whereas visual tasks use a VAE-based latent dynamics model.
  • The task policy uses the maximum-entropy reinforcement-learning algorithm SAC.

V. EXPERIMENTS

Recovery RL is evaluated across simulated navigation, manipulation, and visual tasks, plus a physical image-based obstacle-avoidance task, using task-success and constraint-violation trade-offs. Across these experiments, it generally outperforms comparison methods, with model-based recovery especially effective in object extraction.

  • Domains: The evaluation covers six simulation domains and one physical dVRK obstacle-avoidance task, including navigation, contact-rich manipulation, and image-based settings.Constraint violations terminate episodes, directly limiting further exploration.
  • Comparisons: Recovery RL is compared with unconstrained learning and five methods enforcing safety through constrained optimization or reward shaping.The comparison algorithms are Unconstrained, LR, SQRL, RSPO, RP, and RCPO.
  • Simulation results: In object extraction, model-based Recovery RL significantly outperforms prior algorithms, whereas its model-free recovery mechanism performs substantially worse.The authors hypothesize that model-based recovery better compensates for approximation errors in the safety critic.
  • Trade-off metric: Recovery RL generally achieves a better trade-off between task success and safety, often matching task completions while incurring fewer constraint violations.The evaluation uses the ratio of cumulative task successes to cumulative constraint violations, with higher values preferred.
  • Ablations and sensitivity: Offline pretraining of the recovery policy and safety critic, plus action relabeling for the task policy, materially affects performance.Performance remains strong with 1000 offline transitions on Object Extraction but degrades when the dataset is reduced further; recovery policies are usually activated when violations occur.

VI. CONCLUSION

Recovery RL more effectively balances task performance and constraint satisfaction than five state-of-the-art safe RL algorithms across six simulation domains and one physical robot task.

  • Recovery RL outperforms five state-of-the-art prior safe RL algorithms across six simulation domains and an image-based obstacle avoidance task on a physical robot.
  • Future work includes broader physical-robot evaluation, formal guarantees, improved offline pretraining of the recovery policy, and settings where violations are non-catastrophic.
  • The authors also identify large-scale robot learning as a future application area for Recovery RL.

Zones Supplementary Material

The supplementary material provides theoretical discussion, algorithmic details, additional metrics and comparisons, safety-critic visualizations, and implementation details.

  • Section VIII discusses theoretical motivation and possible Recovery RL variants.
  • Section IX presents algorithmic details for Recovery RL and the comparison algorithms.
  • Sections X–XII report additional metrics and comparisons, visualize navigation safety critics, and provide further implementation details.

VIII. RECOVERY RL THEORETICAL MOTIVATION AND VARIANTS

The supplementary material motivates Recovery RL through a modified-dynamics view and describes safety-value, reachability, recovery-policy, and comparison-method variants and implementations.

  • Theoretical Motivation: Assuming stationary modified dynamics after pretraining, the task policy operates in a stationary MDP with the original reward function.
  • Safety Value Function: A safety-value variant replaces the state-action safety critic with a state-value function, while the paper’s Q-risk version can switch actions earlier.
  • Reachability-based Variant: A model-based variant uses learned dynamics for one- or k-step lookahead to check whether future state-action tuples enter the risk set.
  • Recovery Policy: Recovery policies can be learned with model-free or model-based off-policy reinforcement learning, including DDPG-style optimization and MPC over learned dynamics.
  • Task Policy: The task policy is trained with SAC in experiments and can be pretrained on task-specific demonstrations when exploration is challenging.
  • Comparison Methods: Comparison methods include Lagrangian relaxation, risk-sensitive policy optimization, reward penalties, and off-policy RCPO.

X. ADDITIONAL EXPERIMENTAL METRICS

Recovery RL combines strong task performance with relatively few constraint violations across simulation and physical evaluations. Additional analyses show faster reward convergence when constraints are satisfied, while failure cases indicate the recovery policy can predict but not always prevent violations.

  • Additional Experimental Metrics: Recovery RL is generally successful across simulation domains with relatively few constraint violations, while some more successful comparisons incur many more violations.The reported comparison covers cumulative task successes and constraint violations across simulation experiments and the physical obstacle-avoidance experiment.
  • Additional Experimental Metrics: Recovery RL generally converges to higher-quality solutions more quickly than comparison algorithms when constraints are satisfied.Reward curves are smoothed over 100 episodes and omit episodes with constraint violations, so the comparison is conditioned on safe episodes.
  • Additional Experimental Metrics: In most tasks, the recovery policy activates when violations occur, predicting future violations without always preventing them.This suggests the method can still identify impending failures even when recovery is unsuccessful.
  • Safety Critic Visualizations: Excessively increasing γrisk causes the safety critic to spread across the state-action space, making safe and unsafe states difficult to distinguish.This is a reported tuning limitation of the safety critic.
  • Implementation and Training: Recovery RL uses separate task and recovery policies, with offline pretraining followed by online task-policy learning constrained by the safety critic and recovery policy.The safety critic and recovery policy are also updated during online learning.
  • Safety Critic Visualizations: Increasing γrisk makes safety-critic values rise more gradually near obstacles, preserving more information about possible future constraint violations.The action-conditioned critic can identify some actions as more unsafe than others near walls and obstacles.
Loading 2010.15920v2…