Source-linked AI summary
Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning
Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, Sergey Levine
TL;DR
Real-world deep RL is hindered by unrecoverable states and costly manual resets between attempts. This paper jointly learns forward and reset policies, using reset values for uncertainty-aware safety aborts. Experiments show fewer manual resets, fewer unsafe actions, and an automatically induced curriculum.
Problem
Real-world RL often requires manual resets after failures, while some tasks are not easily or automatically reversible.
Method
The method jointly learns a forward task policy and reset policy, using reset-policy values to abort actions likely to enter non-reversible states.
Results
The approach reduces hard manual resets across simulated robotic skills and can achieve substantially higher final reward in the cliff cheetah task.
Takeaways & Limitations
The framework automates resets, avoids unsafe states, and automatically induces a curriculum for continual reinforcement learning.
Takeaways & Limitations
Hard resets remain necessary because early training and imperfect reset policies can miss dangerous or irreversible states.
Abstract
from arXiv · showhide
Deep reinforcement learning algorithms can learn complex behavioral skills, but real-world application of these methods requires a large amount of experience to be collected by the agent. In practical settings, such as robotics, this involves repeatedly attempting a task, resetting the environment between each attempt. However, not all tasks are easily or automatically reversible. In practice, this learning process requires extensive human intervention. In this work, we propose an autonomous method for safe and efficient reinforcement learning that simultaneously learns a forward and reset policy, with the reset policy resetting the environment for a subsequent attempt. By learning a value function for the reset policy, we can automatically determine when the forward policy is about to enter a non-reversible state, providing for uncertainty-aware safety aborts. Our experiments illustrate that proper use of the reset policy can greatly reduce the number of manual resets required to learn a task, can reduce the number of unsafe actions that lead to non-reversible states, and can automatically induce a curriculum.
1 Introduction
Real-world deep RL is limited by unrecoverable failures and the manual resets needed between attempts. The paper learns task and reset policies together, using reset values to avoid irreversible actions and reduce hard resets.
- Real-world deep RL often requires human intervention because failures can leave systems unrecoverable or require manual environment resets.Examples include collisions, broken objects, and robots waiting for reset after an environment failure.
- Manual resets bottleneck real-world learning by stopping data collection rather than merely slowing experience acquisition.
- The “leave no trace” approach jointly learns how to perform a task and undo it through alternating forward and reset policies.
- A value-based reset policy restricts the forward policy to states from which it can return, intervening before potentially irreversible actions.
- The reset policy’s recoverable-state set grows over time, enabling exploration of more of the environment when it is safe.
2 Related Work
Prior work studies safe exploration, multiple policies, and curriculum generation, but this paper targets real-world learning with uncertainty estimation and preemptive early aborts.
- Earlier safe-exploration methods are limited to small, discrete MDPs where exact planning is tractable.
- Related multi-policy methods learn reset behavior, whereas this work predicts forward-policy failure and engages resetting before failure.
- Reset-policy learning naturally generates a curriculum by engaging the reset controller in increasingly distant states.
- The proposed approach is aimed at streamlining real-world learning through uncertainty estimation and early aborts.
3 Preliminaries
Episodic RL repeatedly samples initial states, but robotics often obtains those starts through hard-coded resets or human intervention. The paper replaces this requirement with a learned reset policy.
- Episodic RL maximizes expected discounted returns from states sampled from an initial distribution p0.
- Training typically samples new episodes and requires a hard-coded reset policy or human intervention to restore the starting state.
- The proposed setup learns an additional reset policy to avoid manual resets between episodes.
4 Continual Learning with Joint Forward-Reset Policies
The method jointly trains forward and reset policies, using reset values and uncertainty-aware early aborts to constrain exploration to recoverable states. Hard resets remain a fallback when learned resetting fails.
- Joint forward-reset learning: The forward policy maximizes task reward, while the reset policy receives rewards for returning to states with high initial-state density.
- Joint forward-reset learning: The method assumes at least one reachable maximum-reward state has a policy capable of resetting the environment without manual intervention.
- Joint forward-reset learning: Off-policy actor-critic learning shares experience between policies, and Q-functions provide signals for early aborts.
- Early aborts: If a proposed forward action has a reset-policy Q value below Qmin, execution is aborted and control switches to the reset policy.
- Early aborts: When reset reward is binary, the reset Q function represents the probability that resetting will succeed.
- Early aborts: Early aborts act as learned, dynamic safety constraints that prevent actions from which the agent cannot recover.
- Hard resets: The algorithm approximates irreversibility by declaring failure after N unsuccessful reset episodes, trading fewer hard resets against more time stuck in that state.The parameter N should depend on the cost of hard resets.
- Uncertainty-aware aborts: Uncertainty-aware Q-functions use bootstrap ensembles to produce Q-value distributions for forward and reset policies.
5 Small-Scale Didactic Example
The didactic examples examine early aborts in gridworld, including settings with an absorbing goal. Increasing the early abort threshold produces more cautious exploration without severely increasing the steps needed to solve.
- Figure 1 presents early aborts in gridworld.
- Increasing Qmin can decrease training time even when it requires more steps to learn in some real-world experiments.
- Figure 2 presents early aborts with an absorbing goal.
- Increasing the early abort threshold causes more cautious exploration without severely increasing the number of steps to solve.
6 Continuous Environment Experiments
Across continuous-control experiments, the jointly learned reset policy reduces hard resets, supports safer early aborts, and can induce a curriculum. Compared with forward-only learning, it also enables task learning where automatic resetting is needed.
- Ball in Cup: The forward-only approach fails to learn ball-in-cup catching, while the reset-policy method solves it without hard resets.The status quo hard-resets after every episode and is therefore impractical outside simulation.
- Manual Resets: The method eventually matches pusher reward with half as many hard resets and achieves substantially higher final reward on cliff cheetah with an order of magnitude fewer resets.Early aborts may prevent wasted exploration in irreversible states.
- Multiple Reset Attempts: On pusher, four reset attempts yield another 2.5x reduction in hard resets while reducing reward by less than 25%; on cliff cheetah, resets approach zero without changing reward.A single reset attempt already uses 64% fewer hard resets than the status quo on pusher.
- Ensembles: A single value-function ensemble fails to learn, whereas increasing ensemble size slightly decreases hard resets without affecting reward.The ensemble-size experiment tests sensitivity to value-function bias during early-abort decisions.
- Automatic Curriculum Learning: The method solves sparse-reward peg insertion by automatically inducing a curriculum as the reset policy improves.The reset policy inserts the peg while the forward policy removes it; hard resets plateau after one million steps.
7 Conclusion
The framework automates resets and uses early aborts to avoid unrecoverable states, reducing manual resets while learning to avoid unsafe states and induce a curriculum. Its main limitations are early uncertainty about dangerous states, equal weighting of manual-reset costs, and simulation-only evaluation.
- The algorithm jointly learns forward and reset policies to automate resets, avoid unsafe states, and induce a curriculum.
- During early learning, dangerous states cannot be avoided until the agent visits them or a similar state and experiences a manual reset.
- The method treats all manual resets as equally costly, although breaking a wine glass can cost more than moving a block.
- The experiments were conducted in simulation, and applying the algorithm to real robots remains the next step.
A Combining an Ensemble of Value Functions
The ensemble study compares optimistic, realistic, and pessimistic value-function combinations. Optimistic exploration succeeds in gridworld, while ensemble choice has little effect in continuous environments.
- Only the optimistic agent efficiently explored gridworld; realistic and pessimistic agents failed when Qmin was too large.
- In continuous control environments, the ensembling method made relatively little difference to reset counts or final performance.
B.1 Does Our Method Reduce Manual Resets? – More Plots
The appendix provides additional plots for experiments comparing manual-reset baselines, early-abort thresholds, and the number of reset attempts across environments.
- Figure 13 compares the method with resetting after every episode across all environments.
- Figure 14 shows the effect of varying the early-abort threshold across all environments.
- Figure 15 shows the effect of increasing the number of reset attempts across all environments.
- Figure 16 also shows the effect of increasing the number of reset attempts across all environments.
C Experimental Details
The experiments use multiple random seeds and ensemble models, with task-specific rewards defining both forward objectives and reset behavior across benchmark environments.
- Figures 1 and 2 average early-abort counts across 10 random seeds, while Figure 3 reports the median across 10 random seeds.
- Both gridworld experiments use 5 models in the ensemble.
- The Ball in Cup reset reward is the negative distance from the initial state, where the ball hangs stationary below the cup.
- For Cliff Cheetah and Cliff Walker, reset rewards combine distance from the origin, standing status, and a control penalty.
- For Pusher, the reset reward uses puck distance from the start instead of distance from the goal.
- For Peg Insertion, insertion receives reward 1 when successful, while removal minimizes distance to a fixed state outside the hole plus a control penalty.
C.3 Continuous Control Experiments
The continuous-control experiments used shared reward normalization and DDPG settings across environments, with fixed ensemble and reset parameters. Results were aggregated across five random seeds without hyperparameter optimization.
- Results used the median number of task-solving runs across 5 random seeds, without hyperparameter optimization.Most experiments were solved by all random seeds.
- Rewards were normalized to [0, 1] across three continuous-control environments, using γ = 0.99 and defining Sreset as states with reset reward greater than 0.7.The cumulative discounted reward therefore lay in [0, 100).
- All continuous-control environments used the same DDPG hyperparameters.
- The actor network used two fully connected layers of sizes 400 and 300 with tanh nonlinearities.
- The critic applied a 400-dimensional state layer, concatenated actions, and then used a 300-dimensional fully connected layer with tanh nonlinearities.
- Unless otherwise noted, experiments used an ensemble of size 20, Qmin = 10, one reset attempt, and early aborts using min(q).Section 6.2 instead used two reset attempts to better illustrate reducing hard resets.