Source-linked AI summary
Failure-Aware RL: Reliable Offline-to-Online Reinforcement Learning with Self-Recovery for Real-World Manipulation
Huanyu Li, Kun Lei, Sheng Zang, Kaizhe Hu, Yongyuan Liang, Bo An, Xiaoli Li, Huazhe Xu
TL;DR
Real-world offline-to-online reinforcement learning can produce Intervention-requiring Failures during exploration. FARL combines a world-model-based safety critic with an offline-trained recovery policy, reducing such failures while improving task performance and generalizability in simulated and real-world evaluations.
Problem
Offline-to-online reinforcement learning faces Intervention-requiring Failures during exploration because exploration introduces randomness into actions, hindering deployment in real-world settings.
Method
FARL integrates a world-model-based safety critic that predicts failures with an offline-trained recovery policy that prevents foreseen failures during exploration.
Results
FARL substantially reduces Intervention-requiring Failures while enhancing task performance and generalizability across simulated FailureBench settings and real-world robot experiments.
Takeaways & Limitations
FARL selectively replaces risky actions with recovery alternatives, providing simultaneous safety and performance benefits during reinforcement learning.
Takeaways & Limitations
The current system does not incorporate additional modalities such as 2D/3D vision or tactile sensing and is not yet extended to mobile manipulation or dual-arm platforms.
Abstract
from arXiv · showhide
Post-training algorithms based on deep reinforcement learning can push the limits of robotic models for specific objectives, such as generalizability, accuracy, and robustness. However, Intervention-requiring Failures (IR Failures) (e.g., a robot spilling water or breaking fragile glass) during real-world exploration happen inevitably, hindering the practical deployment of such a paradigm. To tackle this, we introduce Failure-Aware Offline-to-Online Reinforcement Learning (FARL), a new paradigm minimizing failures during real-world reinforcement learning. We create FailureBench, a benchmark that incorporates common failure scenarios requiring human intervention, and propose an algorithm that integrates a world-model-based safety critic and a recovery policy trained offline to prevent failures during online exploration. Extensive simulation and real-world experiments demonstrate the effectiveness of FARL in significantly reducing IR Failures while improving performance and generalization during online reinforcement learning post-training. FARL reduces IR Failures by 73.1% while elevating performance by 11.3% on average during real-world RL post-training. Videos and code are available at https://failure-aware-rl.github.io.
I. INTRODUCTION
The paper addresses intervention-requiring failures during real-world RL post-training by introducing FARL, which combines offline preparation with safe online refinement. It also introduces FailureBench and reports reduced failures alongside improved task performance and generalization.
- Motivation: Intervention-requiring Failures arise during offline-to-online RL because exploration randomness can produce irreversible damage or unsafe states requiring human intervention.Examples include breaking fragile objects, knocking items out of reach, and damaging the robot arm.
- Approach: FARL refines policies while minimizing failures that would otherwise require human intervention.The framework targets failure-aware post-training in real-world manipulation settings.
- Benchmark: FailureBench incorporates common intervention-requiring failure scenarios into existing simulation environments to evaluate performance, generalization, and failure minimization.Scenarios include objects pushed out of the workspace and robots entering unsafe states.
- Approach: The method combines a latent-world-model safety critic for failure prediction with a recovery policy that prevents failures forecast by the critic.Both components are trained offline using curated demonstrations and deployed during online task-policy post-training.
- Evaluation: FARL reduces intervention requirements while improving task performance and generalization across simulated FailureBench settings and real-world robotic experiments.The experiments include three challenging real robotic tasks on a Franka Emika Panda robot.
II. RELATED WORK
The related work positions FARL at the intersection of safe RL and offline-to-online RL, shifting safety-aware learning toward post-training from demonstrations. Its formulation uses constrained policies and focuses on safe policy exploration and refinement rather than random exploration from scratch.
- Safe RL: Prior safe-RL research primarily studies learning-from-scratch settings, while modern demonstration-trained robotic models motivate safety-aware online post-training.FARL shifts this focus to failure-aware offline-to-online reinforcement learning.
- Safe RL: Safe and recovery RL addresses safety constraints and the formulation of safe actions, often balancing task rewards and constraints through optimization techniques.The passage identifies constrained Markov decision processes as a central framework for safe real-world learning.
- Offline-to-online RL: Offline RL mitigates distributional-shift risks from out-of-distribution state-action pairs using conservatism or constraint-based regularization before online fine-tuning.These approaches discourage exploration in out-of-distribution regions or provide conservative value estimates.
- Problem formulation: FARL formulates post-training as constrained RL with binary constraint costs identifying violating state-action pairs and a threshold defining feasible policies.The optimization is performed over policies whose discounted violation probability remains within the safety bound.
- FARL: The proposed algorithm optimizes safe policy exploration and refinement for models learned from demonstrations instead of relying on unnecessary random exploration from scratch.The approach is inspired by recovery RL but specifically targets post-training.
IV. METHOD
FARL pre-trains task, recovery, and world-model components offline to support safer online policy refinement. The world model predicts near-future failures using latent representations, while the recovery policy helps avoid or escape near-failure states.
- FARL pre-trains a task policy, recovery policy, and world model during its offline phase.The task policy uses successful-task demonstrations, the recovery policy uses recovery demonstrations, and the world model uses task and failure demonstrations.
- The task policy follows offline-to-online training, while the recovery policy is behavior-cloned and not fine-tuned online because failure data are limited.The authors report that keeping the recovery policy fixed enhances safe exploration.
- The model is trained with task and failure demonstrations to predict failures over a limited near-future horizon rather than an entire episode.The constraint and consistency losses support latent dynamics and failure prediction.
- The world model encodes observations into latent representations and predicts next states, reconstructions, constraints, rewards, values, and maximizing actions.These prediction heads enrich the representation and support failure prediction.
- The world model uses value and reward signals to enrich latent representations and regularize against overfitting to constraint patterns.
B. Online fine-tuning with recovery
During online fine-tuning, FARL checks task-policy actions with the world model and replaces unsafe transitions with recovery-policy actions. The task policy is then updated using the resulting safe transitions.
- The task policy is fine-tuned online with PPO initialized from offline policy and value networks.
- Each task-policy state-action pair is checked by the world model through near-future planning for safety.
- When a sampled task action violates the safety threshold, FARL replaces the task transition with a recovery-policy transition.The replacement action is sampled from the recovery policy.
- The task policy is fine-tuned with GAE advantage estimation using the resulting safe transitions.
V. THEORETICAL ANALYSIS
FARL’s theoretical analysis explains its performance through an action-correction mechanism based on classifying states and actions by their H-step constraint-violation risk.
- FARL provides theoretical justification through an action-correction mechanism.
- The analysis classifies states according to H-step constraint violation under the task policy.
- Actions are classified by their H-step constraint-violation status at the action level.
- The risk probability at a state is used to characterize the likelihood of constraint violation.
B. Assumptions
The analysis assumes risky actions occur in a non-negligible set of states, recovery is probabilistically safe, and safe actions have an advantage over risky actions.
- A non-negligible fraction of states is assumed to contain risky actions sampled by the task policy.
- The recovery policy is assumed to provide safe actions with high probability for any state.The recovery failure rate is represented by ϵrec > 0.
- Where safe and risky actions coexist, safe actions are assumed to have better expected advantage.The advantage gap is represented by δ > 0.
C. Main Result
FARL’s action-correction analysis shows that replacing risky actions with recovery alternatives can improve policy learning while enhancing safety. The benefit grows with frequent risky states, stronger safe-action advantages, and higher-quality recovery demonstrations.
- Action Correction: The theorem’s risk term prisk(s) denotes the probability that the task policy samples a risky action at state s.
- Action Correction: FARL’s corrected transitions yield an improvement bound involving risky-action frequency, recovery quality, and the safe-action advantage.The bound is at least E_s∼ρπtask[prisk(s)] · δ · (1 − ϵrec) − O(ϵrec).
- Action Correction: The correction benefit is largest when risky states are frequent, safe actions substantially outperform risky actions, and recovery demonstrations have low error.
- Action Correction: FARL selectively replaces risky actions with recovery alternatives, providing both improved safety and enhanced performance.
VI. EXPERIMENTS
FailureBench evaluates failure-aware offline-to-online RL across four manipulation tasks designed around intervention-requiring failures. The experiments compare FARL with offline-to-online and online safe-RL baselines using task performance, recovery, failure, and world-model training data.
- Benchmark: FailureBench comprises four representative tasks: Bounded Push, Bounded Soccer, Fragile Push Wall, and Obstructed Push.
- Benchmark: The benchmark models intervention-requiring failures such as objects leaving the workspace, unsafe robot states, fragile-object collisions, and obstacle collisions.
- Comparisons: FARL is compared with Uni-O4 and the online safe-RL methods PPO-Lagrangian, P3O, and CPO using the same offline pre-trained policy.
- Data: Offline pre-training uses task demonstrations, recovery demonstrations, and failure sequences collected to approximate failures during online fine-tuning.
- Metrics: Evaluation measures average episodic reward improvement and the number of episodes containing intervention-requiring failures during online fine-tuning.
2) Results and Analysis:
FARL reduces failure episodes across FailureBench while maintaining or improving task performance, and its world-model safety critic plus learned recovery policy outperform the tested alternatives.
- Results: 43.6% average reduction in failure episodes across FailureBench tasks, reaching 65.8% in the most challenging environments.The largest improvements occur in Bounded Soccer and Obstructed Push.
- Results: FARL achieves competitive or higher task rewards despite adding failure-avoidance constraints during exploration.The reported analysis characterizes this as balancing exploration and safety without sacrificing performance.
- Results: Traditional safe RL methods underperform in offline-to-online settings, while FARL exceeds them by an average of over 800% across tasks.The comparison uses policies initialized from offline pre-training and attributes the gap to optimization-induced distribution shifts.
- Ablation Studies: Replacing the world-model safety critic increases failures in every environment, with 92% more failures in Bounded Soccer.The ablation supports explicit future state-action modeling for long-term risk assessment.
- Ablation Studies: The learned recovery policy outperforms MPPI planning, particularly in environments with complex dynamics.The paper links this advantage to recovery demonstrations that provide behaviors difficult to discover through planning alone.
- Ablation Studies: The combined world-model safety critic and pre-trained recovery policy provides the most robust performance across diverse failure scenarios.Figure 6 further reports superior final performance after fine-tuning for FARL.
C. Real-World Experiments
The real-world evaluation deploys FARL on a Franka Emika Panda robot across three challenging manipulation tasks, measuring both failure reduction and performance improvement.
- Real-World Experiments: FARL is evaluated on a Franka Emika Panda robot in three representative real-world tasks.The evaluation assesses failure reduction and performance improvement in challenging scenarios.
1) Experimental Setup: •
The real-world setup tests FARL on fragile-object pushing, dynamic-obstacle avoidance, and bounded soccer, using camera-based perception and 50-episode training sessions.
- Experimental Setup: Franka Fragile Push Wall treats wall contact with a fragile object as an Intervention-requiring Failure requiring replacement.The robot must push the object to a target behind a wall without collision.
- Experimental Setup: Franka Disturbed Push requires avoiding a randomly moved decorative flower while pushing an object to a target.A human operator supplies unpredictable environmental changes by moving the obstacle.
- Experimental Setup: Franka Bounded Soccer defines the ball leaving uneven-surface boundaries as an Intervention-requiring Failure.The robot uses a UMI gripper to kick the ball toward a target.
- Experimental Setup: Perception and control run at approximately 5Hz using an Intel RealSense D435, YOLOv8 detection, and color-based object tracking.The system estimates object positions in the workspace from camera observations.
- Experimental Setup: Each task uses 50 training episodes and 40–80 teleoperated trajectories for three demonstration types.Collecting all demonstration types requires approximately 30–60 minutes of human effort per task.
- Results: FARL dramatically reduces total failures during the 50-episode real-world training phase compared with the baseline.The comparison is presented across all environments in Figure 7.
- Results: Online RL fine-tuning increases returns and reduces standard deviation, with noticeably higher returns in Disturbed Push and Bounded Soccer.The reported result indicates improved robustness while minimizing failures requiring extensive human intervention.
- Method: FARL integrates a world-model safety critic and recovery policy to reduce Intervention-requiring Failures during exploration.The framework is presented as a failure-aware offline-to-online reinforcement learning approach.