Source-linked AI summary
Residual Off-Policy RL for Finetuning Behavior Cloning Policies
Lars Ankile, Zhenyu Jiang, Rocky Duan, Guanya Shi, Pieter Abbeel, Anusha Nagabandi
TL;DR
Behavior cloning is costly and can saturate, while direct real-world RL is difficult because of sample inefficiency, safety concerns, and sparse rewards. ResFiT freezes a BC policy and learns per-step residual corrections with sample-efficient off-policy RL. It improves sparse-reward manipulation across simulation and real-world high-DoF systems, including humanoid dexterous-hand training, but remains constrained by the skills encoded in the base policy.
Problem
Behavior cloning depends on costly human demonstrations and can saturate, while direct real-world RL is challenging for sample efficiency, safety, and sparse-reward high-DoF tasks.
Method
ResFiT treats a BC policy as a frozen black-box base and learns per-step residual corrections with sample-efficient off-policy RL.
Results
ResFiT improves sparse-reward manipulation policies across simulation and real-world high-DoF systems, including real-world RL training on a humanoid robot with dexterous hands.
Takeaways & Limitations
Decoupling BC pretraining from residual fine-tuning keeps long-horizon reasoning while making high-dimensional RL optimization tractable.
Takeaways & Limitations
Learned behaviors remain constrained around the base policy and cannot discover fundamentally different strategies or skills beyond those it encodes while remaining stable.
Abstract
from arXiv · showhide
Recent advances in behavior cloning (BC) have enabled impressive visuomotor control policies. However, these approaches are limited by the quality of human demonstrations, the manual effort required for data collection, and the diminishing returns from offline data. In comparison, reinforcement learning (RL) trains an agent through autonomous interaction with the environment and has shown remarkable success in various domains. Still, training RL policies directly on real-world robots remains challenging due to sample inefficiency, safety concerns, and the difficulty of learning from sparse rewards for long-horizon tasks, especially for high-degree-of-freedom (DoF) systems. We present a recipe that combines the benefits of BC and RL through a residual learning framework. Our approach leverages BC policies as black-box bases and learns lightweight per-step residual corrections via sample-efficient off-policy RL. We demonstrate that our method requires only sparse binary reward signals and can effectively improve manipulation policies on high-degree-of-freedom (DoF) systems in both simulation and the real world. In particular, we demonstrate, to the best of our knowledge, the first successful real-world RL training on a humanoid robot with dexterous hands. Our results demonstrate state-of-the-art performance in various vision-based tasks, pointing towards a practical pathway for deploying RL in the real world.
I. INTRODUCTION
Behavior cloning provides strong visuomotor policies but depends on costly demonstrations and can saturate, while direct real-world RL remains data-intensive and unsafe. ResFiT combines them by learning off-policy residual corrections, enabling sparse-reward improvements across high-DoF tasks and real-world humanoid manipulation.
- Motivation and approach: BC data collection requires substantial manual effort and infrastructure, while performance can saturate as demonstrations increase.
- Motivation and approach: Direct real-world RL is limited by the large interaction demands and safety risks of collecting online data.
- Motivation and approach: ResFiT learns per-step residual corrections on top of a frozen black-box BC policy, avoiding direct optimization of large action-chunked or diffusion-based models.The residual is independent of chunk size and policy parameterization.
- Related work: ResFiT uses demonstrations for BC pretraining and off-policy RL, while retaining them for critic warm-up and continued replay.
- Results and contribution: Compared with prior residual methods, ResFiT addresses higher-DoF, longer-horizon visuomotor tasks and demonstrates bimanual humanoid control in the real world.
III. METHOD
The method begins with a demonstration-trained BC policy and then applies off-policy residual RL to improve it. The frozen base remains a black box while the residual learns corrections that can be applied across policy parameterizations and action-chunk sizes.
- Method overview: ResFiT trains a base policy on demonstration data, freezes it, and learns residual corrections with an off-policy RL recipe.
- Residual fine-tuning: The residual policy is designed to be independent of the base policy’s parameterization and action-chunk size.
- Behavior cloning: The BC base predicts k future actions from each observation, using action chunks to reduce compounding imitation errors and the effective task horizon.
- Residual fine-tuning: The method uses image and proprioceptive observations without access to the underlying system state.
B. fine-tuning with Off-Policy Residual RL
Off-policy residual RL freezes the BC controller and trains a residual policy whose correction is added to the base action. The recipe combines demonstration and online data with critic-based policy optimization to support stable fine-tuning.
- Residual formulation: Freezing the base policy constrains residual magnitude, keeping exploration near the base behavior while allowing corrections to mistakes.
- Residual formulation: The residual policy receives the observation and base action, and the executed full action is their sum.
- Off-policy optimization: The critic approximates the optimal action-value function with a Mean-Squared Bellman Error objective over transition data.
- Off-policy optimization: The residual policy is optimized by gradient ascent on the differentiable critic’s value estimate.
C. Design Decisions
ResFiT combines off-policy data reuse with design choices intended to stabilize and improve residual fine-tuning across tasks and embodiments.
- ResFiT interleaves Bellman-based Q-function training with Q-guided policy training using off-policy data.
- n-step returns propagate sparse rewards beyond one step, while UTD ratios above 1 improve reuse of collected data.
- Critic layer normalization mitigates catastrophic overestimation from out-of-distribution actions without explicitly constraining the policy.
- The implementation additionally uses delayed actor updates, Polyak target averaging, target-policy smoothing, visual augmentation, and symmetric demo sampling.Demonstration batches mix 50% frozen offline data with 50% online-buffer data.
A. Simulation Experiments
The simulation evaluation tests ResFiT on varied single-arm and bimanual manipulation tasks under realistic visual, control, and sparse-reward constraints, using matched demonstration budgets across approaches.
- Experimental setup: Experiments use image and robot-joint observations, sparse binary rewards, and no privileged object-state information in a single simulation environment.
- Tasks: The benchmark spans single-arm Can and Square tasks plus bimanual BoxCleanup, CanSort, and Coffee tasks with higher-dimensional coordinated control.
- Experimental setup: All approaches in the simulation comparison start with the same number of demonstrations.
- Baselines and ablations: The comparison includes ResFiT, Tuned RLPD, IBRL, PPO-based residual RL, and ablations of layer normalization, demo use, UTD, and n-step returns.
B. Real-World Experiments
The real-world evaluation studies ResFiT on a dexterous humanoid platform using bimanual manipulation tasks, safety infrastructure, and matched blind A/B testing.
- Platform: The platform is a wheeled Vega humanoid with two 7-DoF arms, two 6-DoF dexterous hands, two camera streams, and a 29-dimensional action space.
- Tasks: The tasks require placing a ball into a tote and transferring a deformable package between hands before placement.
- Infrastructure: Safety controls combine wrist force-torque limits with collision checking to prevent self-collisions during real-world RL.
- Evaluation protocol: Blind A/B testing randomly assigns policy labels and executes both policies from identical initial states under matched scene configurations.
A. Simulation Results
ResFiT achieves strong simulation performance across tasks, with major sample-efficiency gains over PPO and robustness on harder sparse-reward, long-horizon settings.
- 200k steps versus 40M steps yields a ∼200× sample-efficiency boost for off-policy residual RL over PPO on BoxCleanup.
- ResFiT converges to near-perfect policies across all simulation tasks, while harder-task baselines and ablations often collapse or learn more slowly.
- On Coffee, approaches without action chunking fail, whereas ResFiT remains efficient despite the task’s long horizon and precision demands.
- Filtered BC remains stable but improves minimally over the initial BC policy, indicating limited gains from filtering alone in these tasks.
- For sparse-reward tasks, n-step values above 1 help, while UTD 4 captures most benefits without the diminishing returns of very high UTD.
B. Real-World RL Results
ResFiT substantially improved two real-world manipulation tasks on a 29-DoF bimanual wheeled humanoid, including the first reported fully real-world RL demonstration on a bimanual dexterous humanoid.
- WoollyBallPnP: 14% to 64%: ResFiT improved WoollyBallPnP success after 134 autonomous RL rollouts, or approximately 15 minutes of robot data.The base ACT policy had been trained with around 1,000 demonstrations and mainly failed through hovering and missed dexterous-hand grasps.
- PackageHandover: 23% to 64%: ResFiT improved PackageHandover success after 343 real-world RL episodes, or approximately 76 minutes of data.PackageHandover requires two-arm coordination and has a longer horizon before potential reward.
- Overall real-world results: ResFiT showed a significant performance boost over the base model on the 29-DoF bimanual wheeled humanoid with two five-fingered hands.The reported real-world results concern the residual RL approach applied to this platform.
- Overall real-world results: The work reports the first real-world RL demonstration on a bimanual dexterous humanoid with two five-fingered hands, trained fully in the real world.The claim is stated as being to the authors’ knowledge.
VI. DISCUSSION
The discussion attributes ResFiT’s tractability and deployment behavior to single-step residual corrections around a frozen BC policy, while acknowledging constraints on strategy discovery and remaining human oversight.
- Design implications: Single-step residual corrections preserve long-horizon reasoning while avoiding the 870-dimensional action space created by action chunking.The method decouples BC pre-training from RL fine-tuning and keeps optimization tractable.
- Design implications: The base policy acts as both an implicit safety constraint and an exploration prior, supporting sparse-reward RL in high-DoF spaces.Without this regularization, policies converge to faster but more aggressive behaviors unsuitable for real-world deployment.
- Limitations and future directions: Learned behaviors may remain constrained around the base policy and cannot discover fundamentally different strategies or skills beyond its encoded behavior while remaining stable.The discussion identifies relaxing the frozen-base constraint without sacrificing stability as a path for further improvement.
- Limitations and future directions: Real-world experiments still required human supervision for resets and reward labeling, so autonomous improvement does not yet scale independently of human oversight.Automatic resets, success detection, and safety rails are identified as missing prerequisites.