PaperlayerView exact revision on arXiv

Source-linked AI summary

Object-Centric Residual RL for Zero-Shot Sim-to-Real VLA Enhancement

Kinam Kim, Namiko Saito, Heecheol Kim, Katsushi Ikeuchi, Jaegul Choo, Yasuyuki Matsushita

arXiv:2606.18953v1cs.RO

TL;DR

VLAs remain brittle in precise physical interactions, while existing residual-RL approaches face sim-to-real transfer barriers. This paper uses object-centric residual RL trained solely in simulation and transfers it zero-shot, raising real-robot success from 42% to 76% across five tasks.

  • Problem

    Sim-to-real residual RL for VLAs remains challenging because privileged-state observations require distillation and image-based observations face a visual domain gap.

  • Method

    The framework trains a corrective residual policy on 6-DoF object poses, proprioception, and base VLA actions, using paired simulated and real VLAs aligned through teleoperation replay.

  • Results

    42% to 76% average success rate across five manipulation tasks on a real FR3 robot was achieved through zero-shot transfer without real-world RL or fine-tuning.

  • Takeaways & Limitations

    Residual-corrected real-robot rollouts can be aggregated to retrain a single multi-task VLA, enabling self-improvement without additional teleoperation.

  • Takeaways & Limitations

    The method relies on real-time 6-DoF pose tracking, manual specification of task-relevant objects, and residual corrections that cannot recover states far outside the base VLA’s training distribution.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models can generalize across diverse manipulation tasks, but their imitation-learning-based policies remain brittle in precise physical interactions due to compounding execution errors; Can a reinforcement learning policy trained purely in simulation improve the robustness of real-world VLAs zero-shot? Residual RL, which learns a corrective policy on top of a frozen VLA, offers a natural framework, but existing approaches face a fundamental sim-to-real dilemma: privileged-state methods require lossy distillation for deployment; image-based methods suffer from the visual domain gap; and real-world RL is costly and unsafe. We propose an object-centric residual RL framework that refines VLA actions using object poses, enabling a compact observation space that transfers consistently between simulation and reality. To align the two domains, we additionally replay the same teleoperation demonstrations in simulation to train a sim counterpart of the real-world VLA. The residual RL policy is trained only in simulation with pose noise injection and dropout, and transfers zero-shot to the real robot. Across five manipulation tasks on a real Franka Research 3 (FR3) robot, our method improves the success rate from 42% to 76% zero-shot, and the improved rollouts can be further reused to retrain the base VLA for self-improvement without additional teleoperation. Project page: https://www.microsoft.com/en-us/research/articles/object-centric-residual-rl/

1 Introduction

The paper proposes a zero-shot sim-to-real residual RL framework that reduces domain discrepancies through object-centric, domain-invariant observations and paired sim-real VLAs aligned by teleoperation replay. Successful real-robot rollouts can then support multi-task VLA retraining without additional teleoperation.

  • Motivation: Imitation-trained VLAs accumulate small execution errors over time, causing failures in unseen states.The introduction identifies RL as a potential recovery mechanism but notes that directly applying RL to modern diffusion- or flow-matching-based VLAs is difficult.
  • Limitations of prior work: Prior residual RL methods either distill privileged simulator state into image policies or face visual sim-to-real gaps that prevent zero-shot transfer.The three approaches differ in their failure modes: distillation incurs performance loss, image-based simulation suffers visual domain mismatch, and real-world residual RL is costly.
  • Proposed approach: The proposed framework builds residual policies on domain-invariant object-centric observations to reduce discrepancies between simulation and reality without distillation.Its design pairs simulated and real VLAs by replaying the same teleoperation demonstrations, aligning the residual policy’s interface across domains.
  • Self-improvement: Residual-corrected real-robot rollouts can be aggregated across tasks to retrain a single multi-task VLA without additional teleoperation.This enables automatic VLA self-improvement using higher-quality training data collected from successful deployments.

2 Related Work

Prior residual-RL methods combine frozen-base generalization with learned precision but face deployment, scalability, or real-world training costs. Sim-to-real research instead uses randomization and data-driven adaptation to improve robustness across domains.

  • Residual Reinforcement Learning: Residual RL learns corrective policies atop frozen bases, combining broad base-policy generalization with reinforcement-learning precision.
  • Residual Reinforcement Learning: Privileged-state approaches such as ResiP perform strongly in simulation but require teacher-student distillation for deployment, causing non-trivial transfer performance loss.
  • Residual Reinforcement Learning: RialTo avoids real-world reinforcement learning through a digital twin, but its reconstruction, registration, point-cloud training, and inverse-distillation pipeline has substantial per-task overhead.
  • Residual Reinforcement Learning: Image- and proprioception-based ResFiT requires real-robot training with safe exploration infrastructure, episode resets, and real-world reward detection.
  • Sim-to-Real Transfer: Sim-to-real methods use domain randomization over visual and physical parameters, language-model automation, real-world distribution adaptation, and online human feedback.

3 Method

The method builds a simulation counterpart of a real-world VLA by replaying teleoperation, trains an object-centric residual policy with robustness augmentations in simulation, and deploys both zero-shot on the real robot. Object poses, proprioception, and aligned base actions reduce sim-to-real discrepancies while pose noise and dropout provide fallback behavior.

  • Method overview: The framework has three stages: teleoperation replay to build a simulation VLA, simulation training of an object-centric residual policy, and zero-shot real-robot deployment.The real VLA remains frozen during residual training, and both components are frozen at deployment.
  • Simulation counterpart: Replaying identical teleoperation actions in simulation trains a sim VLA whose action distribution aligns with the real VLA despite different visual domains.The sim VLA provides the base action for residual RL, while the real VLA replaces it during deployment.
  • Object-centric observation: The residual observes task-relevant object 6-DoF poses, proprioceptive state, and the VLA’s current action, with poses concatenated across multiple objects.This compact observation avoids privileged simulator state and image-based visual domain gaps while keeping the base VLA frozen.
  • Robust pose training: Training injects timestep-wise position and orientation noise and zeros the entire object-pose vector with probability ρdrop to handle estimator errors and failures.The dropout condition forces a fallback strategy based on proprioception and the base action.
  • Real-robot deployment: At deployment, low pose-estimator confidence triggers pose zeroing, while the residual actor adds less than 1 ms of forward-pass overhead.FoundationPose runs in tracking mode after SAM2 initialization at approximately 18 ms per frame.
  • Self-improvement: Successful residual-corrected rollouts can be merged with demonstrations to retrain the base VLA without additional teleoperation data.Rollouts from multiple task-specific residuals can be aggregated into a single multi-task VLA while preserving generalist ability.

4 Experimental Setup

The experiments evaluate five tabletop manipulation tasks in MuJoCo simulation and on a real FR3 robot, using matched task definitions across domains. GR00T-N1.5 is fine-tuned on teleoperation demonstrations, while a lightweight TD3 residual policy is trained for efficient action correction.

  • Simulation and Real-World Evaluation: Each task is instantiated in MuJoCo for residual training and on a real FR3 robot for zero-shot deployment.The simulation uses measured real-object dimensions rather than requiring visual realism.
  • Tasks: The evaluation covers five tasks: Cube Lift, Pick-and-Place, Stack Cube, Close Drawer, and Stand Cup Up.These tasks span lifting, placing, stacking, drawer closing, and reorienting a cup upright.
  • Base VLA: GR00T-N1.5 is fine-tuned on 30 teleoperation demonstrations per task for each domain, sim and real.The same open-source VLA serves as the base policy in both domains.
  • Residual Policy: The residual policy is a lightweight 2-layer MLP trained with TD3, taking ∼0.06 ms per GPU forward pass versus ∼140 ms for the VLA.The residual inference time is less than 0.05% of the VLA’s inference time.

5 Results

Simulation-trained residual RL transfers zero-shot to a real robot, improving VLA performance across tasks and architectures. Results attribute transfer to object-centric observations and robustness training, while showing selective correction and downstream self-improvement.

  • 5.1 Zero-shot real-robot performance: 76% average success follows 42% with a simulation-trained residual on the real robot across all five tasks, without real-world RL or fine-tuning.The residual also improves all five tasks in simulation, with largest gains where the base VLA struggles most.
  • 5.1 Zero-shot real-robot performance: Residual RL consistently improves real-robot performance with π0.5, indicating compatibility with different VLA backbones.The shared object-centric observation interface supports this cross-architecture evaluation.
  • 5.2 Transfer-enabling designs: Combined pose dropout and noise injection yield the strongest sim-to-real transfer, with dropout aiding detection-failure resilience and noise helping tight-tolerance tasks.Both mechanisms contribute, while pose dropout contributes most strongly.
  • 5.2 Transfer-enabling designs: Object-centric poses transfer best because image-based observations face the visual sim-to-real gap and privileged-state distillation loses performance.The object-centric design sidesteps the dominant visual domain barrier by using poses directly.
  • 5.3 Residual intervention: 9–22% faster task completion results from selective correction: the residual steers toward the goal when the base is misaligned and contributes less when aligned.This behavior is consistent across all five real-robot tasks.
  • 5.4 VLA self-improvement: SFT on residual-corrected rollouts raises real-robot success and reduces episode length versus SFT on plain base rollouts, enabling self-improvement without additional teleoperation.Residual-corrected trajectories provide higher-quality supervision for retraining the base VLA.

6 Conclusion

Object-centric residual RL enhances frozen VLAs through sim-trained corrective policies that transfer zero-shot to real robots. Using recoverable object-centric observations and robustness training, the method improves real-robot performance and supports further VLA self-improvement.

  • 6 Conclusion: Object-centric residual RL transfers sim-trained residual policies to real robots zero-shot using observations recoverable in both domains.The observation space is constructed from 6-DoF object poses, proprioception, and the base VLA action, without visual rendering.
  • 6 Conclusion: The object-centric design transfers most effectively across tested observation spaces and remains compatible with different VLA backbones.Figure 6 reports performance improvement on π0.5 and compares sim-to-real transfer across observation spaces.
  • 6 Conclusion: SFT on residual-corrected rollouts improves success rate and reduces episode length, enabling further VLA improvement without additional teleoperation.The conclusion identifies improved rollouts as a route for self-improvement, while Figure 6 reports the SFT effects.
  • 6 Conclusion: 9–22%: residual-corrected policies consistently reduce completion time on successful episodes compared with base policies.Figure 7 reports episode-length reductions, with error bars denoting standard error of the mean.
  • 6 Conclusion: 42% to 76%: robustness-trained residual policies improve the GR00T-N1.5-based VLA across five tasks on a real FR3 robot.Training uses noise injection and pose dropout, without real-world reinforcement learning or residual-policy fine-tuning.

7 Limitations and Future Work

The method remains limited by failures in real-time 6-DoF pose tracking, manual task-object specification, and an unresolved simulation-to-reality dynamics gap. Future work includes memory-based pose estimation and automatic object identification from VLA attention maps.

  • Limitations: Real-time 6-DoF pose tracking with FoundationPose and SAM2 can fail under full occlusion or heavy clutter.Memory-based pose estimation could mitigate this limitation.
  • Future Work: Task-relevant objects must be specified manually, limiting scalability to open-world settings.Automatic identification, such as from VLA attention maps, could address this limitation.
  • Limitations: Pose-based observations bridge the visual domain gap but do not resolve the dynamics gap.The passage identifies contact friction and gripper compliance differences between simulation and reality as relevant dynamics factors.

Appendix for: Object-Centric Residual RL for Zero-Shot Sim-to-Real VLA Enhancement · A Appendix · A.1 Reward Design

The reward design uses dense, shaped signals clipped to [0, 1] and decomposed into progressively applied stages. Across tasks, these stages provide continuous feedback based on distance, orientation, or contact metrics.

  • A.1 Reward Design: Rewards are dense and shaped for all tasks, with values clipped to [0, 1].This bounds the reward signal while retaining continuous task feedback.
  • A.1 Reward Design: Each task reward is decomposed into staged subrewards.The staged structure organizes feedback around task progress.
  • A.1 Reward Design: Reward stages are applied progressively as the task advances.Later stages become relevant as execution moves through the task.
  • A.1 Reward Design: Table 3 summarizes the reward structure separately for each task.The appendix uses the table to present task-specific reward stages.
  • A.1 Reward Design: Each reward stage supplies a continuous signal rather than only a terminal outcome.The signals are designed to vary continuously during task execution.
  • A.1 Reward Design: Stage signals are based on distance, orientation, or contact metrics.These metrics provide the underlying measurements for the task-specific stages.

A.2 Simulation Environment Construction … A.11 Failure Case Analysis

The appendix details the simulation, training, deployment, and tracking setup for object-centric residual RL, then validates sim-to-real behavioral consistency, emergent corrections, and real-robot failure modes. It shows that shared sim/real VLA failures enable zero-shot correction, while perception errors remain a key limitation.

  • A.2 Simulation Environment Construction; A.3 Realistic Simulation Rendering: MuJoCo scenes use measured real-object dimensions and workspace layouts, with simple geometric primitives because the residual policy does not observe images.A visually realistic rendering setup with textured meshes, physically based lighting, and domain randomization supports image-based RL and policy-distillation baselines.
  • A.4 Algorithm Pseudocode; A.5 Training Hyperparameters; A.6 Pose Noise and Dropout Parameters: Residual RL training uses VLA action chunking, pose-noise augmentation, pose dropout, and TD3 updates, while deployment replaces stochastic dropout with confidence-gated pose dropout.FoundationPose provides 6-DoF tracking and SAM2 provides instance segmentation; training uses ρdrop = 0.1, H = 16, and τc = 0.5 at deployment.
  • A.7 Sim-to-Real Behavioral Consistency: Shared teleoperation-trained VLA failures in simulation and reality let simulation-trained residuals correct corresponding real-robot failures without observing images.Representative failures include hovering above the cube and stopping short of the target.
  • A.8 Strong Base VLA + Residual RL: A strong base VLA retains its 17/20 real-world success rate with residual RL, while a weak base roughly doubles success in simulation and reality.When the base acts correctly, the residual learns to stay near zero; simulation exploration also discovers failures and situations that demonstrations may omit.
  • A.9 Emergent Behaviors from Residual RL: Residual RL discovers task-specific behaviors absent from demonstrations, including cube pre-rotation for Cube Lift and Pick-and-Place, corrective pushing for Stack Cube, and sustained contact for Close Drawer.These strategies emerge from simulation exploration and are observed during real-robot deployment.
  • A.10 Object Tracking Visualization: During real-robot deployment, FoundationPose tracks each task’s object with projected 6-DoF mesh overlays, while drawer progress is estimated from depth within drawer-face regions.The visualization covers Cube Lift, Pick-and-Place, Stack Cube, Stand Cup Up, and Close Drawer.
  • A.11 Failure Case Analysis: Real-robot failures arise from pose-estimation drift, gripper occlusion, or wrong-object detection, each producing an incorrect pose that passes the confidence gate.Suggested remedies include semantic validation, multi-hypothesis pose estimation, and fallback to base VLA behavior.
Loading 2606.18953v1…