Source-linked AI summary
ReBot: Scaling Robot Learning with Real-to-Sim-to-Real Robotic Video Synthesis
Yu Fang, Yue Yang, Xinghao Zhu, Kaiyuan Zheng, Gedas Bertasius, Daniel Szafir, Mingyu Ding
TL;DR
High-cost real-robot data collection limits the scaling of VLA training data, while synthetic-video approaches face realism and consistency challenges. ReBot replays real trajectories in simulation and combines the movements with inpainted real backgrounds; the paper reports improved VLA performance in simulation and real-world settings.
Problem
Real-robot data collection is costly, and generated robot videos may lack physical realism and temporal consistency, limiting scalable target-domain adaptation.
Source: §I · p. 1Method
ReBot replays real-world robot trajectories in simulation and integrates the simulated movements with inpainted real backgrounds to create synthetic robot videos.
Source: §I · pp. 1–2Results
Takeaways & Limitations
The fully automated pipeline is presented as addressing robot manipulation’s last-mile deployment challenge through target-domain VLA adaptation.
Source: §I · p. 2
Abstract
from arXiv · showhide
Vision-language-action (VLA) models present a promising paradigm by training policies directly on real robot datasets like Open X-Embodiment. However, the high cost of real-world data collection hinders further data scaling, thereby restricting the generalizability of VLAs. In this paper, we introduce ReBot, a novel real-to-sim-to-real approach for scaling real robot datasets and adapting VLA models to target domains, which is the last-mile deployment challenge in robot manipulation. Specifically, ReBot replays real-world robot trajectories in simulation to diversify manipulated objects (real-to-sim), and integrates the simulated movements with inpainted real-world background to synthesize physically realistic and temporally consistent robot videos (sim-to-real). Our approach has several advantages: 1) it enjoys the benefit of real data to minimize the sim-to-real gap; 2) it leverages the scalability of simulation; and 3) it can generalize a pretrained VLA to a target domain with fully automated data pipelines. Extensive experiments in both simulation and real-world environments show that ReBot significantly enhances the performance and robustness of VLAs. For example, in SimplerEnv with the WidowX robot, ReBot improved the in-domain performance of Octo by 7.2% and OpenVLA by 21.8%, and out-of-domain generalization by 19.9% and 9.4%, respectively. For real-world evaluation with a Franka robot, ReBot increased the success rates of Octo by 17% and OpenVLA by 20%. More information can be found at: https://yuffish.github.io/rebot/
I. INTRODUCTION
ReBot addresses costly real-robot data scaling and target-domain adaptation by replaying real trajectories in simulation and combining the simulated movements with inpainted real backgrounds. The paper reports improved VLA performance in simulation and real-world evaluation.
I. INTRODUCTION
- Real-world data collection is costly, while prior generated videos can contain artifacts and instruction mismatches that hinder realistic, temporally consistent target-domain adaptation.These shortcomings can introduce new domain gaps and reliability concerns.Source: §I · p. 1
- ReBot replays real robot trajectories to manipulate new objects in simulation, then combines simulated movements with inpainted real backgrounds to synthesize realistic, temporally consistent videos.Its pipeline includes automated scene setup and trajectory replay, robot and object segmentation and removal, and video synthesis.Source: §I · pp. 1–2
- ReBot grounds action and observation spaces in real robot data while leveraging simulation scalability, and its data pipeline requires no manual intervention.The paper presents this approach as addressing robot manipulation’s last-mile deployment challenge.Source: §I · p. 2
- 21.8% in-domain and 9.4% generalization gains for OpenVLA on SimplerEnv accompany a reported 20% gain in real-world tasks.The introduction also reports that ReBot significantly outperforms ROSIE.Source: §I · p. 2
II. RELATED WORK
Prior data-scaling approaches trade costly real-world collection against simulation gaps and synthetic-video quality issues. ReBot applies a real-to-sim-to-real pipeline to construct synthetic episodes for target-domain VLA adaptation.
II. RELATED WORK
- Real-world robot data is resource-intensive to scale, simulation is more scalable but faces sim-to-real gaps, and generated videos may lack physical realism and temporal consistency.These limitations motivate ReBot’s real-to-sim-to-real approach to scaling robot datasets.Source: §II · p. 2
- Existing real-to-sim-to-real methods face scalability challenges from limited scene and object diversity and the manual effort of building digital twins.ReBot explores this strategy for dataset scaling without manual intervention.Source: §II · p. 2
III. METHOD
- The method starts from real episodes containing video frames, actions, and language instructions, then aims to build a synthetic dataset for target-domain VLA adaptation.The synthetic episodes are based on the original episodes.Source: §III · p. 2
- ReBot’s method has three steps: replay trajectories in simulation, inpaint task-specific content from real backgrounds, and synthesize new video frames.Trajectory replay produces simulated movements for manipulating new objects.Source: §III · p. 2
A. Real-to-Sim Trajectory Replay
ReBot aligns digital twins to a real scene, replays real robot trajectories to manipulate new objects, and validates each replay before using it to synthesize data. Reusing trajectories across object shapes demonstrates the approach’s scalability.
A. Real-to-Sim Trajectory Replay
- Each replay begins by creating spatially aligned digital twins and replaying the real-world robot trajectory to generate simulated movements.The robot, cameras, and table are aligned to the initial video frame; table height is estimated from metric depth.Source: A. Real-to-Sim Trajectory Replay · pp. 2–3
- The new object is placed at the original object’s position, estimated from the gripper position when it closes to grasp.A container can optionally be placed at the gripper position when it opens to place the object.Source: A. Real-to-Sim Trajectory Replay · p. 3
- Replay validation monitors the Cartesian distance between object and gripper from grasp to placement, discarding episodes where manipulation fails.Replay success depends on affordance compatibility between the new object and the original object.Source: A. Real-to-Sim Trajectory Replay · p. 3
- Real-world trajectories can be reused to manipulate objects of different shapes, demonstrating the scalability of trajectory replay.Replays that fail to manipulate the new object are discarded after validation.Source: A. Real-to-Sim Trajectory Replay · p. 3
B. Real-world Background Inpainting
ReBot removes the robot and original object from real video to create a task-agnostic background, then combines that background with simulated movement. The resulting videos are described as realistic, physically plausible, temporally consistent, and better than ROSIE.
B. Real-world Background Inpainting
- The inpainting stage removes task-specific content—the original robot and object—from the real robot video to produce a task-agnostic background.The virtual robot is used in the synthetic video to preserve correct occlusions and realistic physical interactions.Source: B. Real-world Background Inpainting · pp. 3–4
- GroundedSAM2 segments and tracks the robot and object, using the replay-estimated object position as a prompt when locating the object.SAM2 propagates the masks across frames, and ProPainter inpaints the removed regions.Source: B. Real-world Background Inpainting · pp. 3–4
- ReBot’s synthetic videos are described as realistic, physically plausible, and temporally consistent, significantly outperforming ROSIE.The comparison shows examples from DROID, BridgeData V2, and ReBot’s dataset.Source: Figure 3 · p. 4
C. Sim-to-Real Video Synthesis
ReBot combines simulated movements with an inpainted real-world background to form synthetic episodes. The episodes use updated language instructions while retaining the original real-world actions.
C. Sim-to-Real Video Synthesis
- ReBot extracts the simulated robot and manipulated object, then merges them onto the inpainted real-world background to create synthetic video.The robot and object movements come from the simulated replay.Source: C. Sim-to-Real Video Synthesis · p. 4
- The synthetic instruction replaces the original object and container names with those used in trajectory replay.For example, the instruction can change “yellow mug” to “spoon” and “table” to “towel.”Source: C. Sim-to-Real Video Synthesis · p. 4
- Synthetic episodes retain the real-world actions because ReBot faithfully replays the original robot trajectories.The paper evaluates VLA adaptation using the resulting synthetic dataset.Source: C. Sim-to-Real Video Synthesis · p. 4
IV. EXPERIMENTS
The experiments assess ReBot’s synthetic-video fidelity and its effects on VLA performance in simulation and real-world environments.
IV. EXPERIMENTS
- ReBot’s experiments evaluate whether it produces high-fidelity synthetic robot videos.The video-quality evaluation is described in Section IV-B.Source: §IV · p. 4
- The experiments assess how ReBot affects VLA performance in simulation environments.The simulation evaluation is described in Section IV-C.Source: §IV · p. 4
- The experiments also assess VLA performance in real-world environments.The real-world evaluation is described in Section IV-D.Source: §IV · p. 4
A. Experimental Setups
The experiments use real robot episodes from BridgeData V2 and DROID, plus a separately collected dataset for real-world evaluation. ReBot and ROSIE are compared by finetuning Octo and OpenVLA on their generated episodes.
A. Experimental Setups
- BridgeData V2 and DROID provide tabletop pick-and-place episodes, while 220 newly collected episodes support real-world evaluation.DROID includes two exterior videos captured from opposite sides of the robot; simulated objects are kitchen assets from Objaverse.Source: A. Experimental Setups · p. 4
- Isaac Sim 4.1 and Isaac Lab support replay using digital twins matched to the WidowX and Franka robot platforms.The twins use a WidowX 250 6DOF arm for BridgeData V2 and a Franka Panda 7DoF arm with Robotiq 2F-85 gripper for DROID and the authors’ dataset.Source: A. Experimental Setups · p. 4
- Octo and OpenVLA are evaluated zero-shot and after finetuning on ROSIE- or ReBot-generated episodes, using 100 synthetic episodes per task.ROSIE is compared using an implementation based on Stable Diffusion because ROSIE is not open-source.Source: A. Experimental Setups · p. 4
B. Evaluation of Video Quality
Video quality is assessed with VBench and comparisons to ROSIE and original real videos. ReBot’s videos show temporal, imaging, and multi-view consistency advantages described in the evaluation.
B. Evaluation of Video Quality
- ReBot’s average temporal-quality score is 93.0%, compared with 96.1% for real robot videos.ReBot scores 99.2% in motion smoothness and 92.2% in background consistency.Source: Figure 4 · p. 5, B. Evaluation of Video Quality · p. 5
- ReBot’s imaging quality is 3.7% below original videos and 13.0% above ROSIE.The evaluation attributes ROSIE’s object-quality problems to reliance on inpainting masks over text-prompt guidance.Source: B. Evaluation of Video Quality · p. 5
- ReBot preserves multi-view consistency across camera views through its 3D simulation process, while ROSIE lacks this consistency.The figure shows two DROID examples, each captured from two camera views.Source: Figure 5 · p. 5, B. Evaluation of Video Quality · p. 5
C. Evaluation in Simulation Environment
In SimplerEnv, ReBot is evaluated against zero-shot models and ROSIE on WidowX tasks, including in-domain, generalization, and cross-embodiment performance. The reported results show gains across these settings.
C. Evaluation in Simulation Environment
- 7.2% for Octo and 21.8% for OpenVLA are ReBot’s increases in average success rate on WidowX in-domain tasks.The comparison uses four SimplerEnv tasks; grasp rates and task success rates are reported.Source: C. Evaluation in Simulation Environment · p. 5, Table I · p. 6
- ReBot raises Octo’s average generalization success rate from 6.5% to 26.4%, while OpenVLA’s average grasp rate rises from 15.6% to 66.8% and success rate from 0.7% to 11.1%.ReBot improves both models across physical, semantic, and subject generalization types.Source: Figure 6 · p. 6, C. Evaluation in Simulation Environment · p. 6
- On cross-embodiment evaluation, OpenVLA’s average success rate on WidowX rises from 1.4% with ROSIE to 12.5% with ReBot.OpenVLA is finetuned using scaled DROID data for Franka Panda and evaluated on WidowX and Google Robot; ReBot also improves across Google Robot object poses.Source: C. Evaluation in Simulation Environment · p. 6
D. Evaluation in Real-world Environment
Real-world evaluation compares finetuned Octo and OpenVLA models after scaling a real robot dataset with ROSIE or ReBot. ReBot improves average task success rates for both models.
D. Evaluation in Real-world Environment
- 17% for Octo and 20% for OpenVLA are ReBot’s gains in average real-world success rates across four tasks.The evaluation uses 10 trials per task and includes 220 real-world episodes during finetuning.Source: D. Evaluation in Real-world Environment · pp. 6–7, Table II · p. 7
- For the challenging put carrot in blue plate task, where Octo initially had 0% grasp and success rates, ReBot raised grasp rate to 40% and success rate to 20%.Source: D. Evaluation in Real-world Environment · p. 7
V. CONCLUSION AND DISCUSSION
ReBot combines simulated trajectory replay with inpainted real backgrounds to synthesize robot videos and improve VLA performance in simulation and real-world evaluation. The authors identify cross-embodiment settings and more challenging scenarios as directions for future work.
- In SimplerEnv, ReBot improved Octo’s in-domain performance by 7.2% and OpenVLA’s by 21.8%, and their generalization performance by 19.9% and 9.4%, respectively.Source: §V · p. 7
- In real-world evaluation with a Franka Panda, ReBot increased Octo’s success rate by 17% and OpenVLA’s by 20%.Source: §V · p. 7
- ReBot’s VBench temporal consistency and imaging quality scores were 93.0% and 66.4%, versus 96.1% and 70.1% for real robot videos.Source: §V · p. 7
- ReBot replays real trajectories in simulation to diversify manipulated objects, then combines simulated movements with inpainted real backgrounds to synthesize videos.Source: §V · p. 7
- Future work includes diverse data settings such as varying cameras and robots for cross-embodiment learning, and scenarios beyond tabletop manipulation.Source: §V · p. 7