PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboRender: Robot-Oriented Video Generation for Visual Sim-to-Real Transfer

Huang Huang, Wensi Ai, Ziyu Chen, Youhui Wang, Zijian Du, Yang Liu, Jiaolong Yang, Li Fei-Fei, Jiajun Wu

arXiv:2610.09254v1cs.RO

TL;DR

Policies trained in simulation can struggle to transfer because simulated and real visual observations differ, while existing approaches may lose semantic appearance cues or require additional perception modules. RoboRender generates photorealistic RGB trajectories from simulated depth, language, and robot masks, then pairs them with simulator data for policy learning. Across real-world manipulation tasks, its generated data outperforms raw simulation and visual domain randomization.

  • Problem

    Vision-based robot policies face a visual sim-to-real gap, and existing approaches may be scene-specific, discard semantic appearance cues, require extra perception modules, or remain visually mismatched.

    Source: A. Visual Sim-to-Real Transfer · p. 2

  • Method

    RoboRender conditions video generation on simulated depth, language instructions, and robot RGB masks to render photorealistic RGB-action trajectories that retain simulator geometry and robot motion.

    Source: §I · p. 2, A. Video Generation Model · p. 3

  • Results

    RoboRender-generated data yields 7.1× and 3.6× higher real-world success rates than raw simulator RGB and visual domain randomization, respectively.

    Source: §I · p. 2

  • Takeaways & Limitations

    The experiments demonstrate that RoboRender mitigates the visual sim-to-real gap for zero-shot policy transfer.

    Source: §I · p. 2, §VI · p. 8

    RoboRender remains bounded by simulator fidelity in dynamics and contact modeling, making deformable-object tasks especially challenging.

    Source: §VI · p. 8

Abstract

from arXiv · show

Simulation enables large-scale, low-cost robot data generation, but policies trained in simulation often fail to transfer to the real world due to the sim-to-real visual discrepancies. Existing approaches often rely on intermediate representations, which can discard rich semantic information or require additional perception modules at deployment. We address this visual sim-to-real gap with RoboRender, a framework that converts simulated trajectories into photorealistic RGB videos for policy learning. RoboRender trains a robot-oriented video generation model conditioned on simulated depth videos, language instructions, and robot RGB mask videos, preserving simulator geometry, robot motion, and action labels while synthesizing realistic textures, backgrounds, and distractors. The generated RGB videos are paired with simulator-provided states and actions to train policies for zero-shot real-world deployment. On robot video test sets, our video model outperforms depth-conditioned video generation baselines in generation quality. In real-world experiments across pick-and-place, articulated-object manipulation, and mobile manipulation tasks, policies trained on RoboRender-generated data achieve a 71% average success rate, outperforming raw simulation renderings and conventional visual domain randomization by approximately 7.1x and 3.6x, respectively. We further show that policy performance improves with more generated videos per simulation trajectory, increasing opening-task success by 65 percentage points. These results demonstrate that generative video rendering mitigates the visual sim-to-real gap for zero-shot policy transfer. Project website: https://robo-render.github.io/.

I. INTRODUCTION

RoboRender addresses the visual sim-to-real gap by turning simulated trajectories into photorealistic RGB-action data for zero-shot policy transfer. Across manipulation tasks, policies trained on its generated data outperform raw simulation and visual domain randomization.

A. Visual Sim-to-Real Transfer

  • Vision-based manipulation faces a visual sim-to-real gap, while existing approaches can be scene-specific, discard semantic appearance cues, require extra perception modules, or remain visually mismatched.Prior strategies include digital-twin reconstruction, depth or object-state representations, and visual domain randomization.Source: §I · p. 1, A. Visual Sim-to-Real Transfer · p. 2

I. INTRODUCTION

  • RoboRender converts simulated trajectories into photorealistic RGB-action data, conditioning video generation on depth, language, and robot RGB masks while retaining simulator geometry, robot motion, and action labels.The resulting trajectories train policies for zero-shot real-world deployment.Source: §I · p. 2
  • Policies trained on RoboRender-generated data achieve 7.1× and 3.6× higher real-world success rates than raw simulator RGB and visual domain randomization, respectively.The evaluation covers five pick-and-place tasks, two opening tasks, and one mobile manipulation task; generating multiple videos per trajectory improves performance by up to 65 percentage points on opening tasks and 10 points on pick-and-place tasks.Source: §I · p. 2

B. Video Generation for Robot Data Synthesis

RoboRender combines simulator-derived structure and robot-specific conditioning with a video-generation prior to synthesize realistic, multi-view-consistent robot videos. Its design supports prompt-guided appearance changes and distractor insertion while preserving simulated trajectories and action labels.

B. Video Generation for Robot Data Synthesis

  • RoboRender distinguishes itself from prior robot-video synthesis by combining simulator-provided action labels, multi-view depth grounding, distractor insertion, and robot-mask conditioning for unseen-embodiment generalization.The related-work comparison contrasts these features with methods relying on language alone, 2D structural cues, or precollected appearance exemplars.Source: B. Video Generation for Robot Data… · p. 2

A. Video Generation Model

  • The robot-oriented model conditions synthesis on depth for geometry and motion, robot masks for embodiment and motion, language for appearance and semantics, and previous frames for long-horizon generation.During training, condition latents are concatenated with noised RGB latents and stacked across views for diffusion-transformer denoising.Source: A. Video Generation Model · p. 3, Figure 2 · p. 3
  • Masking random depth patches during training lets the model insert prompt-guided distractors by dropping depth over a target region across views, augmenting data with object clutter.The inserted distractors remain consistent across views and are used to make downstream policies more robust to clutter.Source: A. Video Generation Model · p. 3
  • The training-data pipeline prepares aligned 81-frame windows with scene prompts, per-view depth, robot-arm masks, and a pre-window frame from real-robot clips.The final set contains about 130k samples from AgiBot-World-Alpha and DROID, processed at 416×240 resolution.Source: A. Video Generation Model · p. 3
  • RoboRender produces realistic video quality while preserving object consistency across views, unlike the compared depth-conditioned video-generation baselines.Figure 3 compares RoboRender with Cosmos-Transfer2.5 [25] and Wan-Fun-Control [24] across time steps and views.Source: Figure 3 · p. 4

B. Robot Simulation Data Generation

RoboRender builds simulated robot trajectories and converts them into photorealistic training videos. Its video model outperforms generation baselines, maintains quality in autoregressive rollouts, and is preferred for cross-view consistency.

B. Robot Simulation Data Generation

  • The simulation pipeline records multi-view observations, proprioception, actions, and robot masks while scripted planners execute and retain successful task episodes.Task primitives cover pick-and-place, articulated-object manipulation, and mobile manipulation.Source: B. Robot Simulation Data Generation · p. 4

C. RoboRender Data Generation

  • RoboRender conditions generation on simulated depth, robot RGB masks, and task prompts, then pairs synthesized RGB videos with simulator-provided states and actions.Prompts control textures, backgrounds, and scene semantics; depth crops and prompt descriptions can add distractors while preserving geometry and actions.Source: C. RoboRender Data Generation · p. 4

IV. VIDEO GENERATION EXPERIMENTS

  • RoboRender outperforms all video-generation baselines across metrics and settings, including Cosmos-Transfer2.5 despite its higher-resolution inputs.For sim-depth inputs without ground-truth RGB, a human study also preferred RoboRender over all baselines.Source: Table I · p. 4, §IV · p. 5
  • Across 15-second autoregressive rollouts, RoboRender leads both baselines at every sampled time step in PSNR and LPIPS.The reported rollout comparison samples every three seconds.Source: §IV · p. 5, Figure 4 · p. 5
  • Human raters prefer RoboRender over all baselines for cross-view consistency, and distillation reduces its sampler from 50 steps to 6.The distilled model generates a three-view clip in 4.29 seconds on one H200.Source: §IV · p. 5

V. ROBOT PHYSICAL EXPERIMENTS

Physical experiments cover pick-and-place, articulated-object opening, and mobile manipulation, using multiple robot embodiments and task-specific randomized setups.

V. ROBOT PHYSICAL EXPERIMENTS

  • Experiments include five pick-and-place tasks, two opening tasks, and one mobile task, plus a YAM marker-and-bowl pick-and-place task.The main tasks use Franka with a Robotiq gripper or R1Pro; YAM is an additional embodiment.Source: §V · p. 5
  • The video model was trained on DROID and AgiBot robot data, while R1Pro and YAM are unseen embodiments during training.The setups use external, wrist-mounted, or egocentric cameras depending on the robot.Source: §V · p. 5
  • Pick-and-place trials vary object and distractor positions, opening trials vary robot starting pose, and mobile trials vary robot and radio positions.Pick-and-place object positions are held constant across policies for fair comparison.Source: §V · p. 5

A. Policy Training

The policy-training data combines simulated trajectories and generated RGB videos, while policies share a common action-chunk training setup.

A. Policy Training

  • The dataset includes 1,000 trajectories per pick-and-place task, 2,000 for mobile manipulation, 2,000 for drawer opening, and 1,000 for fridge opening.DROID pick-and-place and mobile data come from OmniGibson, while opening and YAM data come from MuJoCo.Source: A. Policy Training · p. 5
  • The generated dataset pairs photorealistic videos with simulator states and actions, while DROID pick-and-place data also includes inserted distractor patches.The distractors are added to external and wrist views using cropped depth patches.Source: A. Policy Training · pp. 5–6
  • Policies finetune pretrained π0.5 for 40,000 steps to predict 32-step action chunks, executing the first 16 predicted steps at inference.One joint policy covers all pick-and-place tasks; opening and mobile manipulation use separate policies.Source: A. Policy Training · p. 6

B. Experiments and Analysis

RoboRender achieves the strongest reported real-world success across task categories, with additional evidence of gains from visual diversity, clutter robustness, and transfer to unseen embodiments.

B. Experiments and Analysis

  • 71% average success across tasks is about 3.6× higher than Sim + DR and 7.1× higher than Sim.RoboRender is the only method with non-zero success on the opening tasks, where handles require fine visual recognition.Source: B. Experiments and Analysis · p. 6, Figure 7 · p. 6, Table III · p. 7
  • RoboRender shows only a small success-rate gap between distractor and no-distractor evaluations, while the baselines degrade substantially with distractors.In the no-distractor setting, Sim, Sim + DR, and RoboRender achieve 48%, 70%, and 80%, respectively.Source: B. Experiments and Analysis · pp. 6–7, Figure 7 · p. 6
  • On unseen R1Pro and YAM embodiments, RoboRender substantially outperforms Sim and Sim + DR.The video model was trained only on DROID and AgiBot embodiments.Source: B. Experiments and Analysis · p. 7
  • RoboRender-trained policies achieve substantially higher success than depth-only policies across all three evaluated setups.The comparison uses depth estimated from RGB at inference for the depth-only policies.Source: B. Experiments and Analysis · p. 7, Table IV · p. 7
  • On two opening tasks, policies trained with RoboRender videos outperform those trained with Cosmos and Wan, consistent with the video-quality comparison.The compared methods generate the same number of videos from the same simulation data.Source: B. Experiments and Analysis · p. 7, Table V · p. 7
  • Opening-task average success rises from 0% at 1× generation to 45% at 1.5× and 65% at 2× per trajectory.Pick-and-place success rises from 48% at 1× to 58% at 2× and saturates at 3×; these comparisons use half the simulation trajectories.Source: B. Experiments and Analysis · pp. 7–8, Figure 8 · p. 7

VI. CONCLUSION AND LIMITATIONS

RoboRender turns simulated trajectories into photorealistic RGB-action-state data for zero-shot transfer, with policies outperforming raw-simulation and domain-randomized training across manipulation tasks. Its scope remains limited by simulator dynamics and contact fidelity, and broader validation is needed.

VI. CONCLUSION AND LIMITATIONS

  • Across pick-and-place, articulated-object opening, and mobile manipulation, RoboRender policies achieve about 7.1× and 3.6× the average success rate of raw-simulation and visual-domain-randomized policies, respectively.The framework pairs simulator states and actions with generated RGB videos while preserving geometry and robot motion.Source: §VI · p. 8
  • RoboRender’s robot-oriented video model conditions on depth, language prompts, and robot RGB masks to synthesize realistic textures, lighting, backgrounds, and distractors.Generating diverse videos from each simulated trajectory further improves policy learning efficiency without proportional increases in simulated data.Source: §VI · p. 8
  • Transfer remains bounded by simulator dynamics and contact fidelity, making deformable-object tasks especially challenging.The paper also calls for broader validation across robot morphologies, camera configurations, and real-world environments, and for scaling to more complex long-horizon tasks.Source: §VI · p. 8

APPENDIX

The appendix describes a preprocessing pipeline that turns robot demonstrations into paired video-model training samples, with aligned prompts and visual conditions across views. The resulting corpus spans many scenes and manipulation task types.

A VIDEO MODEL TRAINING DATA CURATION

  • Each demonstration is converted into an 81-frame window with aligned scene prompts, per-view depth, robot-arm RGB and masks, and a pre-action frame.The pre-action frame anchors appearance and supports autoregressive long-video generation at inference.Source: A VIDEO MODEL TRAINING DATA CURATION · p. 10
  • A two-pass captioning process drafts a first-frame caption, then cross-verifies objects across non-wrist views and outputs structured scene metadata.The passage says this reduces hallucination compared with single-pass captioning.Source: A VIDEO MODEL TRAINING DATA CURATION · p. 10
  • Per-view depth combines relative-depth estimates from Video-Depth-Anything and, when stereo recordings are available, FoundationStereo, with sources randomly mixed during training.The depth outputs use inverse-depth normalization to reduce estimator-specific inductive bias.Source: A VIDEO MODEL TRAINING DATA CURATION · p. 10
  • SAM3 segments robot arms and grippers across views, providing explicit embodiment grounding for pose and motion.The masks isolate the robot from the rest of the scene.Source: A VIDEO MODEL TRAINING DATA CURATION · p. 10
  • The processed corpus contains 126,950 paired samples across 39,692 scenes and 16 manipulation task types.Pick-and-place accounts for 47.8% and fold/deform for 19.1%; the source datasets contribute approximately 70k AgiBot-World-Alpha and 60k DROID samples.Source: A VIDEO MODEL TRAINING DATA CURATION · p. 10

B VIDEO GENERATION BASELINE COMPARISON

RoboRender’s dedicated multi-view training produces realistic videos with cross-view consistency that stacked-input baselines do not achieve. The appendix also reports faster inference with TeaCache while retaining visually similar quality, and a six-step distilled variant.

B VIDEO GENERATION BASELINE COMPARISON

C ON ACCELERATING VIDEO MODEL GENERATION

D DISTRACTOR INSERTION

RoboRender inserts controllable distractors by removing depth constraints in selected regions, while simulation trajectories are generated by a task-and-motion planning pipeline. The resulting data combines realistic appearance with preserved task geometry and robot actions.

D DISTRACTOR INSERTION

  • Cropping regions from simulation depth removes geometric constraints there, letting RoboRender synthesize plausible distractors while preserving task-relevant geometry.The inserted objects remain consistent over time and add visual diversity without geometric artifacts in task-relevant regions.Source: D DISTRACTOR INSERTION · p. 13

E ROBOT SIMULATION DATA GENERATION

  • A simulation pipeline instantiates scenes, robots, and synchronized multi-view cameras, then records RGBD observations, proprioception, actions, and robot segmentation masks.The pipeline is built on MolmoSpaces and CuRobo.Source: E ROBOT SIMULATION DATA GENERATION · pp. 13–14
  • Across eight evaluation tasks, RoboRender training data supplies photorealistic textures and lighting while preserving geometry, unlike raw simulation and domain-randomized renderings.The figure compares the three data-generation approaches visually.Source: Figure 15 · p. 14
  • A scripted planner selects feasible grasps and executes task-specific motion primitives, retaining successful trajectories for policy training.Success is defined by object placement, object lifting, or task-specific joint displacement; episodes fail after retry limits or a 300-step timeout.Source: E ROBOT SIMULATION DATA GENERATION · p. 14

F POLICY TRAINING DATA VISUALIZATION

RoboRender produces photorealistic observations that preserve scene geometry, supporting zero-shot policy transfer across manipulation tasks. The evaluation varies physical configurations by task while keeping pick-and-place object positions consistent across policies.

F POLICY TRAINING DATA VISUALIZATION

  • Raw simulation has synthetic textures, uniform lighting, and unrealistic backgrounds, while domain randomization produces visually incoherent frames that still miss real-world material and lighting statistics.Source: F POLICY TRAINING DATA VISUALIZATION · p. 14
  • RoboRender generates realistic textures, reflections, and lighting while preserving scene geometry, and its data supports zero-shot transfer across opening and mobile robot tasks.Source: F POLICY TRAINING DATA VISUALIZATION · p. 15

G PHYSICAL SETUP

  • Pick-and-place trials use the same randomized object positions across policies; opening trials vary the robot’s initial pose, while radio pickup varies the mobile robot and radio positions.Source: G PHYSICAL SETUP · p. 15
Loading 2610.09254v1…