Source-linked AI summary
Real2Render2Real: Scaling Robot Data Without Dynamics Simulation or Robot Hardware
Justin Yu, Letian Fu, Huang Huang, Karim El-Refai, Rares Andrei Ambrus, Richard Cheng, Muhammad Zubair Irshad, Ken Goldberg
TL;DR
Robot-learning datasets remain limited by the cost, speed, embodiment specificity, and physical requirements of human teleoperation. R2R2R generates robot training data from a smartphone object scan and one human demonstration using reconstruction, object-motion tracking, and parallel rendering, and its policies can match teleoperation-trained performance while avoiding dynamics simulation and robot hardware.
Problem
Human teleoperation data collection is costly, embodiment-specific, and far smaller than the corpora used to train frontier language-and-vision models.
Method
R2R2R reconstructs rigid or articulated objects from smartphone scans, extracts object motion from one human video, randomizes trajectories, and parallel-renders RGB-action demonstrations without dynamics simulation or robot hardware.
Results
Policies trained on one human demonstration can match policies trained on 150 human teleoperation demonstrations across 1,050 physical robot evaluations.
Takeaways & Limitations
R2R2R provides a scalable alternative to human teleoperation for training effective robot policies on the evaluated tasks.
Takeaways & Limitations
R2R2R does not support deformable objects or non-prehensile strategies such as pushing, toppling, or sliding.
Abstract
from arXiv · showhide
Scaling robot learning requires vast and diverse datasets. Yet the prevailing data collection paradigm-human teleoperation-remains costly and constrained by manual effort and physical robot access. We introduce Real2Render2Real (R2R2R), a novel approach for generating robot training data without relying on object dynamics simulation or teleoperation of robot hardware. The input is a smartphone-captured scan of one or more objects and a single video of a human demonstration. R2R2R renders thousands of high visual fidelity robot-agnostic demonstrations by reconstructing detailed 3D object geometry and appearance, and tracking 6-DoF object motion. R2R2R uses 3D Gaussian Splatting (3DGS) to enable flexible asset generation and trajectory synthesis for both rigid and articulated objects, converting these representations to meshes to maintain compatibility with scalable rendering engines like IsaacLab but with collision modeling off. Robot demonstration data generated by R2R2R integrates directly with models that operate on robot proprioceptive states and image observations, such as vision-language-action models (VLA) and imitation learning policies. Physical experiments suggest that models trained on R2R2R data from a single human demonstration can match the performance of models trained on 150 human teleoperation demonstrations. Project page: https://real2render2real.com
1 Introduction
R2R2R addresses robot-learning data scarcity by generating diverse robot training data from smartphone object scans and a single human demonstration, without dynamics simulation or robot hardware. Its data is compatible with modern policies, and one demonstration can match the performance of 150 teleoperation demonstrations.
- Motivation: Robot teleoperation datasets remain constrained by cost, collection speed, embodiment specificity, and scale relative to language-and-vision training corpora.The largest human teleoperation datasets are over 100,000× smaller than corpora used for frontier LLMs and VLMs.
- Research question: R2R2R asks whether robot vision-action data can scale computationally without dynamics simulation or human teleoperation.The framework is motivated by pseudo-labeling successes in other vision-language subfields.
- Approach: R2R2R reconstructs object geometry and appearance, tracks 6-DoF part trajectories, and synthesizes robot executions from smartphone scans and one human video.It supports rigid and articulated objects and uses differential inverse kinematics with randomized object initializations.
- Approach: The pipeline renders varied observations under changes in lighting, camera pose, and object placement for vision-language-action and imitation-learning policies.The resulting data is directly compatible with transformer-based and diffusion-based policy architectures using RGB and proprioceptive input.
- Results: Policies trained from one R2R2R human demonstration can match policies trained on 150 human teleoperation demonstrations across 1,050 physical robot evaluations.The contribution statement also reports significantly less data-generation time for R2R2R.
2 Related Work
Prior robot-data approaches use industrial logs, teleoperation, real robots, simulation, or real-to-sim pipelines, each retaining scalability, hardware, visual-gap, or modeling constraints. R2R2R instead combines object-centric motion extraction, 3D reconstruction, and parallel rendering to generate varied robot data from one demonstration.
- Existing collection paradigms: Industrial logs scale with production throughput but are task- and embodiment-specific, whereas teleoperation offers diversity but remains bottlenecked by human effort and real-time collection.These limitations motivate alternative robot data-generation paradigms.
- Procedural and simulation generation: Procedural methods reduce human intervention but often still require real robot hardware, while simulation removes hardware dependence through parallelized generation.Simulation approaches commonly use motion primitives, reinforcement learning, trajectory optimization, or motion planning.
- R2R2R comparison: R2R2R compares favorably by requiring no teleoperation, reward engineering, reinforcement learning, or accurate asset physics modeling while supporting multiple robot embodiments.It generates object-centric demonstrations directly from human-object interaction video.
- Real-to-synthetic approaches: Real-to-synthetic methods repurpose real RGB data but still require human teleoperation and cannot generate additional diverse trajectories beyond the initial demonstrations.Real-to-sim-to-real methods reduce visual gaps through digital twins or digital cousins but vary in their reliance on teleoperation.
- R2R2R comparison: R2R2R uses 3D Gaussian Splatting and part-level decomposition to reconstruct rigid or articulated objects, then converts assets for scalable parallel rendering.The method also generates multiple varied trajectories from a single demonstration without collision modeling or detailed meshes.
3 Assumptions
R2R2R assumes tabletop, quasistatic manipulation of rigid or articulated objects with low-specularity surfaces, limited mutual occlusion, and approximately known camera poses relative to the robot.
- Object and motion assumptions: The framework covers rigid and articulated objects manipulated prehensilely on a tabletop under quasistatic conditions.These assumptions define the current operating regime of the method.
- Visual assumptions: Objects are assumed to have low-specularity surfaces to support robust geometry reconstruction and visual feature extraction.
- Observation assumptions: Human demonstrations must avoid configurations producing complete mutual occlusion, and approximate camera poses relative to the robot must be available.The camera-pose assumption enables observations from nearby viewpoints during data collection.
4 Method
R2R2R converts smartphone object scans and one human demonstration into diverse robot demonstrations through trajectory extraction, augmentation, and parallelized rendering. It combines 3D object representations, motion interpolation, inverse kinematics, and domain randomization to produce scalable visual-action data.
- Pipeline: R2R2R extracts rigid or articulated object assets and part trajectories from smartphone captures, then renders RGB-action demonstrations in IsaacLab.The pipeline comprises real-to-sim extraction, augmentation, and GPU-parallel rendering.
- Asset generation: 3DGS-based grouping supports object-level and part-level decomposition, including articulated components, while mesh conversion preserves compatibility with trajectory reconstruction and rendering engines.The conversion requires no fiducials or hardware beyond a smartphone camera.
- Trajectory augmentation: Trajectory interpolation adapts one human demonstration to varied object start and end poses while preserving its semantic intent.R2R2R uses spatial normalization for translation and Slerp for keyframe orientations.
- Robot execution: R2R2R identifies grasped parts from hand-keypoint distances and solves differential inverse kinematics for smooth joint trajectories across pre-grasp, grasp, and post-grasp phases.The method does not model object dynamics or simulate physics interactions.
- Rendering and augmentation: Randomized lighting, camera extrinsics, and object initial poses create diverse visual contexts without changing the underlying kinematic rollout.On one NVIDIA RTX 4090, rendering averages 51 demonstrations per minute versus 1.7 via human teleoperation, a 27× speedup.
5 Experiments
Physical experiments evaluate R2R2R-trained policies across five manipulation tasks and compare performance scaling with human teleoperation data. R2R2R performance improves with dataset size and becomes comparable to teleoperation at larger scales.
- Evaluation setup: The study conducts 1,050 physical evaluations on an ABB YuMi across five tasks using policies trained on rendered or teleoperated data.Rendered training sets contain 50, 100, 150, or 1,000 trajectories, while teleoperation provides up to 150 per task.
- Evaluation setup: The tasks span single-object picking, multi-object interaction, articulated manipulation, and bimanual coordination.Examples include picking a toy tiger, placing a mug on a coffee maker, turning off a faucet, opening a drawer, and lifting a package with both hands.
- Statistical comparison: Across evaluated tasks and models, no statistically significant performance differences are found between R2R2R-trained and human-teleoperation-trained policies.TOST indicates observed differences fall within a ±5% margin.
6 Conclusion
R2R2R is presented as a practical, scalable pipeline for real-world dexterous manipulation policy learning. Experiments suggest its generated data can scale with volume and perform comparably to teleoperated demonstrations.
- Conclusion: R2R2R generates diverse robot training data from a multi-view object scan and a single human demonstration without teleoperation, robot hardware, or dynamics simulation.The pipeline supports realistic rendering and robot-policy training across manipulation settings.
- Conclusion: Across five robotic tasks, policies trained on R2R2R data scale with data volume and perform comparably to policies trained on teleoperated demonstrations.The conclusion characterizes R2R2R as practical and scalable for real-world dexterous manipulation policy learning.
7 Limitations
R2R2R remains limited by reconstruction fidelity, collision-unaware trajectory synthesis, restricted manipulation scope, gripper compatibility, and tracking robustness. These constraints bound physical realism and generality despite scalable data generation.
- Reconstruction and Simulation Fidelity: Vision-based reconstruction provides high-fidelity appearance but may lack watertight, physically plausible geometry, limiting contact-rich interaction modeling.R2R2R therefore omits physics simulation, restricting friction, compliance, and force-feedback modeling.
- Scene Diversity and Collision Awareness: Geometric interpolation ignores distractors and obstacles, so synthesized trajectories may intersect scene geometry and produce physically infeasible plans.Fast motion planning is proposed to improve collision avoidance, especially in cluttered or multi-object scenes.
- Scope of Manipulation Tasks: The framework supports only rigid and articulated objects with prehensile manipulation, excluding deformable-object handling and pushing, toppling, or sliding.These excluded interactions require accurate metric depth and finer physical modeling.
- Grasping Generality: Antipodal grasp sampling currently limits R2R2R to parallel-jaw grippers, excluding multi-fingered and anthropomorphic hands.More complex end-effectors require richer grasp representations and contact models.
- Tracking Robustness: Tracking can fail under fast motion, heavy occlusion, poor texture, or reflective surfaces, degrading reconstructions and pose tracks.Resulting invalid grasps or trajectories may transfer poorly to the real world.
- Future Direction: Addressing physical modeling, context-aware planning, manipulation breadth, and reconstruction robustness is presented as a path toward more general and reliable robot learning at scale.
Contents for This Section
This section reports physical policy evaluations, trajectory and appearance ablations, mixed-data training, qualitative rollouts, and implementation details across several manipulation tasks. The results include both successful R2R2R-only rollouts and failure modes associated with the Franka gripper.
- Raw Evaluation Results: Table 2 compares Diffusion Policy and π0-FAST trained exclusively on human teleoperation or R2R2R-generated data across physical task evaluations.Each policy is evaluated on 15 trials per task using binary success.
- Trajectory Interpolation: 80.0% to 0.0%: π0-FAST success falls when trajectory interpolation is disabled, while Diffusion Policy drops from 53.3% to 6.7% on mug placement.The ablation uses 1,000 R2R2R demonstrations.
- Visual Augmentation: Aggressive background and tabletop texture augmentation consistently reduces performance for both π0-FAST and Diffusion Policy on mug placement.The result suggests that overly strong appearance perturbations require careful balancing.
- Sim-and-Real Co-training: Diffusion Policy co-training improves success from 40.0% to 86.7% when combining 1,000 R2R2R demonstrations with 150 human teleoperation demonstrations.π0-FAST co-training reaches 73.3%, on par with either source alone.
- Qualitative Rollouts: Physical rollout figures depict real RGB inputs from successful Diffusion Policy and π0-FAST evaluations trained exclusively on R2R2R synthetic data.
- Task-Specific Qualitative Results: The faucet demonstration differs in manipulation strategy: teleoperation pushes the handle non-prehensilely, whereas R2R2R uses prehensile grasping and twisting.This reflects the current support for prehensile manipulation.
- Implementation Details: The evaluation adapts π0-FAST to predict delta joint positions because R2R2R records joint positions rather than joint velocities.Blocking control with temporal ensembling is used to improve execution accuracy.
- Failure Modes: Franka gripper failures include table collision during grasping, off-center contact causing mug slippage, and imprecise placement.A smaller Robotiq 2F-85 gripper is recommended for future experiments.
9.4 Qualitative Ablations
Qualitative ablations isolate the effects of R2R2R’s visual and trajectory-diversity mechanisms. Removing interpolation fixes the mug configuration, while removing randomized lighting leaves only uniform skybox illumination.
- R2R2R Augmentations: The default R2R2R pipeline augments sphere lighting, camera pose, robot initial joints, and object initialization through manual parameter ranges.Both camera views are shown in the qualitative pipeline figure.
- Trajectory Interpolation: Without trajectory interpolation, multiple rigid bodies follow only the tracked demonstration trajectories, leaving the mug fixed relative to the coffee maker.Additional trajectory diversity would require more tracked demonstration videos.
- Random Lighting: Without randomized sphere lights, rendering uses only uniform skybox or dome-light illumination.
9.5 Upfront Processing Time Until Generation
This section covers preprocessing and generation-time accounting, training configurations, and statistical comparison between R2R2R and teleoperation data. Equivalence testing finds similar performance patterns but does not establish statistical equivalence for any individual task.
- Upfront Processing: Upfront preprocessing converts demonstration videos and scanned assets into renderer-ready formats through segmentation, tracking, meshification, and asset import.
- Generation Time: Table 7 compares time for one human operator to collect 150 teleoperation demonstrations against R2R2R generation of 1,000 synthetic demonstrations.R2R2R generation times exclude upfront preprocessing.
- Training Configuration: Training configurations include diffusion-policy hyperparameters and π0-FAST fine-tuning with LoRA.
- Statistical Comparison: TOST evaluates whether absolute success-rate differences fall within a ±5% practical-equivalence margin.The lower and upper tests assess both directional differences between R2R2R and teleoperation.
- Statistical Comparison: No individual task satisfies both TOST conditions, although the global results show no strong evidence that either R2R2R or teleoperation outperforms the other.The paper interprets this pattern as similar effectiveness without provable equivalence under the chosen threshold.