PaperlayerView exact revision on arXiv

Source-linked AI summary

Real2Edit2Real: Generating Robotic Demonstrations via a 3D Control Interface

Yujie Zhao, Hongwei Fan, Di Chen, Shengcong Chen, Liliang Chen, Xiaoqi Li, Guanghui Ren, Hao Dong

arXiv:2512.19402v2cs.ROcs.CVcs.GR

TL;DR

Collecting diverse demonstrations for spatial generalization in manipulation is costly. Real2Edit2Real reconstructs geometry, edits trajectories, and synthesizes multi-view demonstrations through 3D-controlled video generation. Across four real-world tasks, policies trained on generated data from 1–5 demonstrations matched or surpassed those trained on 50 real demonstrations, with data efficiency improved by up to 10–50×.

  • Problem

    Spatial generalization in manipulation requires diverse demonstrations, making data collection costly for policies handling varied object arrangements.

    Source: §1 · p. 2

  • Method

    Real2Edit2Real reconstructs metric-scale geometry, edits point clouds and robot poses to create trajectories with reliable depth, then generates multi-view videos conditioned on depth and auxiliary signals.

    Source: §1 · p. 2

  • Results

    Across four real-world tasks, policies trained on data generated from 1–5 source demonstrations matched or surpassed those trained on 50 real-world demonstrations, improving data efficiency by up to 10–50×.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Height and texture editing demonstrate the framework’s extensibility and its potential as a unified engine for scalable data generation.

    Source: §5 · p. 8

    Video generation is a computational bottleneck, particularly in resource-constrained scenarios, and object generalization is limited for articulated or deformable objects.

    Source: §12 · p. 18

Abstract

from arXiv · show

Recent progress in robot learning has been driven by large-scale datasets and powerful visuomotor policy architectures, yet policy robustness remains limited by the substantial cost of collecting diverse demonstrations, particularly for spatial generalization in manipulation tasks. To reduce repetitive data collection, we present Real2Edit2Real, a framework that generates new demonstrations by bridging 3D editability with 2D visual data through a 3D control interface. Our approach first reconstructs scene geometry from multi-view RGB observations with a metric-scale 3D reconstruction model. Based on the reconstructed geometry, we perform depth-reliable 3D editing on point clouds to generate new manipulation trajectories while geometrically correcting the robot poses to recover physically consistent depth, which serves as a reliable condition for synthesizing new demonstrations. Finally, we propose a multi-conditional video generation model guided by depth as the primary control signal, together with action, edge, and ray maps, to synthesize spatially augmented multi-view manipulation videos. Experiments on four real-world manipulation tasks demonstrate that policies trained on data generated from only 1-5 source demonstrations can match or outperform those trained on 50 real-world demonstrations, improving data efficiency by up to 10-50x. Moreover, experimental results on height and texture editing demonstrate the framework's flexibility and extensibility, indicating its potential to serve as a unified data generation framework. Project website is https://real2edit2real.github.io/.

1. Introduction

Real2Edit2Real addresses the cost of collecting diverse demonstrations for spatial generalization by connecting 3D editing with 2D multi-view video generation. It reconstructs geometry, edits trajectories, and generates demonstrations, with reported gains in data efficiency across four real-world tasks.

1. Introduction

  • Spatial generalization remains a bottleneck for policy robustness, while randomly arranged manipulation objects require costly demonstrations covering diverse spatial configurations.The paper motivates synthesis from limited existing data as a way to reduce repetitive collection.Source: §1 · p. 2

2.1. Demonstration Generation

  • Prior demonstration-generation methods face visual or physics gaps, require object assets, or cannot augment RGB images for 2D policies.The paper positions its approach as generating real-world 2D multi-view videos with novel trajectories while preserving visual realism and interaction fidelity.Source: §1 · p. 2

1. Introduction

  • Policies trained on data generated from 1–5 source demonstrations matched or surpassed those trained on 50 real-world demonstrations, improving data efficiency by up to 10–50×.The experiments cover four real-world manipulation tasks; height and texture editing also demonstrate flexibility and extensibility.Source: §1 · p. 2

2.2. Geometry Reconstruction

  • The geometry-reconstruction approach fine-tunes VGGT on robotic data, combining simulated multi-view depth and camera poses with real-world geometry metrics.The aligned geometry prediction supports high-quality point clouds and reliable depth maps for demonstration synthesis.Source: §2.2 · p. 3

2.3. Video Generation in Robotics

  • Earlier robotics video-generation work neglects spatial generalization; Real2Edit2Real generates spatially augmented multi-view demonstrations and can combine with first-frame editing.Source: §2.3 · p. 3

3. Method

Real2Edit2Real reconstructs metric-scale geometry, edits point clouds and trajectories with depth correction, then uses multi-view video generation controlled by depth and auxiliary signals.

3.1. Overview

  • The framework takes a source demonstration and robot and camera models as input, then augments it with novel object configurations and trajectories.Its three components are metric-scale geometry reconstruction, depth-reliable spatial editing, and 3D-controlled video generation.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2. Metric-scale Geometry Reconstruction

  • Hybrid VGGT fine-tuning combines simulated camera-pose supervision with real and simulated depth data to predict metric-scale geometry in humanoid scenes.Real depth can be noisy, while simulated depth is geometrically precise but may have scale deviations; the training recipe uses their complementary properties.Source: §3.2 · pp. 3–4

3.3. Depth-reliable Spatial Editing

  • Spatial editing relocates object and robot point clouds during skill segments, plans new motion segments, and corrects robot poses to preserve kinematic validity.Background inpainting and depth filtering mitigate holes and noise; re-rendering arm depth from joint states avoids rigid-body artifacts.Source: §3.3 · pp. 4–5

3.4. 3D-Controlled Video Generation

  • The video model uses intra-view and cross-view attention to capture per-view context and multi-view correspondence while reducing computation relative to global attention at every layer.Source: §3.4 · p. 5
  • Depth is concatenated with image latents as the 3D control interface; Canny edges, action maps, and ray maps provide auxiliary conditioning.The paper assigns these auxiliary signals roles in sharpening object boundaries, grounding motion, and enhancing multi-view consistency, respectively.Source: §3.4 · p. 5, Figure 3 · p. 5
  • Policies trained on data generated from 1–5 source demonstrations can match or outperform those trained on 50 real-world demonstrations, improving data efficiency by up to 10–50×.Source: Table 1 · p. 6
  • Smooth relocation interpolates object translation and rotation before manipulation, turning object placement editing into video generation within the same framework.Source: §3.4 · p. 6

4. Experiments

Experiments evaluate Real2Edit2Real across four real-world manipulation tasks, including policy learning, height and texture editing, component ablations, and generated-video visualizations. Policies trained on generated data from few source demonstrations match or outperform those trained on 50 real demonstrations.

4.1. Implementation Details

  • Across four real-robot tasks, evaluation trains on 200 generated demonstrations from sampled sources; model and policy training used 8 H100 GPUs.The experiments cover single- and dual-arm manipulation and use Go-1 and π0.5 policies.Source: §4.2 · p. 6, §4.1 · p. 6

4.3. More Applications

  • 80% success was achieved using generated demonstrations at tabletop and platform heights, whereas 20 real tabletop demonstrations failed at platform height.Table 2 compares Go-1 height generalization.Source: Table 2 · p. 7, §4.3 · p. 7

4.2. Gen2Real Policy Learning

  • With five source demonstrations, Go-1 reached 78.8% and π0.5 81.3% average success, exceeding 50-real-demo policies by 17.5% and 20%, respectively.The study reports data efficiency gains of 10–50× overall.Source: §4.2 · p. 7

4.3. More Applications

  • Generated demonstrations with different desktop textures improve policy robustness to texture variation.The texture experiments vary the desktop background.Source: Table 3 · p. 7, §4.3 · p. 7

4.4. Ablation Study

  • Our reconstruction yields the cleanest point clouds and most accurate camera poses; real data have inaccurate poses, while VGGT has clutter and noise.Source: §4.4 · pp. 7–8
  • Without robot pose correction, erroneous depth maps produce blurry, inconsistent videos; correction makes depth kinematically consistent and robot motions realistic.Source: §4.4 · p. 8
  • Without smooth object relocation, placements can be unusable; the module enables precise placement at target locations.Source: §4.4 · p. 8

4.5. Visualization

  • Generated videos across all four tasks relocate objects, synthesize the corresponding manipulation trajectories, and maintain multi-view consistency and realistic appearance.Source: §4.5 · p. 8

5. Conclusion

Real2Edit2Real links 3D editability with 2D visual data to generate manipulation demonstrations. Across four real-world tasks, policies trained with generated data from 1–5 demonstrations match or surpass those trained on 50 real demonstrations, with further results on height and texture editing.

  • Across four real-world tasks, generated data from 1–5 demonstrations lets policies match or surpass those trained on 50 real demonstrations, with data efficiency improving up to 10–50×.Source: §5 · p. 8
  • The framework links 3D editability with 2D visual data through metric-scale reconstruction, depth-reliable spatial editing, and 3D-controlled video generation.Source: §5 · p. 8
  • Height and texture editing results highlight the framework’s extensibility and its potential as a unified engine for scalable data generation.Source: §5 · p. 8

Supplementary Material

The supplied passage identifies Table 4 as a comparison of one-to-many demonstration generation methods.

Supplementary Material

  • Table 4 compares other one-to-many demonstration generation methods.Source: Table 4 · p. 13

7. Contribution Clarification

The comparison distinguishes Real2Edit2Real from prior demonstration-generation approaches by its direct use of original RGB observations without a simulator. It combines texture and trajectory augmentation in a unified framework for VLA training.

7. Contribution Clarification

  • Simulation-based MimicGen and SkillMimicGen require simulators and scene or object assets, while 3D-Gaussian-splatting methods require dense scanning.Point-cloud editing methods depend on depth sensors, and video-generation methods enhance visual appearance without diversifying object placements or robot trajectories.Source: §7 · p. 13
  • Real2Edit2Real directly augments original RGB observations without a simulator, improving scalability relative to approaches with those dependencies.It generates new textures and trajectories for VLA training in one framework.Source: §7 · p. 13

8. Real2Edit2Real Implementation Details

The implementation details cover metric-scale geometry reconstruction, depth-reliable spatial editing, and 3D-controlled video generation. The editing pipeline completes backgrounds and aligns reconstructed geometry before producing revised depth and action-related outputs.

8.1. Metric-scale Geometry Reconstruction

  • A hybrid real-and-simulated training paradigm fine-tunes VGGT to improve metric-scale depth maps and camera-pose prediction in humanoid scenes.The reconstruction targets sparse views from the robot’s head and wrist cameras.Source: §8.1 · p. 13

8.2. Depth-reliable Spatial Editing

  • Background inpainting removes foreground objects and the robot before Metric-VGGT reconstructs background geometry, followed by point-cloud alignment to correct scale inconsistencies.The identified inpainting prompt uses “xxx” to denote manipulated objects.Source: §8.2 · pp. 13–14, Figure 10
  • The spatial-editing pipeline separates foreground objects with Grounded-SAM and robot arms through forward kinematics, then generates a novel depth sequence and revised motion-related states.The described algorithm takes source point clouds, joint states, action trajectories, and camera poses as inputs.Source: §8.2 · p. 14, §8.3 · p. 15

8.3. 3D-Controlled Video Generation

  • Video-generation training uses 7K episodes across 64 Agibot-World tasks and globally normalizes depth across all three views within each training chunk.Depth and Canny-edge conditions are independently dropped with probability 0.5 and jointly dropped with probability 0.1 during training.Source: §8.3 · pp. 15–16

9. Experiment Details

The experiments use real-robot workspaces and object sets, with the testing laboratory absent from framework training data. The data-generation setup also specifies spatial-editing and video-generation parameters.

Experiment Details

  • The four task workspaces are bounded by the maximum range in which the robot’s kinematics can perform the intended tasks.Figure 12 visualizes the manipulation workspaces.Source: §9 · p. 16, Figure 12 · p. 16
  • Newly purchased objects were used to minimize potential overlap with the training set, and the real-world testing laboratory was absent from training data.Figure 13 visualizes the manipulation objects.Source: §9 · p. 16, Figure 13 · p. 17
  • Generation used a 40 cm × 40 cm augmentation region and 30°–60° object rotations, with diffusion step 6 and memory length 4 for video generation.The augmentation region is typically centered around the object’s original location.Source: §9 · p. 16

10. Additional Experiments

The additional experiments assess video realism, policy performance, generation time, scaling, and control-condition ablations. Generated demonstrations improve policy success as their quantity grows, while removing depth or edge controls degrades generated-video quality.

10.1. Quantitative Results of Video Generation

  • The video-generation module produces significantly enhanced visual realism compared with GE-Sim in conditional I2V evaluation.Table 7 reports standard metrics for the comparison; the supplied passage does not provide their values.Source: §10.1 · p. 17, Table 7 · p. 17

10.2. Diffusion Policy on Mug to Basket

  • On Mug to Basket, Diffusion Policies trained with data generated from only 1–5 source demonstrations can surpass those trained with 50 real demonstrations.The evaluation uses a ViT-S encoder initialized with DINO-v3 weights.Source: §10.2 · p. 17, Table 8 · p. 17

10.3. Generation Time Analysis

  • Multi-GPU acceleration reduces generation time, allowing Real2Edit2Real to surpass manual teleoperation success rates in a short time.Figure 14 plots success rate against total data-generation time in seconds on a logarithmic scale.Source: §10.3 · p. 17, Figure 14 · p. 17

10.4. Generation Data Scaling Analysis

  • More than 300 demonstrations generated from one source demo yield average success rates above those from 50 real demonstrations.Both evaluated policies improve consistently as the number of generated demonstrations increases.Source: §10.4 · p. 17, Table 9 · p. 18, Figure 15 · p. 18

10.5. Ablation Study of Control Conditions

  • Removing either depth control or the Canny-edge constraint leads to object blurring and incorrect interactions that degrade generated-demonstration quality.The model uses depth as its 3D control interface and Canny edges computed from depth as an auxiliary condition.Source: §10.5 · p. 18

11. Additional Visualizations

The section provides additional visualizations of the four manipulation tasks and a separate visualization of failure cases.

11. Additional Visualizations

  • Figures 21–24 show additional visualizations of the four real-world manipulation tasks.Source: §11 · p. 18
  • Figure 16 presents visualized failure cases.Source: Figure 16 · p. 18

12. Limitation and Discussion

The discussion identifies video-generation time and object generalization as limitations, and the accompanying figures show control-condition ablations and generated videos across the four tasks.

12. Limitation and Discussion

  • Video generation is a computational bottleneck, particularly in resource-constrained scenarios.The authors suggest exploring KV caching and model distillation to improve pipeline throughput.Source: §12 · p. 18
  • Object generalization is limited, especially for articulated or deformable objects absent from the training distribution.The authors report possible motion blurring or structural inconsistency in synthesized videos.Source: §12 · p. 18
  • Figures 17–20 show control-condition ablations for Mug to Basket, Pour Water, Lift Box, and Scan Barcode.Source: Figure 17 · p. 19, Figure 18 · p. 20, Figure 19 · p. 21
  • Figures 21–24 visualize generated videos for Mug to Basket, Pour Water, Lift Box, and Scan Barcode.Source: Figure 21 · p. 23, Figure 22 · p. 24, Figure 23 · p. 25
Loading 2512.19402v2…