PaperlayerView exact revision on arXiv

Source-linked AI summary

Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrations

Shivansh Patel, Shraddhaa Mohan, Hanlin Mai, Unnat Jain, Svetlana Lazebnik, Yunzhu Li

arXiv:2507.00990v3cs.ROcs.AIcs.CV

TL;DR

Robotic video imitation often requires matched demonstrations or adaptation to a robot, while the usefulness of generated videos for real-world manipulation remains uncertain. RIGVid filters and tracks scene-conditioned generated videos, then retargets object motion to a robot. On the evaluated tasks, filtered videos perform on par with human videos, and results favor RIGVid over compact VLM alternatives and other trajectory extractors.

  • Problem

    Video imitation can require closely matched demonstrations, while generated videos may contain implausible geometry or interactions and their usefulness for robotics remains unestablished.

    Source: §1 · p. 1

  • Method

    RIGVid filters task- and scene-conditioned generated videos, extracts 6D object trajectories, and retargets them for robot execution without real-world demonstrations.

    Source: §1 · p. 2, §5 · p. 10

  • Results

    Filtered generated videos perform on par with human videos, and RIGVid outperforms VLM-based trajectory prediction and a range of alternative trajectory extractors.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that synthetic videos can serve as an effective substitute for real data in robotic imitation.

    Source: §1 · p. 2

    Failures on filtered Kling v1.6 videos were attributed mainly to monocular depth-estimation errors that produced inaccurate trajectories and tracking failures.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

This work introduces Robots Imitating Generated Videos (RIGVid), a system that enables robots to perform complex manipulation tasks--such as pouring, wiping, and mixing--purely by imitating AI-generated videos, without requiring any physical demonstrations or robot-specific training. Given a language command and an initial scene image, a video diffusion model generates potential demonstration videos, and a vision-language model (VLM) automatically filters out results that do not follow the command. A 6D pose tracker then extracts object trajectories from the video, and the trajectories are retargeted to the robot in an embodiment-agnostic fashion. Through extensive real-world evaluations, we show that filtered generated videos are as effective as real demonstrations, and that performance improves with generation quality. We also show that relying on generated videos outperforms more compact alternatives such as keypoint prediction using VLMs, and that strong 6D pose tracking outperforms other ways to extract trajectories, such as dense feature point tracking. These findings suggest that videos produced by a state-of-the-art off-the-shelf model can offer an effective source of supervision for robotic manipulation.

1. Introduction

RIGVid asks whether robots can imitate scene- and task-conditioned generated videos without physical demonstrations or task-specific training. Filtered generated videos match human demonstrations on the evaluated tasks, and performance improves with video quality.

1. Introduction

  • Prior video-imitation approaches face domain gaps or laborious matched demonstrations; generated videos can be physically implausible, and video generation is computationally costly.Source: §1 · pp. 1–2
  • RIGVid turns an initial RGB-D scene and free-form command into a generated video, tracked 6D object trajectory, and retargeted robot execution.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Filtered generated videos perform on par with human videos for visual imitation, and RIGVid performance improves with video quality.Source: §1 · p. 2
  • RIGVid outperforms VLM-generated sparse keypoint constraints and alternative trajectory extractors, including point tracking, optical flow, feature fields, and generated-goal supervision.Source: §1 · p. 2

2. Related Work

Related work uses real video demonstrations, offline video datasets, or generated videos to supervise robotic manipulation. RIGVid distinguishes itself by conditioning generated videos on the task and scene without robot data, then tracking objects for execution.

2. Related Work

  • Direct imitation matches visual states without expert action labels or robot states, but requires paired demonstrations in the same setting; methods use tracks or optical flow [5] [35] [125].Source: §2 · p. 2
  • Offline-video methods reduce the need for paired demonstrations, but dataset-based approaches face domain gaps and may need mechanisms for task-conditioned affordances.Source: §2 · p. 3
  • Many prior video-generation robotics methods rely on robot data [3] [29] [30] [72] [110], whereas RIGVid uses task- and scene-conditioned videos and tracks objects rather than tools.Source: §2 · p. 3

3. Our Method: Robots Imitating Generated Videos

RIGVid generates a task-conditioned video, extracts the manipulated object’s 6D pose trajectory, retargets it to the robot, and adjusts execution using real-time feedback.

3. Our Method: Robots Imitating Generated Videos

  • The method takes an initial RGB image, depth map, and free-form command, then generates a conditioned video, estimates object poses, retargets the trajectory, and executes it.Source: §3 · p. 3

3.1. Generating Videos and Corresponding Depth

  • A VLM screens generated videos for command compliance, while predicted video depth is aligned to the initial real depth map to resolve scale-and-shift ambiguity.Source: §3.1 · pp. 3–4

3.2. Identifying Active Object Mask and 6D Object Pose Trajectory

  • GPT-4o identifies the active object, Grounding DINO and SAM-2 produce its initial mask, and FoundationPose tracks its 6D pose using scaled depth.FoundationPose requires a precomputed object mesh in the real-world deployment; mesh-free alternatives are currently too slow for real-time use.Source: §3.2 · p. 4

3.3. Object to Robot Motion Retargeting

  • After grasping, RIGVid applies a fixed end-effector-to-object transform to the object trajectory, so another robot or gripper requires only an updated transform.Source: §3.3 · p. 4, Figure 2 · p. 4

3.4. Closed Loop Execution

  • During execution, real-time pose tracking detects deviations above 3 cm or 20 degrees, prompting the robot to backtrack to its last successful point and resume.Source: §3.4 · pp. 4–5, Figure 3 · p. 4

4. Experiments

Experiments evaluate RIGVid on four real-world tasks and compare video quality and filtering, VLM-based prediction, trajectory extraction, and transfer. Filtered generated videos perform similarly to human videos, while RIGVid outperforms tested trajectory baselines.

4.1. Robot Setup, Tasks, and Evaluation

  • The evaluation used an xArm7 and four tasks—pouring, lifting a lid, placing a spatula, and sweeping—with human-judged success and shared videos across baselines.The tasks vary in depth, object geometry, and occlusion challenges.Source: §4.1 · p. 5

4.2. Quality and Filtering of Generated Videos

  • Filtered Kling v1.6 improved success from 80% to 100% on pouring, 60% to 80% on lifting, 50% to 90% on placing, and 20% to 70% on sweeping.Filtered Kling v1.6 performance was similar to that with real human demonstration videos.Source: §4.2 · p. 6, Figure 7 · p. 7

4.3. RIGVid vs. VLM-Based Trajectory Prediction

  • RIGVid achieved 85% success across four tasks versus ReKep’s 50%, although the authors note other VLM approaches could close the gap.The comparison is limited to the tasks and experimental setup evaluated.Source: §4.3 · p. 7

4.4. Comparison to Alternative Trajectory Extraction Methods

  • RIGVid achieved 85.0% success versus 67.5% for Gen2Act, with its advantage increasing on harder tasks.The comparison covers four tasks; relative improvements are higher for spatula placement and sweeping.Source: Figure 9 · p. 8, §4.4 · p. 9
  • Track2Act diverged from the object path, AVDC accumulated flow errors, and 4D-DPM tracked unstably, while Gen2Act and RIGVid succeeded in the watering-can example.RIGVid’s pose trajectories remained stable and matched object motion in that example.Source: §4.4 · pp. 9–10, Figure 10 · p. 9

4.5. Testing Generalization

  • RIGVid achieved 80% pouring success on ALOHA versus 100% on xArm and also transferred to bimanual shoe placement and preliminary additional tasks.The additional tasks included wiping, mixing, ironing, and tasks involving extreme rotations.Source: §4.5 · p. 10, Figure 11 · p. 10

5. Conclusions

RIGVid performs manipulation without real-world demonstrations by using generated videos and 6D pose estimation. The authors report performance approaching real demonstrations as video quality improves, alongside stronger results than competing approaches.

5. Conclusions

  • Camera-calibration challenges caused a slight performance drop, primarily because ALOHA’s arms produced less accurate pose estimates.Source: §5 · p. 10
  • RIGVid performs manipulation without teleoperation, human demonstrations, or expert policy rollouts by extracting 6D object-pose trajectories from generated videos and retargeting them to a robot.The conclusion characterizes this as a data-efficient approach to robotic skill acquisition.Source: §5 · p. 10
  • As video generation improves, RIGVid approaches real-demonstration performance, and its dense visual and temporal cues yield more reliable results than the compared VLM-based methods.The conclusion also reports that RIGVid outperforms competing trajectory-extraction methods across diverse visual-imitation tasks and is robust to environmental disturbances.Source: §5 · p. 10

A. Best Practices for Video Generation

Reliable video generation favors uncluttered scenes, clear tasks, and concise prompts. For filtering, GPT o1 judges whether sampled video frames show the commanded task being performed.

A. Best Practices for Video Generation

  • Clean backgrounds, few distractors, reasonably large objects, a natural human-like viewpoint, one clear task, and concise prompts supported reliable video generation.A relevance factor of 0.7 with the negative prompt “fast motion” led to the most reliable generations.Source: A. Best Practices for Video Generation · p. 17

B. Prompting for Video Filtering and Filtering Statistics

C. Mesh-Free Object Tracking

The supplement reports a successful mesh-free tracking result for pouring, describes trajectory smoothing and baseline adaptations, and illustrates ReKep’s keypoint prediction failures.

C. Mesh-Free Object Tracking

  • 90% success over 10 pouring trials with BundleSDF matched the default tracking setup, but its official implementation took about 30 minutes per video, limiting real-time deployment.The default tracker operates in real time and supports closed-loop execution and disturbance recovery.Source: C. Mesh-Free Object Tracking · p. 17

D. Smoothing Object Trajectories

  • A centered moving-average filter smooths position and orientation estimates to reduce noise and jitter and produce more stable, realistic trajectories.Translations are smoothed independently by axis; orientations are filtered after conversion from quaternions to rotation vectors.Source: D. Smoothing Object Trajectories · p. 18

E. Description of Baselines

  • Track2Act estimates object motion from predicted 2D point tracks and PnP, while the adaptation uses a generated-video goal frame and interpolates sparse poses for execution.The adaptation excludes Track2Act’s closed-loop residual policy; AVDC estimates motion from optical flow and depth, but its replanning strategy is not used.Source: E. Description of Baselines · p. 18
  • The 4D-DPM adaptation tracks whole objects using clustered 3D Gaussian features, while Gen2Act’s tracks are converted to 6DoF poses without behavior cloning or offline demonstrations.The 4D-DPM adaptation changes part-level tracking to object-level tracking; the Gen2Act adaptation uses initial depth, RANSAC filtering, and PnP.Source: E. Description of Baselines · pp. 18–19

F. ReKep Predictions and Executions

  • ReKep’s pouring execution failed after VLM keypoint predictions placed the spout and tilted can relative to specified keypoints; inaccurate initial keypoints underlie many failures.Examples include no predicted keypoint at the lid handle and keypoints clustered around pan corners, while sweeping keypoints were generally well placed and executions succeeded.Source: F. ReKep Predictions and Executions · p. 19

G. Limitation of Tracking with Point Tracks

Correspondence-based point tracking is vulnerable to extreme object rotations that occlude initially visible points, destabilizing pose estimation and robot execution.

G. Limitation of Tracking with Point Tracks

  • Under extreme rotations, occlusion removes initially visible points, leaving too few 2D–3D correspondences for stable PnP and causing pose drift or jumps that can lead to task failure.The authors report that both Gen2Act variants fail under large out-of-plane rotations, whereas RIGVid’s model-based 6D tracking handles these situations more robustly.Source: G. Limitation of Tracking with Point… · p. 19

H. Additional Robustness Examples

RIGVid recovers from grasp misalignment and object disturbances during execution. The examples also show ReKep keypoint placements are often suboptimal and Gen2Act pose tracking can fail after tracking loss.

H. Additional Robustness Examples

  • RIGVid corrects a misaligned grasp by reorienting the object, and realigns it after human disturbances before completing the task.The robot successfully corrected a second human intervention as well.Source: H. Additional Robustness Examples · p. 19, Figure 16 · p. 20
  • ReKep’s keypoint placements are often suboptimal, with reasonable placements shown for the sweeping task.Source: Figure 14 · p. 19
  • Gen2Act’s tracked points drift from reprojected 3D points in later frames, indicating pose-estimation failure from tracking loss and failed robot execution.Source: Figure 15 · p. 19

I. Errors from Depth Estimation

Depth-estimation errors reduce execution success and include inaccurate values and temporal flicker in both generated and real videos. Replacing depth with RGB-only MegaPose tracking yields even noisier trajectories.

I. Errors from Depth Estimation

  • 100% success with real videos and real depth contrasts with 85% average for real videos with generated depth and 85% for Kling V1.6 videos with generated depth.The authors identify monocular depth estimation as the primary source of error.Source: Figure 17 · p. 20, §I · p. 20
  • Generated-video spatula depth changes only 6.8 cm despite a large visual change, while real-video estimates place the spatula head farther away despite its apparent closeness.Source: §I · p. 20, Figure 18 · p. 20
  • Watering-can depth flickers despite minimal motion, changing from 40.1 cm to 38.2 cm over 0.066 seconds in generated video and from 43.2 cm to 40.9 cm in real video.Source: §I · p. 20
  • Estimating object pose directly from generated RGB frames with MegaPose instead of using depth produces even more unstable, noisy trajectories.Source: §I · p. 20

J. Choice between MegaPose and FoundationPose

FoundationPose produces smoother, more stable trajectories than MegaPose by the reported RMS-jitter measures. The comparison smooths pose sequences before computing residual motion across ten generated-video pouring trajectories.

J. Choice between MegaPose and FoundationPose

  • The comparison applies a Gaussian smoothing filter (σ = 2 frames), measures residuals from the smoothed SE(3) pose sequences, and averages over ten generated-video pouring trajectories.Translational and rotational RMS jitter use translational residuals and angular residuals, respectively.Source: J. Choice between MegaPose and… · pp. 20–21
  • FoundationPose has 0.0029m translational RMS jitter and 14.31° rotational RMS jitter, versus MegaPose’s 0.0045m and 37.47°, respectively.FoundationPose also supports real-time tracking during execution, which the authors say makes RIGVid robust to external disturbances.Source: J. Choice between MegaPose and… · p. 21

K. Comparing Video Generative Models

Kling v1.6 compares favorably with other video generators on most reported quality metrics, although some metrics and qualitative properties tell a more qualified story. The paper also cautions that video-text consistency and I2V subject consistency do not reliably indicate command-following.

K. Comparing Video Generative Models

  • Kling v1.6 surpasses or matches other models on most Table 3 quality metrics, but is similar or worse on video-text consistency and dynamic degree.Table 3 reports metrics from 40 unfiltered videos per model, with 10 videos for each of four tasks.Source: K. Comparing Video Generative Models · p. 21, Table 3 · p. 21
  • Human evaluations suggest video-text consistency and I2V subject consistency are not reliable indicators of whether a video follows the command.Source: K. Comparing Video Generative Models · p. 21
  • VBench++ spans visual consistency, motion smoothness and amount, aesthetic and imaging quality, and text alignment.Source: K. Comparing Video Generative Models · p. 21
  • Figure 20 ranks Kling v1.6 highest on three selected VBench++ scores, while the broader report says its video-text consistency is similar or worse.The selected scores are video-text consistency, I2V subject consistency, and subject consistency.Source: K. Comparing Video Generative Models · p. 21, Figure 20 · p. 22
  • Sora often changes scene layout and objects; Kling v1.5 looks more plausible but includes physically implausible elements, while Kling v1.6 better preserves the scene and follows the command.Source: Figure 21 · p. 23
Loading 2507.00990v3…