PaperlayerView exact revision on arXiv

Source-linked AI summary

Dreamitate: Real-World Visuomotor Policy Learning via Video Generation

Junbang Liang, Ruoshi Liu, Ege Ozguroglu, Sruthi Sudhakar, Achal Dave, Pavel Tokmakov, Shuran Song, Carl Vondrick

arXiv:2406.16862v1cs.ROcs.CV

TL;DR

Visuomotor policies must generalize across diverse physical situations, but behavior cloning requires ground-truth robot actions, making scaling to diverse situations challenging. Dreamitate fine-tunes a video model on human demonstrations, generates an execution video from a scene image, and tracks the tool trajectory to derive robot actions. Across four real-world tasks, the video model consistently outperformed behavior cloning in generalizing to unseen scenarios; on this task, it achieved 92.5% success versus Diffusion Policy’s 12.5%.

  • Problem

    Visuomotor policies must generalize across diverse physical situations, but behavior cloning requires ground-truth robot actions, making scaling to diverse situations challenging.

    Source: §1 · pp. 1–2

  • Method

    Dreamitate fine-tunes a video model on human demonstrations, generates an execution video from a scene image, and tracks the tool trajectory to derive robot actions.

    Source: §1 · p. 2, §3.1 · p. 3, §3.3 · p. 4

  • Results

    Across four real-world tasks, the video model consistently outperformed behavior cloning in generalizing to unseen scenarios; on this task, it achieved 92.5% success versus Diffusion Policy’s 12.5%.

    Source: §1 · p. 2, §4.4 · p. 7

  • Takeaways & Limitations

    Using common tools bridges the embodiment gap between the human hand in demonstrations and the robot manipulator.

    Source: §6 · p. 8

    High video-model computational costs make real-time closed-loop control infeasible.

    Source: §5 · p. 8

Abstract

from arXiv · show

A key challenge in manipulation is learning a policy that can robustly generalize to diverse visual environments. A promising mechanism for learning robust policies is to leverage video generative models, which are pretrained on large-scale datasets of internet videos. In this paper, we propose a visuomotor policy learning framework that fine-tunes a video diffusion model on human demonstrations of a given task. At test time, we generate an example of an execution of the task conditioned on images of a novel scene, and use this synthesized execution directly to control the robot. Our key insight is that using common tools allows us to effortlessly bridge the embodiment gap between the human hand and the robot manipulator. We evaluate our approach on four tasks of increasing complexity and demonstrate that harnessing internet-scale generative models allows the learned policy to achieve a significantly higher degree of generalization than existing behavior cloning approaches.

1 Introduction

Robust visuomotor policies must generalize across physical situations, but behavior cloning requires robot actions that are difficult to scale. Dreamitate fine-tunes a video model on human demonstrations and tracks a tool in generated videos to produce robot actions.

1 Introduction

  • Behavior cloning maps visual observations to actions but requires ground-truth robot actions, making scaling to diverse situations challenging.Source: §1 · pp. 1–2
  • Dreamitate synthesizes a person using a tool, tracks that tool in 3D, and transfers its trajectory into robot actions, using inverse kinematics for the rest of the embodiment.The authors identify the end-effector as the most important part of the embodiment for manipulation.Source: §1 · p. 2
  • Internet-video pretraining can preserve broad manipulation priors during fine-tuning, supporting generalizable skills with the same amount of demonstration data.Source: §1 · p. 2
  • Human demonstration recordings instead of teleoperation make collection more scalable, while predicted execution videos provide an interpretable intermediate representation for human-robot interaction.Source: §1 · p. 2
  • Across four real-world tasks, the video model consistently outperformed behavior cloning in generalizing to unseen scenarios and retained strong generalization with fewer demonstrations.The evaluation included bimanual manipulation, precise 3D manipulation, and long-horizon tasks.Source: §1 · p. 2

2 Related Work

Prior work explores behavior cloning, visual pretraining, and video models for robotics. Dreamitate differs from video world models and planning methods by using predicted video as part of a visuomotor policy for action prediction.

2 Related Work

  • Behavior-cloning research progressed from state-to-action models to energy-based and conditional generative approaches for handling multimodal demonstrations and improving task success.Source: §2 · p. 2
  • Figure 2 outlines human-demonstration capture and video-model fine-tuning before conditioning generation on a novel scene; the generated tool trajectory is tracked and executed by the robot.Source: Figure 2 · p. 3, §2 · p. 3
  • The related-work comparison identifies video prediction combined with 3D tracking as Dreamitate’s route to action prediction.Source: §2 · p. 3
  • Visual pretraining for policy learning includes video prediction, contrastive learning, masked autoencoding, and visual reward-function learning.Source: §2 · p. 3
  • Video models have been used as action-conditioned world simulators or for longer-horizon planning, whereas Dreamitate uses predicted video within a visuomotor policy for action prediction.Source: §2 · p. 3

3 Approach

Dreamitate generates task videos from scene images, then tracks a tool to derive robot actions. It fine-tunes an internet-pretrained video model on human demonstrations and uses known tool geometry to recover 3D trajectories.

3.1 Overview

  • The policy maps a scene frame through video generation and tool-trajectory tracking to robot actions, with the synthesized video serving as an intermediate plan.The authors describe this synthesize-then-track composition as facilitating data collection without teleoperation and aligning with internet video data of human behavior.Source: §3.1 · p. 3

3.2 Video Generation

  • The video model starts from internet-pretrained weights and is fine-tuned per task on human stereo demonstrations using a video-prediction objective.The demonstrations use 3D-printed tools; the encoder and decoder are frozen while spatial and temporal attention layers are fine-tuned.Source: §3.2 · pp. 3–4
  • Custom tools with known CAD models support precise 3D tracking across four real-world tasks, with task-specific designs such as a scooping spoon.Source: Figure 3 · p. 4

3.3 Track then Act

  • At test time, the generated stereo video is tracked using a known CAD model and calibrated cameras to recover each frame’s 6D tool pose, which the robot executes as its action trajectory.The authors state that this tool-based tracking makes the pipeline agent-independent by avoiding agent-specific action spaces.Source: §3.3 · p. 4, §3.2 · p. 4

4 Experiments

The experiments compare Dreamitate with Diffusion Policy on four real-world manipulation tasks using distinct test objects from training. Dreamitate performs better across the reported task metrics, and its rotation performance remains stable with reduced training data.

4.1 Experimental Setup

  • Evaluation spans four tasks, with different objects at test time; data collection and evaluation vary tabletop and lighting conditions while keeping the camera setup the same.Diffusion Policy is trained on the same data with stereo images and a pretrained ResNet-18 variant.Source: §4 · p. 4, Table 1 · p. 5, §4.1 · p. 5

4.2 Object Rotation Task

  • For rotation, the policy must grasp real-world objects and turn them counterclockwise; Diffusion Policy can miss contact or choose unstable grasps that let objects slip.Training uses 31 objects and evaluation uses 10 unseen real-world objects; success requires maintained contact and at least 25° of counterclockwise rotation.Source: §4.2 · pp. 5–6, Figure 4 · p. 5

4.3 Granular Material Scooping Task

  • Scooping tests transferring granular material between bowls while ignoring distractors; evaluation uses unseen bowls, new particles, and five distractor objects per trial.A trajectory counts as successful if any particles reach the empty bowl.Source: §4.3 · p. 6

4.4 Table Top Sweeping Task

  • Sweeping tests multi-modal choices with particles, a randomly placed target, and obstacles; at test time, particles and distractors are unseen.Success means any particle reaches within 50 mm of the target.Source: §4.4 · pp. 6–7

4.5 Push-Shape Task (Long Horizon)

  • Push-Shape requires multi-step pushes that adjust a foam object’s position and orientation to match a target mask; testing uses eight unseen shapes.Performance is scored using the best of four rollouts, with mIoU and average rotation error reported.Source: §4.5 · p. 7

4.6 Performance Scaling Curve

  • On rotation, Dreamitate maintains high success rates with only one-third of the training data, while Diffusion Policy’s performance declines as data is reduced.The reduced-data comparison is evaluated over 40 trials.Source: §4.6 · p. 8

5 Limitations

The approach has limits in visual tracking, tool use, and inference speed. It also maintains strong generalization with less training data than Diffusion Policy.

5 Limitations

  • The approach maintains strong generalization performance with less training data than Diffusion Policy.Source: Figure 8 · p. 8
  • Heavy end-effector occlusion can cause failure, and the implementation is limited to visually trackable robot actions.The authors expect advances in video generation and object tracking to improve accuracy.Source: §5 · p. 8
  • Reliance on rigid tools limits applicability to tasks requiring fine-grained control.Source: §5 · p. 8
  • Video models have higher computational costs, making real-time closed-loop control infeasible, though inference acceleration may mitigate this.Source: §5 · p. 8

6 Conclusion

Dreamitate fine-tunes a video diffusion model on human demonstrations to generate execution-plan videos used to control a robot. The authors report greater generalization than previous behavior-cloning methods, including Diffusion Policy.

6 Conclusion

  • Dreamitate fine-tunes a video diffusion model on human demonstrations to synthesize execution-plan videos that directly control the robot at test time.Source: §6 · p. 8
  • Using common tools bridges the embodiment gap between the human hand in demonstrations and the real-world robot manipulator.Source: §6 · p. 8
  • Experiments validate a much larger degree of generalization than previous behavior-cloning methods such as Diffusion Policy.Source: §6 · p. 8

A Video Model Implementation

The implementation adapts Stable Video Diffusion to produce stereo-conditioned videos and uses them to generate execution plans. It specifies inference settings and provides training hyperparameters in Table 3.

A Video Model Implementation

  • The adapted Stable Video Diffusion model generates 25-frame videos, assigning the first 13 frames to stereo view 1 and the last 12 to stereo view 2.Frame embeddings are modified by viewing angle; the first output frame matches the input and is discarded at test time to address the odd frame count.Source: A Video Model Implementation · p. 12
  • Inference uses 30 denoising steps and constant classifier-free guidance of 1.0.Source: A Video Model Implementation · p. 12
  • Table 3 lists the video model training hyperparameters, including resolution, learning rate, batch size, training steps, clip duration, frame rate, and motion score.Source: Table 3 · p. 12

B Experimental Setup

The experiments use a two-camera stereo setup and task-specific robot platforms, with image preparation and evaluation procedures varying across tasks. A black table covering introduces friction variation and uncertainty in Push-Shape experiments.

B Experimental Setup

  • Two stereo cameras are spaced about 660 mm apart at a 45° angle and positioned about 760 mm from the table; training video is recorded at 1280×720 before cropping and resizing.Source: B Experimental Setup · p. 12
  • A black cloth covers the table, introducing friction variation and increasing uncertainty in Push-Shape experiments; rotation and scooping use xArm 7 robots, while sweeping and Push-Shape use UR5 robots.Source: B Experimental Setup · p. 12
  • Push-Shape mIoU uses stereo camera 1, rotation error uses the highest-IoU resulting image in multi-step trials, and sweeping and Push-Shape limit end-effector height to avoid table collisions.Source: B Experimental Setup · p. 12

C Data Collection

Human demonstrations begin with a scene image, followed by a person using a tool to perform the manipulation. Push-Shape demonstrations also encode a multi-step target position.

  • Each demonstration starts with an image of the scene, followed by frames of a human demonstrator using a tool to manipulate it.Source: C Data Collection · p. 12
  • For Push-Shape, the object is pushed to its target over multiple steps, and its final position is used as a mask blended across the video.Source: C Data Collection · p. 12
  • Training and testing objects for the tasks are shown in Figures 13–16.Source: C Data Collection · p. 12

D Object Tracking

The demonstrations’ tools are tracked in 3D from stereo views, with task-specific choices about which part of the tool to track.

  • MegaPose tracks the tool in the videos, and stereo camera projections are used to estimate its 3D translation and rotation.Translation is the midpoint between the projected lines; rotation averages the two camera-view estimates.Source: D Object Tracking · pp. 12–13
  • For scooping, only the handle is tracked to reduce particle-occlusion inaccuracies; for sweeping and Push-Shape, the handle is excluded because the human hand occludes it.Source: D Object Tracking · p. 13

E Diffusion Policy Baseline

The baseline is a CNN-based Diffusion Policy that processes stereo scene images with two pretrained ResNet-18 encoders. The accompanying figures show qualitative trajectories, task objects, and experiment setups.

Loading 2406.16862v1…