Source-linked AI summary
Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers
Vidhi Jain, Maria Attarian, Nikhil J Joshi, Ayzaan Wahid, Danny Driess, Quan Vuong, Pannag R Sanketi, Pierre Sermanet, Stefan Welker, Christine Chan, Igor Gilitschenski, Yonatan Bisk, Debidatta Dwibedi
TL;DR
Robots often need to infer manipulation intent from demonstrations rather than text, especially when task execution depends on context and differs across embodiments or environments. Vid2Robot learns an end-to-end policy from paired prompt and robot videos using cross-attention and auxiliary representation-alignment losses, and outperforms BC-Z with human prompts while supporting cross-object motion transfer.
Problem
Robots need to infer manipulation tasks from visual demonstrations despite differences in embodiment and environment, while text and final-goal specifications can underspecify how actions should be performed.
Method
Vid2Robot trains an end-to-end video-conditioned policy on paired prompt-video and robot-trajectory data, using cross-attention between video features and robot state plus auxiliary losses for representation alignment.
Results
Vid2Robot outperforms BC-Z by over ∼20% with human prompt videos and by ∼17% for cross-object motion transfer.
Takeaways & Limitations
Prompt videos can specify manipulation tasks and support transferring demonstrated motions to different objects in the robot’s environment.
Takeaways & Limitations
The paper reports a performance gap for video-conditioned policies and a significant drop in grasping success when visual state estimation is incomplete under occlusion or gripper visibility loss.
Abstract
from arXiv · showhide
Large-scale multi-task robotic manipulation systems often rely on text to specify the task. In this work, we explore whether a robot can learn by observing humans. To do so, the robot must understand a person's intent and perform the inferred task despite differences in the embodiments and environments. We introduce Vid2Robot, an end-to-end video-conditioned policy that takes human videos demonstrating manipulation tasks as input and produces robot actions. Our model is trained with a large dataset of prompt video-robot trajectory pairs to learn unified representations of human and robot actions from videos. Vid2Robot uses cross-attention transformer layers between video features and the current robot state to produce the actions and perform the same task as shown in the video. We use auxiliary contrastive losses to align the prompt and robot video representations for better policies. We evaluate Vid2Robot on real-world robots and observe over 20% improvement over BC-Z when using human prompt videos. Further, we also show cross-object motion transfer ability that enables video-conditioned policies to transfer a motion observed on one object in the prompt video to another object in the robot's own environment. Videos available at https://vid2robot.github.io
I. INTRODUCTION
Vid2Robot studies video-conditioned robot learning as an alternative to text or final-goal specifications, aiming to infer task intent from demonstrations across embodiments and environments. It introduces an end-to-end video-conditioned policy and reports improved performance with human prompt videos.
- Video demonstrations can convey task intricacies that are difficult to express through still images or text, motivating robots that learn from human observation.
- Text-only task specifications can struggle when similar verbs require context-dependent motor skills, such as opening different kinds of containers.
- Final goal images can be ambiguous because identical end states may correspond to different actions, whereas multiple sub-goal frames provide action-relevant information.
- Video-conditioned policies face computational, data-collection, and human-demonstration variability challenges, including limited robot-specific motion data.
- Vid2Robot addresses this setting with an end-to-end policy for real-world robotic control and reports better performance than baselines on human prompt videos and cross-object motion transfer.
A. Preliminaries
Vid2Robot maps a prompt video and recent robot observations to task-relevant actions, requiring paired demonstrations of the same task across differing embodiments or environments. Its training data combines robot-robot, hindsight human-robot, and co-located human-robot pairs.
- The policy receives a prompt video and robot state history, infers the demonstrated task, and predicts task-relevant robot actions.
- Paired prompt videos and robot trajectories are constructed so both demonstrations perform the same task despite differing settings or embodiments.
- Robot-robot pairing reuses existing trajectories and supports imitation of the same task across different environments.
- Hindsight human-robot pairing uses robot-trajectory instructions to collect human demonstrations from the robot’s viewpoint, introducing embodiment and speed variability without collecting new robot trajectories.
- Co-located human-robot pairing records both agents performing the task in the same workspace, while the combined dataset spans diverse environments and backgrounds.
Dataset
The system combines paired video data with a four-stage video-conditioned policy that encodes prompt and robot observations, fuses them through cross-attention, and decodes actions. Cross-attention reduces the token-computation burden while preserving task-relevant interactions between prompt and state.
- Dataset: The dataset combines approximately 100k robot videos and 10k human videos, with robot-robot data comprising more than 90% of the training set.
- C. Model Architecture: The policy comprises prompt-video encoding, robot-state encoding, state-prompt fusion, and robot-action decoding.
- C. Model Architecture: The prompt encoder converts demonstration frames into task-relevant tokens, while the state encoder represents recent robot observations, objects, and environment information.
- C. Model Architecture: Cross-attention uses robot-state tokens as queries over prompt tokens to identify which objects and interactions in the current scene match the demonstrated task.
- C. Model Architecture: The action decoder cross-attends to prompt-aware state tokens and predicts binned action dimensions, using one forward pass and a four-step prediction horizon.
- C. Model Architecture: Cross-attention reduces the example attention workload from about 22M entries with full self-attention to about 0.3M entries with Perceiver resampling.
D. Preprocessing
Vid2Robot preprocesses variable-length prompt and robot videos into fixed-size temporal inputs and trains with action prediction plus auxiliary alignment and contrastive objectives.
- D. Preprocessing: Vid2Robot samples 16 prompt frames and constructs an 8-frame robot-state history, repeating the first frame when insufficient history exists.Photometric distortions include cropping, brightness, contrast, hue, and saturation.
- D. Preprocessing: The robot action is an 11-dimensional vector covering termination or movement mode, arm pose and gripper state, and base displacement and rotation.Action values are scaled to [0,1] and discretized into 256 bins.
- D. Preprocessing: Training combines cross-entropy action prediction with temporal video alignment, prompt-robot video contrastive, and video-text contrastive losses.The overall objective is the mean of the four losses.
- D. Preprocessing: Temporal cycle consistency aligns prompt and robot video embeddings so corresponding frames map back to the same task progress.The loss computes soft temporal neighbors in both directions and minimizes cycle-back mean squared error.
- D. Preprocessing: Video-video contrastive learning brings prompt and robot embeddings for the same task together, while video-text contrastive learning links videos to their task descriptions.Both objectives use attention pooling and SigLIP losses.
F. Implementation
The model is trained in Jax with AdamW and cross-attention transformer components for prompt/state resampling, state-prompt fusion, and action decoding.
- F. Implementation: Vid2Robot is trained for 200K iterations with AdamW, an initial learning rate of 8e-5, cosine scheduling, 2,000 warmup steps, and a final learning rate of 1e-6.Prompt and state resamplers use two Perceiver layers with 64 latent tokens each.
- F. Implementation: The state-prompt encoder and action decoder are each four-layer cross-attention transformers.These components operate alongside the Perceiver resamplers.
III. EXPERIMENTS
The experiments evaluate video-conditioned manipulation on real robots, emphasizing prompt embodiment, unseen videos, baseline comparison, and cross-object transfer under varied environments.
- III. EXPERIMENTS: Real-robot evaluation measures whether a rollout reaches, grasps, places, and terminates the demonstrated task correctly without reset or recovery assistance.Success requires all four milestones; partial success is analyzed separately.
- III. EXPERIMENTS: Human evaluators are used because final-state rules can mistake unintended outcomes for successful task execution.Evaluation therefore considers the complete process rather than only the final state.
- III. EXPERIMENTS: Evaluations vary object placement, lighting, backgrounds, and distractors while using unseen prompt videos and a Google mobile manipulator.Initial configurations are matched across policies before randomization.
- III. EXPERIMENTS: Vid2Robot is compared with BC-Z using the same training data, while accounting for BC-Z’s lack of a terminate action through fixed-length rollouts.The comparison tests both robot- and human-performed prompt videos.
- III. EXPERIMENTS: The study organizes experiments around prompt embodiment, unseen task videos, overall comparison with BC-Z, and out-of-distribution object interactions.These questions frame the task-success and motion-transfer evaluations.
A. Task-based success
Task-based evaluations compare Vid2Robot and BC-Z on real-robot rollouts using robot and human prompts, varied visual conditions, and partial-success milestones.
- A. Task-based success: The evaluation compares Vid2Robot and BC-Z on robot- and human-performed prompt videos using the same training data and unseen prompts.Prompt videos cover a subset of training tasks but are not seen during evaluation training.
- A. Task-based success: Task success is assessed across nine manipulation tasks with eight trials per task and four evaluators, totaling 288 rollouts for Table I.The table reports an overall success rate across the nine tasks.
- A. Task-based success: Vid2Robot rollouts succeed despite prompt and execution differences in lighting, backgrounds, distractors, and object placement.Figure 5 presents human task videos alongside corresponding successful robot rollouts.
- A. Task-based success: Partial success compares reaching, grasping, releasing, and terminating between Vid2Robot and BC-Z, whose policy lacks terminate control.The reported comparison follows these four sequential milestones.
- A. Task-based success: Table II adds trials for Vid2Robot and BC-Z to assess statistical significance in real-robot evaluation.The expanded evaluation reports confidence intervals over 314 rollouts on two tasks.
B. Tasks with More Rollouts
Vid2Robot transfers task motions from prompt videos to different objects, outperforming BC-Z on cross-object motion transfer, especially for drawer placement.
- C. Cross-object motion transfer: The experiment tests whether a policy trained with paired prompt and robot videos can apply a demonstrated motion to objects absent from the prompt video.This setting defines the paper’s cross-object motion transfer evaluation.
- C. Cross-object motion transfer: The evaluation compares Vid2Robot and BC-Z across five prompt videos, six unrelated objects, and 240 rollouts by four raters.Each evaluator performs 60 rollouts: two models, five tasks, and six objects.
- C. Cross-object motion transfer: Vid2Robot transfers placing-upright motion from a coke-can prompt to a green can, chips bag, stapler, and soft toy.The policy selects the green can instead of other distractor objects, including the chips bag or banana.
- C. Cross-object motion transfer: 29% →54% success on placing into drawers, while Vid2Robot reaches 25%−29% on several tasks where BC-Z is unable to perform.Overall cross-object transfer success is 34% for Vid2Robot versus 17% for BC-Z, while out-of-distribution object picking is comparable at 45%.
D. Ablations
Prompt videos materially improve task success by resolving ambiguity that robot observations alone cannot determine.
- 1) What is the impact of the prompt for task inference?: A robot observation containing a Coke can does not reveal whether the task is to pick, move, or knock it over.Prompt videos provide the task specification needed to distinguish these possible actions.
- 1) What is the impact of the prompt for task inference?: Conditioning on prompt videos raises BC-Z success from 5% to 52.6% and Vid2Robot success from 23% to 54.6%.With blank prompt frames, the policies achieve 23% and 5% success for Vid2Robot and BC-Z, respectively, across three tasks and 120 rollouts.
- 1) What is the impact of the prompt for task inference?: The no-prompt experiment uses blank input videos, three tasks, 20 rollouts per task, and both policies.This setup directly measures performance without a suitable task prompt.
2) What is the role of auxilliary losses?:
Auxiliary losses improve Vid2Robot performance by constraining learned representations, while the auxiliary language loss contributes only a borderline gain.
- 2) What is the role of auxilliary losses?: The ablation compares full Vid2Robot, a variant without video-text contrastive loss, and a variant without any auxiliary losses across nine tasks.The full approach uses action prediction together with three auxiliary losses.
- 2) What is the role of auxilliary losses?: Vid2Robot reaches 61% success with auxiliary losses versus 45% using only action prediction loss.The ablation attributes the improvement to representation constraints enforced by the auxiliary losses.
- 2) What is the role of auxilliary losses?: Removing video-text contrastive loss yields only a borderline 1-2% success-rate improvement when the language loss is included.The result suggests video alignment and video contrastive loss contribute more substantially to performance than the auxiliary language loss.
- 2) What is the role of auxilliary losses?: Video-conditioned policies distinguish visually different tasks such as opening a jar or drawer, unlike language conditioning that can share the same verb embedding.Vid2Robot does not use language as an input.
C. Imitation via Paired Demonstrations
Vid2Robot addresses video-conditioned imitation through paired demonstrations while documenting remaining failure modes and scope limits. The system uses cross-attention and auxiliary losses, improves over BC-Z, and supports cross-object motion transfer, but performance remains constrained by perception and demonstration coverage.
- The paper reports improved performance over previous work but acknowledges a remaining performance gap for video-conditioned policies.
- Policy rollouts can fail because of self-occlusion, grasping errors, and distractors.
- Grasping success drops when camera observations incompletely capture depth because of occlusion or an out-of-view gripper.The authors suggest multimodal sensor fusion as a possible way to improve grasp success.
- The evaluation uses short, carefully collected 5–20-second demonstrations, leaving effective pairing for long-horizon or in-the-wild videos as future work.
- Vid2Robot outperforms BC-Z by over ∼20% with human videos and by ∼17% for cross-object motion transfer.Cross-object transfer means producing valid actions for the same verb on a different object than the one shown in the prompt.
- The policy uses cross-attention to fuse prompt-video and robot-state encodings, then decodes actions with auxiliary losses that encourage generalizable representations.
APPENDIX
The appendix provides implementation, dataset, qualitative, and deployment details for Vid2Robot. It covers rare-task completion, chained long-horizon prompting, robot hardware and execution, training data, and supporting architecture and hyperparameter tables.
- The dataset mixture contains robot, human, and co-located human–robot trajectories, with approximately 90% Robot-Robot, 5% Hindsight Human-Robot, and 5% Co-located Human-Robot pairs.
- Vid2Robot completes rare tasks such as opening a glass jar, picking up a green microfiber cloth, and pulling out a napkin.
- Experiments use mobile manipulators with 7-DoF arms, two-fingered grippers, and fixed mobile bases.
- The policy runs in a client-server setup at approximately 5–7 Hz using an eight-frame history from the robot head camera.
- Chaining prompt videos lets Vid2Robot complete long-horizon tasks without explicitly training on long-horizon videos or resetting between subtasks.The demonstrated cleanup task chains opening a drawer, placing objects inside, and closing the drawer.
- The appendix includes detailed architecture, hyperparameter, and data-augmentation tables.