Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers

I. INTRODUCTION

• People use demonstrations to convey nuanced novel skills, and video observation can reveal intent in tasks difficult to specify through text or still images [6] [32].The introduction motivates learning from how-to videos and human demonstrations.Source: §I · p. 1
• Visual imitation lets robots acquire skills from differently embodied demonstrators, infer tasks across environments, and learn tasks difficult to describe in words.Source: §I · p. 1
• Text-only task specifications can obscure context-dependent motor differences, even for tasks sharing a verb, while policies should generalize across object variations within a task.Source: §I · pp. 1–2
• A final goal image may not specify how to act—for example, holding and waving a flag can share the same final image—whereas sub-goal frames convey motion.Source: §I · p. 2
• Video-conditioned policies face high-dimensional inputs, scarce robot-specific video and motion data, and variation in objects, lighting, and background distractions.The cited prior approaches also assume limited workspace variability and lag language-conditioned policies.Source: §I · p. 2
• Vid2Robot applies end-to-end learned video representations to real-world robot control and enables rapid adaptation to tasks specified by video demonstrations.The paper distinguishes this from prior work focused on object and verb recognition or simulated motor control.Source: §I · p. 2

II. APPROACH

• The policy encodes video task demonstrations with a transformer and uses three auxiliary contrastive losses to align prompt and robot video representations.Demonstrations may come from either human or robot embodiments.Source: §I · p. 2

A. Preliminaries

• The policy π(a_t|S_t, V) uses a prompt video V and recent robot-camera frames S_t to infer the task and predict task-relevant actions.The prompt may show a different setup or embodiment from the robot, which must perform the inferred task with objects in its own environment.Source: A. Preliminaries · p. 2

B. Datasets

• Training pairs prompt videos with robot trajectories that perform the same task, including examples of both humans and robots performing it.The paper uses three pairing classes: Robot-Robot, Hindsight Human-Robot, and Co-located Human-Robot.Source: B. Datasets · p. 2
• Robot-Robot pairing matches existing robot videos of the same instruction-defined task across different settings, using already collected and labeled trajectories without extra data collection.Instructions are used to pair videos and as an input to the robot policy.Source: B. Datasets · p. 2
• Hindsight Human-Robot pairing asks one to five people to perform existing robot-trajectory instructions, recording from the robot’s viewpoint without collecting new robot trajectories.Different hands, speeds, and randomized camera angles add embodiment and speed variability, but this approach does not increase task diversity.Source: B. Datasets · pp. 2–3
• Co-located Human-Robot pairing records people and robots doing the same task in the same workspace, across diverse spaces including kitchens, living spaces, and workstations.Although described as a presumed gold standard, this approach is more time- and labor-intensive and forms a small fraction of the training set.Source: B. Datasets · pp. 2–3
• The human-prompt datasets introduce background and distractor differences, as well as a change from robot to human actors, compared with Robot-Robot pairs.Figure 2 illustrates examples of the three paired-data approaches.Source: B. Datasets · p. 2, Figure 2 · p. 3
• The combined datasets contain approximately 100k robot videos and 10k human videos covering tasks introduced in RT-1 [8] and RT-2 [9].Robot-Robot pairing requires no additional data collection, while hindsight human demonstrations avoid robot teleoperation data but do not broaden task diversity.Source: B. Datasets · pp. 2–3

dataset.

• The paired video–robot trajectory dataset is publicly available, with a Supplementary Material notebook for visualizing its training video pairs.Source: dataset · p. 3

C. Model Architecture

• Vid2Robot maps prompt videos and current robot observations to actions through prompt-video, robot-state, state-prompt, and action-decoding modules.Figure 3 depicts the pipeline from video and state token embeddings through cross-attention to the robot action at the current timestep.Source: C. Model Architecture · p. 3, Figure 3 · p. 4
• The prompt encoder uses a per-frame ViT [15] and Perceiver Resampler [1] [20], while the state encoder processes the current and recent robot frames.Both encoders share image-encoder weights; the resamplers learn across frames and reduce the number of video tokens passed to the action decoder. The prompt encoder outputs N tokens of dimension d.Source: C. Model Architecture · p. 3
• The state-prompt encoder uses robot-state tokens as queries and prompt-video tokens as keys and values to produce prompt-aware state tokens.For example, when a prompt shows picking up an apple, cross-attention can select the apple among objects in the robot’s scene.Source: C. Model Architecture · p. 3
• The action decoder uses fixed action-position tokens as queries to predict binned values for each action dimension in one forward pass.Each token corresponds to a mode, arm, or base dimension; token embeddings are projected to 256 dimensions before softmax. Unlike autoregressive decoding [8] [9], the policy predicts a four-step horizon [22] [53] and executes the highest-probability bin; action-position tokens follow ACT [53].Source: C. Model Architecture · pp. 3–4
• With ViT-B/16, two Perceiver Resamplers with 64 latent tokens reduce the illustrated attention-matrix size from about 22M entries to about 0.3M.The example uses a 16-frame prompt video and an 8-frame robot-state video at 224×224 resolution; the reduced attention computation enables training with paired videos.Source: C. Model Architecture · p. 4

D. Preprocessing

• Training samples 16 ordered frames including the first and last, uses the eight frames before a random robot-state timestep, and repeats the first frame when needed; frames are resized, normalized, and photometrically distorted.Photometric distortions include cropping, brightness, contrast, hue, and saturation.Source: D. Preprocessing · p. 4
• Actions encode mode, arm and gripper, and base motion as an 11-dimensional vector scaled to [0,1] and discretized into 256 bins; the study keeps the base stationary.Source: D. Preprocessing · p. 4

E. Training

• Vid2Robot uses behavior cloning with cross-entropy over 256-bin discretized expert actions to train all model parameters.The loss compares predicted actions with expert actions for the robot trajectory.Source: E. Training · p. 4
• The training objective averages the action-prediction loss and three auxiliary losses, which are added because the dataset is insufficient for large transformers and to reduce overfitting.The four terms are action prediction, temporal-cycle consistency, video-video contrastive, and video-text contrastive losses.Source: E. Training · pp. 4, 6
• Temporal-cycle consistency aligns prompt and robot video frames for the same task, encouraging embodiment-robust features while encoding task progress [18] [51].It uses soft-neighbor matching in both directions and minimizes the mean squared error between each frame’s cycle-back position and its original position.Source: E. Training · pp. 4–5
• VVCL applies SigLIP to attention-pooled prompt and robot video embeddings, pulling same-task videos with similar motions and objects together and separating other batch videos [52].Same-task pairs are positive examples, while off-diagonal pairs are negatives.Source: E. Training · p. 5
• VTCL uses attention-pooled video embeddings and SigLIP to align prompt and robot videos with their task-text embeddings, encoding object names and verbs [22] [52].Each video is encouraged to match its textual description and differ from other text embeddings in the batch.Source: E. Training · pp. 5–6

F. Implementation

• The model was trained in JAX for 200K iterations.Source: F. Implementation · p. 6
• Training used AdamW with an initial learning rate of 8e-5, 2,000 warmup steps, cosine decay, and a final learning rate of 1e-6.Source: F. Implementation · p. 6
• The architecture used two Perceiver Resampler layers with 64 latent tokens each, plus four-layer cross-attention transformers for the state-prompt encoder and action decoder.Source: F. Implementation · p. 6

Evaluation metrics

• A rollout succeeds only when a human evaluator confirms all four task milestones, in sequence and without assistance.A task’s Success Rate averages its rollouts; Overall Success Rate aggregates across tasks.Source: §III · p. 6
• Human evaluation judges the process, since final-state rules can misclassify an unintended action that happens to leave an object in the target state.For knocking over a bottle, an unintended pinch that knocks it down could look successful under final-state verification, unlike the intended grasp-and-turn behavior.Source: §III · p. 6

Setup

• Policies are tested on the Google Robot with unseen prompt videos while object placement, lighting, background, and distractors vary.Initial object configurations are kept comparable across policies and randomized only after their rollouts; the robot has an egocentric camera, a seven-degree-of-freedom arm, and a two-finger soft gripper [8].Source: §III · p. 6

Baselines

• Vid2Robot is compared with BC-Z [22] using the same training data, while BC-Z rollouts use a fixed step limit because it lacks a terminate action.The passage describes BC-Z [23] as using a FiLM [38]-conditioned ResNet encoder and a ResNet-based policy network.Source: §III · p. 6

Key Questions and Results

• The experiments ask whether prompt embodiment affects success, how policies perform on unseen task videos, whether Vid2Robot beats BC-Z, and whether motion representations handle out-of-distribution interactions.These questions correspond to Sections III-A1, III-A2, III-B, and III-C, respectively.Source: §III · p. 6

• Vid2Robot and BC-Z use the same mixed training data, while evaluation prompts depict training tasks but are unseen by both models.Source: A. Task-based success · p. 6
• Successful Vid2Robot rollouts span visually different human prompts and robot scenes with varied lighting, backgrounds, and distractor objects.Source: Figure 5 · p. 7
• Vid2Robot outperforms BC-Z in reaching, grasping, releasing at the correct location, and terminating correctly; BC-Z lacks terminate control.Source: Figure 6 · p. 8

B. Tasks with More Rollouts

• The more-rollouts evaluation reported mean success rates with confidence intervals from 314 real-robot rollouts on two tasks.The tasks were ‘place coke can upright’ and ‘close middle drawer’; results were reported in Table II.Source: B. Tasks with More Rollouts · p. 8

C. Cross-object motion transfer

• Despite training on paired scenarios where the prompt object appeared in robot observations, Vid2Robot performed learned actions on objects absent from the prompt video.The authors call this emergent behavior cross-object motion transfer.Source: C. Cross-object motion transfer · p. 8
• The cross-object evaluation paired five prompt videos with six unrelated objects, chosen to vary in shape, size, and deformability.Each of four raters ran 60 rollouts, comparing two models across five tasks and six objects, for 240 total rollouts.Source: C. Cross-object motion transfer · p. 8
• With a ‘place coke can upright’ prompt, Vid2Robot transferred the upright-placement motion to a green can, chips bag, stapler, and soft toy, selecting the green can over other objects.The qualitative example is shown in Fig. 7.Source: Figure 7 · p. 8, C. Cross-object motion transfer · p. 8
• For picking out-of-distribution objects, Vid2Robot was comparable to BC-Z at a 45% success rate.For picking from drawers, placing on counters, and knocking over, Vid2Robot succeeded 25%–29% of the time while BC-Z was unable to perform.Source: C. Cross-object motion transfer · p. 8
• Drawer-placement tasks improved from 29% to 54% with Vid2Robot in the cross-object setting.The reported comparison is for tasks involving placing into drawers.Source: C. Cross-object motion transfer · p. 8
• 34% versus 17%: Vid2Robot achieved higher cross-object motion-transfer success than BC-Z across the reported evaluation.The comparison covers two models, five tasks, six objects, and four raters (240 rollouts total).Source: C. Cross-object motion transfer · p. 8, Table III · p. 9

D. Ablations

• The ablations examine whether state observations alone can support learning environment interactions and how auxiliary losses affect performance.The no-prompt evaluation measures task success without a suitable prompt video.Source: D. Ablations · p. 9, 1) What is the impact of the prompt… · p. 9

1) What is the impact of the prompt for task inference?:

• A Coke can in view leaves the task ambiguous among picking it, moving it near other objects, or knocking it over.As the robot acts, new states can make task inference from the current state alone even more difficult.Source: 1) What is the impact of the prompt… · p. 9
• With prompt videos, success rose from 5% to 52.6% for BC-Z and from 23% to 54.6% for Vid2Robot.The no-prompt evaluation used blank-frame prompts and 20 rollouts per task per policy across three tasks.Source: 1) What is the impact of the prompt… · p. 9
• The no-prompt experiment tested three tasks with 20 rollouts per task per policy, totaling 120 robot rollouts.Both policies received blank frames as prompt videos.Source: 1) What is the impact of the prompt… · p. 9

2) What is the role of auxilliary losses?:

• The auxiliary-loss study covered nine tasks and 162 rollouts across three policy variants, with three human evaluators running two rollouts per variant.The variants were the original Vid2Robot, a model without video-text contrastive loss, and a model using only action prediction loss.Source: 2) What is the role of auxilliary… · p. 9
• 61% success with all auxiliary losses exceeded 45% with action prediction alone, showing the benefit of representation constraints.Fig. 8 compares the full approach with a version using no auxiliary losses.Source: Figure 8 · p. 9, 2) What is the role of auxilliary… · p. 9
• Language loss yielded only a borderline 1–2% success-rate improvement, while video alignment and video contrastive loss contribute significantly to performance.The comparison was against a variant trained with all losses except Video-Text contrastive loss.Source: 2) What is the role of auxilliary… · p. 9

A. Task Specifications for Robots

• Vid2Robot infers tasks such as opening a jar or drawer from video visuals, without language input; prompt videos are framed as complementary task specifications for rapid adaptation.The passage contrasts visual task identification with language-conditioned policies that use the same embedding for the verb “open” across tasks.Source: A. Task Specifications for Robots · p. 9

B. Learning from Human Demonstrations

• Human-video robot learning faces embodiment differences; one cited approach translates a human image into the robot [44].The passage presents this as one approach to leveraging videos of people performing tasks for robot learning.Source: B. Learning from Human Demonstrations · p. 9
• Rollouts can fail when IK solutions block the camera, grasping transparent or deformable objects proves difficult, or distractors and scene differences cause object-recognition errors.The policy may predict the correct motion but apply it to the wrong object.Source: Figure 9 · p. 10

C. Imitation via Paired Demonstrations

• Prior paired-demonstration methods use a task-demonstration video and agent observations, with later approaches training transformer policies on these pairs.Early work [17] used temporal convolution and neighborhood attention in a demonstration network conditioning a manipulation policy; the passage also mentions additional constraints in recent approaches [30] [21].Source: C. Imitation via Paired Demonstrations · p. 10

V. LIMITATIONS AND FUTURE DIRECTIONS

• The authors note a performance gap remains for video-conditioned policies despite improvement over previous work.They introduce the limitations discussion as an investigation of rollout failures and directions for future work.Source: §V · p. 10
• Self-occlusion, grasping errors, and distractors can cause policy rollout failures.The paper qualitatively illustrates three failure examples in Fig. 9.Source: §V · p. 10
• Incomplete camera-based state estimates under occlusion or when the gripper leaves view are associated with reduced grasp success; multimodal sensor fusion may help.The authors note a significant drop in grasping success in Figure 6 and say robot-camera observations implicitly learn depth estimation, but that information is often incomplete.Source: §V · pp. 10–11
• Training uses carefully collected 5-to-20-second demonstrations, so long-horizon or in-the-wild videos require effective pairing with a few robot trajectories.The demonstrations come from three sources; the paper identifies pairing strategies as necessary for extending to these video types.Source: §V · p. 11

VI. CONCLUSION

• Vid2Robot outperforms BC-Z by over ∼20% with human-video prompts and by ∼17% for cross-object motion transfer.The transfer setting uses a prompt video with a different object from the one the robot manipulates, while retaining the same verb.Source: §VI · p. 11
• Vid2Robot uses paired videos of the same task across varied settings and cross-attention to encode prompts and robot state and decode actions.The model is trained for action prediction with cross-entropy and three auxiliary losses intended to encourage generalizable latent representations.Source: §VI · p. 11

Supplementary qualitative results

• Supplementary videos show successful rollouts, task-based successes, cross-object motion transfer, and failures involving self-occlusion, grasping errors, and ambiguous prompt interpretation.The videos also include qualitative long-horizon task composition and rollouts for tasks rare in the training dataset.Source: APPENDIX · p. 11

Implementation details

• The appendix provides detailed architecture and training hyperparameter tables, video augmentations, and video normalization for the pretrained ViT.The authors also state that they will release model code and trained checkpoints.Source: APPENDIX · p. 11

Rare tasks

• Vid2Robot completes rare training-set tasks including opening a glass jar, picking up a green microfiber cloth, and pulling out a napkin.Qualitative examples are shown in Fig. 10.Source: APPENDIX · p. 11, Figure 10 · p. 12

Experimental setup

• Experiments use mobile manipulators with a 7-degree-of-freedom arm, two-fingered gripper, and fixed base; client-server deployment uses 8 recent head-camera frames and runs at approximately 5–7 Hz.The base remains fixed during each episode because experiments focus on object manipulation.Source: APPENDIX · p. 11

Long-horizon tasks

• Vid2Robot completes long-horizon tasks by chaining prompt videos for subtasks, switching prompts when each episode terminates without resetting the robot.The example chains prompts to open a drawer, place a table object inside, and close the drawer; the task requires no explicit training on long-horizon videos.Source: APPENDIX · p. 11, Figure 11 · p. 12