Learning Transferable World-Action Models from Task-Paired Human Videos

1 Introduction

• Within-video prediction uses context and targets sharing scene appearance, viewpoint, and collection source, so it need not separate transferable interactions from demonstration-specific cues.This entanglement can make robot-control post-training more difficult across environment, embodiment, and viewpoint differences.Source: §1 · pp. 1–2
• WATCH conditions on one human demonstration and predicts another’s actions and future frames, using task-matched human–human pairs without requiring matched robot demonstrations.The human pairs vary in demonstrator, viewpoint, and environment, and robot data collection is reserved for adaptation.Source: §1 · p. 2
• Unlike task-matched human–robot pairing, human demonstrations can be collected without robot hardware or teleoperation, offering a potentially more scalable, cost-effective pre-training source.The paper compares WATCH with within-video learning and co-training baselines and reports gains from more paired human data.Source: §1 · p. 2
• MimicDroid success rises from 40.9% to 68.3% with Cosmos Policy, with gains across backbones and embodiments and the largest gains on unseen levels.WATCH also shows gains across all three context-free benchmarks.Source: §1 · p. 2, Figure 1 · p. 2
• WATCH’s cross-video pre-training aligns motions across collection sources, preserves motion information, and shifts attention toward hand–object interactions.These analyses are conducted before robot post-training.Source: §1 · p. 2, Figure 2 · p. 3

2.1 Preliminary

• World-action models jointly predict actions and their visual outcomes; the within-video formulation conditions and predicts within the same demonstration or episode.That means the conditioning and target share scene, viewpoint, and demonstrator or embodiment.Source: §2.1 · p. 3

2.2 Motivation

• The motivation is that the video backbone can attend broadly to edges and background; cross-video prediction shifts attention toward the acting hand and manipulated object.The paper describes this as encouraging capture of interactions shared across demonstrations with different object placements and demonstrators.Source: §2.2 · p. 3

2.3 Cross-Video World-Action Modeling

• The model conditions on a reference video and actions plus the query’s current frame and state, then predicts the query’s future frames and actions.Slot-type and physical-time tags allow the sequence layout to carry over to post-training; the joint velocity-matching loss applies to generated tokens.Source: Figure 3 · p. 4, §2.3 · p. 4
• Pairs come from different EgoVerse episodes, are gated by task taxonomy, and are ranked by standardized trajectory distance to match motion despite differences in reach.Each query’s reference set can include matches from its own and other collection sources.Source: §2.3 · p. 4
• 10.99M query–reference pairs yield 10.0K hours of paired 3-second windows; selected matches exceed same-task random cross-source pairs in trajectory-shape cosine similarity by 0.36.Each query has 10.52 references on average, including 6.35 cross-source matches; 90.7% have at least one cross-source reference.Source: §2.3 · p. 4
• In-context post-training uses a human reference, while context-free post-training removes reference slots and predicts from the robot’s observation, state, and task instruction.Both retain the same backbone and joint prediction objective.Source: §2.3 · pp. 4–5

• Action MSE rises from 0.0322 under matched references to 0.0335 with a source change and 0.0367 with a task change, with only the task-change increase significant.Attention remains on the interacting hands and object when the reference source changes.Source: §3 · pp. 5–6
• WATCH ranks matching motions from different human sources and robots earlier than both baselines, while its features predict future motion at Spearman correlations of 0.18 on human and 0.24 on robot windows.The original video model reaches 0.06 and 0.01, respectively, on those windows.Source: §3 · p. 6
• Joint action–video training concentrates both output slots’ attention on the acting hand and manipulated object, unlike action-only or video-only supervision.The compared variants use the same cross-video pairing.Source: §3 · p. 6, Figure 6 · p. 6
• On unseen RoboMIND, WATCH lowers action MSE from 0.046 to 0.039 and raises accuracy from 55.6% to 68.5% versus within-video pre-training.Video FVD and FID also improve over the original video model, while within-video pre-training worsens both.Source: §3 · pp. 6–7, Table 1 · p. 6
• Human-trained WATCH achieves mean Match@1 of 0.76 on human and 0.60 on robot windows, versus 0.21 and 0.19 for the original video model, when matched to robot-trained features.The means are over all layers.Source: Figure 7 · p. 6, §3 · p. 7

4.1 Experimental Setup

• The evaluation covers in-context GR1 MimicDroid with L1–L3 shifts and infeasible 2×-speed references, plus reference-free MimicDroid, TableTop-Sim, and RoboCasa settings.Backbones are Cosmos Policy and Cosmos3; trained methods use the same robot demonstrations.Source: §4.1 · p. 7

4.2 Comparison with Prior Methods

• With Cosmos Policy, WATCH raises MimicDroid in-context average success from 40.9% to 68.3%; with Cosmos3, it rises from 59.8% to 77.7%.The largest gains occur on the shifted or unseen levels: Cosmos Policy L2 rises from 29.4% to 72.5% and L3 from 18.3% to 40.8%.Source: Table 2 · p. 7, §4.2 · p. 8
• Without reference demonstrations, WATCH improves MimicDroid success from 49.3% to 69.0% with Cosmos Policy and from 49.7% to 63.4% with Cosmos3.Benefits also extend to TableTop-Sim and RoboCasa across robot embodiments.Source: §4.2 · p. 8, Table 3 · p. 8

4.3 Ablation Studies

• WATCH’s cross-demonstration objective outperforms within-video and co-training comparisons, with larger gains on L2 and L3 and advantages on all three context-free benchmarks.Under 2×-speed references, within-video reaches 30.3% versus 30.7% with no pre-training, while WATCH retains most of its gain.Source: §4.3 · pp. 8–9, Table 4 · p. 8
• Joint action–video pre-training has the highest average success in both in-context MimicDroid and context-free TableTop-Sim; action-only and video-only rankings differ across those settings.On TableTop-Sim, joint pre-training reaches 65.4% versus 64.6% for video-only.Source: §4.3 · p. 9, Figure 8 · p. 9
• Performance improves with more paired human videos, and WATCH matches all-data within-video pre-training on unseen in-context levels using 25% of the cross-video data.The scaling comparison uses more training steps for larger datasets; WATCH surpasses the baseline at 50% on L3 and 75% on L2.Source: §4.3 · p. 9, Figure 9 · p. 9

• Robot-policy pre-training scales cross-embodiment skills, but robot data collection remains a constraint; human videos offer additional experience through inferred actions or explicit motion supervision.Source: §5 · p. 10
• Prior co-training and alignment approaches ground human experience in executable actions but use robot data before target-domain adaptation, whereas WATCH pre-trains on task-matched human demonstrations alone.Source: §5 · p. 10
• Demonstration-conditioned learning supports task adaptation from robot demonstrations or human videos; WATCH instead studies human-to-human pre-training for representations used in both in-context control and reference-free behavior cloning.Source: §5 · p. 10

6 Conclusion

• WATCH’s cross-video prior aligns motion across people, rigs, and robots, attends to hand–object interactions, and improves in-context and context-free post-training across two backbones and three benchmarks.The largest gains occur where the scene departs from the demonstration, and performance grows with paired human video.Source: §6 · p. 10
• Extending the pairing beyond one human dataset to less-structured sources remains future work.Source: §6 · p. 10

Appendix

• The appendix covers data and implementation, extended representation analyses, human-versus-robot pre-training comparisons, supervision and scaling ablations, and qualitative rollouts.Source: Appendix · p. 14

A Limitations

• A shared Cartesian action space supports human–robot trajectory pairing, reference-action conditioning, and action-head warm-starting, but joint-space policies are not demonstrated.All post-training targets, including the 20-d bimanual ALOHA space, are Cartesian; a mobile base or moving camera would also require re-anchoring or expressing motion relative to the current view.Source: A Limitations · p. 14
• Pairing requires task labels and hand tracking, and a taxonomy class that mixes distinct motions can admit pairs that agree in task but not motion.The ranking separates such pairs better than same-class random pairs, but not perfectly.Source: A Limitations · p. 14

B.1 Action and State Representation

• The 57-dimensional action representation describes poses and fingertip positions, with human frames anchored at the waist at the episode’s first frame.Robot poses are base-relative; each source’s channels are normalized using robust per-channel statistics estimated over its windows.Source: §B.1 · p. 15

B.2 Pair Construction

• The pairing distance standardizes each motion descriptor by its window’s channelwise mean and standard deviation, then uses banded warping and a handedness penalty.Standardization reduces sensitivity to motion magnitude but is not strict invariance because position, rotation, and hand channels do not all scale with reach.Source: §B.2 · p. 15
• The reference set is task-gated and ranked by hand-motion descriptors; 9.3% of queries have no cross-source reference, all of them MECKA queries.Each query has an average of 10.5 references, including 6.4 from other sources; one reference is sampled uniformly from its set during training.Source: §B.2 · pp. 15–16

B.3 Data Statistics

• Pre-training uses 1,045,007 human query windows from 14,137 episodes, with roughly three-second windows and query episodes excluded from their references except in the human play set.The data snapshot comprises EgoVerse human sources and MimicDroid human play; robot subsets are excluded from the query and reference roles.Source: §B.3 · p. 16
• The 10.99M query-reference links span 2.0 sources and 10.4 episodes per query on average, and the sampled reference stream retains 95% of the pool’s motion diversity.90.6% of episodes serve as references at least once; the RBF Vendi score is 247.8 for the stream versus 260.5 for the pool.Source: §B.3 · p. 16

B.4 Held-Out Data

• Held-out analyses use 150 whole episodes, while the separate EgoDex set is an unseen human source paired only against itself for Table 6.Representation analyses use 122 episodes and 120 cross-source pairs; robot analyses use 396 RoboMIND windows across three robots and six tasks.Source: §B.4 · p. 17

B.5 MimicDroid Robot Data

• MimicDroid robot training episodes are generated by retargeting three canonical human demonstrations per task into sampled scenes and replaying them on the GR1.The evaluation holds out styles and objects in L2 and layouts in L3; each task is evaluated over 30 episodes, except one L2 task with 40.Source: §B.5 · pp. 17–18

B.6 Pair Examples

• Each query window is paired with same-task references from its own source and, by design, from other sources that differ in person, camera, and scene.Figure 10 illustrates three such query-reference pairs across Scale, MECKA, and Aria.Source: §B.6 · p. 18, Figure 10 · p. 19

C.1 Architecture

• The pre-training sequence conditions on reference actions and video plus the query’s current frame, then generates query actions and future video.Continuous time encodings and slot-type embeddings accommodate variable slot layouts; the query frame sits immediately before the generation boundary to preserve the video continuation prior.Source: §C.1 · p. 20

C.2 Training

• For RoboCasa Cosmos3 pre-training, RoboMIND Franka and TienKung streams make up 12.5% of training samples alongside EgoVerse pairs.The robot windows use 16-frame action chunks and are served without references because RoboCasa is a context-free benchmark.Source: §C.2 · p. 21
• Cosmos Policy post-training updates the video backbone to Predict2.5 and uses future-video and action prediction without auxiliary future-proprioception prediction or rollout-data augmentation.Pre-training runs for 8K iterations; the MimicDroid and TableTop-Sim robot action chunks are 32 frames at 20 Hz.Source: §C.2 · p. 21

C.3 Pre-Training Variants

• The within-video baseline predicts a window’s actions and future frames from its own observation, while co-training adds robot windows as queries without reference content.The baseline uses 29 latents versus WATCH’s 50, so it costs less per step at the same 8K iterations.Source: §C.3 · p. 21

D.1 More Frames for the Motivation

• Cross-video pre-training places attention on the hand and object, whereas the original backbone spreads attention over edges and background and within-video pre-training shifts it to the arm.In the original model’s successful rollout, attention is near the hand and manipulated object; in the failed rollout, it focuses on the wrist and forearm.Source: §D.1 · p. 21, Figure 12 · p. 22

D.2 Prediction on More Held-Out Data

• WATCH leads on every action metric and on FVD and LPIPS on unseen EgoDex and RoboMIND, and matches or exceeds co-training on Humanoid Everyday except FVD.Co-training was trained on Humanoid Everyday, whereas WATCH, within-video, and the original model had not seen it.Source: Table 6 · p. 22, §D.2 · p. 22
• In the held-out clips, WATCH tracks the ground-truth gripper on Humanoid Everyday and lifts the object on EgoDex as the hand does.The original model and within-video baseline drift on Humanoid Everyday; the other models leave the object on the table in the EgoDex example.Source: Figure 13 · p. 23, §D.2 · p. 24

D.3 Swapped Reference on More Queries

• With swapped references, WATCH attends to hands and manipulated objects in both reference and query frames, even when the reference comes from another source and place.The video backbone instead attends to edges, ceiling lights, and background, with little change between reference sources.Source: §D.3 · p. 24, Figure 14 · p. 25

D.4 Attention on Robot and Human Frames, Three Models

• On robot and human frames, attention shifts from background under the original model, to the arm under within-video training, to the hand and object under WATCH.The examples include four MimicDroid robot frames and three EgoVerse queries.Source: §D.4 · p. 24, Figure 15 · p. 26

D.5 Analysis Protocol for the Representation Analyses

• Retrieval AUC is 0.78 on human windows and 0.74 on robot windows for WATCH, above within-video at 0.66 and 0.69 and the original model at 0.50 and 0.65.Pairs are matched across human collection sources or across robots, and candidate partners are ranked among windows from the partner’s source or robot.Source: §D.5 · pp. 24, 27

D.6 Additional Probing Results

• WATCH improves three-way probes on all four motion attributes in both human and robot windows; four-way probes preserve the model ordering on every attribute.The largest gains over the original model are for human lateral direction and robot grasp state in the three-way probes.Source: §D.6 · p. 27, Figure 17 · p. 28
• Held-out-embodiment regression reaches 0.18 on the human set and 0.24 on the robot set for WATCH, versus 0.06 and 0.01 for the original video model.The robot-set gain is larger even though no model has seen robot data; WATCH also receives the reference’s future actions as input, which bears directly on this probe.Source: §D.6 · pp. 27–28

E.1 Setup

• The human-video and robot-data arms use the same recipe and about 1K hours of paired video each, with only the robot arm seeing RoboMIND during pre-training.Both are compared on RoboMIND windows, including 300 windows spanning Franka, UR5e, and TienKung.Source: §E.1 · p. 28

E.2 Analysis

• After alignment, human-video and robot-data features for the same robot windows lie 0.03 to 0.06 apart on average, versus 0.12 to 0.31 for the original model.The pattern holds across three robots and at layers 12 and 27, though the human-video arm has never seen a robot.Source: §E.2 · p. 28, Figure 18 · p. 29
• Across layers 10–27, match@1 averages 0.50 on robot windows and 0.71 on human windows for the human-video arm, versus 0.13 and 0.11 for the original model.The human-video arm preserves the robot-data arm’s nearest neighbors across layers, while the original model falls to 0.20 or less from layer 10 onward.Source: §E.2 · pp. 28–29, Figure 19 · p. 29
• On both Franka and MECKA queries, the human-trained and robot-trained models attend to hands or grippers and manipulated objects across reference and query frames.Neither spreads attention over the background as the original video model does.Source: §E.2 · p. 30, Figure 20 · p. 30

E.3 Downstream Transfer from Human and Robot Pre-Training

• On in-context L2, human-pair pre-training raises success from 45.0% to 72.5%, while the largest robot-pair setting reaches 49.0% at about twice the estimated budget.The comparison assumes an 8× robot-to-human collection-cost ratio, with sensitivity shown from 4× to 16×.Source: §E.3 · pp. 30–31, Figure 21 · p. 31

F.1 Baselines

• Baseline evaluation uses the same robot demonstrations for post-training, while π0.5 and MolmoAct2 use 20-dimensional MimicDroid actions rather than the 57-dimensional target used by most models.On TableTop-Sim, every model uses ALOHA’s 20-dimensional action target.Source: §F.1 · p. 31

F.2 Per-Task Results

• Cross-video models improve multiple MimicDroid tasks on L2 and L3, with only those models reaching double digits on L3 PnPSinkToMicrowaveTop; on TableTop-Sim, unseen box-into-pot separates Cosmos3 from Cosmos Policy variants.Context-free MimicDroid gains are largest on L2 pick-and-place tasks; TableTop-Sim lift box is the hard case on both splits.Source: §F.2 · p. 31, Table 7 · p. 32, Table 8 · p. 32

F.3 Data Scaling Setup

• The scaling sweep uses 25%, 50%, and 75% of query episodes and increases training iterations with data, so each arm sees the same 0.49 epochs.Data and compute grow together in the sweep.Source: §F.3 · pp. 32–33

F.4 Supervision Ablation: Post-Training Target as Well

• Video-plus-action pre-training is the best target under both post-training targets on both benchmarks; action-only pre-training falls below no pre-training on TableTop-Sim.Action-only post-training is competitive on L1 and close on L3, but trails on L2 and collapses on TableTop-Sim, which has no reference.Source: Figure 22 · p. 33, §F.4 · p. 33

G.1 MimicDroid, In-Context

• In in-context MimicDroid rollouts, WATCH tracks the demonstrated path as the reference window advances, while the unpretrained policy hovers or stalls after grasping.The figure compares the same task at the same time steps, with the failing episode continuing to the time limit.Source: §G.1 · p. 34, Figure 23 · p. 35

G.2 MimicDroid, In-Context with 2× Speed Demonstrations

• On 2×-speed demonstrations, WATCH follows task progress and succeeds at closing the cabinet door and placing an object, while the unpretrained policy stalls or times out.The demonstrated hand closes the door within one chunk; in the two-stage task, WATCH places the object in the cabinet.Source: §G.2 · p. 34, Figure 24 · p. 36, Figure 25 · p. 37

G.3 TableTop-Sim, Context-Free

• In context-free TableTop-Sim rollouts, WATCH completes tasks within the same window where unpretrained policies reach or lift objects but miss the completing step.Unpretrained examples include dropping a plate, failing to lift a reached box, or not releasing shoes; WATCH finishes on both splits.Source: §G.3 · p. 34, Figure 26 · p. 38