Source-linked AI summary
Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization
Jiaming Zhou, Qihang Zhang, Gangwei Xu, Cunxin Fan, Yujie Zhao, Ruilin Wang, Yiming Luo, Shuai Yang, Xing Zhu, Yujun Shen, Junwei Liang, Yinghao Xu
TL;DR
Zero-shot cross-task generalization requires robots to execute unseen manipulation tasks without task-specific parameter updates, while language can underspecify intended visual evolution. Zero-WAM uses human videos as in-context task specifications, combines them with scalable HumanGen data and IFP training, and achieves strong simulation and real-world generalization. On seven unseen RoboTwin tasks, it reaches 46.95% average success and surpasses LingBot-VA by 29.50 percentage points.
Problem
Zero-shot cross-task generalization requires robotic policies to execute unseen manipulation tasks without collecting corresponding robot data or updating model parameters, while language often underspecifies task evolution.
Method
Zero-WAM is a causal video-action model that uses human videos or language as in-context task specifications, trained with HumanGen data and an IFP objective.
Results
46.95% average success across seven unseen RoboTwin tasks outperforms LingBot-VA by 29.50 percentage points, while real-world tests cover three unseen task-configuration families.
Takeaways & Limitations
Human video can provide an in-context task interface for zero-shot robotic manipulation across unseen simulated tasks and real-world configurations.
Takeaways & Limitations
Experiments focus primarily on stationary tabletop manipulation; future work must extend the paradigm to dynamic, unstructured, mobile, and substantially longer-horizon environments.
Abstract
from arXiv · showhide
Zero-shot cross-task generalization, where a policy must execute manipulation tasks never seen during training, remains a central challenge in robot learning. In large language models, a novel task can be performed simply by specifying it in the context, without any parameter update. This form of in-context learning (ICL) turns generalization into a problem of task specification. To achieve cross-task generalization, we bring this paradigm to robotic manipulation, and argue that the natural task specification for manipulation is a human video: unlike language, it provides rich visual cues about the intended task evolution. We present Zero-WAM, a causal video-action model that executes unseen tasks by following in-context human video guidance. To address the scarcity of task-rich paired human-robot data, we propose an automatic pipeline that converts task-sampled robot trajectories into semantically matched human videos, yielding HumanGen, a dataset of 74.2K human-robot ICL pairs across 8.6K tasks. For model training, we further introduce an in-context future chunk prediction (IFP) objective that suppresses shortcuts learned from seen tasks and forces the policy to draw task information from the video prompt. On seven unseen tasks in RoboTwin 2.0 simulation, Zero-WAM achieves a 47.0% average success rate, an absolute improvement of 29.5 percentage points over the strongest video-action baseline. In real-world evaluations, it follows human video guidance to generalize to unseen task configurations involving multi-object scenes, long-horizon manipulation, and fine-grained insertion.
1 Introduction
Zero-WAM frames zero-shot robotic task generalization as in-context world-action modeling, using human videos to specify unseen manipulation tasks and guide executable behavior. It combines scalable paired-data construction with an IFP training objective and improves unseen-task performance in simulation and real-world configurations.
- Motivation and formulation: Human videos provide visual evidence of spatial constraints, intermediate states, and temporal scene evolution that language often underspecifies.They serve as natural in-context specifications of intended manipulation tasks.
- Data construction: 74.2K human-robot ICL pairs spanning 8.6K tasks are produced by converting task-sampled robot trajectories into semantically matched human manipulation videos.The same task-level sampling produces Task-diverse VA data for task-balanced robotic video-action pre-training.
- Model training: IFP supervises multiple strided future robot-video chunks, encouraging the policy to encode longer-term task evolution from the human video rather than rely on local-history shortcuts.Standard next-chunk prediction can otherwise be solved from robot history, particularly on training tasks.
- Evaluation: 46.95% average success across seven unseen RoboTwin tasks exceeds LingBot-VA by 29.50 percentage points.The comparison is reported for zero-shot cross-task generalization in RoboTwin 2.0 simulation.
- Evaluation: Real-world experiments show human-video-guided generalization to unseen configurations involving multi-object scenes, long-horizon manipulation, and precision-demand insertion.Zero-WAM outperforms LingBot-VA across all three task families without corresponding robot-data collection or parameter updates.
- Motivation and formulation: Zero-shot robotic task generalization is formulated as in-context world-action modeling, with language and human videos serving as task instructions.The policy infers an unseen task from deployment-time context without parameter updates.
2 Data Curation
The paper curates task-balanced robotic video-action data and constructs HumanGen by automatically converting task-sampled robot trajectories into semantically matched human video instructions. HumanGen combines multiple sources and task settings to broaden paired data for cross-task generalization.
- Task-diverse VA data: Task-diverse VA data are curated by re-sampling public robotic pre-training trajectories at the task level to reduce dominance by repeated executions.The source datasets include AgiBot, InternData-A1, Open-X-Embodiment, RoboCOIN, and RoboMIND.
- In-context human video generation: HumanGen pairs generated human manipulation videos with corresponding robot trajectories that retain executable actions.The pipeline uses task-sampled robot trajectories and preserves task semantics while varying background, viewpoint, environment style, object instance, and object placement.
- In-context human video generation: The generation pipeline analyzes each robot video with a VLM, synthesizes a human video from an edited initial frame, and filters outputs for semantic preservation and physical plausibility.The VLM extracts task name, object states, and state changes before video synthesis and evaluation.
- HumanGen dataset: HumanGen is organized into Pre-train ICL (External), Pre-train ICL (In-house), Simulation ICL, and Real-world ICL sources.These sources cover public and in-house data, RoboTwin simulation tasks, and real-world bimanual Franka evaluation tasks.
3 Zero-WAM: In-Context World Action Modeling
Zero-WAM is a causal video-action policy that uses language or human videos as task specifications for unseen robotic manipulation tasks. It predicts future robot video and aligned executable actions, while IFP encourages the model to encode longer-term task evolution from human-video context.
- Human video as task specification: Zero-WAM uses either a language instruction or a human video demonstration as the task specification for unseen-task execution.The policy is pretrained on human-robot ICL pairs and task-balanced robotic video-action data.
- Causal video-action modeling: At each control step, the causal model predicts the next robot video chunk and its temporally aligned executable action chunk.The joint prediction is factorized into video prediction followed by action decoding.
- Human video as task specification: Human video is prepended as prefix memory so robot-video prediction can use demonstrated task evolution despite differences in embodiment, viewpoint, background, and object placement.Action decoding instead uses robot-domain history and the predicted next robot video chunk, without directly attending to the human video.
- In-context future chunk prediction: IFP addresses a shortcut in which immediate next-chunk prediction can rely on recent robot history rather than the human-video prompt.Its design prevents the auxiliary branch from independently using the prompt, encouraging the main video Transformer to encode task information from it.
- In-context future chunk prediction: IFP predicts multiple strided future robot-video chunks from the current robot-video representation, using weighted flow-matching losses for those targets.The auxiliary modules operate on representations produced after the main branch has interacted with the human video and are removed at inference.
4 Experiments
Experiments test Zero-WAM on unseen RoboTwin tasks and real-world configurations, using controlled baselines and ablations to assess human-video instructions, IFP, and task-balanced pretraining.
- Simulation setting: Zero-WAM is evaluated on seven RoboTwin 2.0 tasks held out at the task level, covering unseen-object, articulated-object, bimanual, and long-horizon manipulation.The simulation split uses 43 tasks for post-training and reserves 7 tasks for evaluation.
- Simulation results: 46.95% average success rate across seven unseen tasks exceeds LingBot-VA’s 17.45% and WAN-Action’s 10.98%.Zero-WAM outperforms both baselines on all seven tasks.
- Simulation results: 84.87% success on place empty cup more than doubles both baselines, while stack blocks three remains difficult but receives non-zero success.The results span unseen articulated-object, relocation, and long-horizon dynamics.
- Real-world experiments: Real-world evaluations cover unseen object-container configurations, multi-object long-horizon ordering, and precise two-table-leg insertion.The evaluations use human video instructions and test configurations involving unseen objects, containers, target holes, or manipulation orders.
- Ablation studies: 36.36% average success from adding human video instructions rises from WAN-Action’s 10.98%, while all three variants score zero on stack blocks three without full pretraining.This isolates the contribution of in-context human videos and highlights the role of large-scale task-diverse ICL pretraining for the hardest long-horizon task.
- Ablation studies: IFP raises stack blocks three success from 0.00% to 9.00%, indicating that future-chunk supervision helps elicit human-video-following capability.The task is an unseen long-horizon manipulation task.
- Ablation studies: 39.44% seven-task average for the text-only task-balanced variant exceeds LingBot-VA by 21.99 percentage points, supporting balanced robotic pretraining as an additional contributor.This variant masks the human-video condition to isolate task-balanced robotic data.
5 Conclusions and Discussions
The paper frames human video as a scalable task interface for zero-shot robotic manipulation, supported by semantically matched data and task-balanced robotic pretraining. Experiments provide evidence for unseen-task execution, while the current scope remains primarily stationary tabletop manipulation.
- Conclusions: HumanGen turns task-sampled robot trajectories into semantically matched human video instructions paired with executable robot actions.The dataset contains 74.2K human-robot ICL pairs spanning 8.6K tasks.
- Conclusions: Zero-WAM supports both language and human video as task specifications within a causal video-action model.The model predicts future robot videos and executable actions from either instruction form.
- Discussions: The experiments focus primarily on stationary tabletop manipulation, leaving extension to dynamic, unstructured, mobile, and substantially longer-horizon environments for future work.This is the paper’s stated scope boundary rather than a claim that the method fails in those settings.
- Discussions: The approach is motivated by bridging abundant egocentric human video and comparatively scarce robot trajectories through semantically aligned human-robot data.The paper identifies embodiment, observation, and action gaps between humans and robots as complications for this bridge.
6 Related Works
Related work studies cross-task generalization, video-action and world-action models, human-video learning, and test-time adaptation. Zero-WAM differs by automatically generating paired human-video data from robot trajectories and using it without test-time parameter or memory updates.
- Cross-task generalization: Cross-task generalization requires executing an unseen manipulation task without collecting robot demonstrations for that task.It is harder than visual generalization because the model must infer unseen task-conditioned dynamics from instructions and observations.
- Vision-language-action models: VLA models scale language-conditioned manipulation over heterogeneous robot data, but the vision-language-to-action mismatch remains a cross-task challenge.The cited discussion reports that representative VLA policies fail under zero-shot cross-task manipulation.
- Video-action and world-action models: World-action models shift unseen-task generalization toward predicting future robot videos, from which executable actions can be decoded.This direction builds on prior evidence that accurate actions can be decoded from correctly predicted future videos.
- Video-action and world-action models: Video-action models couple causal video prediction with action prediction, while prior work also explores visual-domain transfer and embodied control.The cited examples include LingBot-VA and DreamZero.
- Test-time adaptation: Test-time training methods execute unseen tasks by updating deployment-time memory or fast weights, unlike Zero-WAM’s no-adaptation path.WAM-TTT updates lightweight memory from raw human videos, whereas RoboTTT compresses execution histories through fast-weight updates.
- Human videos for robotics: Human videos offer diverse object interactions for learning visual representations, manipulation priors, and policy pretraining, with varying grounding in robot actions.The paper positions human video as an important data source because robot-only collection is difficult to scale.
- Human videos for robotics: Zero-WAM automatically generates HumanGen from task-sampled robotic video-action data, pairing scalable human instructions with task-diverse robot dynamics.This targets the missing regime of large-scale, task-rich paired human-robot data.