Source-linked AI summary
Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization
Jiaming Zhou, Qihang Zhang, Gangwei Xu, Cunxin Fan, Yujie Zhao, Ruilin Wang, Yiming Luo, Shuai Yang, Xing Zhu, Yujun Shen, Junwei Liang, Yinghao Xu
TL;DR
Zero-shot cross-task generalization requires robots to execute unseen manipulation tasks from deployment-time task specifications. Zero-WAM uses human videos as visual guidance and achieves 46.95% average success across seven unseen RoboTwin 2.0 tasks, outperforming LingBot-VA by 29.50 percentage points.
Problem
Zero-shot robotic manipulation needs task interfaces that convey unseen tasks’ spatial, intermediate-state, and temporal requirements beyond language alone.
Method
Zero-WAM combines human-video task specification with a causal video-action policy, trained using semantically matched HumanGen pairs and task-balanced robotic data.
Results
46.95% average success across seven unseen RoboTwin 2.0 tasks, outperforming LingBot-VA by 29.50 percentage points; real-world tests generalize across three unseen task families.
Takeaways & Limitations
Human video guidance supports zero-shot cross-task generalization in simulation and real-world manipulation without corresponding robot data collection or model updates.
Takeaways & Limitations
Experiments focus primarily on stationary tabletop manipulation, leaving mobile manipulation and substantially longer-horizon, dynamic, and unstructured environments for future work.
Abstract
from arXiv · showhide
Zero-shot cross-task generalization, where a policy must execute manipulation tasks never seen during training, remains a central challenge in robot learning. In large language models, a novel task can be performed simply by specifying it in the context, without any parameter update. This form of in-context learning (ICL) turns generalization into a problem of task specification. To achieve cross-task generalization, we bring this paradigm to robotic manipulation, and argue that the natural task specification for manipulation is a human video: unlike language, it provides rich visual cues about the intended task evolution. We present Zero-WAM, a causal video-action model that executes unseen tasks by following in-context human video guidance. To address the scarcity of task-rich paired human-robot data, we propose an automatic pipeline that converts task-sampled robot trajectories into semantically matched human videos, yielding HumanGen, a dataset of 74.2K human-robot ICL pairs across 8.6K tasks. For model training, we further introduce an in-context future chunk prediction (IFP) objective that suppresses shortcuts learned from seen tasks and forces the policy to draw task information from the video prompt. On seven unseen tasks in RoboTwin 2.0 simulation, Zero-WAM achieves a 47.0% average success rate, an absolute improvement of 29.5 percentage points over the strongest video-action baseline. In real-world evaluations, it follows human video guidance to generalize to unseen task configurations involving multi-object scenes, long-horizon manipulation, and fine-grained insertion.
1 Introduction
Zero-WAM frames zero-shot robotic task generalization as in-context world-action modeling, using human videos or language to specify unseen tasks without parameter updates. It combines scalable human–robot video pairing with an in-context future chunk prediction objective and achieves strong simulation and real-world generalization.
- Approach: Zero-WAM is a causal video-action model that supports both language instructions and human demonstration videos as task specifications.It autoregressively predicts future robot videos and executable actions, conditioning on demonstrated visual state evolution when human videos are provided.
- Data: The HumanGen pipeline converts task-sampled robot trajectories into semantically matched human videos, producing 74.2K human–robot ICL pairs across 8.6K tasks.The paired robot trajectories retain executable actions; the same task-level sampling also yields Task-diverse VA data for autoregressive robotic video-action pre-training.
- Training objective: The in-context future chunk prediction objective discourages shortcut learning from robot history and text by encouraging representation of longer-term task evolution from the human-video prompt.Standard next-chunk prediction can often rely only on local robot history, especially for tasks seen during training.
- Evaluation: Real-world experiments show human-video-guided generalization to unseen multi-object, long-horizon, and precision-demand insertion configurations, outperforming LingBot-VA across all three task families.These evaluations do not require corresponding robot data collection or model-parameter updates.
2 Data Curation
Section 2 curates task-diverse robot video-action data by sampling trajectories at the task level and introduces HumanGen, an automatically generated collection of semantically matched human-robot in-context learning pairs. The pipeline uses vision-language analysis, video generation, and quality filtering to provide visually diverse human video instructions from robot trajectories.
- Task-diverse VA data: Task-level resampling reduces the dominance of repeated executions and creates task-balanced robotic video-action pre-training data from public datasets.The source datasets include AgiBot [23], InternData-A1 [24], Open-X-Embodiment, RoboCOIN [25], and RoboMIND [26].
- HumanGen generation: HumanGen automatically converts task-sampled robot trajectories into semantically matched human manipulation videos, avoiding manual paired-data collection while supporting diverse visual alignment.Generated videos vary background, viewpoint, environment style, object instance, and other visual factors; Figure 2 summarizes the composition and pipeline.
- HumanGen generation: The generation pipeline extracts task names and object-state transitions with a VLM, transforms the initial robot frame, synthesizes a human video, and filters outputs for semantic preservation and physical plausibility.VLMs include Gemini 3.1 Pro and Qwen3.6-Plus, while video generation uses Wan 2.7 or Kling AI 3.0.
- HumanGen composition: HumanGen is organized into Pre-train ICL (External), Pre-train ICL (In-house), Simulation ICL, and Real-world ICL, with source trajectories sampled by task rather than raw frequency.The composition spans public, in-house, simulation, and real-world sources, preserving executable robot actions alongside generated human instructions.
- Dataset comparison: HumanGen differs from prior task-level paired human-robot datasets by using automatic human-video generation and combining multiple data sources for scalable in-context learning data construction.The comparison emphasizes automatic generation rather than manual human collection and multi-source construction; Table 1 provides the dataset-comparison scope.
3 Zero-WAM: In-Context World Action Modeling
Zero-WAM is a causal world-action model that uses language or human video as in-context task specifications for executing unseen manipulation tasks. It transfers task-level dynamics from human demonstrations through robot video prediction and action decoding, while future-chunk prediction discourages history-based shortcuts.
- 3.1 Causal Video-Action Modeling: Zero-WAM executes unseen manipulation tasks without finetuning by conditioning a pretrained world-action model on either language instructions or human video demonstrations.The model is pretrained on large-scale human-robot in-context learning pairs and task-balanced robotic video-action data.
- 3.2 Human Video as In-Context Task Specification: Human video serves as an in-context task specification, with the video Transformer transferring task semantics into predicted robot video before inverse-dynamics action decoding.The human and robot demonstrations may differ in embodiment, viewpoint, background, and object placement, requiring task-level correspondence rather than motion copying.
- 3.3 In-Context Future Chunk Prediction: Teacher-forcing training can encourage predicting the next robot chunk from recent history, causing unseen-task failures because the model ignores the human video.This shortcut arises because immediate next-chunk prediction on seen tasks can often be solved by extrapolating robot history.
- 3.3 In-Context Future Chunk Prediction: In-context future chunk prediction adds strided future robot-video targets and flow-matching modules, forcing the model to model longer-range task evolution beyond immediate history.Each module predicts a future chunk using fused intermediate representations, robot history, action history, and the language instruction.
4 Experiments
Zero-WAM achieves 47.0% average success on seven unseen RoboTwin 2.0 tasks, improving 29.5 percentage points over the strongest video-action baseline. It also generalizes to unseen configurations involving multi-object scenes, long-horizon manipulation, and fine-grained insertion, while ablations support the roles of large-scale ICL pre-training and IFP.
- RoboTwin 2.0 results: 47.0% average success on seven unseen RoboTwin 2.0 tasks represents a 29.5-point improvement over the strongest video-action baseline.Zero-WAM outperforms both baselines on all seven unseen tasks, with gains on articulated-object, relocation, and long-horizon manipulation; it achieves 84.87% on place empty cup and is the only main method with non-zero success on stack blocks three.
- Real-world evaluations: Real-world evaluations test human-video guidance across unseen object-container configurations, three-object sequential manipulation, and precise two-table-leg insertion.These settings respectively probe unseen objects or containers, long-horizon action order, and fine-grained target-hole specification.
- Ablations: Human-video ICL raises average success from 10.98% to 36.36% over WAN-Action, while Zero-WAM without pre-training reaches 17.45% despite LingBot-VA’s robotic video-action pre-training.The comparison indicates that human-video instructions provide task information beyond text-only conditioning, while small-scale ICL training remains insufficient for unseen long-horizon tasks.
- Ablations: IFP improves stack blocks three from 0.00% to 9.00%, breaking the zero-success barrier on an unseen long-horizon manipulation task.This supports IFP’s role in eliciting the human-video-following capability learned from ICL data.
5 Conclusions and Discussions
The work frames zero-shot robotic cross-task generalization as an in-context task-specification problem addressed through causal video-action modeling, scalable human-video instructions, and task-balanced data. It argues that semantically aligned human–robot data can help bridge abundant egocentric human video and scarce robot trajectories.
- Conclusions: Zero-WAM targets unseen robotic manipulation tasks without corresponding robot data collection or model-parameter updates, using causal video-action modeling and a scalable in-context task interface.The interface uses human video instructions generated at scale and task-balanced data construction.
- Conclusions: HumanGen converts task-sampled robot trajectories into semantically matched human video instructions, while task-level trajectory sampling curates task-diverse video-action data from robotic pre-training data.These data sources support Zero-WAM’s use of language and human video as task interfaces.
- Discussions: Zero-shot cross-task generalization requires stronger transferable priors in robotic foundation policies and richer multimodal task interfaces that convey intent.The discussion identifies human demonstrations as natural task specifications and emphasizes continued growth in their scale and diversity.
- Discussions: Egocentric human video is promising because it can be collected at far greater scale than robot demonstrations, but embodiment, observation, and action gaps complicate its use.The semantically aligned human–robot data introduced by this work may bridge abundant human video and comparatively scarce robot trajectories.
6 Related Works
Related work frames zero-shot cross-task manipulation as harder than visual generalization because policies must infer unseen task-conditioned dynamics. Prior VLA, WAM, test-time-training, and human-video approaches motivate Zero-WAM’s adaptation-free use of automatically generated, action-grounded human-video instructions.
- Cross-task generalization: VLA models [42] [43] [3] [44] [5] [6] [45] [46] [47] scale language-conditioned manipulation over heterogeneous robot data, but vision-language-to-action mismatch leaves the cross-task gap unresolved.Cross-task manipulation requires executing unseen tasks without robot demonstrations for those tasks and is more challenging than changing scenes or object attributes around a seen task.
- World-action models: WAMs [48] [49] [50] [51] [52] [53] [54] [55] [56] [57] shift unseen-task generalization toward predicting future videos, building on evidence that accurate actions can be decoded from correctly predicted future dynamics.Video-action modeling, including LingBot-VA [11] [12] and DreamZero [13], couples or exploits predicted future visual dynamics for embodied control.
- Test-time adaptation: Unlike WAM-TTT [66] and RoboTTT [67], which adapt at test time through memory or fast-weight updates, Zero-WAM pursues unseen-task execution without such adaptation.WAM-TTT [66] updates lightweight memory from raw human videos, while RoboTTT [67] compresses execution histories into the policy state.
- Human-video learning: Human videos provide diverse object interactions for robot learning, supporting visual representations, manipulation priors, and policy pre-training with varying degrees of action grounding [68] [69] [70] [71] [34] [73] [74] [75].Zero-WAM targets the more task-aligned regime by automatically pairing generated human videos with robot trajectories.
- Human-video data: Zero-WAM automatically generates HumanGen from task-sampled robot video-action data, pairing human videos with executable robot trajectories to scale task-diverse video instructions.This makes human-video task specification a scalable component of WAM training for zero-shot cross-task generalization.