PaperlayerView exact revision on arXiv

Source-linked AI summary

InternW0-$Δ$: A World Action Model Bridging Predictive Dynamics and Actions with 20K+ Hours of Open Data

Xingyu Miao, Zizun Li, Baole Fang, Kaiwen Song, Tenghui Wang, Hanxue Zhang, Yating Wang, Xudong Li, Yuping He, Xueyuan Wei, Chao Gao, Xijie Yang, Yingxiang Xu, Kerui Ren, Wenqi Guo, Jianjun Zhou, Xinzhe Wang, Weiguang Zhao, Ni Yang, Zetao Cai, Yufei Xue, Hengjie Li, Zeyu He, Yuanzhen Zhou, Rong Fu, Jianyang Zhang, Siwei Cui, Fuxian Huang, Yunsong Zhou, Xing Gao, Yifei Yao, Qiaojun Yu, Kailin Li, Ming Zhou, Mu Huang, Xinyue Li, Wenze Cui, Bingqi Jiang, Xueyue Zhu, Junting Dong, Haoyu Guo, Tao Lu, Mulin Yu, Bowen Zhou, Bin Zhao, Tianfan Xue, Weinan Zhang, Chunhua Shen

arXiv:2609.31394v1cs.ROcs.CV

TL;DR

Robot policies need to turn pretrained visual-dynamics knowledge into action-relevant representations without requiring future-video generation during online control. InternW0-∆ combines video and action experts, predictive and geometric priors, and heterogeneous-data pretraining; it reports leading results across simulation benchmarks and deployment on real robots.

  • Problem

    Predicting future observations from video does not by itself provide the task-relevant changes, geometry, motion, and scene grounding needed for action generation without online future generation.

    Source: §1 · p. 2

  • Method

    InternW0-∆ couples pretrained video and action experts, using Causal Imprint and training-only 4D-aware distillation to provide predictive, geometric, and motion representations.

    Source: §1 · p. 2

  • Results

    InternW0-∆ leads reported results across four simulation benchmarks and is deployed on gripper-based and dexterous-hand robots.

    Source: §1 · p. 3

  • Takeaways & Limitations

    A shared pretrained checkpoint can be adapted through post-training across task distributions, embodiments, and control interfaces, including gripper-based and dexterous-hand manipulation.

    Source: §8 · p. 38

    The contribution of egocentric human demonstrations remains limited and has not been systematically examined across sources, conversion strategies, scaling ratios, or supervision forms.

    Source: §8 · p. 38

Abstract

from arXiv · show

World Action Models (WAMs) jointly model visual dynamics and action generation for generalist robot manipulation. A central challenge is to integrate priors from large-scale pretrained models---including visual dynamics, scene semantics, geometry, and motion---into a unified framework for robot action generation. We introduce InternW0-$Δ$, a unified WAM pretrained on a heterogeneous corpus that outperforms prior methods across simulation benchmarks and real-robot platforms. InternW0-$Δ$ combines pretrained visual dynamics, scene-level semantics, 4D geometric and motion priors, and action generation within a Mixture-of-Transformers (MoT) framework. A pretrained video expert and an action expert interact under semantic guidance from a frozen VLM, while a pretrained 4D foundation model injects geometric and motion priors through training-only distillation. We further introduce Causal Imprint, which learns future-relevant scene changes from training-only future supervision and provides predictive representations directly to the action expert without future-video rollout at inference. For large-scale joint training, we construct a heterogeneous corpus of robot demonstrations, UMI data, egocentric human demonstrations, and Ego2Robot data, curated and aligned under a common state-action representation. The resulting corpus contains over 20K hours of processed training data, to our knowledge the largest open-source corpus of its kind. We pretrain InternW0-$Δ$ on this corpus and demonstrate strong performance across simulation benchmarks and real-robot platforms. We will open source the training code, model weights, infrastructure, data-processing pipeline, and processed data where licenses permit. Project page: https://internrobotics.github.io/InternW0-Delta/

DATA FILTERING & VERIFICATION

The section names checks involving signal, state-action, and visual quality, plus boundary trimming and action-magnitude review.

DATA FILTERING & VERIFICATION

  • Filtering references signal, state-action, and visual-quality checks.The source fragment does not specify how these checks are performed.Source: DATA FILTERING & VERIFICATION · p. 1
  • Boundary trimming uses instruction and video information.The passage names both instruction and video as inputs to trimming.Source: DATA FILTERING & VERIFICATION · p. 1
  • Action magnitude is manually sampled within canonical 80-D state-action processing.The passage associates manual sampling with action magnitude and a canonical 80-D state-action representation.Source: DATA FILTERING & VERIFICATION · p. 1

A R C

The section lists three manipulation tasks and names code, recipes, and weights.

A R C

  • Cup filling is listed as a task.The section also lists cup stacking and dropper transfer.Source: A R C · p. 1
  • Cup stacking is listed as a task.The other named tasks are cup filling and dropper transfer.Source: A R C · p. 1
  • Dropper transfer is listed as a task.The other named tasks are cup filling and cup stacking.Source: A R C · p. 1
  • Code, recipes, and weights are listed as materials.The passage gives no further detail about these items.Source: A R C · p. 1

1 Introduction

InternW0-∆ addresses the challenge of making predictive visual knowledge useful for robot action generation without future-video generation during online control. It combines video and action experts, predictive representations, and heterogeneous-data pretraining, with reported results across simulation benchmarks and real robots.

1 Introduction

  • InternW0-∆ couples pretrained video and action experts through directed Mixture-of-Transformers blocks, with Causal Imprint and training-only 4D-aware distillation.The action expert receives predictive representations without future observations at inference; future information serves only as training supervision.Source: §1 · pp. 2–3
  • The corpus contains over 20K hours of robot, UMI, egocentric-human, and Ego2Robot data aligned to a canonical state-action representation.Training first jointly pretrains visual dynamics and action generation, then adapts the checkpoint to target embodiments and tasks.Source: §1 · pp. 2–3
  • Caching frozen-encoder features and compiling model components speed iteration; deployment optimizations yield 152.8 ms average round-trip latency on an RTX 5090.The reported latency corresponds to a 5.11× speedup over the standard runtime on physical dexterous-hand deployment.Source: §1 · pp. 2–3
  • The authors plan to release code, checkpoints, recipes, infrastructure, and processed data where licenses permit.Versioned indices will record original samples and filtering decisions to support reproducibility.Source: §1 · p. 3
  • InternW0-∆ achieves leading results across four simulation benchmarks and is deployed on gripper-based and dexterous-hand robots.It reports best success rates under distribution shift on LIBERO-Plus and RoboTwin 2.0, the highest EBench overall score, and the best RoboDojo average success rate.Source: §1 · p. 3

2 Related Work

Related work spans vision-language-action and world-action policies, predictive and geometric visual representations, and open robot-learning infrastructure. InternW0-∆ positions its approach within those lines while also emphasizing an inspectable training and deployment stack.

2.1 Vision–Language–Action and World Action Models

  • VLA models transfer pretrained vision-language semantics to robot control, while WAMs additionally model visual-world evolution; Fast-WAM separates future-video supervision from action inference [Yuan et al., 2026b].The section contrasts semantic and instruction-following foundations with predictive video–action approaches.Source: §2.1 · pp. 3–4

2.2 Visual Representations for Robot Control

  • Prior work uses predictive visual features and varied geometric supervision, while InternW0-∆ combines future-latent change and alignment with training-only Track4World geometry and motion distillation.Causal Imprint representations inform the action expert; the teacher and distillation branch are discarded at inference.Source: §2.2 · p. 4

2.3 Open-Source Systems for Robot Learning

  • Open robot-learning systems differ in supported models, data pipelines, and deployment settings; InternW0-∆ plans to release its representation, training and evaluation code, checkpoints, and deployment utilities.The release also includes filtering code and versioned filtered-data indices.Source: §2.3 · pp. 4–5

3 InternW0-∆Model Design

InternW0-∆ couples pretrained video and action experts through directed information flow, combining sparse visual context, proprioception, and task-conditioned scene semantics. Training-only future supervision and 4D distillation add predictive, geometric, and motion information without future-video rollout at inference.

3.1 Architecture Overview

  • The directed Mixture-of-Transformers couples video and action experts, while Causal Imprint and training-only 4D distillation provide predictive scene-change and geometric-motion representations without future-video rollout at inference.The video expert uses anchor, recent, and current observations; proprioceptive states condition both experts.Source: §3.1 · p. 5

3.2 Multimodal Context Encoding

  • Canonical proprioceptive states are projected for both experts, and embodiment-specific robot controls map to canonical action vectors that are embedded for action prediction.The canonical state includes joint positions, absolute end-effector pose, and gripper or hand configuration.Source: §3.2 · p. 6
  • T5 preserves pretrained language conditioning, while a frozen VLM relates the instruction to current views and supplies scene-level semantics for observation-conditioned control.The VLM pathway complements T5, which does not directly perceive the current environment or its objects and spatial relationships.Source: §3.2 · pp. 6–7

3.3 Sparse Memory Context

InternW0-∆ uses sparse visual context and training-only predictive and geometric supervision to inform action generation. Its directed expert interaction keeps realized future observations out of the action path, while training combines video, action, and representation objectives.

3.3 Sparse Memory Context

  • Sparse memory combines the episode-start observation, a reference from before the previous action chunk, and the current observation, retaining global and recent context without dense history.Nearby history supports short-term motion and interaction continuity, while distant history serves as a coarse global reference.Source: §3.3 · p. 7

3.4 Causal Imprint

  • Causal Imprint tokens aggregate recent and current visual features, learning future-relevant scene changes from training-only future supervision for use by the action expert.Clean future latents construct adjacent-difference targets but are not exposed to the action prediction path.Source: §3.4 · pp. 7–8

3.5 4D-Aware Representation Distillation

  • A student branch aligns observation-derived features with cached Track4World descriptors to distill geometric and motion priors, with neither branch required at inference.The auxiliary MSE objective adds no policy inference cost.Source: §3.5 · p. 8, Figure 2 · p. 8

3.6 Mixture-of-Transformers and Attention Mask

  • Separate video and action experts exchange information through masked joint attention; the mask prevents Causal Imprint and action tokens from accessing realized future frames.Causal Imprint and action tokens instead use observed context and learned predictive representations.Source: §3.6 · p. 9

3.7 Training Objectives

  • Training jointly applies flow matching to future-video and action generation, direct adjacent-latent-difference supervision to Causal Imprint, and alignment with detached future-video features.Causal Imprint alignment pairs block-8 features with spatially corresponding features from the terminal future slice at block 20.Source: §3.7 · p. 10

3.8 Efficient Inference

Inference computes visual context once per observation, caches it, then iteratively denoises actions without sampling future video. After execution, the next observation triggers a refreshed cache and cycle.

Visual prefill and caching

  • One video-expert prefill caches visual key/value features, which remain reusable because video-side inputs are fixed and the directed mask blocks observation tokens from attending to actions.The cache includes anchor, recent, and current observation tokens plus Causal Imprint tokens.Source: §3.8 · pp. 11–12

Action-only denoising and execution

  • At each flow step, the action expert attends to cached visual features and action tokens to predict action-flow velocity, without recomputing video features or decoding future video.The action-only denoising loop integrates from σ = 1 to σ = 0 to produce the action chunk.Source: §3.8 · p. 12
  • Each policy cycle executes the predicted action chunk, then updates sparse memory and conditioning when a new observation arrives and recomputes visual context.A cycle comprises one video-expert prefill and N action-expert updates.Source: §3.8 · pp. 11–12

4 Data

This section describes how robot, egocentric human, and UMI data are filtered, curated, and converted into compatible training representations for joint use, including retained data volumes and Ego2Robot output.

4.1 Unified Representation

  • An 80-dimensional canonical action space assigns fixed physical meanings to slots, while robot states preserve joint, end-effector, and gripper configuration and actions encode joint targets or relative end-effector motion.End-effector state uses a 3D position and continuous 6D rotation; its action uses a 3D translation increment and 3D local-frame rotation vector.Source: §4.1 · p. 12

4.2.1 Data Sources

  • Robot manipulation supervision combines 15 simulated and real-world datasets spanning single- and dual-arm, dexterous, mobile, and humanoid manipulation settings.The collection includes varied embodiments, tasks, sensing configurations, and control interfaces.Source: §4.2.1 · p. 13

4.2.2 Processing Rules

  • Robot demonstrations undergo ordered signal, boundary, visual, action-magnitude, and instruction checks, followed by manual verification of sampled dataset–embodiment pairs.The consistency filter uses a default agreement threshold of 0.65; the action guard removes single-step end-effector translations above 0.2 m or rotations above 0.5 rad. Video–instruction checks use VLM-described action clips and episode frames to assess task match and completion.Source: §4.2.2 · pp. 16–17

4.3.2 Processing Rules

  • Ego2Robot conversion combines action alignment, coarse-to-fine kinematic alignment, and rendered visual alignment; egocentric data use action alignment and speed alignment only.Human wrist poses become end-effector targets, while coupled base search and trajectory IK validate robot trajectories; human regions are removed and the robot is composited into the scene with depth-aware occlusion. EgoVerse and EgoDex trajectories are slowed by a factor of two while preserving their spatial paths.Source: Figure 6 · p. 17, §4.3.2 · pp. 18–20

4.3.1 Data Sources

  • EgoDex (Hoque et al., 2025) contributes 829 hours of egocentric demonstrations, while EgoVerse (Punamiya et al., 2026) contributes 3,819 hours across 1.48 million segmented episodes.These collections pair egocentric video with 3D pose or hand-keypoint information and task descriptions.Source: §4.3.1 · p. 18

4.4.2 Filtering

  • UMI filtering rejects invalid gripper-pose quaternions, segments with both hands static, and trajectories with excessive speeds or abrupt adjacent-frame jumps.Both hands are deemed static when each has position variance below 5 × 10^-4 m^2 and rotation variance below 0.1 rad^2.Source: §4.4.2 · p. 20

4.5 Data Filtering and Conversion Statistics

  • 5,633.77 robot-hours across 3,730,513 training episodes were exported from EgoDex and EgoVerse, covering 1,101.55 hours from 283,450 unique source episodes.Robot-hours sum usable durations across embodiments before the factor-of-two slowdown; one source episode can yield multiple robot episodes, and exported segments contain at least 32 action steps.Source: §4.5 · p. 21
  • Filtering retained 11,302.20 of 13,867.71 robot-hours and 1,247,656 of 1,370,003 robot episodes, plus 4,061.35 of 4,640.04 EgoDex and EgoVerse hours.The retained robot collection contains 1,101.047 million frames; EgoDex and EgoVerse retain 1,622,756 of 1,818,396 episodes and 438.626 million frames, with 772.16 and 3,289.19 hours retained, respectively.Source: §4.5 · pp. 20–21

4.4.1 Data Sources

  • Hy-UMI-10K pairs head- and wrist-camera RGB video with 6-DoF gripper poses, gripper openness, and task descriptions; filtering retained 2,075.42 of 2,161.74 hours.Its public release contains approximately 250K episodes spanning 2,162 hours; retained trajectories are segmented into training episodes, raising the count to 416,053 across 224.146 million frames.Source: §4.4.1 · p. 20, §4.5 · p. 21

5 Training

InternW0-∆ is trained first on heterogeneous robot trajectories and then specialized to target embodiments and task distributions. The training pipeline converts varied sources into a canonical state-action representation and adapts the pretrained checkpoint for simulation and real-robot use.

5 Training

  • Training proceeds in two stages: joint pretraining of video and action experts on heterogeneous robot trajectories, followed by specialization to a target embodiment and task distribution.The video expert is pretrained and the action expert is randomly initialized.Source: §5 · p. 21, §5.1 · p. 22

5.1 Pretraining

  • Pretraining mixes robot, Ego2Robot, UMI, and egocentric demonstrations in an 80-dimensional canonical space, with 80%, 10%, 8%, and 2% of data, respectively.Training uses 33 video frames and a 32-step action block; a later phase adds Track4World representation distillation.Source: §5.1 · pp. 21–22

5.2 Post-Training for Simulation

  • Simulation post-training starts from the same checkpoint and maps benchmark-specific observations and actions into the canonical representation, masking unused embodiment dimensions.LIBERO-Plus evaluates controlled shifts across seven dimensions; RoboTwin training uses clean demonstrations before randomized evaluation.Source: §5.2 · pp. 23–24

5.3 Post-Training for Real Robots

  • Real-robot post-training adapts the same pretrained checkpoint across four gripper-based and dexterous-hand setups, using prefix-conditioned training for action continuation.Training simulates inference delay with a committed prefix of up to 16 clean actions and computes loss on the remaining valid suffix.Source: §5.3 · pp. 24–25

6 Infrastructure

The infrastructure reuses frozen-encoder features and optimizes MoT execution for either throughput or activation memory. It also supports asynchronous real-robot replanning under a bounded response window.

6.1 Training Infrastructure for Model Development

  • Layerwise compilation and checkpointing complement feature caching by reducing backbone execution cost and repeated frozen-encoder work during model development.The cache serves VAE latents and VLM contexts through a common interface for online and offline materialization.Source: §6.1 · p. 25, Figure 8 · p. 26

6.1.1 Feature Caching

  • A unified cache reuses frozen VAE latents and VLM contexts, while online and offline materialization let training amortize feature extraction across repeated samples.Offline caching prepares persistent feature shards before training; online caching populates entries as samples are traversed.Source: §6.1.1 · pp. 26–27, Figure 8 · p. 26

6.1.2 Layerwise MoT Optimization

  • The 30-block MoT backbone compiles layers independently; three consecutive layers can be grouped for external checkpointing when activation memory is limiting.VLM contexts are padded before action cross-attention, and the checkpointed path disables CUDA Graph capture because validation found incorrect gradients with that combination.Source: §6.1.2 · pp. 27–28, Figure 9 · p. 27

6.1.3 Training Efficiency

  • Combining VAE/VLM feature caching with workload-specific layerwise MoT optimization speeds end-to-end training by 3.02× on LIBERO and 2.11× on RoboTwin over unoptimized baselines.Layerwise optimization alone improves throughput by 20–30% when feature reuse is unavailable or cache materialization is impractical.Source: §6.1.3 · p. 28

6.2 Real-Robot Deployment

  • At 30 Hz, asynchronous replanning has a 533 ms round-trip window for the next action chunk, including transport, inference, alignment, and dispatch.The deployment uses committed action prefixes and launches a background inference request every 16 execution steps.Source: §6.2 · p. 28

7 Experiments

Experiments evaluate InternW0-∆ across four simulation benchmarks and real-robot platforms, examining robustness, pretraining, model components, data sources, and execution-time correction. The reported results include leading benchmark scores and improvements on selected real-robot tasks.

7.2.1 LIBERO-Plus; 7.2.2 RoboTwin 2.0

  • 92.8% overall success on LIBERO-Plus and 71.9% on RoboTwin Clean2Random are best reported; RoboTwin Clean2Clean reaches 90.0%.Evaluations follow official simulator criteria without adaptation on evaluation environments, with baselines matched to published protocols.Source: §7.2.1 · p. 29, §7.1 · p. 29, Table 9 · p. 30

7.2.3 EBench; 7.2.4 RoboDojo

  • 66.0 overall score on EBench leads; RoboDojo reaches 23.91% average SR and 30.77 score, exceeding GPT-6 Astra’s 22.48% and 28.97.On EBench Long Horizon tasks, InternW0-∆ records the highest success rate (49.4%) and score (76.5).Source: §7.2.3 · p. 30, Table 11 · p. 31, §7.2.4 · p. 31

7.3 Real-Robot Experiments

  • Pretraining raises toast-bread success from 20% to 95% and Luminol from 0% to 95%; the drink policy also succeeds at unseen marked water levels.The real-robot evaluation covers eleven tasks on four platforms; the drink-task demonstrations all use one target level.Source: §7.3 · pp. 32–33, Table 13 · p. 33, Figure 11 · p. 34
  • Training-time conditioning keeps the prefix-to-suffix boundary change near ordinary within-plan changes, with a ratio of 1.12 versus 7.13 for hard prefix and 8.94 for VJP.The diagnostic ratios come from recorded AC-One traces; the evaluated boundary differs by strategy.Source: §7.3 · p. 34, Figure 12 · p. 35

7.4.1 Component-Wise Ablations

  • SMC raises success from 49.59% to 53.47%; with SMC fixed, Qwen3.5-2B reaches 60.37%, and RynnBrain1.1-2B reaches 69.08%.All variants use the same training and evaluation settings; the authors adopt RynnBrain1.1-2B for remaining experiments.Source: §7.4.1 · pp. 34–35, Table 14 · p. 35
  • Causal Imprint raises success from 69.08% to 70.80%, Lalign to 76.45%, and L4D to 78.37%; future observations provide training supervision only.Track4World distillation reaches 78.37%, compared with 76.15% for CoWTracker and 73.63% for Pi3X; the student branch is removed at inference.Source: Table 14 · p. 35, §7.4.1 · p. 36, Table 15 · p. 36

7.4.2 Data Source Ablations

  • UMI pretraining reaches 83.24% on LIBERO-Plus and raises RoboTwin Clean2Random from 2.80% to 13.90%; robot data reaches 83.73% and raises it from 4.38% to 32.34%.These results are specific to the current data scale and evaluation setting, which focuses primarily on gripper-based manipulation and downstream success rate.Source: §7.4.2 · pp. 36–37, Table 16 · p. 37, Table 17 · p. 37

7.5 Exploration of GPT-Guided Policy

  • On five selected RoboDojo tasks, GPT-6 Astra correction raises average success from 10.80% to 47.20% and average score from 13.92 to 52.00.The reasoning-setting comparison is based on five episodes for the evaluated language-based classification task.Source: §7.5 · pp. 37–38, Table 18 · p. 38

8 Conclusion

InternW0-∆ combines predictive visual dynamics, scene semantics, action generation, and sparse visual memory, with training-only supervision and distillation supporting action generation without future-video rollout at inference. Its scalable data and deployment recipe supports adaptation across tasks, embodiments, and control interfaces, while the authors identify limits in their study of egocentric data and agent-assisted control.

8 Conclusion

  • InternW0-∆ couples pretrained visual dynamics and a frozen VLM with action generation, while Causal Imprint and training-only 4D-aware distillation add predictive, geometric, and motion information without future-video rollout at inference.Sparse visual memory supplies episode-level context and recent interaction history.Source: §8 · p. 38
  • Across simulation benchmarks and real-robot platforms, a shared pretrained checkpoint can be adapted to different task distributions, embodiments, and control interfaces.The interfaces include gripper-based and dexterous-hand manipulation.Source: §8 · p. 38
  • The training and deployment recipe uses over 20K hours of processed robot, UMI, egocentric, and Ego2Robot data, with canonical representations, filtering, alignment, and target-embodiment post-training.The authors also developed infrastructure for efficient iteration and online execution, and plan to release associated tools and materials.Source: §8 · p. 38
  • The study does not systematically examine how egocentric data sources, conversion strategies, scaling ratios, and supervision forms affect downstream performance, or broadly evaluate agent-assisted control.Agent-assisted experiments cover only a limited corrective setting; invocation policies, long-horizon planning, hierarchical decision making, and tighter integration remain understudied.Source: §8 · pp. 38–39

9 Team

The team is organized across data, modeling and infrastructure, deployment, and leadership roles. The listed contributors span real-robot data, Ego and UMI data, model pretraining, and real-robot deployment.

Core Contributors

A Human-to-Robot IK Details

For a fixed robot base placement, each local IK iteration combines position and approach tracking, contact-point and orientation refinement, and projected joint-space regularization. The update is bounded and accepted iterates are selected using task-priority criteria.

A Human-to-Robot IK Details

  • Each fixed-base local IK iteration combines position and approach tracking, contact-point and orientation refinement, and projected joint-space regularization.The update separates these contributions into ∆qp, ∆qs, and Np gaux, respectively.Source: A Human-to-Robot IK Details · p. 49
  • Position and approach tracking reduces TCP position and approach-direction errors beyond tolerance using a weighted position-and-approach Jacobian and residual.Source: A Human-to-Robot IK Details · p. 49
  • Optional contact-point and full-orientation targets are refined after tracking, with corrections accounting for residual error and the position-and-approach projection attenuating interference.Contact-point alignment moves the gripper-pad centers toward virtual points on the human pinch axis.Source: A Human-to-Robot IK Details · p. 49
  • Projected joint-space regularization encourages continuity, a preferred home configuration, and separation from joint limits, using a warm-reference configuration when available or a continuation seed otherwise.The respective weights control these contributions, and disabled terms are set to zero.Source: A Human-to-Robot IK Details · p. 49
  • The combined increment is clipped before updating, then joint limits and displacement bounds are enforced; accepted iterates use task-priority criteria rather than one aggregate loss.Continuity depends on reference attraction, solution reuse, step bounds, and trajectory-level validation.Source: A Human-to-Robot IK Details · p. 49
Loading 2609.31394v1…