PaperlayerView exact revision on arXiv

Source-linked AI summary

Magic-W0: A Structured World-Action Foundation Model for Physical Intelligence

Xuhua Chen, Zhenhan Yin, Yuan Zhang, Lingfeng Zhang, He Zheng, Tong Mu, Shun Zuo, Dian Zhou, Di Wu, Xuan Zhou, Shaojie Wan, Rongtian Shen, Qiulong Xu, Yiduo Li, Yinglong Wang, Yanqian Wang, Kun Wang, Tao Zhang

arXiv:2609.39870v1cs.RO

TL;DR

Robot world-action models often predict future observations or generic latents without structured, control-oriented representations. Magic-W0 jointly models structured world transitions and continuous actions; on RoboDojo-Sim, it ranks first among compared WAMs in average Score.

  • Problem

    Existing approaches often use future-observation reconstruction or generic latent prediction, leaving control-oriented world structure and tight coupling to action generation unaddressed.

    Source: §1 · pp. 3–4

  • Method

    Magic-W0 models Current State, Transition, and Future State using 3D Geometry, 3D Motion, and Future Semantics, with layer-aligned interaction between world representations and continuous action generation.

    Source: §1 · p. 5, §3.1 · p. 8

  • Results

    27.10 average Score on RoboDojo-Sim ranked first among compared WAMs, with average SR 20.84%.

    Source: §6.1.1 · p. 14

  • Takeaways & Limitations

    The reported conclusion supports the potential of structured world modeling for robot manipulation.

    Source: §7 · p. 19

    Dexterous precision manipulation, tactile and force inputs, and reducing the inference-speed gap with state-of-the-art VLA models remain future directions.

    Source: §7 · pp. 19–20

Abstract

from arXiv · show

World-action models (WAMs) augment robot policies with action-conditioned environment dynamics, yet existing approaches largely rely on future observation reconstruction or generic latent prediction and lack structured, control-oriented world representations tightly coupled with action generation. We introduce Magic-W0, a world-action foundation model that jointly models structured physical state evolution and continuous actions. Magic-W0 represents interaction as a Structured World Transition consisting of Current State, Transition, and Future State. Current State combines vision-language context with Current 3D Geometry; Transition is represented by 3D Motion capturing action-induced three-dimensional changes; and Future State is represented by Future Semantics describing task-relevant outcomes. To couple prediction and control, we propose a layer-aligned world-action interaction architecture in which evolving action hypotheses condition world-transition prediction, while predicted world representations continuously inform action generation. Magic-W0 is pre-trained on large-scale egocentric human manipulation, UMI, real-robot, and simulation data, with latent supervision for geometry, 3D motion, and future semantics from pre-trained visual models. Inference-time interventions show that structured world representations respond systematically to changes in candidate actions and that action-related information propagates through shared 3D representations into future semantic predictions. On RoboDojo-Sim, Magic-W0 achieves an average Score of 27.10, the highest among the compared WAMs. Across multiple real-robot tasks, it also demonstrates strong downstream performance after fine-tuning with limited downstream data, supporting generalization and rapid adaptation.

1 Introduction

The introduction motivates explicit, control-oriented prediction of action-induced world changes and asks how to structure those predictions and couple them to action generation. Magic-W0 addresses both with a three-part world representation, bidirectional interaction, and a shared interface for heterogeneous embodied data.

1 Introduction

  • VLA research combines vision-language knowledge with robot demonstrations for continuous action generation, while WAMs add prediction of how the world evolves under candidate actions.The cited VLA progression spans large-scale robot data, heterogeneous datasets and embodiments, and continuous action experts.Source: §1 · pp. 3–4
  • Existing policies rarely supervise action-induced state changes explicitly, while image prediction retains control-irrelevant appearance and generic latent prediction leaves future-world structure unresolved.The introduction also notes that semantic generalization does not automatically transfer to unseen physical motions and that static visual representations do not fully capture temporal dynamics.Source: §1 · pp. 3–4
  • Control-oriented world modeling distinguishes current physical conditions, action-induced transitions, and task-relevant future outcomes, while requiring action hypotheses and predictions to mutually condition one another.Prior approaches may predict futures without evolving action hypotheses informing prediction or predicted changes continuously feeding back into action generation.Source: §1 · pp. 4–5
  • Magic-W0 represents interaction as Current State, Transition, and Future State, using 3D Geometry, 3D Motion, and Future Semantics with layer-aligned bidirectional action coupling.Its action hypotheses inform transition prediction, while predicted world representations continuously inform action updates.Source: Figure 2 · p. 4, §1 · p. 5
  • Magic-W0 pre-trains across egocentric human manipulation, UMI, real-robot, and simulation data through a unified 34-dimensional state-action interface.The interface maps available end-effector poses, gripper information, and joint states to corresponding dimensions across differing embodiments and action spaces.Source: §1 · p. 5

2 Related Work

Related work develops robot policies, future-world representations, three-dimensional modeling, and tighter links between prediction and control. Magic-W0 combines explicit geometry, motion, and future semantics with multi-depth interaction between world transitions and action hypotheses.

2 Related Work

  • General-purpose VLAs scale task coverage and continuous control through robot data, heterogeneous training sources, and action experts, but leave explicit three-dimensional world transitions within policies open.The surveyed approaches include RT-1/RT-2, Open X-Embodiment, Octo, OpenVLA, and newer continuous-action systems.Source: §2 · pp. 5–6
  • Future-world methods range from observable video or image generation to predictive latent features, while Magic-W0 learns structured geometry, motion, and semantic targets in latent space.Prior work uses visual rollouts or intermediate representations; Magic-W0 targets current-frame geometry, cross-time motion, and future-frame semantic features.Source: §2 · p. 6
  • Three-dimensional policy and world-modeling work encodes spatial information or predicts observable structure, while Magic-W0 learns latent geometry and motion alongside future semantic features.Its geometry and motion targets draw on Track4World representations, with future DINOv3 features supplying semantic supervision.Source: §2 · p. 6
  • World-action coupling progresses from latent planning and predictive-feature conditioning to shared generation; Magic-W0 exchanges structured transitions and action hypotheses across multiple network depths.Its layer-aligned multi-stream architecture supports both action-conditioned world transitions and world-informed action generation.Source: §2 · pp. 6–7

3 Magic-W0

Magic-W0 structures robot interaction as current context and geometry, action-induced 3D motion, and future task semantics, while jointly modeling continuous actions. Its aligned streams couple action hypotheses with world prediction and train world representations against teacher features.

3 Magic-W0

  • Magic-W0 jointly models continuous action generation and structured world representations, with a 3D stream, semantic stream, and action expert interacting through layer-aligned attention.The 3D stream shares hidden states for current geometry and motion; Future Semantics has a separate stream, and teacher models provide latent supervision only during training.Source: §3 · p. 7, Figure 3 · p. 7

3.1 Structured World Representations

  • Current State combines VLM context and Current 3D Geometry; 3D Motion represents action-induced changes, and Future Semantics captures task-relevant outcomes.The geometry and motion targets come from Track4World, while future semantic targets use DINOv3 features; geometry targets remain fixed across candidate actions.Source: §3.1 · p. 8

3.2.1 Inputs and Bidirectional Interaction

  • Action hypotheses inform future motion and semantic predictions, while world-stream features update the action velocity field at successive layers.The geometry and motion heads read final 3D hidden states; the semantic head reads final semantic hidden states.Source: §3.2.1 · p. 8

3.2.2 Layer-Aligned Joint Attention

  • The model aligns three expert streams across 24 layers, exchanging information through joint attention after every three Gated DeltaNet layers.The VLM retains its causal path; expert streams can access one another, while the VLM does not attend to them.Source: §3.2.2 · p. 9, Figure 4 · p. 9

3.3 Training Objectives

  • World losses align predicted features with teacher targets, using cosine distance for semantics and channel-normalized mean squared error for geometry and motion.Continuous action training interpolates normalized action chunks with Gaussian noise and regresses the target velocity; these losses are optimized jointly.Source: §3.3 · pp. 9–10
  • Pre-training also uses FAST discrete-action supervision, optional VQA cross-entropy, and standard VLM autoregressive cross-entropy on vision-language batches.VQA loss applies only when enabled and valid question–answer annotations are available.Source: §3.3 · p. 10

4 Pre-Training

Magic-W0 pre-training combines heterogeneous manipulation and vision-language data, maps robot states and actions into a shared interface, and aligns future-world supervision with each action chunk. It jointly trains continuous control, structured world representations, and vision-language modeling.

4.1 Pre-Training Corpus and Unified Representation

  • Approximately 2.014 million manipulation episodes across four acquisition domains are combined with 2.61 million vision-language samples at a 9:1 manipulation-to-vision-language batch ratio.The manipulation corpus spans egocentric human, UMI, real-robot, and simulation data; the vision-language corpus supervises spatial localization, task semantics, and interleaved image-text-action modeling.Source: §4.1 · p. 10, Figure 5 · p. 11
  • Heterogeneous sources map to a fixed 34-dimensional state-action interface, with unavailable slots zero-padded and masked; egocentric data omit joint targets and UMI targets use kinematic retargeting.The interface lets new embodiments populate corresponding slots without changing model or checkpoint parameter shapes.Source: §4.1 · p. 10

4.2 Action-Aligned Training Samples

  • Future-frame offsets vary with each source’s action-sampling multiplier, aligning world supervision with the interaction interval covered by each 50-step action chunk.For EgoDex, the stated multiplier of 1.95 yields an offset of 26 frames after rounding.Source: §4.2 · p. 11

4.3 Joint Pre-Training

  • Joint pre-training combines flow-matched continuous actions and geometry, motion, and semantic targets with FAST action and autoregressive vision-language objectives.Continuous action and world losses do not backpropagate through VLM context or its key-value projections; only the vision-language and FAST objectives update the VLM backbone.Source: §4.3 · p. 12
  • FAST, semantic, and 3D auxiliary weights are higher early and decay, while occasional masking of one world branch reduces the action expert’s reliance on either branch alone.Branch masking leaves the 3D and semantic queries’ visibility and supervision unchanged.Source: §4.3 · p. 12

5 Post-Training and Inference

Magic-W0 adapts its full pre-trained checkpoint to each target domain by jointly fine-tuning world representations and action generation. During inference, evolving action hypotheses and predicted world representations interact throughout flow-based action generation.

5.1 Supervised Fine-Tuning in the Target Domain

  • Each target-domain policy starts from the full pre-trained checkpoint and jointly fine-tunes on all demonstrations for a unified multi-task policy, retaining model topology and world representations.The adaptation adjusts pre-trained world representations and action generation together rather than training a separate action policy from scratch.Source: §5.1 · p. 13, §5 · p. 13
  • Fine-tuning retains continuous-action and world losses but removes FAST supervision, vision-language corpus mixing, and pre-training gradient isolation.The action and world objectives jointly update the VLM backbone, 3D stream, semantic stream, and action expert; world-loss weights remain fixed.Source: §5.1 · p. 13
  • Target-domain validity masks select observable, controllable dimensions, and endpoint values are held when future times exceed a trajectory’s end.This retains terminal samples for action and world supervision.Source: §5.1 · p. 13

5.2 Inference-Time Execution

  • An action chunk is initialized from Gaussian noise and generated by integrating a flow-matching velocity field, with 10 uniform Euler steps as the default.The VLM context is computed and cached once per decision because the VLM does not attend to the 3D, semantic, or action streams.Source: §5.2 · p. 13
  • At every flow-integration step, evolving action hypotheses condition 3D-motion and semantic predictions, whose updated representations feed back into action generation.The layer-aligned world-action interaction operates throughout continuous action generation, not only as a training-time constraint.Source: §5.2 · p. 13

6 Experiments

Experiments assess RoboDojo-Sim performance, LIBERO transfer, five real-robot tasks, and whether action inputs and cross-stream pathways shape structured world predictions.

6.1.1 RoboDojo

  • On RoboDojo-Sim, Magic-W0 achieved an average Score of 27.10 and average SR of 20.84%, ranking first among compared WAMs.In Generalization, its Score was 28.20 and SR was 22.01%, both highest among listed WAMs.Source: §6.1.1 · p. 14

6.1.2 LIBERO

  • 99.1% average SR on LIBERO accompanied the highest Spatial and Goal SRs, 99.4% and 99.6%, respectively.The reported results indicate downstream transfer across task types after fine-tuning from pre-trained weights.Source: §6.1.2 · p. 14, Table 2 · p. 15

6.2 Real-Robot Experiments

  • 94.6% average SR across five real-robot tasks exceeded π0.5’s 91.8% by 2.8 percentage points, with no task-level decrease.Each task used 100 independent trials; both methods matched at 100% on bottle uprighting, while Magic-W0 improved on four tasks.Source: §6.2 · p. 16

6.3.1 Action Sensitivity

  • Action replacement changed all three structured world outputs, with smaller responses as τ increased and less ground-truth action information remained.From τ ∈[0, 0.2) to τ ∈[0.8, 1.0), Dm fell from 0.50 to 0.14 for Future Semantics, 0.37 to 0.11 for 3D Motion, and 0.26 to 0.07 for Current 3D Geometry; Future Semantics loss rose 73.5% in the lowest-noise interval and 8.0% in the highest.Source: §6.3.1 · pp. 17–18, Figure 10 · p. 18

6.3.2 Cross-Stream Dependence

  • Masking 3d →sem increased Future Semantics loss by 60.8%, versus 2.6% for the reverse connection, indicating stronger dependence on 3D information.The 3D stream shares hidden states for Current 3D Geometry and 3D Motion.Source: §6.3.2 · p. 18
  • Future Semantics depends on action information both directly and through the 3D stream: masking action →sem or action →3d increased its loss.The latter removes the route by which action information enters 3D representations and then reaches the semantic stream.Source: §6.3.2 · p. 18
  • Removing all cross-stream connections increased semantic loss by 95.3%; the 0.0748 joint increase was approximately 1.0% below the 0.0755 sum of individual increases.Because connections may affect shared intermediate representations, the numerical relationship is not strict additivity.Source: §6.3.2 · pp. 18–19, Figure 11 · p. 19
  • The complementary interventions indicate that action conditions alter structured world outputs and that Future Semantics depends on cross-stream information, especially 3d →sem.The effect of masking 3d →sem varied relatively little across τ, while masking action →sem had a slightly increasing effect; these interventions change different components.Source: §6.3.2 · p. 19

7 Conclusion

Magic-W0 jointly models structured world evolution and robot action generation using heterogeneous data and a layer-aligned architecture. It ranks first among compared WAMs on RoboDojo-Sim and performs well on multiple real-robot tasks after fine-tuning with limited data.

7 Conclusion

  • 27.10 average Score on RoboDojo-Sim ranks Magic-W0 first among the compared WAMs.The paper also reports strong performance across multiple real-robot tasks after fine-tuning with limited downstream data.Source: §7 · p. 19
  • Magic-W0 represents interaction as Current State–Transition–Future State, including action-conditioned 3D transitions and task-relevant future states.Its layer-aligned architecture couples world prediction and action generation continuously across multiple network depths.Source: §7 · p. 19
  • Diagnostics show structured world representations adjust to candidate-action changes and shared 3D representations convey action-related information to future-state predictions.Source: §7 · p. 19
  • The model is trained on heterogeneous data from egocentric human manipulation, UMI, real robots, and simulation, with latent supervision from frozen geometry, motion, and semantic vision models.Source: §7 · p. 19
  • Dexterous-hand precision manipulation, tactile and force signals, and reducing the inference-speed gap with state-of-the-art VLA models remain future directions.The authors identify additional computational cost from multilayer world–action interaction.Source: §7 · pp. 19–20

Contributions and Acknowledgments

The acknowledgments identify four people as contributors to Magic-W0.

Contributions and Acknowledgments

A Pre-Training Data and Processing

The appendix describes offline processing that standardizes trajectories and associates camera calibration with each one.

A Pre-Training Data and Processing

  • Offline processing unifies units, arm ordering, gripper definitions, and media indices, and associates camera calibration with each trajectory.Processed trajectories contain task text, timestamps, multi-camera videos, states and actions, camera intrinsics and poses, and per-frame quality flags.Source: A Pre-Training Data and Processing · p. 24

A.1 Pre-Training Corpora

The appendix describes the pre-training corpora and how supervision is constructed across data sources. It details trajectory processing and source-specific pose and action labels.

A.1 Pre-Training Corpora

  • Tables 3 and 4 list manipulation and vision–language supervised fine-tuning corpora used for pre-training; their counts are approximate valid samples after filtering and aggregation.The counts should not be equated with official dataset sizes.Source: §A.1 · p. 24, Table 3 · p. 24, Table 4 · p. 24

A.2 Cross-Source Supervision Construction

  • End-effector poses are transformed into each frame’s camera coordinates, with poses matched to the same frame and rotations represented in 6D.Validity is assessed separately for each arm, and source orientation and axis conventions are recorded.Source: §A.2 · p. 24
  • Egocentric human records lack robot proprioceptive states, so hand keypoints provide virtual end-effector poses and thumb-to-middle-finger distance provides gripper-opening supervision.These trajectories are learned at the end-effector level during pre-training, then adapted to a target robot during fine-tuning.Source: §A.2 · p. 24, Figure 13 · p. 25
  • Figure 12 projects end-effector poses using source-specific camera poses and intrinsics, with red, green, and blue marking local x, y, and z axes.UMI poses come from tracked devices and inverse kinematics, real-robot poses from joint states and URDF, and simulation poses from simulator records.Source: Figure 12 · p. 25
  • UMI tracked poses are registered to the robot base and retargeted through inverse kinematics to produce joint and gripper supervision.The construction retains end-effector poses and records solver errors and collision checks for filtering.Source: §A.2 · p. 25, Figure 13 · p. 25

A.3 Training Samples

Training samples are anchored at time t, with observations, action chunks, and validity masks aligned around each anchor. Actions are normalized relative to the anchor state, while future targets and valid timesteps are handled according to source-specific rules.

A.3 Training Samples

  • Samples are anchored at time t, and state and action validity masks cover the action chunk from t to t+H−1.World targets are extracted from temporally aligned observations during training forward passes rather than stored in advance.Source: §A.3 · p. 25
  • Three cameras are used, with anchors sampled every 25 frames and additional keyframe-based anchors where available.The duration represented by the frame stride varies with each source’s native frame rate.Source: §A.3 · p. 25
  • Action chunks preserve native control targets when available; otherwise, next-timestep states provide supervision, and joint angles and end-effector poses become anchor-relative.Gripper openings remain absolute; actions are scaled by source and dimension using the 1st and 99th percentiles, then clipped to [−1, 1].Source: §A.3 · p. 25
  • Future-frame offsets use source-configured sampling multipliers, and pre-training excludes anchors whose future targets exceed the episode endpoint.For EgoDex, ρi = 1.95 and H = 50 yield ∆i = 26; fine-tuning instead holds endpoint values and reuses the final frame for world targets.Source: §A.3 · p. 26

B Data Processing and Quality Control

This material describes source-specific calibration and quality-control procedures, then explains how data processing and shared-grid world targets support training losses. Checks, thresholds, and filtering or masking actions vary by source.

B Data Processing and Quality Control

  • The section focuses on calibration when camera parameters are missing, alongside source-specific quality checks and the process by which those checks’ outputs enter training.Source: B Data Processing and Quality Control · p. 26

B.1 Camera Calibration for Sources without Metadata

  • For sources without camera metadata, calibration is estimated from video and metric poses: silhouette overlap fits bimanual real-robot views, while tracked-pose/image correspondences fit UMI views.Real-robot calibration depends on foreground extraction quality and is spot-checked across frames and tasks; validated parameters are shared within recording groups.Source: §B.1 · p. 26, Figure 14 · p. 26

B.2 Quality Checks

  • Quality checks cover signal completeness, media, temporal consistency, and geometric validity, with source-specific checks and thresholds.Spike flags combine smoothing residuals with second- or third-order differences; quantile bands identify soft anomaly candidates. Table 5 lists checks in execution order.Source: §B.2 · p. 26, Table 5 · p. 27

B.3 Filtering, Repair, and Validity Masks

  • Offline quality outputs can filter trajectories, repair signals, exclude anchors, or mask future timesteps; not every source uses all five mechanisms.Short, physically invalid, or unsuccessfully retargeted trajectories are discarded; enabled repair interpolates flagged spikes or end-effector discontinuities, while anchor and timestep validity are independent.Source: §B.3 · pp. 27–28, Figure 15 · p. 27

C World-Representation Targets

  • Geometry, motion, and future semantics are rearranged onto a common spatial grid so their losses can use common indexing.Rgeo resamples 18 × 18 geometry to 16 × 16; Rmot rearranges 32 × 32 motion features to 16 × 16 × 1024, while DINOv3 supplies 16 × 16 patch tokens.Source: C World-Representation Targets · p. 28

D Loss Definitions

  • Action chunks are trained with flow matching between normalized actions and Gaussian noise, while the semantic loss uses cosine distance and geometry and motion use mean squared error.The masked action mean squared error is normalized by valid-element count, preventing action dimensionality or valid-timestep counts from directly setting its scale.Source: D Loss Definitions · p. 28

E Model and Training Configuration

The training configuration schedules selected auxiliary loss weights over training progress, while the action loss weight and inner geometry/motion weights remain fixed. Table 6 and Table 7 summarize model specifications and pre-training settings.

E Model and Training Configuration

  • Table 6 summarizes Magic-W0’s principal configuration, and Table 7 lists pre-training optimization and execution settings.The supplied passages identify these tables’ scope but do not provide their cell values.Source: E Model and Training Configuration · p. 28, Table 6 · p. 29, Table 7 · p. 29
  • The outer weights λFAST, λsem, and λ3d stay at maximum for the first 20% of training, cosine-decay over the middle 60%, then stay at minimum for the final 20%.Training progress is p = g/G, where g is the current step and G is the total number of steps.Source: E Model and Training Configuration · p. 29
  • The action loss weight and inner geometry/motion weights remain fixed, while vision–language batches contribute autoregressive cross-entropy with fixed weight 0.1.The vision–language weight is independent of the auxiliary weight schedule.Source: E Model and Training Configuration · p. 29

Related papers

Loading 2609.39870v1…