Native Action-Prior Learning from Videos for World Action Models

1 Introduction

• Action-annotated robot trajectories are costly to collect and difficult to standardize across embodiments, while observation-only videos provide abundant evidence about physical interactions and state transitions.The challenge is to use those videos to learn priors that improve action-policy learning.Source: §1 · p. 1
• Existing video-based approaches either transfer visual representations to control downstream or infer latent actions, adding an intermediate stage that can constrain supervision and control learning.Latent-action methods may require architectural or optimization constraints that restrict latent-space expressiveness.Source: §1 · pp. 1–2
• NAVA-WAM directly pretrains the action policy through future-video prediction, then uses joint video–action flow matching on labeled demonstrations to adapt it for robot control.Transition-structured joint attention propagates future-video supervision into the Action-DiT; asymmetric attention supports cached visual representations and action-only inference.Source: §1 · p. 2
• NAVA-WAM outperforms prior approaches across in-distribution and out-of-distribution settings, reports 93.3% real-robot success, and shows larger native-prior gains at low action-label budgets.The reported real-robot rates are 66.7% for DreamZero and 53.3% for π0.5.Source: §1 · p. 2

2.1 World Action Models

• Cascaded world action models use predictive visual information to drive a separate action module, while joint models represent future observations and actions within a shared architecture.NAVA-WAM follows the joint paradigm and enables action-only inference.Source: §2.1 · p. 3

2.2 Learning from Observation-Only Videos

• NAVA-WAM differs by directly pretraining the action-generating expert from observation-only videos, without an intermediate representation or latent-action interface.Its training uses future-video supervision during pretraining, then joint video and action objectives on robot demonstrations.Source: §2.1 · p. 3, §2.2 · p. 3, Figure 2 · p. 4
• Observation-only-video methods typically transfer visual representations during downstream policy learning or use inferred latent actions as intermediate supervision.These routes can entail indirect representation-to-control transfer or extra modeling constraints on latent actions.Source: §2.2 · p. 3

3 Method

• NAVA-WAM is a world action model that learns action-relevant priors within the robot policy from observation-only videos, without an intermediate interface.Its MoT architecture pairs a visual-dynamics Video-DiT with an Action-DiT that generates continuous robot actions; both retain modality-specific parameters and exchange information through joint attention.Source: §3 · p. 4, §3.1 · p. 4

3.1 Preliminaries

• The policy predicts a continuous action chunk from visual observations, language instructions, and proprioceptive state, using both video-only and action-labeled robot datasets.Video clips are encoded as visual states, and adjacent encoded states define transitions used for pretraining before adaptation to embodiment-specific control.Source: §3.1 · p. 4

3.2 Native Action-Prior Pre-training

• During pretraining, transition-structured joint attention lets future-video prediction update the Action-DiT while the pretrained Video-DiT remains frozen.The Action-DiT accesses the future visual state only in noised form, avoiding direct copying of the clean target; its transition representation remains implicit rather than materializing latent actions.Source: §3.2 · pp. 5–6

3.3 Post-training and Inference

• Post-training jointly applies video and action flow-matching objectives to demonstrations, and asymmetric attention lets cached visual representations condition action-only iterative denoising.The Video-DiT is independent of the evolving action sample; after an initial joint pass, denoising runs only through the Action-DiT without generating future videos.Source: §3.3 · pp. 6–7

4.3 Ablation Studies

• 72.0% success with 25 demonstrations per task, versus 60.6% and 65.0% for representation-based and latent-action-based pre-training; it also exceeds every baseline trained with 50 demonstrations.The comparison covers varying action-label budgets on RoboTwin 2.0.Source: Table 2 · p. 8, §4.3 · p. 9

4.1 Experimental Setup

• Simulation evaluation uses LIBERO and LIBERO-Plus ID/OOD settings and RoboTwin Clean/Random settings; physical evaluation uses three tabletop tasks with five trials each and no task-specific fine-tuning.The RoboTwin policy is trained using 50 Clean-domain demonstrations per task and no Random-domain demonstrations.Source: §4.1 · pp. 7–8, §4 · p. 7

4.2 Main Results

• 99.0% success on LIBERO and 83.5% on LIBERO-Plus; RoboTwin reaches 88.5% Clean and 73.6% Random, beating its strongest baseline by 3.5 and 17.9 points.The paper reports a smaller Clean-to-Random performance drop for NAVA-WAM than for the strongest baseline.Source: §4.2 · p. 8
• 14/15 trials (93.3%) succeeded on the Franka FR3, compared with 10/15 (66.7%) for DreamZero and 8/15 (53.3%) for π0.5.On T1, NAVA-WAM succeeded in 4/5 trials while both baselines failed all five; the results demonstrate transfer to physical deployment.Source: §4.2 · pp. 8–9

4.3 Ablation Studies

• Random-domain success consistently improves as observation-only pre-training data scales up under both action-label budgets.The authors conclude that native action-prior learning benefits from more observation-only data.Source: Figure 6 · p. 9, §4.3 · p. 9
• Action-DiT features achieve the highest ridge-regression R2 across probe-data budgets, while attention focuses on interaction-relevant regions rather than static regions.The visual comparison is with a latent-action baseline whose attention is more diffuse over static regions.Source: Figure 4 · p. 8, §4.3 · p. 10
• Transferred states follow source motion across simulation, real-world, and sim-to-real examples despite substantial changes in appearance and scene configuration.The transfer uses Action-DiT activations from a source transition to predict a next state on a different target observation.Source: Figure 5 · p. 9, §4.3 · p. 10
• 4.30 TFLOPs per action chunk yields 73.6% Random-domain success, with inference compute comparable to Image-WAM at 4.09 TFLOPs and Fast-WAM at 3.56 TFLOPs.The paper characterizes this as strong out-of-domain generalization with low inference overhead.Source: Table 2 · p. 8, §4.3 · p. 10

5 Conclusion

• NAVA-WAM directly pre-trains the action policy from observation-only videos without an intermediate representation or latent-action interface.It learns action-relevant priors from visual transitions and adapts them to robot control using action-labeled demonstrations.Source: §5 · p. 10
• The unified model combines action-free pre-training and downstream robot-policy learning, with efficient action-only inference.Experiments show strong generalization in simulation and real-robot deployment.Source: §5 · p. 10
• The authors conclude that directly learning action priors from scalable observation-only videos offers a simple and effective path toward scaling beyond costly action-labeled robot data.Source: §5 · p. 10

A Theoretical Analysis of Native Action-Prior Learning

• The transition representation carries future-state information beyond the preceding state, while future-state corruption prevents the prediction objective from being solved by copying the clean target.The analysis characterizes these as complementary roles in native action-prior learning.Source: A Theoretical Analysis of Native… · pp. 16–17

B Additional Implementation Details

• LIBERO and RoboTwin 2.0 downstream training use action-free target-domain cold start followed by joint Video-DiT and Action-DiT post-training with labeled demonstrations.RoboTwin 2.0 post-training uses Clean-domain demonstrations.Source: Table 3 · p. 18, Table 4 · p. 18

B.1 Native Action-Prior Pre-training

• Pretraining uses 235,790 observation-only episodes totaling 1,054.3 video hours, converted into approximately 7.0 million latent blocks.The Action-DiT alone is optimized for 14,000 steps with future-video flow matching, while the Video-DiT, text encoder, and VAE remain frozen.Source: §B.1 · p. 18

B.2 Downstream Post-training

• Downstream adaptation first trains the Video-DiT on target-domain videos without action labels while freezing the pretrained Action-DiT, then jointly post-trains both experts.The cold-start stage provides target-domain visual initialization and adds no action supervision.Source: §B.2 · p. 18

B.3 Detailed Transition-Structured Attention Masks

• Attention masks isolate each transition segment; pretraining lets noised-future and action queries attend within the segment, while post-training restricts noised-future queries to visual tokens.Action queries retain access to visual and action tokens during post-training.Source: Figure 7 · p. 19, §B.3 · p. 19

B.4 Benchmark and Evaluation Details

• Evaluation details specify LIBERO and LIBERO-Plus task and perturbation protocols, Clean-to-Random evaluation on RoboTwin 2.0, and the baseline sources.RoboTwin training uses Clean-domain demonstrations, with Random-domain evaluation representing an out-of-distribution shift.Source: §B.4 · pp. 19–20

B.5 Action-Only Inference

• Action-only inference caches Video-DiT keys and values in one joint forward pass, then reuses them while only the Action-DiT performs iterative denoising.The future-video latent is not iteratively denoised or decoded, and the visual cache is recomputed at each replanning step.Source: §B.5 · p. 20

C Additional Experimental Results

• The appendix adds analyses of native action-prior pretraining, visualizations of learned transition priors, benchmark breakdowns, and RoboTwin 2.0 policy rollouts.Source: C Additional Experimental Results · p. 20

C.1 Analysis of Native Action-Prior Pre-training

• Training and held-out future-video flow-matching losses decrease rapidly in the first few thousand steps and decline gradually thereafter.The reported curves indicate stable optimization.Source: §C.1 · pp. 20, 22, Figure 8 · p. 21
• Held-out future-video predictions capture hand motion and manipulated-object evolution across diverse egocentric interactions.The visualizations cover block stacking, two-handed assembly, cloth folding, and card manipulation.Source: Figure 9 · p. 21, §C.1 · p. 22

C.2 Qualitative Analysis of Learned Transition Priors

• Transferred target states follow source-specified motion across simulation, real-world, and sim-to-real examples despite differences in appearance and scene configuration.The examples include transfers across changes in scene layout and camera viewpoint.Source: Figure 10 · p. 22, §C.2 · p. 23

C.3 Detailed Benchmark Results

• 99.0% success on LIBERO-Long accompanies consistent performance across the four LIBERO suites, where several compared methods show their largest drop.Source: §C.3 · p. 23
• NAVA-WAM achieves the highest LIBERO-Plus macro-average, despite Image-WAM outperforming it on camera, lighting, background, and sensor-noise perturbations.NAVA-WAM performs best on robot-initialization, language, and object-layout shifts, with success rates of 65.6%, 93.8%, and 87.1%, respectively.Source: §C.3 · p. 23
• 44 of 50 RoboTwin 2.0 tasks have best or tied-best Random-domain performance, and 32 have best or tied-best Clean-domain performance.The aggregate Clean-to-Random performance drop is also substantially smaller than those of the compared baselines.Source: §C.3 · pp. 23–24, Table 7 · p. 24

• Each rollout row shows four keyframes, with a head-camera view and two wrist-camera views; blue and red outlines mark success and failure, respectively.Source: Figure 11 · p. 25

Successful rollouts

• Seven successful Random-domain rollouts span varied manipulation skills and complete instructed behaviors despite distractors, appearance, and scene variation.Tasks include bimanual coordination, precise placement, articulated-object manipulation, and long-horizon multi-object sequences.Source: §C.4 · p. 26

Failure cases

• The two representative failures involve unstable grasping in Move Can Pot and an incomplete second-arm subtask in Hanging Mug.In both cases, the episode reaches its step limit.Source: §C.4 · p. 26
• Both failures illustrate how a local execution error can propagate when the policy does not recover, despite recognizing the target and task progression.Source: §C.4 · p. 26

DROID post-training

• The policy is post-trained on approximately 76K DROID teleoperated trajectories and deployed directly on a physical Franka FR3 without evaluation-task or test-environment fine-tuning.Source: §C.5 · p. 26

Deployment setup

• Evaluation covers three tabletop tasks with five trials each, comparing π0.5, DreamZero, and NAVA-WAM under a shared observation pipeline and 15 Hz controller.NAVA-WAM predicts 32 absolute-action steps and executes 24 before replanning; π0.5 executes 8 of 24 relative-action steps, while DreamZero executes 24 absolute-action steps.Source: §C.5 · pp. 26–27

Detailed task analysis

• NAVA-WAM succeeds in 14 of 15 trials (93.3%), versus 10/15 (66.7%) for DreamZero and 8/15 (53.3%) for π0.5.Source: §C.5 · p. 27
• On T1, NAVA-WAM succeeds in 4/5 trials while both baselines fail all five; on T2 all policies succeed in five trials, and on T3 NAVA-WAM and DreamZero succeed in five while π0.5 succeeds in three.The T1 baseline rollouts show successful localization and grasping but failure to place the cube left of the bowl.Source: §C.5 · p. 27

Inference cost

• NAVA-WAM requires 15.8 ms per executed action, compared with 24.5 ms for π0.5 and 146.8 ms for DreamZero.The normalized measure accounts for the different numbers of actions executed before replanning.Source: §C.5 · p. 27

• NAVA-WAM can struggle with precise grasping, complex interactions, and other fine-grained contact-rich tasks; complex-instruction understanding and local-error recovery remain limitations.The authors identify mechanisms to detect failures and recover during execution as future work.Source: D Limitations · p. 27