Adapting Video Models for Robot Control via Motion-guided Asymmetric Prediction

1. Introduction

• The visual prediction objective shapes WAM representations for control, but its inherited video-generation formulation treats visual prediction as generation rather than robot control.Removing visual prediction substantially degrades control performance.Source: §1 · p. 1
• Prior approaches add semantic or geometric supervision, often relying on external pretrained models or modalities such as depth or 3D information.The paper instead asks how native latent-space visual supervision can better support action learning.Source: §1 · p. 1
• A motivating study found that shifting visual supervision toward dynamic regions consistently improves policy performance across WAMs.Dynamic regions tend to capture robot motion and object interactions, while static regions can contain complex but less action-relevant details such as background textures.Source: §1 · p. 1
• MoGAP treats dynamic regions as prediction targets and static regions as scene context, applying more noise to dynamic regions based on motion scores computed from training videos.It requires neither external teacher models nor additional supervision modalities.Source: §1 · p. 1
• MoGAP supplements the standard WAM objective through a prediction path in the shared video diffusion backbone that is used only during training.Inference is unchanged.Source: §1 · p. 2
• MoGAP raises GR-1 Tabletop success from 38.8% to 47.5% for Fast-WAM (Wan2.2-5B) and from 52.3% to 62.3% for Cosmos3-Nano.The paper also reports consistent improvement across WAM architectures in simulated and real-world manipulation benchmarks.Source: §1 · p. 2

2.1. Preliminaries: World Action Models

• WAMs jointly predict future visual observations and action chunks using pretrained video-diffusion backbones and visual latents encoded by a pretrained video autoencoder.Source: §2.1 · pp. 2–3

2.2. Motivating Study

• Motion scores from temporal differences in clean video latents are normalized by spatial mean and converted to weights that emphasize high-change regions while retaining low-change supervision.Source: §2.2 · pp. 3–4
• Redistributing visual supervision toward dynamic regions improved action-policy performance on simulated GR-1 tabletop, while the action-prediction objective stayed unchanged.Source: §2.2 · p. 4

2.3. Motion-Guided Asymmetric Prediction

• MoGAP extends error reweighting into asymmetric corruption, using less corrupted static regions as context for predicting more heavily corrupted dynamic regions.The task uses latent motion scores to assign regional noise levels and applies its visual loss to moving tokens.Source: §2.3 · p. 4
• MoGAP augments standard WAM training through an auxiliary path in the shared backbone, with an asymmetric attention mask and tokens omitted at inference.Source: §2.3 · p. 5

3.1. Setup

• The evaluation spans two multitask simulation benchmarks and an OpenArm real-world setup, covering different embodiments, observation configurations, and environmental conditions.RoboCasa GR-1 Tabletop has 24 humanoid hand-manipulation tasks; RoboCasa-Kitchen has 24 single-arm gripper tasks and three camera views.Source: §3.1 · p. 5

3.2. Simulated benchmark results

• 62.3% versus 52.3% average success on GR-1 Tabletop for Cosmos3-Nano, a 10.0-point gain that surpasses the compared VLA and WAM methods.Pick-and-place success rises from 48.3% to 60.8% (+12.5 points).Source: §3.2 · p. 6, Table 1 · p. 6
• MoGAP outperforms teacher-based representation methods on GR-1 Tabletop while using the same Cosmos3-Nano baseline and no external teacher.The comparison includes visual- and 3D-foundation-model teachers.Source: §3.2 · p. 6, Table 2 · p. 6
• Replacing standard visual supervision with MoGAP raises success from 52.3% to 54.4%, while the auxiliary branch yields a larger gain; removing or reversing noise asymmetry gives 55.3% and 55.2%.The authors describe standard full-noise-range and asymmetric supervision as complementary signals.Source: §3.2 · pp. 6–7, Table 3 · p. 7
• MoGAP raises Fast-WAM (Wan2.2-5B) success from 38.8% to 47.5%, extending the gains beyond Cosmos3 to another WAM architecture and video backbone.Fast-WAM omits future visual prediction at inference.Source: §3.2 · p. 7
• On RoboCasa-Kitchen, MoGAP outperforms the Cosmos3 baseline while retaining longer execution horizons, including with a moving wrist camera.Motion-guided noise is applied to external views; all wrist-camera tokens are assigned to the high-noise group because camera motion can dominate wrist-view motion scores.Source: §3.2 · p. 7, Figure 5 · p. 7

3.3. Real-world experiment results

• On OpenArm, MoGAP improves results across in-domain, OOD Objects & Environments, and OOD Object Relocation tasks, with particularly large gains on relocation despite no relocation demonstrations during training.The baseline often reaches toward the object’s former location; MoGAP more successfully adjusts its trajectory toward the relocated object.Source: §3.3 · p. 7, Figure 6 · p. 8, Figure 7 · p. 8

3.4. Further Analysis

• Motion scores localize moving arms, hands, and target objects over background, while trained policies attend more to those dynamic regions and generate clearer moving-arm structure; MoGAP’s training adds 1.26× wall-clock time.The analysis reports region-wise AUROC and attention shifts; the authors identify additional training tokens as the source of the time increase.Source: §3.4 · p. 8

Related work

• Other approaches use semantic encoders or depth and 3D geometry, often adding pretrained representation models or geometric and motion estimators.The passage contrasts these added sources of supervision with the paper’s native-latent-space direction.Source: §4 · pp. 8–9
• Prior video-generation representation work aligns diffusion or flow representations with external encoders, encourages physically plausible predictions, or incorporates 3D geometry.The passage cites Yu et al. (2025), Zhang et al. (2025), Bhowmik et al. (2026), and Wu et al. (2026).Source: §4 · p. 9
• MoGAP builds on token-wise noise assignment and introduces motion-guided asymmetry for action modeling without external teachers or additional modalities.The passage positions this as complementary to prior representation-learning approaches.Source: §4 · p. 9

• MoGAP adapts pretrained video diffusion models for robot control through motion-guided regional noise allocation.Source: §5 · p. 9
• MoGAP consistently improves performance over strong WAM baselines in simulated and real-world manipulation settings.Source: §5 · p. 9
• MoGAP improves adaptation to target object relocation tasks during execution, motivating visual supervision focused on task-relevant dynamics.Source: §5 · p. 9

A. Computational Details

• Cosmos3 training for GR-1 Tabletop takes 14.7 hours for 16K steps, while MoGAP takes 18.5 hours, or 15.0 hours with cached VAE latents.Both main runs use 16K steps and batch size 256; a 22K-step Cosmos3 run is also reported for the compute-matched comparison in Table 5.Source: A. Computational Details · p. 13, Table 5 · p. 13

B.1. Dataset

• The simulated benchmarks use 1,000 demonstrations per GR-1 Tabletop task and 300 per RoboCasa-Kitchen task from publicly available datasets.Source: §B.1 · p. 13

• Cosmos3 trains for 16K steps with batch size 256, while Fast-WAM trains for 100K steps with batch size 64; both predict 16 actions and 8 future frames.Cosmos3 uses a peak learning rate of 2 × 10^-5; Fast-WAM uses 10^-4.Source: §B.2 · p. 13
• Each method is evaluated over 50 fixed-seed trials per task suite, totaling 1,200 trials across the 24 suites in each benchmark.At execution, Cosmos3 uses all 16 generated actions without classifier-free guidance, while Fast-WAM uses 12 actions.Source: §B.2 · p. 13

• AGRA aligns per-token cosine features from denoiser patches with a frozen DINOv2 encoder, resizing its 32 × 32 feature grid to the 8 × 8 latent grid.It extracts hidden states after transformer block 8 and uses alignment loss weight 0.5.Source: §B.3 · p. 13
• Geometry Forcing aligns projected denoiser value features with frozen VGGT aggregator tokens using mean per-token cosine distance.Features come from transformer block 8; the alignment loss weight is 0.5.Source: §B.3 · p. 14
• VideoREPA aligns pairwise cosine-similarity matrices of projected denoiser features and frozen video-model features with a margin-based hinge loss.It uses VideoMAEv2 ViT-B with margin 0.1 or V-JEPA 2.1 ViT-L with margin 0.05.Source: §B.3 · p. 14
• Masked TSA restricts alignment queries to the top 20% of tokens per temporal slice, ranked by patch-wise frame differences, and minimizes teacher-to-student KL divergence.It uses VideoMAEv2 ViT-B, temperature 0.1, and alignment loss weight 0.5.Source: §B.3 · p. 14

B.4. Analysis Details

• AUROC evaluates whether latent motion scores identify regions, treating inside locations as positives and outside locations as negatives; 0.5 corresponds to chance-level discrimination.Ground-truth regions come from simulator segmentation masks; the moving arm, hand, and objects are selected using centroid displacements of at least two pixels.Source: §B.4 · pp. 14–15
• Attention ratio measures a region’s mean action-token attention share relative to its spatial coverage; values above one indicate concentration beyond uniform attention.The analysis aggregates across evaluation windows and conditioning and future-latent blocks, comparing both models on the baseline’s top four dynamic-attention layers.Source: §B.4 · p. 15

B.5. Hyperparameters and further details

• MoGAP uses auxiliary loss weight λMoGAP = 1.0; Wan2.2’s first predicted latent uses a forward-difference motion score, and zero scores yield uniform loss weights.The normalization denominator is lower-bounded by 10^-8; the all-zero case was not encountered in experiments.Source: §B.5 · p. 15

C.1. Hardware setup

• OpenArm has two 7-DoF arms, 6-DoF hands, a 2-DoF neck-mounted stereo camera, and a fixed table height; only the left egocentric view trains and evaluates the model.Both camera views are recorded; an exoskeleton and gloves support teleoperation and fingertip tracking.Source: §C.1 · p. 16

C.2. Real-Robot Tasks

• The six real-robot settings comprise in-domain pick-and-place and air-fryer manipulation, plus generalization to unseen objects, unseen environments, and dynamic object relocation under human interference.The two in-domain settings assess basic manipulation capabilities.Source: §C.2 · p. 16
• In-domain evaluation covers three object-destination pairs from three initial locations and separately tests opening and closing the air fryer.The pairs are banana to basket, mango to air fryer, and apple to air fryer.Source: §C.2 · p. 16
• Unseen-object tests replace training objects with a box, doll, cube, or tennis ball and require placing each in a basket or air fryer.The doll differs in shape and texture from dolls seen during training; the three predefined initial locations are reused.Source: §C.2 · p. 16
• Unseen-environment evaluation changes the tabletop covering while otherwise keeping the manipulation setup unchanged.Source: §C.2 · p. 16
• Object-relocation tests introduce human interference during pick-and-place, with set II displacing the object farther and later, when the hand is closer.Set II therefore requires stronger adaptation to task-relevant visual changes than set I.Source: §C.2 · p. 16

C.3. Real-World Training and Evaluation Details

• The OpenArm dataset contains teleoperated expert demonstrations across 21 real-world tasks, including in-domain pick-and-place and air-fryer open-and-close tasks.Source: §C.3 · p. 16
• Because the real-robot dataset is substantially smaller than the simulated benchmark’s, training uses four historical frames and image augmentations to help mitigate overfitting.The setup uses video-to-video generation, 8K optimization steps, and global batch size 256.Source: §C.3 · p. 16
• Evaluation uses 24 trials per task except for in-domain pick-and-place, which uses 48, totaling 168 trials per method.Success requires placing the target object at its designated destination or moving the air fryer fully open or closed, as appropriate.Source: §C.3 · p. 17
• Figures 10–13 show task examples for in-domain, unseen-object, unseen-environment, and relocation settings; Figure 14 visualizes the OpenArm dataset distribution.Source: Figure 10 · p. 17, Figure 11 · p. 17, Figure 12 · p. 17