PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning 4D Geometric Priors for Inference-Efficient World Action Models

Jianjun Zhang, Jian Zhu, Taiyi Su, Chong Ma, Zitai Huang, Yi Xu, Hanli Wang

arXiv:2607.05468v1cs.ROcs.AI

TL;DR

Appearance-oriented video latents may not capture the evolving geometry needed for precise manipulation. MECo-WAM uses a training-only 4D expert and action-aware geometric distillation to transfer those priors into video-action representations. The reported experiments show improved manipulation performance without increased inference cost.

  • Problem

    Appearance-oriented video latents may support plausible future synthesis without preserving the evolving spatial relations needed for manipulation.

    Source: Introduction · p. 1

  • Method

    MECo-WAM trains a lightweight 4D expert with frozen VGGT targets and transfers action-relevant temporal geometry into the video-action pathway through decayed read-mask attention and geometric distillation.

    Source: Introduction · p. 2

  • Results

    Experiments on LIBERO, RoboTwin 2.0, and ARX-R5 real-world tasks, together with ablations, show that action-relevant 4D supervision improves geometric reasoning.

    Source: Conclusion · p. 7

  • Takeaways & Limitations

    The reported gains in geometric reasoning do not increase deployment cost because auxiliary geometry modules are removed at inference.

    Source: Conclusion · p. 7

    Clean VGGT targets are used only for training losses and never as input to the deployed policy.

    Source: Multi-Expert Co-Training Architecture · p. 4

Abstract

from arXiv · show

World Action Models (WAMs) have shown strong potential for robotic manipulation by jointly modeling visual future dynamics and executable action sequences. However, existing video-action co-training methods primarily optimize appearance-oriented video latents, which may insufficiently capture the temporally evolving geometry required for precise manipulation. We propose MECo-WAM, a Multi-Expert Co-Training World Action Model that injects action-relevant 4D geometric priors into video-action representations while preserving the original lightweight inference graph. During training, MECo-WAM combines video and action experts with a lightweight 4D expert supervised by relational targets from a frozen VGGT encoder. Asymmetric expert visibility prevents non-causal shortcuts from auxiliary geometry to action generation. To transfer geometric knowledge into the deployed video-action pathway, we introduce decayed 4D read-mask attention, which provides restricted current-frame geometric guidance early in training and progressively removes this dependency. We further propose action-aware temporal geometric distillation, which aligns within-frame geometric relations and their temporal evolution while emphasizing visual regions most relevant to robot actions. At deployment, all auxiliary 4D components are removed. Experiments on LIBERO (98.2%), RoboTwin 2.0 (92.6%), and challenging real-world manipulation tasks show that MECo-WAM improves manipulation performance without increasing inference cost.

Introduction

Video-action co-training offers motion and interaction priors, but appearance-oriented video latents may not preserve the evolving spatial relations needed for manipulation. MECo-WAM adds training-time 4D geometric supervision while retaining a lightweight video-action inference pathway.

Introduction

  • Appearance-oriented video latents may omit spatial relations that determine grasp reachability, object alignment, and contact outcomes, even when they synthesize plausible futures.These relations evolve as the robot approaches, contacts, moves, and releases objects.Source: Introduction · p. 1
  • Explicit deployment-time geometry can raise inference cost, while generic geometric supervision may overlook relations relevant to the robot’s current action.Source: Introduction · p. 1
  • MECo-WAM adds a lightweight 4D expert supervised by frozen VGGT features alongside video and action denoising, while restricting geometric access to early current-frame guidance.Future geometry remains loss-side supervision, limiting non-causal shortcuts to action generation.Source: Introduction · p. 2
  • Decayed 4D read-mask attention removes current-frame geometric access before deployment, while temporal distillation aligns keyframe relations and their changes with action-relevant token pairs.Source: Introduction · p. 2
  • Experiments on LIBERO, RoboTwin 2.0, and real-world manipulation tasks demonstrate gains in task success and execution efficiency.Source: Introduction · p. 2

Related Work

World action models use video-based temporal supervision to complement direct action policies, and deployment-oriented designs can avoid future imagination at test time. Geometry-aware approaches add spatial structure in several ways; MECo-WAM instead transfers frozen 4D priors during training into a lightweight video-action path.

World Action Models

  • Video-based temporal supervision exposes policies to physical evolution and action-conditioned scene changes, complementing direct observation-to-action learning.Fast-WAM keeps future video prediction in training while allowing test-time action generation without explicit future imagination.Source: World Action Models · p. 2

Geometry-Aware Embodied Models

  • Geometry-aware VLA and WAM methods use spatial inputs, alignment, constraints, depth, or 4D world synthesis to support manipulation.MECo-WAM uses frozen VGGT structure as a training-time representation constraint and removes auxiliary geometry components at inference.Source: Geometry-Aware Embodied Models · p. 2, Figure 2 · p. 3

Methodology

MECo-WAM adds a training-only 4D expert to the video-action pathway, using frozen VGGT geometry targets and restricted information flow. Action-aware relational and temporal distillation transfers geometric supervision while the deployed policy retains only its video and action experts.

Multi-Expert Co-Training Architecture

  • Training combines video and action experts with a 4D expert supervised by frozen VGGT features from current and future frames, while clean VGGT targets remain training-loss targets rather than deployed-policy inputs.The video expert predicts future VAE latents, the action expert predicts future action chunks, and the 4D expert predicts selected future geometry keyframes.Source: Problem Formulation · p. 3, Multi-Expert Co-Training Architecture · pp. 3–4

Decayed 4D Read-Mask Attention

  • The decayed read mask temporarily lets future video and action queries read current-frame geometry g0, while future 4D slots remain auxiliary targets and all 4D tokens are removed at inference.Current anchors are self-only; future action tokens cannot read noisy future video slots, preventing future-information shortcuts.Source: Decayed 4D Read-Mask Attention · p. 4, Figure 3 · p. 4

Action-Aware Temporal Geometric Distillation

  • Action-aware temporal geometric distillation matches pairwise geometry relations and their changes across keyframes, weighting visual token pairs by their coupling to contemporaneous robot motion.The weighting mixes learned relevance with a uniform prior to prevent collapse onto a single location while retaining scene context.Source: Action-Aware Temporal Geometric… · pp. 4–5

Training Objective and Inference

  • The video and action experts use the base WAM conditional flow-matching objective, alongside an auxiliary 4D objective during training.The auxiliary 4D components are removed at inference, leaving the original video and action experts.Source: Training Objective and Inference · p. 5
  • At inference, MECo-WAM removes the 4D expert, VGGT encoder, and alignment modules, leaving the original video-action pathway without an extra geometric decoder, sensor input, or denoising stage.The training-time 4D objective is intended to transfer geometric priors into the WAM representation.Source: Training Objective and Inference · p. 5

Experiments

MECo-WAM is evaluated on LIBERO, RoboTwin 2.0, and real-world tabletop tasks, with ablations testing how its 4D components affect performance. The reported results show gains over Fast-WAM across benchmarks and real-robot tasks.

Experimental Setup

  • Evaluation covers LIBERO’s four suites, RoboTwin 2.0’s clean and randomized conditions, and real-robot cube stacking and size sorting.The tabletop evaluation reports success rate, progress rate, correction count, and completion time.Source: Experimental Setup · pp. 5–6

Main Results

  • 98.2% average LIBERO success is 0.6 points above Fast-WAM and close to pretrained LingBot-VA’s 98.5%.MECo-WAM scores 98.8% on Spatial, 100.0% on Object, and 98.2% on Goal.Source: Main Results · p. 6, Table 1 · p. 6
  • 92.62% RoboTwin 2.0 average success improves on Fast-WAM’s 91.83%, with gains in both clean and randomized evaluation.The clean results are 93.26% vs. 91.88%; randomized results are 91.98% vs. 91.78%.Source: Main Results · p. 6, Table 2 · p. 6
  • Across two ARX-R5 tabletop tasks, MECo-WAM gains 5.0 success-rate points and 2.5 progress-rate points over Fast-WAM, with about 39% fewer corrections and 12% shorter completion time.On sorting by size, success rate rises from 60.0% to 70.0%; on stacking, success and progress rates match Fast-WAM while corrections and completion time decrease.Source: Main Results · p. 6, Table 3 · p. 7
  • Probes excluding auxiliary 4D tokens show sharper depth and stronger sensitivity to action-relevant 3D position and pose in the video-action features.The grasp comparison concerns cube position, grasp pose, and contact geometry.Source: Main Results · p. 6, Figure 6 · p. 7

Ablation Studies

  • Full MECo-WAM reaches 92.62% average success and 0.013 action MSE, versus 91.83% success for Fast-WAM; the uniform-weight variant reaches 92.38%.An isolated 4D expert raises success only to 91.87% and does not improve action MSE; decayed read access raises success to 92.14% and lowers MSE to 0.026.Source: Ablation Studies · pp. 6–7, Table 4 · p. 7

Conclusion

MECo-WAM injects 4D geometry during training and transfers frozen VGGT priors into the deployed video-action representation. Across LIBERO, RoboTwin 2.0, and ARX-R5 tasks, results and ablations show improved geometric reasoning without increased deployment cost.

Conclusion

  • MECo-WAM uses a training-only 4D expert, decayed read-mask attention, and action-aware temporal geometric distillation to transfer frozen VGGT priors into the deployed video-action representation.Auxiliary geometry modules are removed at inference.Source: Conclusion · p. 7
Loading 2607.05468v1…