PaperlayerView exact revision on arXiv

Source-linked AI summary

MVG-WAM: Multiple View Geometry-Aware World-Action Modeling for Robotic Manipulation

Wenbo Chen, Tianfu Li, Haoxuan Xu, Zhihao Cao, Zhenghan Chen, Zhengming Zhu, Zizhou Luo, Guosheng Yang, Yuan Liu, Lujia Wang, Wen Chen, Haoang Li

arXiv:2609.37793v1cs.RO

TL;DR

Multi-view WAM interfaces commonly leave geometric relationships among synchronized cameras implicit, despite the need to connect global context with local interaction geometry. MVG-WAM structures the views with epipolar-constrained and view-indexed geometric states, camera-aware routing, and metric-depth supervision. It reports results on LIBERO, RoboTwin 2.0, and real-world Cobot Magic trials.

  • Problem

    Common multi-view WAM interfaces tile images or concatenate tokens, leaving calibrated relationships among synchronized cameras implicit despite the need to connect global scene context with local interaction geometry.

    Source: §I · pp. 1–2

  • Method

    MVG-WAM combines an epipolar-constrained global state, jointly inferred view-indexed geometric states routed to camera regions, and multi-horizon future-depth supervision for metric grounding.

    Source: §I · p. 2

  • Results

    Without additional embodied pretraining, MVG-WAM achieves 99.1% average success on LIBERO and 92.07% on RoboTwin 2.0, and 91.3% across 150 Cobot Magic trials.

    Source: B. Simulation Benchmark Results · p. 5, §V · p. 8

  • Takeaways & Limitations

    The reported results support explicitly organizing multi-view geometry as an effective representation for world–action modeling in robotic manipulation.

    Source: §V · p. 8

Abstract

from arXiv · show

World-Action Models (WAMs) couple visual dynamics with action prediction, bringing the rich priors of pretrained video models to robotic manipulation. However, their multi-view interfaces typically tile images or concatenate tokens, leaving the geometric relationships among synchronized cameras implicit. This makes it harder to connect global scene context with the local geometry required for interaction. We introduce the Multi-View Geometry-Aware World-Action Model (MVG-WAM), which organizes these observations as related projections of one physical world rather than separate images on a canvas. Our model combines an epipolar-constrained global state with view-indexed geometric states jointly inferred from synchronized observations. Camera-aware routing supplies each video region with its corresponding geometric context and the shared global state, explicitly structuring the representation used for action prediction. We further ground the geometry-aware representation in metric scale through multi-horizon future-depth supervision, without requiring depth decoding during action rollout. MVG-WAM achieves average success rates of 99.1% on LIBERO and 92.07% on RoboTwin 2.0, demonstrating competitive performance across both benchmarks. Real-world experiments on Cobot Magic further demonstrate a 91.3% success rate across 150 trials spanning three manipulation tasks.

I. INTRODUCTION

Multi-view WAMs must connect global workspace context with local interaction geometry, but common video interfaces leave calibrated relationships among synchronized cameras implicit. MVG-WAM structures these observations as projections of one world through geometric states, camera-aware routing, and metric-depth supervision.

I. INTRODUCTION

  • Tiled or concatenated camera views leave calibrated cross-view relations implicit, although manipulation requires linking global scene context to local interaction geometry.The model must recover correspondences while learning visual dynamics and actions, which the paper says can weaken geometric coherence and spatial reasoning.Source: §I · pp. 1–2
  • MVG-WAM represents synchronized camera observations as geometrically related projections of one physical world rather than separate images on a canvas.Its global state aggregates information under epipolar constraints, and jointly encoded per-view states condition corresponding camera regions.Source: §I · p. 2
  • Multi-horizon future-depth supervision grounds the geometry-aware representation in metric scale without requiring depth decoding at deployment.The stated aim is to encourage physically scaled geometry across views and time.Source: §I · p. 2
  • Without additional embodied pretraining, MVG-WAM is reported as competitive on LIBERO and RoboTwin 2.0 and strong in real-world manipulation.The introduction does not give the benchmark or real-world success-rate values.Source: §I · p. 2

A. World–Action Modeling

  • WAMs build on pretrained video-generation backbones, while multi-view inputs are still commonly handled through generic interfaces that leave calibrated relations implicit.The passage situates this interface problem alongside video-prior approaches for robot control, [4].Source: A. World–Action Modeling · p. 2

B. Geometry-Aware Robot Learning

The paper distinguishes its focus from prior geometry-aware robot learning: it studies how synchronized camera observations relate within a multi-view WAM. Its formulation retains a video interface while adding geometric context to video and action modeling.

B. Geometry-Aware Robot Learning

A. WAM Formulation and Method Overview

  • The model predicts future video and action trajectories from synchronized calibrated-camera observations, with action tokens attending to the full video sequence.Future video latents and action tokens are independently perturbed during training; the base objective contains video and action losses.Source: A. WAM Formulation and Method Overview · pp. 2–3
  • Calibrated epipolar constraints aggregate cross-view features into a global state, while view-indexed geometric states route context to corresponding camera regions.The global and per-view conditions enter the video–action backbone, and action tokens access its geometry-enhanced video representation through mixed attention.Source: A. WAM Formulation and Method Overview · p. 3, Figure 2 · p. 3

C. Multi-View Geometric Modeling

B. Epipolar-Constrained Cross-View Global State

MVG-WAM uses calibrated epipolar geometry to relate dense features across camera views, then compresses the enhanced features into a compact cross-view memory.

B. Epipolar-Constrained Cross-View Global State

  • Dense DINOv3 patch features support geometry-constrained retrieval, addressing the information-capacity limits of compact video VAE latents.All cameras share a feature projector, and learned view embeddings identify source cameras.Source: B. Epipolar-Constrained Cross-View… · p. 3
  • Epipolar lines restrict each query patch’s source matches to geometrically admissible locations, where masked attention selects candidate features.A learnable null candidate offers a no-match option.Source: B. Epipolar-Constrained Cross-View… · pp. 3–4
  • A query receives an exact zero update when it has no valid real candidates, while a null candidate and confidence gate reduce unsupported matches.Each view can serve as both query and source; the residual is zero when no valid source views exist.Source: B. Epipolar-Constrained Cross-View… · p. 4
  • A learnable query bank pools epipolar-enhanced dense features into a compact cross-view memory.The residual gate is learned during training.Source: B. Epipolar-Constrained Cross-View… · p. 4

C. View-Indexed Geometric Conditioning

A frozen geometry foundation model jointly processes synchronized views into view-indexed states, which are routed to matching video regions alongside shared geometric context.

C. View-Indexed Geometric Conditioning

  • A frozen geometry foundation model jointly processes synchronized observations to provide compact, view-indexed geometric context.The implementation instantiates the encoder with VGGT-Ω and retains its per-view register banks.Source: C. View-Indexed Geometric Conditioning · p. 4
  • A shared adapter maps each view’s registers into WAM features, and a learnable query bank summarizes the adapted states.A camera embedding identifies each stream; calibrated epipolar geometry enters through the separate epipolar branch.Source: C. View-Indexed Geometric Conditioning · p. 4
  • Each video token is associated with the camera region containing its latent-grid center, and that region’s state conditions its temporal tokens.The routing is repeated across latent temporal groups.Source: C. View-Indexed Geometric Conditioning · p. 4
  • Geometric conditioning enters the video expert after mixed attention and before text cross-attention and the block FFN; action tokens access the resulting video representation through mixed attention.The epipolar and view-indexed branches use independent cross-attention parameters with layer-specific residual gates initialized to zero.Source: C. View-Indexed Geometric Conditioning · p. 4

D. Future Metric-Depth Supervision

Future metric-depth supervision anchors the geometry-aware representation to absolute scale during training, while the depth decoder is not used during rollout.

D. Future Metric-Depth Supervision

  • Multi-horizon metric-depth supervision encourages the evolving representation to preserve absolute spatial scale alongside cross-view structure.Targets are camera-axis depths in future camera frames, expressed in the same physical unit across views and horizons.Source: D. Future Metric-Depth Supervision · p. 4
  • Metric depth is auxiliary training supervision, and the depth decoder is disabled during action rollout.Rollout uses RGB, proprioception, language, and calibration without depth decoding.Source: D. Future Metric-Depth Supervision · pp. 4–5
  • The depth objective reads denoising features from noised future latents, grounding the representation rather than acting as a standalone forecasting module.The decoder produces log-depth and metric depth; invalid pixels and unavailable horizons are excluded by validity masks.Source: D. Future Metric-Depth Supervision · pp. 4–5
  • The auxiliary objective combines valid-pixel log-depth Huber loss, multiscale log-depth gradient matching, and an RGB-weighted gradient residual.The boundary weight comes from the normalized grayscale gradient of the corresponding future ground-truth RGB frame.Source: D. Future Metric-Depth Supervision · p. 5

IV. EXPERIMENTS

The experiments are framed around simulation and real-robot performance, robustness to distribution shifts, geometric contributions to control, and representation support for prediction and metric multi-view geometry.

IV. EXPERIMENTS

  • The first experiment question asks how MVG-WAM performs in simulation and on a real robot.The supplied experiment framing poses this as a question rather than reporting an outcome.Source: §IV · p. 5
  • The experiments ask how robust MVG-WAM is to distribution shifts and which geometric components contribute to control.These are posed as evaluation questions in the supplied passage.Source: §IV · p. 5
  • The experiments also ask whether the learned representation supports short-horizon prediction and metric multi-view geometry.The passage identifies this as a question, not a reported finding.Source: §IV · p. 5

A. Experimental Setup

The experiments evaluate jointly trained policies across all tasks in LIBERO and RoboTwin 2.0, with separate training configurations for each benchmark. The auxiliary depth decoder is excluded from control-time inference.

A. Experimental Setup

  • LIBERO covers 40 tasks across four suites with 50 demonstrations per task; RoboTwin 2.0 covers 50 tasks with 50 clean and 500 randomized demonstrations per task.One policy is trained jointly across the LIBERO suites and a separate policy covers all RoboTwin 2.0 tasks.Source: A. Experimental Setup · p. 5
  • Evaluation uses 50 trials per LIBERO task and 100 trials per RoboTwin 2.0 task and domain.These trial counts follow FastWAM [6].Source: A. Experimental Setup · p. 5
  • Training optimizes video and action experts end-to-end with MVG modules while freezing the geometry encoder, video VAE, and text encoder.LIBERO and RoboTwin 2.0 are trained for 30k and 100k updates, respectively; inference uses ten denoising steps and 32-action chunks, without the auxiliary depth decoder.Source: A. Experimental Setup · p. 5

B. Simulation Benchmark Results

MVG-WAM is evaluated against representative VLA and WAM policies, including comparisons that distinguish additional embodied pretraining. It attains strong benchmark success rates without such pretraining, while its RoboTwin 2.0 inference latency is lower than LingBot-VA's reported latency.

B. Simulation Benchmark Results

  • 99.1% average SR on LIBERO and 92.07% on RoboTwin 2.0 are achieved without additional embodied pretraining, gains of 0.6 and 1.49 percentage points over FastWAM-Joint.The comparison is with representative VLA and WAM policies and distinguishes whether additional embodied pretraining is used.Source: B. Simulation Benchmark Results · p. 5
  • 0.24 points over FastWAM and 0.17 points below embodied-pretrained LingBot-VA are reported on RoboTwin 2.0.LingBot-VA reports 5.23 s, while MVG-WAM takes 0.74 s per 32-step chunk on an H100.Source: B. Simulation Benchmark Results · pp. 5–6
  • The LIBERO-Plus table reports success rates under seven distribution shifts and marks the highest and second-highest value in each row.The cited caption describes the table's scope and markings, not its cell values.Source: Table II · p. 6
  • The 0.94-point Clean–Randomized gap is not an OOD evaluation because randomized demonstrations are included during training.Source: B. Simulation Benchmark Results · p. 6

C. Out-of-Distribution Generalization

The distribution-shift evaluations show gains on some conditions but not uniformly across all shifts. In clean-only RoboTwin 2.0 training, MVG-WAM improves on FastWAM-Joint, although randomized test conditions remain harder.

C. Out-of-Distribution Generalization

  • 70.4% pooled SR on LIBERO-Plus exceeds FastWAM-Joint's 68.7%, with the largest gains under Camera (+12.0 points) and Background (+4.2 points) shifts.Performance decreases under Language and Light perturbations, so gains do not extend uniformly across shifts.Source: C. Out-of-Distribution Generalization · p. 6
  • The Clean2Random comparison trains both multi-task policies on the same 2,500 clean demonstrations, without exposure to randomized RoboTwin 2.0 data.Both are trained for 100k updates and evaluated over 100 trials per task in each test domain; Table III reports C2C, C2R, and average success rates.Source: C. Out-of-Distribution Generalization · p. 6, Table III · p. 6
  • 72.6% C2C and 29.2% C2R SR are reported for MVG-WAM, improving over FastWAM-Joint by 2.8 and 27.9 percentage points, respectively.The remaining C2C–C2R gap indicates that unseen randomized conditions remain substantially more challenging.Source: C. Out-of-Distribution Generalization · p. 6, Table III · p. 6

D. Ablation Studies

The ablations support complementary roles for epipolar modeling, view routing, and metric-scale depth supervision. On Cobot Magic, MVG-WAM achieves 91.3% success across 150 trials, and removing the epipolar residual reduces mean success rate.

D. Ablation Studies

  • 2.61 points across all 50 tasks and 5.69 points on Hard-13 separate the complete model from FastWAM-Joint in the reduced-regime ablation.Removing metric-depth supervision decreases performance by 1.09 points on all tasks and 2.84 points on Hard-13; the ablation table uses 50 clean and 200 randomized demonstrations per task.Source: D. Ablation Studies · p. 6, Table IV · p. 7

E. Real-World Experiments

  • The Cobot Magic evaluation fine-tunes one three-task policy using 100 RGB-D training and ten validation demonstrations per task.Each task is evaluated over 50 trials, with interventions and safety stops counted as failures; Figure 3 shows representative execution sequences recorded from an external viewpoint.Source: E. Real-World Experiments · p. 6, Figure 3 · p. 7
  • 91.3% success across 150 Cobot Magic trials is reported for MVG-WAM, with success rates of 96%, 90%, and 88% across the three tasks.Under the same evaluation protocol, π0.5 achieves a mean SR of 89.3%; Table V reports success rate over 50 trials per task.Source: E. Real-World Experiments · pp. 6–7, Table V · p. 7
  • A 4.0-point reduction in mean real-world SR follows removal of the epipolar residual.The real-world evaluation uses radish-to-plate, cup stacking, and pot opening followed by pepper placement.Source: E. Real-World Experiments · pp. 6–7
  • On RoboTwin 2.0, MVG-WAM's short-horizon RGB predictions preserve sharper object boundaries and more coherent gripper–object geometry than FastWAM-Joint.The comparison uses the same current observation and future target, with gripper–object interaction regions enlarged.Source: Figure 4 · p. 7

F. Multi-View Geometric Analysis

The multi-view analysis reports RGB-prediction gains and metric-depth diagnostics on RoboTwin 2.0; the conclusion reports benchmark and real-world success rates.

F. Multi-View Geometric Analysis

  • The RoboTwin 2.0 analysis uses 625 clips from 125 validation trajectories, with current-observation-only RGB prediction and a separate depth diagnostic from noisy future latents.Table VI averages Clean and Random task-macro scores equally; depth metrics are not applicable to FastWAM-Joint, which lacks a trained native depth head.Source: F. Multi-View Geometric Analysis · p. 7, Table VI · p. 8
  • 3.74 dB higher average RGB PSNR and 0.072 higher SSIM than FastWAM-Joint, with better preservation of local object boundaries and gripper–object structure.The qualitative comparison focuses on gripper–object interaction regions against the same future target.Source: F. Multi-View Geometric Analysis · p. 8
  • 2.12 cm MAE, 5.80% AbsRel, and 1.68 cm cross-view z error characterize MVG-WAM’s teacher-forced depth readout under corrupted future latents.Reprojection coverage is 16.70%, versus 19.00% for the GT reference, and should be interpreted with reprojection error rather than alone; FastWAM-Joint has no trained depth head for direct comparison.Source: F. Multi-View Geometric Analysis · p. 8

V. CONCLUSION

  • 99.1% average success on LIBERO and 92.07% on RoboTwin 2.0; real-world evaluation achieves 91.3% success across 150 Cobot Magic trials.The reported benchmark rates are without additional embodied pretraining.Source: §V · p. 8
Loading 2609.37793v1…