PaperlayerView exact revision on arXiv

Source-linked AI summary

PAIWorld: A 3D-Consistent World Foundation Model for Robotic Manipulation

Yuhang Huang, Xuan Lv, Junyan Xu, Zhiyuan Yu, Jiazhao Zhang, Ruizhen Hu, Wancheng Feng, Shilong Zou, Hewen Xiao, Ziqiao Zhou, Kaiyun Huang, Zhiyu Peng, Juzhan Xu, Hang Zhao, Chenyang Zhu, Renjiao Yi, Yifei Huang, Douhui Wu, Yan Zhang, Kexu Cheng, Chunhe Song, Yunzhi Xue, Xiuhong Zhang, Leitao Guo, Yunji Chen, Bin Wu, Haibin Yu, Kai Xu

arXiv:2606.18375v3cs.RO

TL;DR

Multi-view world models lack explicit geometric reasoning to maintain 3D consistency across robotic camera views. PAIWorld combines cross-view communication with geometric supervision and achieves state-of-the-art benchmark consistency, ranking 1st on WorldArena and 2nd on AgiBot-Challenge2026.

  • Problem

    Current multi-view world models lack explicit inter-view communication and a 3D geometric prior for maintaining coherent observations across robotic camera views.

    Source: §1 · p. 2

  • Method

    PAIWorld combines Geometry-Aware Cross-View Attention and Geo-RoPE for geometrically informed inter-view exchange with Latent 3D-REPA supervision from frozen 3D foundation models.

    Source: §1 · p. 3

  • Results

    Across text- and action-conditioned settings, PAIWorld leads on reconstruction, geometric, and scene-consistency metrics and ranks 1st on WorldArena (EWMScore 72.31%).

    Source: §1 · p. 3, §5 · p. 16

  • Takeaways & Limitations

    The reported consistency gains carry over to downstream embodied tasks, with benefits for model-based planning, world action models, and multi-view policy post-training.

    Source: §5 · p. 16

    The cross-view pathway alone does not ensure 3D-consistent content, so PAIWorld also requires a geometric learning objective.

    Source: §3.4 · p. 8

Abstract

from arXiv · show

World foundation models (WFMs) are powerful simulators, yet they predominantly operate in a single-view setting and lack the multi-view 3D consistency required for robotic manipulation. While robotic systems rely on multiple cameras (egocentric, eye-to-hand, and wrist-mounted) for policy learning, current multi-view world models simply concatenate view tokens without explicit geometric reasoning. This causes cross-view object drift, depth inconsistency, and texture misalignment. We trace these failures to two deficiencies: the absence of an explicit inter-view communication mechanism and the lack of a 3D geometric prior. We argue that resolving both simultaneously is necessary and sufficient. To address this, we present PAIWorld, a framework that augments diffusion-transformer world models via three core components: (1) Geometry-Aware Cross-View Attention blocks that establish an explicit pathway across views, (2) Geometric Rotary Position Embedding that encodes camera ray directions and extrinsic poses into the attention mechanism, and (3) Latent 3D-REPA, which distills 3D-aware features from frozen 3D foundation models to ensure 3D consistency. Built upon a DiT-based world foundation model, PAIWorld achieves state-of-the-art multi-view 3D consistency on robotic manipulation benchmarks, ranking 1st on the WorldArena leaderboard and 2nd on the AgiBot-Challenge2026 leaderboard, while enabling downstream applications such as model-based planning, world action models, and multi-view policy post-training.

1 Introduction

PAIWorld addresses multi-view world models’ failures to maintain 3D consistency by combining explicit inter-view communication with a geometric learning signal. The paper reports leading benchmark results and downstream gains in embodied tasks.

Applications

  • The paper positions PAIWorld as a backbone for multi-view world generation, world action modeling, robotic planning, and policy post-training, maintaining coherent object placement, depth, and texture across viewpoints.Figure 1 describes pre-training on 2.5M multi-view video clips.Source: Figure 1 · p. 2

Problem

  • Existing multi-view methods often concatenate view tokens without explicit geometric reasoning, while single-view WFMs cannot generate observations across the views robotic manipulation requires.These limitations risk cross-view object drift, depth contradictions, and texture misalignment that undermine imagined trajectories and downstream planning and control.Source: §1 · p. 2

Motivation

  • The paper identifies missing inter-view communication and missing 3D geometric priors as complementary deficiencies, arguing both remedies are necessary and sufficient for multi-view 3D consistency.Communication without geometric supervision can lead to shortcuts such as texture copying, while a geometric prior without communication cannot resolve cross-view inconsistencies.Source: §1 · pp. 2–3

PAIWorld

  • PAIWorld combines Geometry-Aware Cross-View Attention and Geo-RoPE for inter-view communication with Latent 3D-REPA, which aligns DiT features to 3D-aware representations from frozen 3D foundation models.Geo-RoPE encodes camera ray directions and extrinsic poses; Latent 3D-REPA supplies geometric supervision for exchanged content.Source: §1 · p. 3

Results

  • PAIWorld ranks 1st on WorldArena (overall EWMScore 72.31%, best Motion Quality) and 2nd on AgiBot-Challenge2026 (EWMScore 82.45%, best Scene Consistency 90.41%).The paper also reports gains in model-based robotic planning and world action model fine-tuning.Source: §1 · p. 3

2 Related Work

Related work spans world foundation models, multi-view visual generation, and geometric representation learning. PAIWorld addresses the gap between these fields by targeting dynamic, cluttered manipulation scenes with sparse fixed cameras and explicit geometric supervision.

2.1 World Foundation Models for Physical AI

  • World models have shifted from latent-state planning toward pixel- and video-space WFMs, but current systems remain single-view or lack explicit cross-view geometric reasoning [2] [10].Interactive models include action-conditioned robotic simulators and manipulation planners, while AgiBot World and WorldArena [33] provide a multi-view platform and benchmark.Source: §2.1 · pp. 3–4

2.2 Multi-View and 3D-Aware Visual Generation

  • Prior multi-view generation targets clean, object-centric assets, static or short orbital motion, and dense smooth views, unlike cluttered dynamic manipulation under sparse fixed wide-baseline cameras.Examples include SyncDreamer [35], MVDream [36], and SV4D [39]; CameraCtrl and ViewCrafter [41] address single-view camera trajectory control rather than multi-view consistency.Source: §2.2 · p. 4

2.3 3D Representations and Geometric Reconstruction

  • Geometric vision encoders offer 3D-aware supervision targets, and Latent 3D-REPA extends REPA from image diffusion to multi-view video using Depth Anything 3 features.Related priors include dense point-map models DUSt3R [44] and MASt3R [45], monocular depth models [46] [47] [48], and VGGT [49], which jointly predicts camera poses, depth, and point maps.Source: §2.3 · p. 5

3 Method

PAIWorld injects 3D consistency into multi-view robotic manipulation world models by combining an inter-view pathway—Geometry-Aware Cross-View Attention shaped by Geo-RoPE—with Latent 3D-REPA supervision on a DiT flow-matching backbone.

3.1 Problem Formulation

  • Multi-view generation requires views to admit a coherent 3D scene, with corresponding points respecting epipolar geometry across camera views.Source: §3.1 · p. 6

3.2 Flow Matching DiT for Video Generation

  • A flow-matching DiT operates on spatial-temporal VAE latents; AdaLN injects text, while rendered spatial action maps condition action generation and preserve projected geometric structure.Source: §3.2 · p. 6

3.3 Geometric Rotary Position Embedding

  • Geo-RoPE splits attention features between pixel-level ray directions and camera poses, giving corresponding tokens similar rotations while encoding view identity.Source: §3.3 · pp. 6–7

3.4 Geometry-Aware Cross-View Attention

  • Geometry-Aware Cross-View Attention uses view-specific Geo-RoPE to weight corresponding tokens, while periodic spatial-concatenation attention broadens cross-view context; its gate starts at zero.Source: §3.4 · pp. 7–8

3.5 Latent 3D-REPA (3D Geometric Prior)

  • Latent 3D-REPA distills pairwise token relations, not absolute features, from Depth Anything 3’s 3D-supervised representations across spatial and temporal scales using anchor sampling with O(MK) cost.Source: §3.5 · pp. 8–9

3.6 Joint Mechanism

  • The two pillars are complementary: geometry-guided pathways route information between views, while 3D-REPA supervises its content, yielding a non-additive consistency gain neither achieves alone.Source: §3.4 · p. 8, §3.6 · p. 9, §3.5 · p. 9

3.7 Training Objective

  • Training combines flow-matching and Latent 3D-REPA losses, with λ = 0.5 balancing generation quality against 3D alignment.Source: §3.7 · p. 9

4 Experiments

Experiments evaluate action-conditioned rollouts on WorldArena and AgiBot-Challenge2026, text-conditioned multi-view generation on AgiBot-World, and ablations of the geometric components. PAIWorld ranks first on WorldArena, second on AgiBot-Challenge2026, and leads on most text-conditioned metrics, including measures of 3D consistency.

4.2.1 WorldArena Benchmark

  • 72.31 EWMScore earned PAIWorld 1st on WorldArena, ahead of UNIS (72.16) and BWM-Fast (72.15), with second-best Motion Quality (80.45).The paper attributes the overall lead to a strong profile across metrics rather than a single standout dimension.Source: Table 1 · p. 10, §4.2.1 · p. 11

4.1 Implementation Details

  • PAIWorld builds on Cosmos-Predict2.5 [53] as an approximately 14B-parameter DiT and uses Cosmos-Reason1 [54] for physically grounded text conditioning.It is trained on approximately 2.5M multi-view clips from five datasets, with task-specific fine-tuning for action-conditioned benchmarks; Depth Anything 3 [48] is frozen during training.Source: §4.1 · p. 10

4.2.2 AgiBot-Challenge2026 Benchmark

  • 0.8245 EWMScore ranked PAIWorld second on AgiBot-Challenge2026, where its best Scene Consistency was 0.9041 and its nDTW was 0.9531.NeoVerse-ABot leads on EWMScore and PSNR, while PAIWorld surpasses it on Scene Consistency by 0.67%.Source: Table 2 · p. 12, §4.2.2 · p. 13

4.3 Text-Conditioned Multi-View Generation

  • 6 of 7 metrics favored PAIWorld on AgiBot-World, including SSIM 0.7683, LPIPS 0.1844, and FID 45.04.SSIM and LPIPS surpass Genie-Envisioner by 3.2% and 45%, respectively; FID improves 20% over Wan2.1 (56.47). PAIWorld also records the best MEt3R score (14.20), a 10% improvement over Genie-Envisioner (15.75), and a Geometric score of 0.4056; Genie-Envisioner leads Scene Consistency at 0.9231 versus PAIWorld’s 0.9041.Source: Table 3 · p. 14, §4.3 · pp. 14–15, Figure 7 · p. 15

4.4 Ablation Study

  • The full model improves MEt3R by 2.64, exceeding the combined individual gains of 0.93 from the cross-view pathway and 0.72 from geometric supervision.Each component alone has a limitation: the pathway can use texture-copying shortcuts, while per-view supervision cannot propagate across viewpoints; together, SSIM, LPIPS, and FID improve most.Source: §4.4 · pp. 15–16, Table 4 · p. 16

5 Conclusion

PAIWorld pairs geometric inter-view communication with 3D-aware training to generate 3D-consistent multi-view worlds for robotic manipulation. The conclusion reports benchmark and downstream-task benefits, while identifying future directions in physics-aware modeling, long-horizon consistency, embodied learning, and manufacturing.

5 Conclusion

  • Geometry-Aware Cross-View Attention with Geometric Rotary Position Embedding enables inter-view exchange, while Latent 3D-REPA supplies geometric supervision; neither remedy alone suffices.The conclusion attributes consistency to addressing both the architectural pathway and training objective together.Source: §5 · p. 16
  • PAIWorld achieves state-of-the-art multi-view 3D consistency across reconstruction-based, geometric, and scene-consistency metrics in text- and action-conditioned settings.The framework is built upon a DiT-based world foundation model.Source: §5 · p. 16
  • Improved consistency benefits model-based planning, strengthens action-visual causal alignment in world action models, and enables more effective manipulation policies through multi-view policy post-training.Planning benefits from physically plausible imagined trajectories.Source: §5 · p. 16
  • Future work would extend geometric consistency to physics-aware modeling with contact dynamics, deformable objects, and fluid simulation, and maintain it over long-horizon rollouts using hierarchical or recurrent architectures.These are proposed directions, not reported results.Source: §5 · p. 16
  • The authors aim to couple the world model with a World Action Model in a data loop where imagined experiences improve policies, which collect better real-world data to refine the model.The proposed loop is intended to enable continuous self-improvement and autonomous evolution of embodied agents.Source: §5 · p. 16
  • The authors plan industrial manufacturing foundation models for dynamic production-line scheduling and real-time process control, where accurate physical simulation and multi-view monitoring are critical.This direction builds upon the world-modeling framework.Source: §5 · p. 16

Related papers

Loading 2606.18375v3…