Source-linked AI summary
PAIWorld: A 3D-Consistent World Foundation Model for Robotic Manipulation
Yuhang Huang, Xuan Lv, Junyan Xu, Zhiyuan Yu, Jiazhao Zhang, Ruizhen Hu, Wancheng Feng, Shilong Zou, Hewen Xiao, Ziqiao Zhou, Kaiyun Huang, Zhiyu Peng, Juzhan Xu, Hang Zhao, Chenyang Zhu, Renjiao Yi, Yifei Huang, Douhui Wu, Yan Zhang, Kexu Cheng, Chunhe Song, Yunzhi Xue, Xiuhong Zhang, Leitao Guo, Yunji Chen, Bin Wu, Haibin Yu, Kai Xu
TL;DR
Multi-view world models lack the geometric consistency robotic manipulation requires, so PAIWorld adds explicit cross-view communication and 3D supervision to a DiT-based model. It achieves state-of-the-art multi-view 3D consistency across robotic manipulation benchmarks, ranking first on WorldArena and second on AgiBot-Challenge2026.
Problem
Multi-view world models lack explicit inter-view communication and 3D geometric priors, limiting coherent future observations across cameras for robotic manipulation.
Method
PAIWorld combines Geometry-Aware Cross-View Attention, Geo-RoPE, and Latent 3D-REPA to exchange geometrically informed features and supervise 3D-consistent content.
Results
PAIWorld achieves state-of-the-art multi-view 3D consistency across robotic manipulation benchmarks, ranking 1st on WorldArena with a best overall EWMScore of 72.31%.
Takeaways & Limitations
The improved multi-view consistency supports physically plausible model-based planning, stronger action-visual alignment, and more effective manipulation policies.
Takeaways & Limitations
The architectural pathway alone can exploit shortcuts such as texture copying or feature averaging without an explicit geometric objective.
Abstract
from arXiv · showhide
World foundation models (WFMs) are powerful simulators, yet they predominantly operate in a single-view setting and lack the multi-view 3D consistency required for robotic manipulation. While robotic systems rely on multiple cameras (egocentric, eye-to-hand, and wrist-mounted) for policy learning, current multi-view world models simply concatenate view tokens without explicit geometric reasoning. This causes cross-view object drift, depth inconsistency, and texture misalignment. We trace these failures to two deficiencies: the absence of an explicit inter-view communication mechanism and the lack of a 3D geometric prior. We argue that resolving both simultaneously is necessary and sufficient. To address this, we present PAIWorld, a framework that augments diffusion-transformer world models via three core components: (1) Geometry-Aware Cross-View Attention blocks that establish an explicit pathway across views, (2) Geometric Rotary Position Embedding that encodes camera ray directions and extrinsic poses into the attention mechanism, and (3) Latent 3D-REPA, which distills 3D-aware features from frozen 3D foundation models to ensure 3D consistency. Built upon a DiT-based world foundation model, PAIWorld achieves state-of-the-art multi-view 3D consistency on robotic manipulation benchmarks, ranking 1st on the WorldArena leaderboard and 2nd on the AgiBot-Challenge2026 leaderboard, while enabling downstream applications such as model-based planning, world action models, and multi-view policy post-training.
1 Introduction
PAIWorld addresses the failure of existing multi-view world models to maintain 3D-consistent robotic manipulation simulations by combining explicit inter-view communication with a 3D geometric learning signal. Its three modular components establish and supervise this pathway, yielding state-of-the-art benchmark performance and downstream utility.
- Motivation: Robotic manipulation requires future observations from multiple cameras with geometrically compatible object locations, depth, and texture across viewpoints.
- Limitations: Existing methods either remain single-view or concatenate view tokens without explicit cross-view geometric reasoning, leaving viewpoints unable to coordinate predictions.
- Diagnosis: The paper identifies two deficiencies—missing inter-view communication and missing 3D geometric supervision—and argues that both are necessary and sufficient for multi-view 3D consistency.
- Method: PAIWorld combines Geometry-Aware Cross-View Attention and Geometric Rotary Position Embedding with Latent 3D-REPA to transmit and supervise 3D-consistent information across views.
- Results: 1st on WorldArena, PAIWorld achieves best overall EWMScore 72.31%; 2nd on AgiBot-Challenge2026, it achieves EWMScore 82.45% and best Scene Consistency 90.41%.The reported consistency gains also support downstream embodied applications, including model-based robotic planning.
2 Related Work
Prior work spans latent world models, pixel-space world foundation models, interactive simulators, and multi-view 3D generation. PAIWorld targets robotic manipulation’s dynamic, cluttered, wide-baseline multi-view regime through explicit geometric communication and 3D-aware supervision.
- World Models: World-model research progressed from compact latent dynamics for imagined robot-control rollouts to pixel- and video-space world foundation models trained with large-scale generative modeling.Dreamer and DayDreamer established latent-state planning, while Sora and Cosmos advanced video-space physical simulation and physical-AI modeling.
- Interactive World Models: Interactive world models support environment generation and action-conditioned simulation, including single-image generation, autoregressive scaling, and robotic video-language-action modeling.The related systems include UniSim, Genie, Genie 2, iVideoGPT, GR-2, and IRASim.
- Robotic Multi-View Regime: PAIWorld differs from object-centric multi-view generation by modeling cluttered dynamic scenes with manipulators and fixed, wide-baseline cameras having little view overlap.It injects geometry through an explicit communication pathway and a 3D supervisory objective rather than dense view sampling.
- Camera-Aware Generation: CameraCtrl and ViewCrafter provide camera-aware generation foundations but focus on single-view trajectory control or novel-view synthesis rather than multi-view consistency.CameraCtrl uses Plücker ray coordinates for camera poses in video diffusion models.
- 3D-Aware Supervision: Latent 3D-REPA extends representation alignment from 2D diffusion to multi-view video world models by using Depth Anything 3’s 3D-aware features as the alignment target.The stated goal is to inject geometric consistency directly into the diffusion process.
3 Method
PAIWorld injects 3D consistency into flow-matching world foundation models for multi-view robotic manipulation through an explicit inter-view pathway and a geometric training objective. Built on a DiT-based backbone, it combines Geometry-Aware Cross-View Attention, Geo-RoPE, and Latent 3D-REPA because neither architectural communication nor geometric supervision is sufficient alone.
- Backbone and baseline: The method uses a DiT-based flow-matching backbone with a pretrained spatial-temporal video VAE, while naive multi-view token concatenation lacks geometric inductive bias.Concatenation treats view tokens like temporal tokens, forcing the model to learn cross-view correspondences entirely from data.
- Problem setting: PAIWorld targets multi-view video generation in robotic systems, requiring views at each time step to admit a coherent 3D scene explanation.Corresponding points across views must respect the underlying epipolar geometry.
- Geometric encoding: Geo-RoPE separates pixel-level ray directions from view-level camera poses, encoding both geometric signals in distinct query-key subspaces.The ray subspace captures fine-grained correspondences, whereas the pose subspace represents camera identity and avoids interference between spatially varying and uniform signals.
- Inter-view pathway: Geometry-Aware Cross-View Attention and periodic spatial-concat self-attention establish explicit inter-view communication, with view-specific geometry guiding attention toward corresponding tokens.Geo-RoPE rotates queries and keys using each camera’s ray directions and pose, while zero-initialized gating preserves the pretrained single-view model at initialization.
- Geometric objective and complementarity: Latent 3D-REPA distills token relations from frozen 3D foundation models, complementing the pathway because architecture alone permits shortcuts and geometric priors alone cannot coordinate views.Their combination addresses deficiencies that neither component can resolve independently, yielding a non-additive improvement in 3D consistency.
4 Experiments
PAIWorld is evaluated as an action-conditioned world simulator on WorldArena and AgiBot-Challenge2026, with results showing strong overall performance and multi-view 3D consistency. Ablations further indicate that explicit inter-view communication and geometric supervision reinforce each other.
- Implementation: PAIWorld builds on Cosmos-Predict2.5 with approximately 14B parameters, inserting Geometry-Aware Cross-View Attention and Geo-RoPE into the pretrained backbone.Cosmos-Reason1 provides physically grounded text conditioning, while REPA projection heads are randomly initialized.
- Implementation: The training dataset contains approximately 2.5M multi-view robotic-manipulation video clips from five sources spanning diverse embodiments, tasks, and camera configurations.The sources are AgiBot-World, RoboMIND, Galaxea, RoboTwin, and RoboCOIN.
- Action-conditioned generation: 72.31 EWMScore ranks PAIWorld first on WorldArena, ahead of UNIS at 72.16 and BWM-Fast at 72.15, while achieving second-best Motion Quality at 80.45.The aggregate score reflects performance across seven fine-grained dimensions rather than dominance on one metric.
- Action-conditioned generation: 0.8245 EWMScore ranks PAIWorld second on AgiBot-Challenge2026, while its best Scene Consistency score is 0.9041 and its nDTW score is 0.9531.These results indicate strong multi-view coherence and close tracking of ground-truth action sequences.
- Multi-view consistency: PAIWorld achieves the best MEt3R score of 14.20, improving 10% over Genie-Envisioner at 15.75, with a Geometric consistency score of 0.4056 where lower is better.It also attains the best score on 6 of 7 metrics, including SSIM of 0.7683, LPIPS of 0.1844, and FID of 45.04.
- Ablation: 2.64 MEt3R improvement from the full model exceeds the individual gains of 0.93 and 0.72, whose sum is 1.65, demonstrating super-additive coupling.The communication pathway transmits information, while geometric supervision makes it 3D-consistent across views.
5 Conclusion
PAIWorld addresses 3D-consistent multi-view generation for robotic manipulation through explicit inter-view communication and a training signal enforcing 3D-consistent content. Built on a DiT-based world foundation model, it achieves state-of-the-art multi-view 3D consistency and supports downstream embodied tasks, while motivating physics-aware, long-horizon, and industrial extensions.
- Contributions: PAIWorld achieves 3D-consistent multi-view generation for robotic manipulation by combining an architectural inter-view communication pathway with a 3D-consistency training signal.Geometry-Aware Cross-View Attention, shaped by Geometric Rotary Position Embedding, enables geometrically corresponding token exchange, while Latent 3D-REPA supplies geometric learning.
- Results: PAIWorld attains state-of-the-art multi-view 3D consistency on robotic manipulation benchmarks across reconstruction-based, geometric, and scene-consistency metrics in text- and action-conditioned settings.The improved consistency benefits model-based planning, world action models, and multi-view downstream applications.
- Future Work: Future work includes physical interaction modeling, long-horizon planning, and extending 3D consistency beyond geometry into physics-aware world modeling.Planned physics extensions include contact dynamics, deformable objects, and fluid simulation.
- Future Work: Coupling PAIWorld with a World Action Model could create a closed loop in which imagined experiences improve policies, real-world data, and the world model.The proposed loop supports continuous self-improvement and autonomous evolution of embodied agents.
- Future Work: The framework is also intended to support industrial manufacturing foundation models for production-line scheduling and real-time manufacturing-process control.This direction targets applications requiring accurate physical simulation.
Contributions
The paper lists its core contributors and additional contributors, with Kai Xu identified as the corresponding author.
- The core contributors are Yuhang Huang, Jiazhao Zhang, Xuan Lv, Junyan Xu, Zhiyuan Yu, Ruizhen Hu, and Kai Xu.
- Additional contributors include Wancheng Feng, Shilong Zou, Hewen Xiao, Ziqiao Zhou, Kaiyun Huang, Zhiyu Peng, Juzhan Xu, Hang Zhao, Zhibin Zhu, Chenyang Zhu, Renjiao Yi, Yifei Huang, Douhui Wu, Yan Zhang, Kexu Cheng, Chunhe Song, Yunzhi Xue, Xiuhong Zhang, Leitao Guo, Yunji Chen, Bin Wu, and Haibin Yu.
- Kai Xu is identified as the corresponding author.