Source-linked AI summary
ABot-PhysWorld: Interactive World Foundation Model for Robotic Manipulation with Physics Alignment
Yuzhi Chen, Ronghan Chen, Dongjie Huo, Yandan Yang, Dekang Qi, Haoyun Liu, Tong Lin, Shuang Zeng, Junjin Xiao, Xinyuan Chang, Feng Xiong, Xing Wei, Zhiheng Ma, Mu Xu
TL;DR
Embodied video world models need stronger physical consistency and zero-shot evaluation than existing benchmarks and generic visual-data training provide. ABot-PhysWorld addresses this with curated, physics-annotated data, physics-aware DPO, spatial action injection, and EZSbench, achieving state-of-the-art results on PBench and EZSbench against Veo 3.1 and Sora v2 Pro.
Problem
Existing embodied-video benchmarks emphasize visual quality or in-distribution accuracy, with limited assessment of physical consistency and zero-shot generalization.
Method
ABot-PhysWorld combines curated embodied-video data, physics-aware DPO, parallel spatial action injection, and EZSbench’s decoupled evaluation protocol.
Results
The model achieves state-of-the-art physical fidelity and improved trajectory consistency compared with Veo 3.1 and Sora v2 Pro on PBench and EZSbench.
Takeaways & Limitations
EZSbench provides a training-independent benchmark for evaluating physical fidelity and action alignment under distribution shift.
Takeaways & Limitations
The model currently relies on fixed-viewpoint data and lacks closed-loop evaluation.
Abstract
from arXiv · showhide
Video-based world models offer a powerful paradigm for embodied simulation and planning, yet state-of-the-art models often generate physically implausible manipulations - such as object penetration and anti-gravity motion - due to training on generic visual data and likelihood-based objectives that ignore physical laws. We present ABot-PhysWorld, a 14B Diffusion Transformer model that generates visually realistic, physically plausible, and action-controllable videos. Built on a curated dataset of three million manipulation clips with physics-aware annotation, it uses a novel DPO-based post-training framework with decoupled discriminators to suppress unphysical behaviors while preserving visual quality. A parallel context block enables precise spatial action injection for cross-embodiment control. To better evaluate generalization, we introduce EZSbench, the first training-independent embodied zero-shot benchmark combining real and synthetic unseen robot-task-scene combinations. It employs a decoupled protocol to separately assess physical realism and action alignment. ABot-PhysWorld achieves new state-of-the-art performance on PBench and EZSbench, surpassing Veo 3.1 and Sora v2 Pro in physical plausibility and trajectory consistency. We will release EZSbench to promote standardized evaluation in embodied video generation.
1 Introduction
ABot-PhysWorld targets physically plausible, visually realistic, and action-controllable embodied video generation. It combines curated training data, physics-aware preference alignment, spatial action injection, and EZSbench for zero-shot evaluation.
- The two-stage training pipeline first applies SFT to predict future frames, then uses physics-checklist scoring and DPO with LoRA on frozen DiT weights.
- EZSbench is a training-independent zero-shot benchmark that evaluates action controllability, physical consistency, and generalization across unseen robot-task-scene combinations.
- The model achieves new state-of-the-art results on PBench and EZSbench, surpassing Veo 3.1 and Sora v2 Pro in physical plausibility and action trajectory consistency.
- The data pipeline improves embodied-video diversity and balance through curated sampling and physics-aware annotation.
- ABot-PhysWorld jointly optimizes visual realism, physical plausibility, and action controllability through physics-aware DPO and parallel spatial action injection.
2 Data Curation
The data curation system builds a diverse, physically grounded training resource from nearly three million real-world clips. It combines embodied-specific filtering, hierarchical balancing, and multi-level annotation to support dynamics-aware world-model training.
- 2 Data Curation: The curation pipeline has three stages: embodied-specific filtering, hierarchical distribution balancing, and physically grounded caption generation.
- 2 Data Curation: The dataset contains nearly three million real-world clips integrated from five public datasets for physically consistent embodied-manipulation modeling.
- 2.1 Embodied-Specific Data Filtering: Filtering removes unsuitable videos using quality gates, motion analysis, temporal coherence checks, and vision-action alignment verification.These checks discard abnormal or irrelevant clips, near-zero or unphysical motion, visual corruption, and mismatches between observed motion and control signals.
- 2.2 Hierarchical Distribution Balancing: Hierarchical sampling balances video, robot type, task, and macro-dataset levels while preserving rare interaction patterns and long-tail tasks.Head tasks are capped at 8–15%, body tasks are sampled at 40–50%, and long-tail tasks are fully preserved.
- 2.3 Physics-Aware Video Captioning: The annotation system captures task intent, action structure, trajectories, gripper states, physical relations, outcomes, and causal phenomena such as gravity-induced dropping and force feedback.Its four-stage narrative structure organizes scene construction, action flow, final-state confirmation, and camera summary.
- 2.3 Physics-Aware Video Captioning: The resulting language supervision captures events and their underlying causes, providing a semantic foundation for causal understanding in world models.
3 Method
ABot-PhysWorld combines physics-aware preference alignment with spatial action injection to generate controllable manipulation videos while preserving pretrained physical knowledge.
- Physics Alignment: ABot-PhysWorld builds on Wan2.1-I2V-14B and uses Diffusion-DPO to suppress physically violating predictions that supervised fine-tuning cannot distinguish from correct ones.The preference pipeline uses a decoupled VLM discriminator to identify compliant and violating video variants.
- Physics Alignment: The discriminator separates checklist generation from scoring, using physical criteria to select optimal and worst candidates for preference training.Qwen3-VL proposes task-specific checklists, while Gemini 3 Pro scores candidates using tournament-based sampling.
- Physics Alignment: The physical preference objective reduces denoising error for physics-compliant latents while increasing it for physics-violating latents relative to the reference model.LoRA adapters allow reference-loss computation without maintaining a second complete 14B computation graph.
- Action Injection: The model converts robotic commands into spatial action maps encoding position, orientation, depth, and gripper openness before injecting them through parallel context blocks.The representation supports 7D single-arm actions and 14D dual-arm actions with red and blue arm channels.
- Action Injection: Selective context blocks process action maps and add projected outputs residually to corresponding DiT blocks, with zero initialization preserving pretrained physical priors.Context blocks replicate every fifth DiT block and gradually learn action controllability without initially disturbing the backbone.
4 Embodied-ZeroShot Benchmark
EZSbench evaluates embodied video generation under unseen robot-task-scene combinations, combining synthetic and real imagery with physics-grounded descriptions and decoupled physical evaluation.
- Benchmark Design: EZSbench tests zero-shot generalization by composing previously unseen combinations of robot morphologies, environments, and tasks without training overlap.The benchmark targets fully out-of-distribution physical fidelity and cross-embodiment generalization.
- Benchmark Design: Its initial observation pool uses synthetic image generation and real-image scene editing to vary robots, scenes, tasks, perspectives, and backgrounds.Synthetic scenarios target morphological, scene, and task generalization, while real-image editing preserves foreground mechanical-arm content.
- Benchmark Construction: Each benchmark sample pairs an initial image with a dense physics-grounded description covering the initial state, action trajectory, and final state.The description framework combines visual anchoring, kinematically compliant action simulation, and narrative synthesis.
- Evaluation Protocol: The evaluation uses a decoupled dual-model protocol that generates task-specific physical checklists and scores videos against checklist ground truth.The physical score is the fraction of checklist questions whose VQA predictions match the ground truth.
5 Experiments
Experiments evaluate text- and action-conditioned generation using physical, visual, and trajectory metrics against multiple baselines. ABot-PhysWorld achieves strong benchmark performance, preserving visual quality while improving physical plausibility and action fidelity.
- Evaluation setup: Experiments use PAI-Bench/PBench for text-conditioned generation and PSNR, SSIM, and nDTW for action-conditioned evaluation.PBench assesses spatial, temporal, and physical dimensions; nDTW compares detected gripper trajectories with ground truth.
- PBench evaluation: 0.8491 average score and 0.9306 Domain Score make the DPO-augmented model best on PBench.It outperforms the base model’s Domain Score of 0.8785 and all baselines.
- PBench evaluation: 0.7676 Quality Score shows competitive visual quality while the model enforces physical constraints.Veo 3.1 and Sora v2 Pro achieve higher Quality Scores of 0.7740 and 0.7679 but lower Domain Scores of 0.8350 and 0.7626.
- EZSbench evaluation: 0.8030 overall average, 0.7694 Quality Score, and 0.8366 Domain Score establish state-of-the-art performance on out-of-distribution EZSbench.The reported results indicate that physical-fidelity improvements generalize beyond the training distribution.
- Qualitative analysis: Qualitative comparisons show fewer target-identification, grasping-penetration, deformation, and geometric-distortion errors than competing models.The reported comparisons include Sora v2 Pro, Veo 3.1, GigaWorld-0, Cosmos, WoW, UnifoLM, and Wan 2.5.
- Action-conditioned results: The method outperforms action-conditioned baselines in both visual quality and action fidelity.Table 3 reports consistent advantages over the compared methods.
6 Conclusion
The conclusion presents ABot-PhysWorld as a physically grounded, action-controllable embodied world model and introduces EZSbench for zero-shot evaluation. It reports improved physical fidelity and trajectory consistency, while identifying fixed viewpoints and absent closed-loop evaluation as current boundaries.
- Conclusion: ABot-PhysWorld combines a 14B Diffusion Transformer with curated data, Diffusion-DPO, and spatial action injection.The design targets reduced physical violations and control across different embodiments.
- Conclusion: EZSbench evaluates out-of-distribution scenarios with a decoupled protocol for physical fidelity and action alignment.The benchmark is presented as a zero-shot evaluation resource.
- Conclusion: The model reports state-of-the-art physical fidelity and improved trajectory consistency compared with Veo 3.1 and Sora v2 Pro.These results are stated as the conclusion’s main experimental outcome.
- Limitations: The current system relies on fixed-viewpoint data and lacks closed-loop evaluation.Future work is described as exploring multi-view generation and real-world deployment.
7 Contributions
The contribution record assigns work across data curation, model training, evaluation, writing, project leadership, and advising. The supplied passage identifies contributors for each area.
- Author contributions: Data curation is credited to Yuzhi Chen, Ronghan Chen, Dongjie Huo, Haoyun Liu, Yandan Yang, Dekang Qi, Tong Lin, Shuang Zeng, and Junjin Xiao.
- Author contributions: Model training is credited to Yuzhi Chen and Ronghan Chen, while evaluation is credited to Yuzhi Chen, Ronghan Chen, and Dongjie Huo.
- Author contributions: Writing is credited to Yuzhi Chen, Yandan Yang, Ronghan Chen, Dongjie Huo, and Dekang Qi.
- Author contributions: Project leadership is credited to Xinyuan Chang and Feng Xiong, with Zhiheng Ma, Xing Wei, and Mu Xu serving as advisors.
Appendix
The appendix provides a code repository and lists three supplementary sections covering alignment verification, physics-aware captioning, and additional qualitative results.
- Appendix resources: The code repository is available at https://github.com/amap-cvlab/ABot-PhysWorld.
- Appendix contents: Supplementary material includes Vision-Action Alignment Verification, a Two-Stage Physics-Aware Captioning Pipeline, and Additional Qualitative Results.
A. Vision-Action Alignment Verification
The method verifies vision-action alignment by projecting calibrated robot signals onto video frames and removing samples with spatial disagreement between trajectories and observed motion.
- Misaligned action-video pairs can arise from calibration drift, clock errors, or coordinate-frame inconsistencies and introduce spurious correlations into world-model training.
- Calibrated joint positions, end-effector poses, and gripper states are rendered as semi-transparent, color-coded action maps over corresponding video frames.
- Samples with spatial deviation between projected trajectories and observed robot motion are identified and removed.
B. Two-Stage Physics-Aware Captioning Pipeline
The physics-aware captioning pipeline first extracts structured scene and interaction attributes, then converts them into four-phase narratives covering manipulation dynamics and state changes.
- Stage 1: Structured Perception and Attribute Extraction: Stage 1 uses a vision-language module to extract robot morphology, object properties, spatial layout, contact events, and state transitions.Its structured representation captures the scene’s “what” and “where” for the subsequent writing stage.
- Stage 1: Structured Perception and Attribute Extraction: Stage 1 examples show extracted physical attributes, object identities, spatial relations, robot morphology, object properties, contact events, and state transitions.
- Stage 2: Physics-Grounded Narrative Synthesis: Stage 2 converts the structured perception output into a four-phase caption covering scene setup, action detail, state transition, and camera summary.The narrative captures fine-grained manipulation dynamics and causal state transitions.
C. Additional Qualitative Results
Across zero-shot EZSbench and action-conditioned evaluations, ABot-PhysWorld handles compositional and contact-intensive manipulation while preserving physical plausibility, geometry, and spatiotemporal coherence better than the compared baselines.
- Zero-Shot Qualitative Comparison on EZSbench: Baseline models make placement errors, contactless grasps, or geometric collapse on long-horizon EZSbench tasks, whereas ABot-PhysWorld follows compositional instructions coherently.The comparison covers Wan-2.5, Veo 3.1, WoW, Sora v2, and Giga R0.
- Case Study on Zero-Shot Test Set: ABot-PhysWorld correctly binds object attributes to target containers and handles deformable, articulated, rigid-body, and multiobject manipulation tasks in unseen settings.The case study includes dual-arm towel folding, object relocation, placement, and contact-intensive interactions.
- Action-to-Video Qualitative Comparison: In action-to-video generation, ABot-PhysWorld preserves object geometry and visual integrity during contact-intensive manipulation.Genie-Envisioner and Enerverse-AC instead show deformation, contactless grasping, and localization errors that can result in task failure.