Source-linked AI summary
PhysX-Anything: Simulation-Ready Physical 3D Assets from Single Image
Ziang Cao, Fangzhou Hong, Zhaoxi Chen, Liang Pan, Ziwei Liu
TL;DR
Existing 3D generation methods often omit the physical and articulation information needed for simulation and embodied AI. PhysX-Anything addresses this gap by generating explicit geometry, articulation, and physical attributes from a single in-the-wild image, with strong reported performance and direct use in robotic simulation.
Problem
Existing 3D generation methods often lack essential physical and articulation information, limiting their direct use in simulation and embodied AI.
Method
PhysX-Anything uses a unified VLM-based pipeline and compressed voxel representation to generate geometry, articulation, physical properties, and simulation-ready assets from one image.
Results
PhysX-Anything achieves strong generative quality and robust generalization, with absolute-scale error reduced from 43.44 to 0.30 compared with PhysXGen.
Takeaways & Limitations
The generated assets can be directly imported into simulators and used for contact-rich robotic policy learning.
Abstract
from arXiv · showhide
3D modeling is shifting from static visual representations toward physical, articulated assets that can be directly used in simulation and interaction. However, most existing 3D generation methods overlook key physical and articulation properties, thereby limiting their utility in embodied AI. To bridge this gap, we introduce PhysX-Anything, the first simulation-ready physical 3D generative framework that, given a single in-the-wild image, produces high-quality sim-ready 3D assets with explicit geometry, articulation, and physical attributes. Specifically, we propose the first VLM-based physical 3D generative model, along with a new 3D representation that efficiently tokenizes geometry. It reduces the number of tokens by 193x, enabling explicit geometry learning within standard VLM token budgets without introducing any special tokens during fine-tuning and significantly improving generative quality. In addition, to overcome the limited diversity of existing physical 3D datasets, we construct a new dataset, PhysX-Mobility, which expands the object categories in prior physical 3D datasets by over 2x and includes more than 2K common real-world objects with rich physical annotations. Extensive experiments on PhysX-Mobility and in-the-wild images demonstrate that PhysX-Anything delivers strong generative performance and robust generalization. Furthermore, simulation-based experiments in a MuJoCo-style environment validate that our sim-ready assets can be directly used for contact-rich robotic policy learning. We believe PhysX-Anything can substantially empower a broad range of downstream applications, especially in embodied AI and physics-based simulation.
1. Introduction
PhysX-Anything addresses the lack of physical and articulation information in existing 3D generation by producing simulation-ready assets from single in-the-wild images. It combines a unified VLM pipeline, efficient geometry representation, and the PhysX-Mobility dataset to support generation, generalization, and simulation.
- Existing 3D generators often omit density, scale, joint constraints, or other physical information, making their assets difficult to deploy directly in simulators.
- PhysX-Anything jointly predicts geometry, articulation structure, and essential physical properties from a single in-the-wild image.
- Its geometry representation reduces token requirements by 193×, enabling explicit geometry learning within VLM budgets without special tokens or a new tokenizer.
- PhysX-Mobility broadens physical 3D data coverage across 47 categories by collecting and annotating common real-world objects.
- Experiments report strong generative quality and robust generalization, while MuJoCo-style tests show direct use in contact-rich robotic policy learning.
2. Related Works
Prior 3D generation methods trade off efficiency, structural detail, articulation, physical modeling, and generalization. PhysX-Anything instead uses a compressed explicit representation and targets jointly articulated, physically grounded, simulation-ready generation.
- 2.1. 3D Generative Models: Existing 3D generation spans GANs, optimization-based diffusion, feed-forward, autoregressive, and VLM-based approaches, each addressing efficiency or quality with different trade-offs.
- 2.2. Articulated and Physical 3D Object Generation: PhysX-Anything is presented as the only compared approach supporting articulation, physical modeling, strong generalization, and simulation-ready deployment simultaneously.
- 2.1. 3D Generative Models: Unlike representations that require additional special tokens, its compressed representation preserves explicit structural information without introducing a new tokenizer during fine-tuning.
- 2.1. 3D Generative Models: PhysX-Anything uses a voxel-based representation and specialized merging strategy to reduce token count by 193× relative to the original mesh format.
- 2.2. Articulated and Physical 3D Object Generation: Many articulated-object methods retrieve existing meshes or use graph-structured representations, limiting their ability to generate novel, physically grounded assets.
3. Methodology
PhysX-Anything uses a global-to-local VLM pipeline and a compact coarse-to-fine voxel representation to generate physical 3D assets. A controllable flow transformer refines geometry before format decoding produces simulator-ready outputs.
- Pipeline: A global-to-local pipeline generates overall physical structure first, then part-level geometry while retaining global information to reduce context forgetting.The system uses multi-round dialogue and preserves only overall information during per-part geometry generation.
- 3D Representation: The voxel-based representation reduces token length while preserving explicit geometry, avoiding an additional tokenizer through coarse-to-fine modeling.High-resolution geometry is represented through coarse voxels followed by fine-grained refinement.
- Geometry Refinement: A controllable flow transformer converts coarse voxel geometry into fine-grained geometric information for downstream asset reconstruction.Its conditioning includes the coarse voxel representation and image information.
- Physical Representation: Tree-structured physical descriptions encode articulation parameters such as motion direction, axis location, and motion range alongside geometry.The JSON-style format is designed to support VLM understanding and reasoning about physical assets.
- Asset Decoding: The decoder combines global physical information with refined geometry to produce meshes, URDF, XML, and other simulator-oriented formats.A structured latent diffusion model generates 3D assets, which are segmented into part-level components before final formatting.
4. Experiments
Experiments show that PhysX-Anything outperforms competing methods on physical 3D generation, generalizes to in-the-wild images, and supports contact-rich robotic policy learning in simulation.
- 4.1. Evaluation on PhysX-Mobility: PhysX-Anything consistently outperforms all compared methods across geometric and physical metrics on PhysX-Mobility, with especially large gains on physical properties.Absolute-scale error decreases from 43.44 to 0.30 compared with PhysXGen, a reported improvement of over 99%.
- 4.1. Evaluation on PhysX-Mobility: Qualitative PhysX-Mobility results show more faithful geometry, articulation, and physical attributes than competing methods.The comparison particularly highlights stronger generalization than retrieval-based approaches and more plausible physical attributes than PhysXGen.
- 4.2. In-the-Wild Evaluation: In-the-wild VLM evaluation reports substantially higher geometry and kinematic-parameter scores than competing methods, indicating stronger real-life generalization.The results are reported in Table 4 and concern geometry and kinematic parameters.
- 4.2. In-the-Wild Evaluation: User studies based on 1,568 scores from 14 volunteers find stronger human preference for PhysX-Anything in geometry quality and physical plausibility.The study uses a 0 to 5 rating scale and reports a clear margin over other methods.
- 4.3. Representation Ablation: Representation ablations show that higher token compression can preserve complete, detailed geometry for complex structures, unlike alternatives constrained by token budgets.Original mesh and vertex-quantization representations require too many tokens for feasible end-to-end training.
- 4.4. Robotic Policy Learning in Simulation: Generated assets can be directly imported into a MuJoCo-style simulator and used for contact-rich robotics policy learning across everyday objects.The experiments report physically plausible behavior and accurate geometric structure across diverse tasks.
5. Conclusion
PhysX-Anything combines a unified VLM pipeline with a compressed physical 3D representation to generate sim-ready assets from images. The framework reports over 193× token compression and broad real-world category coverage.
- 5. Conclusion: The representation achieves over 193× token compression while preserving explicit geometric structure for efficient physical 3D generation.PhysX-Mobility contains 47 common real-life categories with detailed physical attributes.