PaperlayerView exact revision on arXiv

Source-linked AI summary

PhysX-Omni: Unified Simulation-Ready Physical 3D Generation for Rigid, Deformable, and Articulated Objects

Ziang Cao, Yinghao Liu, Haitian Li, Runmao Yao, Fangzhou Hong, Zhaoxi Chen, Liang Pan, Ziwei Liu

arXiv:2605.21572v1cs.CVcs.RO

TL;DR

Existing 3D generators often omit physical attributes or support only one asset type, limiting simulation-ready use. PhysX-Omni unifies generation across rigid, deformable, and articulated objects with new data and evaluation resources, and achieves strong generation and understanding performance.

  • Problem

    Existing 3D generation methods often omit essential physical attributes or model only limited properties for specific asset types, while datasets and benchmarks remain scarce.

  • Method

    PhysX-Omni unifies simulation-ready generation across rigid, deformable, and articulated objects using a VLM-tailored geometry representation, PhysXVerse, and PhysX-Bench.

  • Results

    PhysX-Omni achieves strong generative quality, robust generalization, and performance across generation and understanding evaluations on PhysX-Bench and conventional benchmarks.

  • Takeaways & Limitations

    The framework’s simulation-ready assets support evaluation in standard simulation environments and downstream embodied AI and robotic manipulation applications.

Abstract

from arXiv · show

Simulation-ready physical 3D assets have emerged as a promising direction owing to their broad applicability in downstream tasks. However, most existing 3D generation methods either neglect physical properties or are limited to a single asset category, e.g., rigid, deformable, or articulated objects. To address these limitations, we introduce PhysX-Omni, a unified framework for simulation-ready physical 3D generation across diverse asset types. Specifically, we develop a novel and efficient geometry representation tailored for Vision-Language Models, which directly encodes high-resolution 3D structures without compression, significantly improving generation performance. In addition, we construct the first general simulation-ready 3D dataset, PhysXVerse, covering diverse indoor and outdoor categories. Furthermore, to comprehensively and flexibly evaluate both generative and understanding capabilities in the wild, we propose PhysX-Bench, which encompasses six key attributes: geometry, absolute scale, material, affordance, kinematics, and function description. Extensive experiments with conventional metrics and PhysX-Bench show that PhysX-Omni performs strongly in both generation and understanding. Moreover, additional studies further validate the potential of PhysX-Omni for applications in simulation-ready scene generation and robotic policy learning. We believe PhysX-Omni can significantly advance a wide range of downstream applications, particularly in embodied AI and physics-based simulation.

1 S-Lab, Nanyang Technological University, 2 ACE Robotics

PhysX-Omni exploits PhysXVerse’s diversity to generate detailed, general, simulation-ready 3D assets across rigid, deformable, and articulated objects for downstream applications.

  • PhysX-Omni exploits PhysXVerse’s high diversity to generate detailed and general 3D assets.The generated assets span multiple physical object categories.
  • The framework covers rigid, deformable, and articulated objects.
  • The generated assets are simulation-ready and suitable for downstream applications.

1 Introduction

PhysX-Omni addresses the lack of comprehensive physical attributes and category coverage in 3D generation with a unified framework for rigid, deformable, and articulated assets. It introduces a VLM-tailored geometry representation, the PhysXVerse dataset, and the six-attribute PhysX-Bench evaluation framework.

  • Motivation: Existing 3D generation methods prioritize photorealistic appearance and detailed geometry but often omit physical attributes needed for real-world deployment.These assets nonetheless target applications including gaming design, robotics, embodied AI, and interactive simulation.
  • Problem: Articulated and deformable-object methods model limited physical attributes for specific asset types, while pioneering physical 3D methods remain constrained by data scarcity.
  • Framework: PhysX-Omni unifies simulation-ready physical 3D generation across rigid, deformable, and articulated assets using a VLM-tailored representation that directly models high-resolution 3D structures without extra special tokens.The representation is designed to improve generation performance and generalization.
  • Dataset: PhysXVerse is the first general simulation-ready physical 3D dataset, containing over 8K assets across more than 2K indoor and outdoor categories with physical attribute annotations.Examples include helicopters, tanks, racing cars, skyscrapers, and toys.
  • Benchmark and Results: PhysX-Bench is the first benchmark for simulation-ready physical 3D generation, integrating physics-based simulation and VLMs to evaluate real-world performance across six key attributes.The paper reports strong generative quality and robust generalization on PhysX-Bench and conventional benchmarks, alongside deployability in standard simulation environments.

2 Related Works

Prior 3D generation progressed from GAN-based methods to diffusion-model approaches, while simulation-ready generation remains constrained by segmentation bottlenecks and limited retrieval coverage. PhysX-Omni addresses these limitations with an efficient geometry representation, a unified framework, and a general simulation-ready dataset.

  • General 3D Generation: GANs established early 3D generation, but their instability and limited robustness hinder scaling to complex, diverse data distributions.
  • General 3D Generation: DreamFusion shifted the field toward score distillation sampling using priors from pretrained 2D diffusion models.
  • Simulation-Ready Generation: PhysX-Anything uses pure text representations and VLM priors, but explicit segmentation creates a bottleneck that constrains overall quality.
  • Simulation-Ready Generation: PhysX-Omni introduces direct high-resolution 3D geometry modeling in a unified framework, improving generation performance over the baseline.It also provides the first general high-quality simulation-ready 3D dataset and supports complex topologies and accurate physical attributes.
  • Articulated Object Generation: Retrieval-based articulated generation assembles meshes from predefined libraries, limiting applicability to the coverage of known source categories.

3 Methodology

PhysX-Omni uses a coarse-to-fine VLM pipeline with hierarchical global-to-local reasoning and a template-based RLE representation for high-resolution physical 3D assets. Its methodology also introduces PhysXVerse and PhysX-Bench for broad simulation-ready data construction and multimodal evaluation.

  • Generation Paradigm: PhysX-Omni first infers object category, semantic identity, absolute scale, component hierarchy, and physical properties before generating detailed part-level geometry and attributes.The pipeline uses holistic global understanding to provide semantic priors and consistency for subsequent local generation.
  • Geometry Representation: The proposed template-based RLE directly encodes part-level voxel geometry through z-axis slices, compact 2D masks, shared templates, and residual differences.Template reuse exploits spatial redundancy across neighboring slices while preserving explicit structural information.
  • Geometry Representation: Template-based RLE achieves stronger compression efficiency and geometric fidelity than conventional 2D RLE and existing text-based explicit representations, producing more detailed structures and better physical alignment than text-based voxel indices.The comparison is reported qualitatively in Figure 3a.
  • PhysXVerse: PhysXVerse contains more than 8.7K simulation-ready 3D assets spanning over 2.9K categories, with object types ranging from furniture and vehicles to robots and large-scale scene components.Its part counts range from 1 to 65, alongside richer category diversity and more comprehensive physical annotations than existing datasets.
  • PhysX-Bench: PhysX-Bench evaluates six dimensions using rendered images or videos and an open-source VLM, while free-fall and water-drop simulations visually assess material properties.Free-fall behavior reflects Young’s modulus and Poisson’s ratio, whereas water-drop behavior primarily evaluates density.

4 Experiments · 4.1 Implementation details · 4.2 Datasets

The experiments evaluate PhysX-Omni with conventional metrics, PhysX-Bench, human alignment, ablations, and downstream applications. Implementation uses a Qwen2.5-VL-7B-Instruct backbone, while training combines over 42K simulation-ready assets spanning diverse object types and viewpoints.

  • 4 Experiments: The experiments cover conventional metrics, PhysX-Bench, human alignment, ablations, and downstream simulation-ready scene generation and robotic policy learning.These evaluations analyze framework components and demonstrate potential applications beyond generation.
  • 4.1 Implementation details: PhysX-Omni adopts Alibaba Cloud Qwen2.5-VL-7B-Instruct as its vision-language backbone.The model is trained for 5 epochs on 64 NVIDIA A100 GPUs over approximately 14 days.
  • 4.1 Implementation details: Training uses a peak learning rate of 2×10−5, cosine decay, a 0.03 warmup ratio, and an effective batch size of 128.These settings are reported for the five-epoch training configuration.
  • 4.1 Implementation details: The maximum sequence length is 16,384 tokens to support high-resolution structures and long-context physical descriptions.This setting is part of the model configuration for simulation-ready generation and physical description processing.
  • 4.2 Datasets: Training combines assets from PhysXNet, PhysX-Mobility, and PhysXVerse into a corpus containing more than 42K simulation-ready physical 3D assets.The combined corpus supports training across the datasets named in the implementation description.
  • 4.2 Datasets: Qualitative results show PhysX-Omni generating complex geometries and rich physical attributes compared with existing generative methods.The comparison is presented in Figure 6.
  • 4.2 Datasets: PhysXVerse spans diverse indoor and outdoor categories and includes rigid, articulated, and deformable objects with rich geometric structures and physical attributes.The dataset is described as covering multiple physical asset types and environments.
  • 4.2 Datasets: The training pipeline renders 25 images per object from different viewpoints to improve view consistency and visual-understanding robustness.These multi-view images are used as conditioning inputs during training.

4.3 Conventional evaluation metrics · 4.4 Evaluations with conventional metrics

The evaluation protocol measures geometry, physical attributes, and articulation properties using conventional metrics across PhysXVerse and PhysX-Mobility. PhysX-Omni achieves leading results in geometry, scale, material, affordance, and description evaluations, demonstrating strong simulation-ready generation performance.

  • 4.3 Conventional evaluation metrics: Geometry evaluation uses PSNR for rendered appearance and Chamfer Distance (CD) with F-score for reconstructed 3D geometry accuracy.The protocol combines conventional geometric metrics with physical attribute metrics to assess visual fidelity, structural quality, and physical correctness.
  • 4.3 Conventional evaluation metrics: Kinematic evaluation measures MSE for joint axis positions, joint directions, joint types, and motion limits to assess physically plausible articulated behavior.This evaluation targets capabilities required for downstream simulation and robotic interaction.
  • 4.4 Evaluations with conventional metrics: PhysX-Omni is compared with PhysXGen, Articulate-Anything, MonoArt, and PhysX-Anything on PhysXVerse and PhysX-Mobility using conventional geometric and physical-attribute evaluations.The experiments follow the evaluation protocols of PhysXGen and MonoArt.
  • 4.4 Evaluations with conventional metrics: On PhysXVerse, PhysX-Omni achieves PSNR 21.52, CD 2.95, and F-score 91.28, substantially surpassing previous best geometric results.The method consistently achieves the best performance across nearly all evaluation metrics on both datasets.
  • 4.4 Evaluations with conventional metrics: Absolute scale error on PhysXVerse decreases from 309.31 in PhysXGen and 298.19 in PhysX-Anything to 2.79 in PhysX-Omni.On PhysX-Mobility, the error decreases to 2.78.
  • 4.4 Evaluations with conventional metrics: On PhysXVerse, PhysX-Omni improves the material score from 15.65 to 27.23 and consistently achieves the best material, affordance, and description results across both datasets.The reported gains extend beyond geometric generation to physical attribute prediction.
  • 4.4 Evaluations with conventional metrics: The quantitative results show superior geometry and physical reasoning, with higher visual fidelity, stronger physical consistency, and more accurate articulation modeling.The paper attributes these results to the VLM-based global-to-local framework and high-resolution geometry representation.

4.5 Evaluations on PhysX-Bench

PhysX-Omni demonstrates strong generalization on PhysX-Bench, outperforming existing approaches across most physical attributes and achieving particularly strong kinematic performance. Its explicit high-resolution 3D representation improves structural coherence, articulation consistency, and the physical plausibility of generated simulation-ready assets.

  • Benchmark evaluation: PhysX-Bench evaluates methods on real photographs and rendered images spanning common object categories and challenging real-world scenarios.The benchmark assesses generalization ability using diverse conditioning images; construction and metrics are described in the supplementary material.
  • Quantitative results: 80.72 kinematic score, outperforming PhysX-Anything (65.99), PhysXGen (69.17), MonoArt (68.32), and Articulate-Anything (71.25).PhysX-Omni achieves the best results on most physical attributes, including absolute scale, material, affordance, kinematics, and description.
  • Representation analysis: PhysX-Omni’s explicit high-resolution 3D structures model interdependencies among geometry, articulation, and physical attributes more coherently than segmentation-based representations.This representation improves structural coherence and articulation consistency, especially for complex articulated assets.
  • Representation analysis: The explicit geometry representation preserves detailed structures while avoiding segmentation-induced ambiguities and artifacts, producing more coherent motions and reliable physical attributes.These improvements substantially enhance the generation of physically plausible simulation-ready assets.
  • Qualitative results: Qualitative comparisons show robust generation of complex structures and articulated objects, with more accurate geometry, plausible material and affordance predictions, and coherent articulated behaviors than PhysX-Anything.The comparison is presented through visualized results in Fig. 6.

4.6 Validating human alignment of PhysX-Bench

PhysX-Bench’s automatic evaluation scores are compared with human preference scores using Spearman’s rank correlation coefficient. Strong correlations across all evaluation dimensions support the benchmark’s reliability and its alignment with human perception and judgment.

  • Alignment evaluation: The study measures alignment between PhysX-Bench’s automatic scores and human preference scores using Spearman’s rank correlation coefficient.Higher Spearman correlation indicates stronger consistency between benchmark evaluations and human preferences.
  • Alignment evaluation: Strong correlations across all evaluation dimensions validate PhysX-Bench’s reliability, robustness, and effectiveness.
  • Alignment evaluation: PhysX-Bench provides automatic evaluation results that closely match human perception and judgment.The findings support its use as a trustworthy framework for evaluating simulation-ready physical 3D generation.

4.7 Ablation Studies

The ablation studies show that PhysX-Omni’s template-based geometry representation substantially outperforms text-based voxel indices on conventional metrics and PhysX-Bench. Qualitative results further show more detailed, structurally coherent assets, especially for articulated objects and strong part interactions.

  • Geometry Representation: Template-based geometry representation substantially improves PhysX-Omni’s performance over text-based voxel indices on conventional metrics and PhysX-Bench.The strongest gains occur on kinematic and absolute-scale evaluation.
  • Qualitative Comparison: PhysX-Omni produces more detailed and structurally coherent simulation-ready assets than PhysX-Anything, which relies on text-based voxel indices and additional segmentation.The baseline frequently exhibits structural ambiguities, incomplete local geometry, and inconsistencies in highlighted regions.
  • Articulated Objects: The representation’s benefits are particularly evident for articulated objects and strong part interactions, improving structural reasoning and motion consistency.Explicit structural encoding helps capture intrinsic relationships between geometry and kinematics, enhancing robustness, fidelity, and generalization.

4.8 Application: Robotic Policy Learning in Simulation · 4.9 Application: Sim-Ready Scene Generation

PhysX-Omni’s generated simulation-ready assets are deployed in physics simulation for robotic interaction and policy learning, while its assets and layouts are extended toward scene-level simulation-ready generation. These applications support physically grounded simulation and suggest broader uses in embodied AI, robotic learning, and world generation.

  • 4.8 Application: Robotic Policy Learning in Simulation: Generated simulation-ready 3D assets are directly deployed in a physics simulator for robotic interaction and policy learning.The assets are imported for use in real simulation environments and downstream robotic tasks.
  • 4.8 Application: Robotic Policy Learning in Simulation: The simulator receives generated assets together with their geometric structures, physical properties, and articulated parameters.This enables physically grounded interaction within the simulator.
  • 4.8 Application: Robotic Policy Learning in Simulation: The applications connect simulation-ready generation with embodied AI training and robotic policy learning.These downstream uses are explicitly identified as promising application areas.
  • 4.9 Application: Sim-Ready Scene Generation: The scene-generation pipeline uses image-to-depth estimation and 2D segmentation to reconstruct an initial 3D scene layout from input images.Estimated depth, segmentation masks, and scene geometry provide the basis for coarse object placements and spatial relationships.
  • 4.9 Application: Sim-Ready Scene Generation: The reconstructed 3D layout is integrated with PhysX-Omni’s simulation-ready assets to explore scene-level simulation-ready generation.The approach begins from coarse scene structure inferred from images before incorporating simulation-ready content.
  • 4.9 Application: Sim-Ready Scene Generation: Together, the studies indicate potential for future physically grounded world generation applications.This direction follows from combining simulation-ready assets with reconstructed scene layouts.
  • 4.9 Application: Sim-Ready Scene Generation: PhysX-Omni supports high-quality simulation-ready asset generation in addition to its scene-level simulation construction potential.The reported results extend the framework’s application scope beyond individual assets.
  • 4.9 Application: Sim-Ready Scene Generation: The framework provides a promising foundation for scalable scene-level simulation construction and physically grounded world generation.The passage identifies these as future application directions supported by the demonstrated capabilities.

5 Conclusion

PhysX-Omni is a unified framework for simulation-ready physical 3D generation across rigid, deformable, and articulated objects. Its tailored geometry representation directly models detailed 3D structures without special tokens or segmentation modules, improving generation quality and robustness.

  • Unified framework: PhysX-Omni unifies simulation-ready physical 3D generation across rigid, deformable, and articulated objects.The framework is designed to support diverse asset types within one generation approach.
  • Geometry representation: Its tailored geometry representation for vision–language models directly models detailed 3D structures.The representation is designed specifically for vision–language models and preserves detailed 3D structure.
  • Generation design: The representation avoids additional special tokens and segmentation modules, improving generation quality and robustness.These design choices eliminate reliance on both mechanisms while supporting the reported improvements.
Loading 2605.21572v1…