Source-linked AI summary
PhysForge: Generating Physics-Grounded 3D Assets for Interactive Virtual World
Yunhan Yang, Chunshi Wang, Junliang Ye, Yang Li, Zanxin Chen, Zehuan Huang, Yao Mu, Zhuo Chen, Chunchao Guo, Xihui Liu
TL;DR
Existing 3D generation methods largely produce static geometry while overlooking physics needed for interaction. PhysForge plans hierarchical physical blueprints with a VLM, then realizes them through diffusion with KineVoxel Injection, producing functionally plausible, simulation-ready assets.
Problem
Existing 3D generation methods largely overlook physics information needed to make assets interactive and deployable in embodied-AI simulators or virtual environments.
Method
PhysForge decouples VLM-based hierarchical physical planning from diffusion-based realization, supported by PhysDB’s 150,000 assets with four-tier physical annotations.
Results
Experiments validate PhysForge’s performance on planning and generation and demonstrate direct applicability of its assets in robotic simulators and interactive virtual worlds.
Takeaways & Limitations
PhysForge provides a foundational data engine for embodied AI and interactive virtual worlds.
Abstract
from arXiv · showhide
Synthesizing physics-grounded 3D assets is a critical bottleneck for interactive virtual worlds and embodied AI. Existing methods predominantly focus on static geometry, overlooking the functional properties essential for interaction. We propose that interactive asset generation must be rooted in functional logic and hierarchical physics. To bridge this gap, we introduce PhysForge, a decoupled two-stage framework supported by PhysDB, a large-scale dataset of 150,000 assets with four-tier physical annotations. First, a VLM acts as a "physical architect" to plan a "Hierarchical Physical Blueprint" defining material, functional, and kinematic constraints. Second, a physics-grounded diffusion model realizes this blueprint by synthesizing high-fidelity geometry alongside precise kinematic parameters via a novel KineVoxel Injection (KVI) mechanism. Experiments demonstrate that PhysForge produces functionally plausible, simulation-ready assets, providing a robust data engine for interactive 3D content and embodied agents.
1. Introduction
PhysForge addresses the gap between high-fidelity 3D geometry generation and the functional, hierarchical physics required for interactive assets. It combines VLM-based physical planning, diffusion-based realization with KineVoxel Injection, and the 150k-asset PhysDB dataset to generate functionally complete, physically interactive 3D assets.
- Motivation: Existing 3D generation methods primarily produce appearance and geometric detail, leaving functional properties needed for interaction insufficiently addressed.The paper frames this as a critical gap for embodied AI and virtual game environments seeking large-scale, high-quality 3D content.
- Motivation: PhysForge shifts asset generation from holistic shape synthesis toward functional logic and hierarchical physics, treating structure as a manifestation of intended physical interaction.Examples include television buttons and cabinet doors and handles with distinct materials, functions, and kinematic definitions.
- Framework: PhysForge decouples physical planning from realization through a VLM-based planning stage and a diffusion-based generation stage.The VLM provides physical planning and world knowledge, while diffusion models synthesize geometry and precise kinematic parameters.
- Framework: KineVoxel Injection encodes articulation parameters such as origin, axis, and limit into kinematic voxels jointly generated with geometry-representing voxels.This mechanism is introduced in the diffusion-based generation stage to forge high-fidelity geometry alongside planned kinematic parameters.
- Dataset: 150k assets comprise PhysDB, whose four-tier annotations capture holistic, static, functional, and interactive physical properties.The tiers include scale and scene, part semantics and materials, intrinsic functions and state machines, plus joint types and atomic affordances.
- Validation and Applications: PhysForge generates functionally complete, physically interactive assets from a single-view image, with experiments and demonstrations validating applications in robotic simulators and interactive virtual worlds.The stated downstream applications include robotic manipulation and game development.
2. Related Work
Related work spans 2D-prior and 3D-native generation, with newer efforts adding part-aware control and beginning to incorporate interactive physics. However, existing approaches retain limitations in view consistency, geometric fidelity, or comprehensive physical grounding.
- 3D Content Generation: 3D content generation follows two main philosophies: optimizing 3D representations with 2D priors or training directly on large-scale 3D data.SDS, pioneered by DreamFusion, enables text-to-3D synthesis without 3D supervision, while 3D-native methods learn shape distributions directly from 3D datasets.
- 2D-Prior Generation: 2D-prior methods use diffusion models to generate multi-view imagery and reconstruct 3D geometry through multi-view consistency.The passage identifies Score Distillation Sampling as a foundational strategy and describes subsequent multi-view reconstruction approaches.
- Part-Aware Generation: Part-aware generation enables part-level control but often produces view-inconsistent, low-fidelity geometry or surface-level segmentations rather than distinct objects.OmniPart advances this direction with a two-stage framework built on TRELLIS for semantic decoupling and structural cohesion.
- Interactive Physics: Recent work begins bridging static geometry and interactive physics through comprehensive interactive-scene systems and physical-property datasets with corresponding generation models.EmbodiedGen integrates generative modules including layout generation, while PhysX-3D introduces PhysXNet on top of PartNet and a generation model based on TRELLIS.
3. Physics-Grounded, Part-Aware 3D Assets Generation
PhysForge generates physics-grounded, part-aware 3D assets through a two-stage pipeline that plans hierarchical physical structure before diffusion-based realization. Its PhysDB dataset supplies large-scale, fine-grained physical annotations for training.
- Two-Stage Generation: PhysForge combines VLM-based planning with diffusion-based realization to generate geometry, texture, and precise articulation parameters from a hierarchical physical blueprint.The planner defines part structure and physical properties, while KineVoxel Injection guides the diffusion model during synthesis.
- PhysDB Dataset: PhysDB annotates holistic, static, functional, and interactive properties at object and part levels, including scale, usage scene, semantic labels, material, and mass.Its human-in-the-loop pipeline combines multimodal LLM annotation with manual screening and correction, emphasizing rich physical properties and joint types.
- PhysDB Dataset: 150k 3D objects comprise PhysDB, spanning seven categories and selected for meaningful part structures suitable for physics annotation.The dataset is sourced from Objaverse and covers household, industrial, weapons, personal, vehicles, tech and electronics, and cultural items.
- VLM Planner: The VLM planner autoregressively predicts hierarchical part structure, bounding boxes, parent-child relationships, and semantic joint types from image and voxel inputs.It uses 66 special tokens, representing each 3D axis-aligned bounding box with six quantized coordinate tokens.
- Diffusion Realization: KineVoxel Injection represents each part’s articulation as an 8-dimensional vector and processes it alongside geometry latents for joint geometry-kinematics denoising.The vector contains joint origin, axis, and motion limits; joint-type embeddings connect planned semantics with continuous kinematic parameters.
4. Experiments
Experiments evaluate PhysForge on part planning, physics-property generation, and articulated-object generation across multiple datasets and compare it with task-specific baselines. Results show stronger planning, geometry and physics-property prediction, kinematic accuracy, and simulation-oriented utility, while ablations validate the two-stage design.
- Evaluation Protocol: Evaluation uses PartObjaverse-Tiny, PhysXNet, PhysDB, and articulated-object sets from PartNet-Mobility and Infinite-Mobility.The protocol includes 200 PartObjaverse-Tiny objects, 1,000 PhysXNet test objects, 1,000 PhysDB cases, and 340 articulated objects.
- Part Structure Planning: PhysForge achieves state-of-the-art part-structure planning, and its mask-free variant remains semantically reasonable while outperforming the physics-free PhysForge-bbox model.Evaluation uses BBox IoU, Voxel Recall, and Voxel IoU to assess bounding-box and voxel-level planning precision.
- Physics Properties: PhysForge surpasses PhysXGen and TRELLIS in physics-property generation, benefiting from VLM world knowledge to predict textual and numerical physical properties.Geometry is evaluated with Chamfer Distance and F1-Score, while part-level properties include scale, material, affordance, function, and interaction.
- Kinematic Parameters: PhysForge generates articulated objects with better image consistency and joint-parameter accuracy than Articulate Anything, Singapo, and URDFormer.The evaluation measures mesh quality, image matching, joint-axis error, and joint-pivot error on validation and in-the-wild images.
- Ablation Analysis: Removing joint-type embeddings or the dedicated kinematic subnetwork degrades joint accuracy and mechanical-constraint synthesis, confirming both components are important.The joint-type embedding transfers Stage 1 articulation planning to Stage 2, while the independent kinematic encoder and decoder support precise constraints.
- Downstream Applications: Generated assets support robotic manipulation in RoboTwin and physics-based interaction in game engines and virtual worlds.Detailed part geometry and precise kinematic parameters enable realistic object interaction in simulation and interactive environments.
5. Conclusion
PhysForge generates interactive, physics-grounded 3D assets through a decoupled VLM Planning + Diffusion Realization framework. Supported by PhysDB, it combines hierarchical physical blueprints with geometry and kinematic-parameter synthesis for embodied AI and interactive virtual worlds.
- Framework: PhysForge introduces a decoupled “VLM Planning + Diffusion Realization” architecture for generating interactive, physics-grounded 3D assets.The framework separates physical planning from diffusion-based realization.
- Physical Planning: A tuned VLM generates “Hierarchical Physical Blueprints” defining an asset’s complete physical profile.These blueprints provide the physical specifications used during realization.
- Diffusion Realization: KineVoxel Injection enables diffusion models to jointly generate geometry and precise kinematic parameters.The algorithm realizes the physical blueprints through synergistic geometry and kinematic-parameter synthesis.
- Dataset: PhysDB is a large-scale, 150k-asset dataset with rich annotations supporting PhysForge.The dataset serves as the framework’s data foundation.
- Applications: PhysForge provides a foundational data engine for embodied AI and interactive virtual worlds.Its generated assets are intended to support these applications.