Source-linked AI summary
Gen2Physics: Grounding Generated 3D Meshes in Physics via Multi-View Material Decomposition
Mauro Comi, Jordi Serrano Berbel, Kevis-Kokitsi Maninis, Philipp Henzler, Manuel Sanchez
TL;DR
Generated 3D meshes often lack the physical properties needed for interactive simulation, while existing physics-grounding methods rely on volumetric representations that do not directly fit standard physics engines. Gen2Physics post-processes arbitrary meshes through multi-view material segmentation, 2D-to-3D consistency projection, VLM refinement, and internal-structure inference. It more than doubles material segmentation accuracy, matches volumetric methods on mass estimation, and produces watertight per-material sub-meshes.
Problem
Generated meshes lack the physical properties required for interactive simulation, while volumetric grounding representations are incompatible with standard physics engines.
Method
Gen2Physics post-processes arbitrary meshes into material-based components using multi-view Vision Transformer segmentation, 2D-to-3D projection, VLM refinement, and amodal completion.
Results
Gen2Physics outperforms prior baselines in 3D material segmentation, matches volumetric methods in mass-estimation accuracy, and produces per-material sub-meshes.
Takeaways & Limitations
The framework grounds generated meshes directly in physics and outputs components usable in standard physics engines.
Takeaways & Limitations
The framework relies on external HoloPart completion, focuses primarily on surface-visible materials, and evaluates on a 1,100-image manually annotated test set.
Abstract
from arXiv · showhide
While state-of-the-art generative models produce high-fidelity 3D meshes, these outputs lack the physical properties required for interactive simulation, gaming, or robotics. We introduce Gen2Physics, a unified and automated framework that grounds generated meshes in physics by automatically decomposing them into their constituent material components. Unlike prior approaches, which focus on volumetric representations incompatible with standard physics engines, Gen2Physics operates directly on meshes to produce immediately simulation-ready assets. Our pipeline integrates a fine-tuned Vision Transformer for dense material segmentation, a robust 2D-to-3D consistency projection, and a Vision-Language Model (VLM) guided refinement that leverages contextual reasoning to assign physical properties and infer internal geometry (solid vs. hollow). By converting surface patches into volumes with distinct densities, our method enables physically plausible dynamic simulations. Experimental results on the ABO-500 and PartNet-Material benchmarks demonstrate that Gen2Physics more than doubles the material segmentation accuracy of prior physics-grounding pipelines (15.6 to 48.3 mIoU), while matching the mass-estimation accuracy of volumetric methods and being the only approach to output watertight per-material sub-meshes.
1. Introduction
Gen2Physics addresses the gap between visually compelling generated meshes and the physical properties required for interactive simulation. It post-processes arbitrary meshes into material-based, simulation-ready components using multi-view segmentation, 2D-to-3D projection, and contextual refinement.
- Generated meshes remain static assets because they lack physical properties needed for realistic interactive behavior.
- Volumetric physics-grounding methods are incompatible with standard rigid-body and articulated physics engines expecting discrete sub-meshes.
- Gen2Physics operates post-hoc on arbitrary generated meshes, avoiding retraining and producing material-based components usable in standard physics engines.
- Its pipeline combines dense Vision Transformer segmentation, robust 2D-to-3D projection, watertight material decomposition, and physical-property assignment.
2. Related Work
Prior work decomposes 3D shapes or estimates physical properties using representations and boundaries that do not fully match material-aware rigid-body simulation. Gen2Physics instead maps generated meshes directly into distinct material sub-meshes compatible with standard physics engines.
- Semantic 3D segmentation can underdetermine simulator parameters because one semantic part may span multiple materials.
- Clustering-based 3D methods require the number of parts in advance, whereas distinct material counts are often unknown.
- Open-world segmenters follow object or topological boundaries, while Gen2Physics predicts dense material classes across geometry-agnostic boundaries.
- Gen2Physics fuses multi-view predictions on mesh faces and assigns mechanical rather than optical properties, without requiring an existing UV parameterization.
- Volumetric approaches support continuum or particle simulation but remain difficult to convert into distinct multi-material mesh partitions for rigid-body engines.
- Gen2Physics operates directly on generated meshes, making its outputs inherently compatible with standard physics engines.
3. Method
Gen2Physics decomposes an input mesh into watertight, material-specific components with plausible physical properties through a four-stage pipeline.
- The pipeline targets physically grounded, simulation-ready components, each represented as a watertight sub-mesh corresponding to a distinct material.
- It performs dense multi-view material segmentation using a specialized Vision Transformer.
- A robust 2D-to-3D projection establishes geometrically consistent material labels on the mesh.
- A Vision-Language Model semantically refines the labels before amodal completion and physical-property assignment.
3.1. Multi-View Material Segmentation
The method learns dense material segmentation from a curated object-centric dataset and aggregates predictions from multiple rendered views to improve coverage and resolve ambiguity.
- 500,000 multi-view renderings were curated from Coohom to address the scarcity of object-centric datasets with detailed material annotations.
- 20,000 assets were relabeled with physically grounded material categories after shader parameters proved unreliable for physical labeling.
- Each isolated part was rendered from randomly sampled spherical viewpoints, and a multimodal VLM assigned one of 12 predefined material categories.
- TIPS was selected because its spatially coherent representations are suited to dense segmentation.
- At test time, the mesh is rendered from 11 diverse viewpoints, and TIPS predicts a 2D material map for each view.
- The multi-view design captures comprehensive appearance information and helps resolve lighting, reflection, and occlusion ambiguities.
3.2. 2D-to-3D Consistency Projection
Gen2Physics lifts noisy multi-view material segmentations onto mesh faces through visibility-aware majority voting, producing geometrically consistent labels and resolving unseen faces with neighborhood information.
- Consistency: Multi-view aggregation filters single-view errors and converges on the globally consistent material assignment.The method projects dense material identities onto explicit mesh faces, consistently unifying separated parts that share a material.
- Multi-view aggregation: Face-index buffers map pixels in 11 rendered views directly to visible mesh faces.This establishes the pixel-to-face correspondence needed for cross-view aggregation.
- Multi-view aggregation: Majority voting assigns each mesh face the material receiving the most votes across viewpoints where it is visible.Votes pair each rendered pixel’s predicted material label with its visible face index.
- Label selection: The final label Lf is selected by maximizing the frequency of projected material labels over views.Proj(f, i) returns the centroid pixel coordinates for face f in view i, and M denotes the material classes.
- Unseen faces: Flood fill labels faces invisible in every rendered view using the majority label among neighboring faces.This supplies labels for otherwise unobserved mesh regions.
3.3. 3D Contextual Refinement with a Vision-Language Model
Gen2Physics uses a Vision-Language Model to critique and repair semantically implausible material labels after geometrically consistent multi-view aggregation.
- Motivation: A VLM refines initial 3D material assignments when out-of-distribution assets or lighting cause semantic errors.The refinement complements geometric consistency with contextual semantic reasoning.
- Inputs: The VLM receives original multi-view RGB images, reprojected color-coded segmentation maps, and a color-to-material legend.These inputs support visual verification of the mesh’s material assignments.
- Contextual refinement: Chain-of-Thought prompting instructs Gemini-2.5 Pro to correct assignments that are implausible given the object’s identity and context.The model uses contextual reasoning rather than relying only on the initial segmentation labels.
- Outcome: The refined assignments are intended to be both geometrically consistent and coherent with the object’s identity and function.The VLM stage follows the multi-view consistency projection and repairs remaining semantic inconsistencies.
- Contextual refinement: For example, the VLM may relabel thin chair legs as metal rather than wood when specular highlights mislead segmentation.This illustrates semantic correction while preserving the object’s contextual interpretation.
3.4. Part Completion and Property Assignment
Gen2Physics converts material-labeled surface patches into watertight components, then assigns density and internal-structure estimates for simulation-ready physical properties.
- Part completion: The mesh is partitioned into connected face groups sharing the same material label, producing non-watertight surface patches.These patches form the material-specific components that are completed for simulation.
- Part completion: HoloPart completes each incomplete material shell into a watertight mesh.Watertight completion supports volumetric property estimation and stable physics-engine interactions.
- Property assignment: For each completed part, a VLM infers plausible density and whether its internal structure is solid or hollow.For hollow parts, it also predicts an average wall thickness; simulator-specific parameters may optionally be inferred.
- Internal structure: Distinguishing solid from hollow components avoids applying a uniform-thickness heuristic to solid objects.Gen2Physics uses thickness-based approximation only for predicted hollow parts and volume-based calculation for solid parts.
- Mass calculation: For solid parts, mass is computed as mi = ρi · Vi; for hollow parts, it is approximated as mi ≈ ρi · Ai · ti.The total object mass sums the component masses, and dynamic properties such as inertia can be derived from the grounded parts.
- Output: The final output is a collection of simulation-ready 3D components that remain visually faithful while being physically plausible for interactive applications.Each component is a watertight, material-aware part with inferred physical properties.
4. Results
Gen2Physics is evaluated on mass estimation, simulation readiness, and material segmentation. It achieves competitive mass estimation while substantially improving material identification and producing decomposed, simulation-ready assets.
- Evaluation setup: Gen2Physics is evaluated on downstream physical property estimation, rigid-body simulation readiness, and material segmentation fidelity.The evaluation uses ABO-500 for mass estimation and PartNet-Material for segmentation.
- Mass estimation: Gen2Physics achieves comparable mass-estimation results within 0.2% ADE of the best method, places second on ALDE, and obtains the best MnRE.PUGS achieves a better APE, which prior work identifies as sensitive to errors on lightweight objects.
- Mass estimation: The mesh-based volume calculation is more principled than volumetric methods’ thickness heuristics, but it overestimates scores for very small objects.The limitation is reported alongside the method’s physical-volume calculation comparison.
- Material identification: 48.27% mIoU more than doubles PUGS at 15.56% and NeRF2Physics at 14.54% on PartNet-Material.The improvement is consistent across nearly all reported material categories.
- Material identification: Local CLIP-based baselines produce fragmented and noisy masks, whereas Gen2Physics yields more coherent segmentations through specialized ViT features and VLM refinement.Local patch features can misclassify materials because appearance cues and object-level semantics do not always align.
- Ablations: VLM refinement raises mIoU from 22.58% with the ViT alone to 48.27% in the full model.Ceramic improves from 0.00 IoU without refinement to 43.49 IoU after refinement, illustrating recovery of ambiguous materials.
- Ablations: Assuming every part is solid improves ADE but degrades ALDE, APE, and MnRE, indicating a trade-off in internal-structure reasoning.The full model’s relative metrics support the authors’ claim that solid/hollow reasoning improves robustness across diverse objects.
- Simulation readiness: Gen2Physics decomposes composite meshes into watertight per-material sub-meshes, enabling dynamic behavior in MuJoCo while estimating comparable total mass to PUGS.For the illustrated asset, PUGS estimates 4.1 kg and Gen2Physics estimates 3.8 kg, but PUGS treats the mesh as monolithic.
5. Limitations and Conclusions
Gen2Physics combines multi-view material decomposition, VLM refinement, and watertight volume completion to produce simulation-ready assets with accurate physical properties. Its main limitations are reliance on external amodal completion, limited internal-material modeling, and a small manually annotated test set.
- 5. Limitations and Conclusions: Gen2Physics combines multi-view segmentation, 2D-to-3D consistency projection, VLM refinement, and watertight volume completion for physically grounded meshes.
- 5. Limitations and Conclusions: The framework relies on the external HoloPart model for amodal part completion, which future work could integrate into an end-to-end pipeline.
- 5. Limitations and Conclusions: The current method focuses primarily on surface-visible material decomposition, leaving internal material structures for future work.
- 5. Limitations and Conclusions: The manually annotated PartNet-Material test set contains 1100 images, motivating the creation of a larger benchmark.