PaperlayerView exact revision on arXiv

Source-linked AI summary

PhysCreator: Creating Physically-Attributed 3D Assets at Will

Shiyan Zheng, Ming Li, Lei Han, Xiangyu Xu

arXiv:2610.34473v1cs.CV

TL;DR

Image-based 3D representations capture appearance but often lack physically plausible behavior under dynamic interactions. PhysCreator jointly models geometry and spatially varying physical properties in a voxel-aligned representation, using vision-language distillation for voxel-level supervision. The paper reports efficient generation of high-quality, physically plausible, simulation-ready assets across diverse objects.

  • Problem

    Image-based 3D representations are often static and existing joint methods can lack explicit 3D structure for modeling spatially varying physical properties.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    PhysCreator lifts image observations into a voxel-aligned latent space, jointly predicts geometry and physical attributes, and distills material-aware VLM knowledge into voxel-level supervision.

    Source: §1 · pp. 1–2, §3 · p. 3

  • Results

    PhysCreator efficiently generates high-quality, physically plausible, simulation-ready assets across diverse objects, and its single-view model approaches multi-view performance while outperforming PhysGM.

    Source: §4.2 · p. 8, §5 · p. 10

  • Takeaways & Limitations

    The shared representation supports simulation-ready assets with spatially varying physical attributes and semantic control over coupled physical properties.

    Source: §5 · p. 10

Abstract

from arXiv · show

Significant advancements in 3D asset creation have enabled high-fidelity 3D generation from images. However, most existing representations remain inherently static, lacking the ability to model physically plausible behaviors under dynamic interactions. We present \textbf{PhysCreator}, a unified framework that directly generates simulation-ready 3D representations with spatially varying physical properties from images. Unlike prior approaches that treat physical attributes as simple post-hoc predictions or rely on expensive per-instance optimization, we reformulate physical property estimation within a unified 3D representation learning framework. We introduce a voxel-aligned latent space by lifting image observations into a spatially structured 3D representation, in which geometry, appearance, and physical properties are jointly modeled and inferred in a feed-forward manner. To enable scalable learning without densely annotated physical ground truth, we leverage vision-language distillation to transfer material-aware semantic priors from powerful vision-language models into spatially grounded voxel-level supervision. Experiments demonstrate that our method achieves superior visual quality, physical plausibility, and generalization compared to prior approaches.

1 Introduction

Image-based 3D assets capture appearance but not physical behavior, and existing approaches often separate geometry from physical-property estimation or lack explicit 3D structure. PhysCreator jointly predicts geometry and spatially varying physical properties using a shared 3D representation and reports high-quality, physically plausible assets across diverse scenes.

  • Image-based 3D representations capture geometry and appearance but remain static, omitting properties such as stiffness and density needed for physical interactions.Source: §1 · p. 1
  • Prior methods commonly separate geometry reconstruction from physical-property estimation, while 2D-feature approaches lack spatial structure for heterogeneous materials.These approaches can have inefficient inference, limited generalization, and inconsistent geometry–physics coupling.Source: §1 · p. 1
  • PhysCreator lifts image observations into a shared 3D representation to jointly predict geometry and spatially varying physical properties.Source: §1 · p. 1
  • Vision-language distillation transfers material-aware knowledge into voxel-level supervision, supporting fine-grained physical learning without dense manual annotations.The VLM identifies semantic object parts and associated attributes, which are lifted into shared 3D space.Source: §1 · p. 2
  • The framework supports feed-forward inference for multi-view reconstruction and single-view generation, with optional semantic conditioning of physical attributes.Source: §1 · p. 2
  • PhysCreator produces high-quality 3D assets with physically plausible behaviors and realistic dynamic simulation across a wide range of scenes.Source: §1 · p. 2

2 Related Work

Related work covers image-based 3D reconstruction and generation, then methods that add physical simulation to neural 3D representations. PhysCreator contrasts with these methods through a unified voxel-aligned representation for spatially varying physical attributes.

2.1 3D Reconstruction and Generation

  • Reconstruction methods range from multi-stage SfM and stereo pipelines to learned predictions of point clouds, depth maps, meshes, neural fields, and 3D Gaussians.Learned methods simplify reconstruction while maintaining strong geometric accuracy.Source: §2.1 · p. 2
  • 3D generation spans diffusion and autoregressive approaches, but existing methods primarily focus on geometry and appearance rather than physical properties.Source: §2.1 · pp. 2–3

2.2 Physics-Based 3D Simulation

  • Physics-based asset methods use per-instance optimization, modular pipelines, or feed-forward prediction, with limitations in efficiency, generalization, physical consistency, or spatial material detail.PhysCreator instead models geometry and physical attributes in a unified voxel-aligned latent representation.Source: §2.2 · p. 3

3 Method

PhysCreator combines voxel-aligned 3D features with vision-language-derived voxel supervision to jointly predict geometry and spatially varying physics. It uses a shared latent for multi-view reconstruction and single-view generation, producing simulation-ready assets through feed-forward inference.

3.1 Unified Geometry-Physics Learning

  • A shared voxel representation gives geometry and physical predictions explicit 3D locality, supporting spatially varying materials aligned with reconstructed geometry.Source: §3.1 · pp. 3–4

3.1.1 Voxel-Based Feature Construction

  • Multi-view image features are projected into occupied voxels and average-pooled, forming spatially localized features that combine geometric structure with visual detail.Source: §3.1.1 · pp. 4–5

3.1.2 Joint Geometry and Physics Prediction

  • The shared latent is decoded into 3D Gaussian primitives and per-voxel physical attributes, intrinsically aligning geometry with material properties.Optional semantic conditioning can modulate physical predictions using descriptions such as “soft” or “rigid,” without manual numerical tuning.Source: §3.1.2 · p. 5

3.1.3 Single-View Feature Generation

  • Conditional flow matching generates the shared 3D latent from a single image, allowing the same geometry and physical-property decoders to serve single-view inputs.Source: §3.1.3 · pp. 5–6

3.2 Vision-Language Distillation

  • VLM part annotations are matched to lifted voxel features and simulation-verified; inferred descriptions can optionally condition physical-property predictions.Source: §3.2 · pp. 6–7

4 Experiments

PhysCreator compares favorably with prior approaches while producing physically plausible simulations through feed-forward generation. Experiments examine its runtime, spatially varying material modeling, geometry and property predictions, and text-conditioned control.

4.1 Experimental settings

  • Experiments use MPM simulation on Objaverse assets and compare optimization-based baselines with feed-forward methods using CLIPsim and user-study scores.The blind user study involved 26 participants and a 5-point Likert scale.Source: §4.1 · p. 7, §4.2 · p. 7

4.2 Results

  • PhysCreator compares favorably with state-of-the-art approaches, outperforms PhysGM, and produces simulation-ready assets in seconds without object-specific optimization.DreamPhysics and OmniPhysGS require several hours of per-instance computation.Source: §4.2 · pp. 7–8, §4.1 · p. 7
  • Under gravity, PhysCreator produces stable dynamics, while baselines show collapse, excessive bouncing, or implausible local deformation.The qualitative comparison reports structural collapse for OmniPhysGS and unrealistic local deformation for DreamPhysics and PhysGM.Source: Figure 3 · p. 8, §4.2 · p. 8

4.3.1 Spatially Varying Material Modeling

  • Spatially varying voxel-level properties let composite-object parts move independently, unlike PhysGM’s implausible deformation with uniform properties.Examples include a swaying flower with a stable pot and deforming snow with a preserved hat.Source: §4.3.1 · p. 8, Figure 4 · p. 9

4.3.2 Geometry and Physical-Property Learning

  • PhysCreator reduces Chamfer Distance and improves F-score, supporting accurate geometry learning with its voxel-aligned representation.Chamfer Distance measures surface discrepancy; F-score at a 2% threshold evaluates surface accuracy and completeness.Source: §4.3.2 · pp. 8–9, Table 2 · p. 9
  • PhysCreator achieves the lowest prediction errors for evaluated physical properties against manually verified reference annotations.Young’s modulus and density are evaluated in logarithmic space, while Poisson’s ratio is evaluated in its original scale.Source: §4.3.2 · p. 9, Table 3 · p. 10

4.3.3 Controllable Physical Generation

  • Under identical wind forces, PhysCreator produces prompt-consistent soft and hard bending, while DreamPhysics responds less distinctly.The qualitative comparison shows stiffer motion for “hard” and more flexible deformation for “soft”.Source: Figure 5 · p. 9, §4.3.3 · p. 10, Table 4 · p. 10

5 Conclusion

PhysCreator jointly models geometry, appearance, and spatially varying physical properties in a voxel-aligned representation. It distills material-aware VLM knowledge into voxel-level supervision and uses simulation-based verification to improve pseudo-label reliability.

  • PhysCreator generates simulation-ready 3D assets from images by jointly modeling geometry, appearance, and spatially varying physical properties.Its voxel-aligned latent representation grounds the predictions in a shared 3D space.Source: §5 · p. 10
  • VLM distillation and simulation-based verification address scarce physical annotations, while the learned representation supports semantic control over coupled physical attributes.Experiments report efficient generation of high-quality, physically plausible assets across diverse objects.Source: §5 · p. 10

Related papers

Loading 2610.34473v1…