PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoPhyX: Automatic Text-Condition Physics Property Generation

Bei Huang, Yixin Chen, Hongbin Zha, Yuru Pei, Siyuan Huang

arXiv:2610.29517v1cs.CVcs.GR

TL;DR

Physical-property assignment for simulation remains difficult because optimization-based approaches can be costly and visual-only prediction cannot resolve material ambiguity. AutoPhyX predicts spatially dense properties from 3D assets and text using a part-aware dataset and cross-modal modulation. It reports state-of-the-art accuracy on Text2PhyX and ABO-500.

  • Problem

    Prior approaches either require costly per-scene optimization or infer properties from visual cues that cannot disambiguate visually similar materials.

    Source: §1 · p. 2

  • Method

    AutoPhyX uses text-conditioned cross-modal modulation to predict dense physical properties in a voxel field, trained with part-aware Text2PhyX annotations.

    Source: §1 · pp. 2–3

  • Results

    AutoPhyX achieves state-of-the-art accuracy on Text2PhyX and ABO-500, improving PSNR from 23.421 to 26.012 on Text2PhyX and reducing ADE by 20% versus the second-best method on ABO-500.

    Source: §1 · p. 3, §6 · p. 15

  • Takeaways & Limitations

    Predicted properties support MPM and FEM simulations and transfer across diverse 3D representations.

    Source: §1 · p. 2, §6 · p. 15

    Test-time optimization methods can require hours per scene and fail to generalize across diverse scenes.

    Source: §2.2 · p. 5

Abstract

from arXiv · show

High-fidelity 4D physical simulation requires accurate physical parameter assignments that align with diverse material properties. Previous methods struggle with critical bottlenecks: they either rely on computationally intensive per-scene optimization or predict parameters from purely visual inputs. The latter suffers from inherent ambiguity, e.g., the inability to distinguish the stiffness of rubber from its appearance alone. To bridge this gap, we propose AutoPhyX, a text-conditioned framework for predicting spatially-varying physical properties. Since training robust feed-forward models requires fine-grained supervision, we first introduce a novel part-controllable data generation pipeline that decomposes complex 3D assets into semantically distinct components and pairs them with physically plausible, diverse parameters and corresponding text descriptions. Leveraging this dataset, AutoPhyX employs a cross-modal modulation mechanism where text features dynamically modulate visual features for precise physical grounding. By formulating property prediction within a voxel field, AutoPhyX ensures compatibility with diverse 3D representations, including meshes, point clouds, Gaussian Splatting, and NeRF. Experiments demonstrate that our method enables physically plausible, text-driven parameter assignment in a single forward pass. It achieves high accuracy and robust generalization to in-the-wild objects, paving the way for future physics-intensive embodied and robotic applications.

1 Introduction

AutoPhyX predicts dense physical properties from 3D assets and text, addressing the cost and ambiguity of prior approaches. It combines cross-modal prediction with a part-aware dataset, and reports state-of-the-art results on Text2PhyX and ABO-500.

1 Introduction

  • Prior methods either require costly per-scene optimization or rely on visual cues that cannot reliably distinguish physically different, visually similar materials.Optimization can take hours and generalizes poorly, while feed-forward visual methods lack explicit semantic guidance.Source: §1 · p. 2
  • AutoPhyX predicts spatially dense properties in a unified voxel field from a 3D asset and text, without per-scene optimization.Text features modulate visual representations to distinguish visually similar materials and assign spatially coherent properties to distinct parts.Source: §1 · pp. 2–3
  • The voxel-field formulation transfers to diverse 3D representations, and the paper presents AutoPhyX as a plug-and-play solution across representations, platforms, and rendering engines.The figure illustrates different simulated responses under identical boundary conditions for text-conditioned predictions.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • Text2PhyX contains 1,700 assets, each paired with eight physical-parameter and text-description annotations generated through an automated part-controllable pipeline.The pipeline uses engineering databases and VLMs to identify parts, assign plausible properties, and produce descriptions.Source: §1 · p. 3
  • PSNR improves from 23.421 to 26.012 on Text2PhyX, while ADE falls by 20% versus the second-best method on ABO-500.The reported results also support simulation workflows using MPM and FEM.Source: §1 · p. 3

2 Related Work

Related work addresses physical-property assignment through VLM prediction, test-time optimization, and feed-forward regression. AutoPhyX is positioned as using text to resolve visual ambiguity while avoiding the per-scene costs and generalization limits of optimization-based methods.

2 Related Work

  • FEM and MPM depend on accurate physical parameters; Young’s modulus, Poisson’s ratio, and density govern stiffness, lateral deformation, and inertial response.These properties are identified as fundamental to a broad range of solid and deformable simulations.Source: §2 · p. 4

2.1 Physics Property Assignment via Vision-Language Model

  • VLMs prompted to predict exact physical parameters can produce random outputs, lack reliable physics knowledge, and struggle with visual ambiguity.The described alternative uses VLMs for semantic reasoning, database-supported property ranges, repeated queries, and explicit material descriptions.Source: §2.1 · p. 4

2.2 Test-time Physics Property Optimization

  • Observation-based optimization minimizes differences between simulated and observed motion, but often needs dense, high-quality supervision that is difficult to obtain in real settings.Video-diffusion-guided methods reduce dependence on such supervision but can take hours per scene and fail to generalize across diverse scenes.Source: §2.2 · pp. 4–5

2.3 Feedforward Physics Property Prediction

  • Feed-forward methods such as Pixie and VoMP regress physical attributes in one pass, but visual-only inputs leave ambiguity between similar-looking objects with different properties.AutoPhyX adds textual constraints and targets efficient, controllable, physically plausible predictions compatible with meshes, point clouds, and Gaussian Splatting.Source: §2.3 · p. 5

3 Method

AutoPhyX lifts multi-view visual features into a volumetric representation, conditions physical-property prediction on text, and uses a part-controllable pipeline to generate training data.

3 Method

  • AutoPhyX extracts multi-view features into a dense 3D voxel grid, fuses them with text, and decodes a unified field of physical parameters for simulation.The method also generates a dataset for training and testing.Source: §3 · pp. 5–6

3.1 3D Feature Extraction from 2D Images

  • NeRF volume rendering suppresses occluded features, while ray-based interior classification and nearest-neighbor feature transfer produce a volumetric representation for simulation.Filling uses six-direction boundary checks followed by ray-surface intersection refinement for complex manifolds.Source: §3.1 · p. 6, Figure 3 · p. 7

3.2 Text-conditioned Physical Parameter Prediction

  • Text embeddings modulate 3D U-Net feature channels at multiple scales, guiding parameter regression when visual appearance alone cannot distinguish physical properties.Dedicated MLPs predict layer-specific scaling and shifting parameters for the channel-wise affine transformation.Source: §3.2 · pp. 6–7

3.3 Dataset Generation

  • Part segmentation uses text keywords and OpenCLIP feature similarity, with five candidate segmentations generated and the VLM selecting the highest-quality result.Fifteen upper-hemisphere views are rendered per object to reduce ambiguity from single-view observations.Source: §3.3 · p. 8
  • Generated descriptions guide plausible part-property assignments grounded in engineering databases, and 100-frame simulations filter assignments; the dataset covers density, Young’s modulus, and Poisson’s ratio.Unmentioned parts receive mean values from valid ranges, and failed cases may be resampled up to five times before an object is discarded.Source: §3.3 · pp. 8–9, Figure 4 · p. 9

4 Method

Text2PhyX combines 3D assets with text-conditioned physical variants in an object-level train-test split.

4 Method

  • Approximately 1,000 Objaverse meshes and 700 reconstructed Gaussian Splatting models form a base of 1,700 unique 3D assets.The Gaussian Splatting models are reconstructed from images using SAM 3D Objects.Source: §4 · p. 9
  • Eight text descriptions with corresponding physical parameters per object yield 13,600 object-text-parameter triplets in Text2PhyX.Each triplet contains a 3D object representation, text description, and ground-truth physical parameters.Source: §4 · p. 9
  • The object-level split assigns 1,575 objects and 12,600 samples to training, and 125 objects and 1,000 samples to testing.Source: §4 · p. 10

5 Experiment

Experiments evaluate physical-property prediction on Text2PhyX and real-world ABO-500, alongside simulation quality and runtime. AutoPhyX outperforms baselines quantitatively and supports varied simulation pipelines.

Dataset and Settings

  • Evaluation compares predicted parameters with ground truth and assesses simulation renderings using physical and visual metrics.Text2PhyX uses MSE for E, ν, and ρ, with E and ρ evaluated in log-space; rendered frames are assessed with PSNR, SSIM, and LPIPS.Source: §5 · pp. 10–12
  • On ABO-500, the 500-object out-of-distribution test uses object-name-only prompts because the dataset lacks detailed physical descriptions.The dataset provides real-world density and mass ground truth; additional human-annotated and cross-dataset evaluations appear in the supplement.Source: §5 · p. 10

Quantitative Results

  • AutoPhyX with OpenCLIP reaches PSNR 26.012, approximately 2.6 dB above Pixie, and outperforms baselines across rendering and physics metrics.Table 1 covers PSNR, SSIM, LPIPS, and physical-property errors; Pixie is evaluated under its default unconditional setting because it does not accept text.Source: Table 1 · p. 11, §5 · p. 11, Quantitative Result · p. 12

Runtime

  • AutoPhyX reaches PSNR 26.01 with 2s inference, while optimization-based baselines require minutes to hours and yield lower PSNR.The runtime plot also includes a point accounting for approximately 3 minutes of CLIP voxel-feature extraction as preprocessing.Source: Runtime · p. 13, Figure 5 · p. 13

Qualitative Results

  • Paired simulations hold boundary conditions fixed while changing descriptions, showing contrasting behaviors for propellers, plant stems, toast, and soft bodies.The first six qualitative rows use MPM with Gaussian Splatting; the final two use FEM in Isaac Sim and change both solver and representation.Source: Figure 6 · p. 14, Qualitative Results · pp. 14–15

6 Conclusion and Future Work

AutoPhyX automates dense, text-conditioned physical-property assignment for diverse 3D assets, supported by the multi-part Text2PhyX dataset. Experiments report state-of-the-art accuracy on Text2PhyX and ABO-500 and demonstrate MPM and FEM simulation use.

Conclusion

  • AutoPhyX automates dense physical-property assignment for diverse 3D assets using a unified voxel field and an automated VLM-driven pipeline.Text2PhyX is a large-scale, multi-part dataset spanning diverse object categories for training and testing.Source: §6 · p. 15
  • Experiments report state-of-the-art accuracy on Text2PhyX and real-world ABO-500, with qualitative integration into MPM and FEM pipelines.Demonstrated objects include a bonsai, propeller, and bread; the authors describe larger, complex interactive environments as a potential application.Source: §6 · p. 15

Future Work

  • Future work plans to add surface friction and damping coefficients and explore higher-resolution fields for objects with very fine structures.Source: §6 · p. 15

A Text2PhyX Dataset Details

Text2PhyX pairs diverse 3D assets with part-level text descriptions and voxel-wise physical properties. Its pipeline builds semantic voxel features, fills object interiors for simulation, and generates grounded text–property variants.

A.1 Overview

  • Text2PhyX contains 1,700 assets—about 1,000 Objaverse meshes and 700 in-the-wild Gaussian Splatting models—with eight text–parameter descriptions per object, totaling 13,600 triplets.Each triplet pairs dense OpenCLIP semantic voxels, a part-focused material or physical-state description, and voxel-level Young’s Modulus, Poisson’s Ratio, and Density labels; the object-level split holds out 125 objects for testing.Source: §A.1 · p. 20

A.2 Gallery with Physics Property

  • Multi-view OpenCLIP features and VLM-generated keywords segment objects into semantic parts, which receive localized, voxel-wise physical-property annotations.The galleries show part-specific values, including a stiffer tree trunk than leaves and bag handles with different density and elasticity from the fabric body.Source: §A.2 · pp. 20, 22, 23, Figure A1 · p. 21

A.3 OpenCLIP-Driven 3D Semantic Segmentation

  • The 3D feature pipeline distills OpenCLIP features from multi-view images into a continuous field using volumetric rendering, then fills internal voxels for simulation.The distillation matches rendered ray features to 2D OpenCLIP features with negative cosine similarity; even-odd ray casting identifies interior voxels, which inherit features by nearest-neighbor interpolation from the surface field.Source: §A.3 · p. 23, 2D Feature Extraction · p. 23, 3D Feature Distillation via Volumetric… · pp. 23–24

A.4 VLM labeling

  • For text-guided property augmentation, a VLM uses external knowledge to ground a material change and assign a corresponding parameter multiplier.In the soda-can example, the VLM assigns a 0.113 multiplier to transition from aluminum to hard plastic; the test descriptions are human-written after viewing simulated behavior, providing an evaluation beyond generated descriptions.Source: §A.4 · pp. 25–26, Table A1 · p. 25

B Additional Evaluation and Analysis

A human-annotated test set evaluates text generalization beyond descriptions generated for Text2PhyX. It is built from simulated behavior and independently sourced material-property ranges.

B.1 Human-Annotated Text Generalization

  • The 30-object human-annotated test set uses descriptions written after annotators watch simulated object behavior, independently of Text2PhyX’s text-generation pipeline.For each material category, the authors query plausible property ranges, reconstruct assets from retrieved images, sample two configurations, and simulate each one.Source: §B.1 · p. 26
  • Two material-property configurations are sampled from queried ranges and simulated for each reconstructed 3D asset.The assets are reconstructed from retrieved object images using SAM 3D Objects.Source: §B.1 · p. 26
  • Annotators describe the simulated videos, making this evaluation independent of the text-generation pipeline used to build Text2PhyX.The set contains 30 objects and is intended to test text generalization beyond the dataset descriptions.Source: §B.1 · p. 26

B.2 Unconditional Cross-Dataset Validation

The appendix describes an unconditional cross-dataset protocol and a resolution trade-off, alongside implementation details for the 3D U-Net and feature projector.

B.2 Unconditional Cross-Dataset Validation

  • The unconditional comparison evaluates Pixie, VoMP, and AutoPhyX on their respective datasets with empty text input, testing feed-forward competitiveness rather than text controllability.The datasets are Pixie, VoMP, and Text2PhyX, respectively.Source: §B.2 · p. 26, Table A2 · p. 27

B.3 Resolution Discussion

  • The model predicts a 64^3 voxel field, a resolution the authors consider often sufficient for current property prediction and simulation, though finer grids may help fine structures at significant GPU-memory cost.The discussion compares this with PhysGaussian’s effective simulation resolution of around 50^3 and leaves higher-resolution prediction as future work.Source: §B.3 · p. 26

C.1 Overview

  • The architecture uses a feature projector, hierarchical 3D U-Net, and FiLM conditioning to inject text embeddings into residual blocks across resolution levels.The projector maps semantic inputs to a base dimension before spatial processing; the output supports either 8-channel discrete material classification or 3-channel continuous parameter regression.Source: §C.1 · p. 27, §C.2 · p. 27, §C.3 · p. 28

C.2 Feature Projector

  • The feature projector maps 768-dimensional OpenCLIP features to 32 channels, then the U-Net processes 64^3 grids through an encoder–decoder with skip connections.The U-Net uses four resolution levels and nearest-neighbor upsampling in the decoder.Source: §C.2 · p. 27, §C.3 · p. 28

C.4 FiLM-Conditioned 3D U-Net Architecture

AutoPhyX injects text conditioning throughout a 3D U-Net using FiLM-modulated residual blocks. The appendix also reports that architectures downsampling or pooling high-resolution inputs perform worse in its ablation setting.

C.4 FiLM-Conditioned 3D U-Net Architecture

  • FiLM layers in residual blocks use text embeddings to generate channel-wise scale and shift parameters for intermediate 3D features.The modulation is applied across encoder, bottleneck, and decoder levels, from 64^3 spatial resolution down to the 8^3 bottleneck.Source: §C.4 · pp. 28–29
  • In the architecture ablation, models that downsample or average-pool the input show a significant predictive-capability drop relative to the high-resolution-input setting.The ablation uses a Text2PhyX subset with 2,700 training and 1,000 test samples for 30 epochs.Source: Table A3 · p. 29
  • Multi-scale text modulation guides feature extraction across the U-Net and aligns visual geometry with specified physical attributes for continuous parameter regression.The modulated feature map proceeds through the residual block’s subsequent normalization, activation, dropout, and convolution before the skip connection.Source: §C.4 · p. 29

C.5 Comparison with Other Architectures

AutoPhyX’s FiLM-conditioned 3D U-Net outperforms sequence-based architectures on physical-parameter prediction. The comparison attributes this advantage to preserving high-resolution 3D geometry and topology rather than flattening pooled voxels into sequences.

C.5 Comparison with Other Architectures

  • The FiLM U-Net achieves significantly lower errors than the 3D Transformer and VisualRWKV across log E, ν, and log ρ.The passage refers to Table A3 but does not provide its error values.Source: §C.5 · p. 29
  • Transformer inputs are downsampled from 64^3 to 16^3, while VisualRWKV uses 32^3 inputs, sacrificing fine-grained geometric information.The Transformer’s original grid would produce 262,144 tokens under global attention; VisualRWKV’s flattened sequence contains 32,768 tokens.Source: §C.5 · p. 30
  • Flattening voxel features forces sequence models to relearn 3D topology, whereas the U-Net’s 3D convolutions preserve and exploit it.The passage notes that this relearning is challenging with the 2,700-sample training set and is prone to overfitting.Source: §C.5 · p. 30
  • The U-Net processes 64^3 grids through a hierarchical encoder-decoder, retaining fine geometry while extracting semantics at an 8^3 bottleneck.Multi-scale FiLM injects text conditioning across geometric levels without destroying spatial structure.Source: §C.5 · p. 30

D.1 Material Point Method

The appendix describes MPM’s particle-grid simulation cycle and how it evolves Gaussian splats, then outlines FEM’s mesh-based deformation and force computations. These methods provide simulation pathways for the paper’s predicted material properties.

D.1 Material Point Method

D.2 Finite Element Method

  • FEM operates directly on a connected tetrahedral or hexahedral mesh, with vertices carrying kinematic state and elements carrying deformation state.The passage describes FEM as preserving volume and capturing elastic behavior without the numerical dissipation often associated with grid-based transfers.Source: §D.2 · p. 35
  • FEM computes each element’s deformation gradient from current and rest shapes, evaluates stress with a hyperelastic model, distributes internal forces, and advances vertex motion.The stated time-integration example is Symplectic Euler.Source: The Finite Element Method Algorithm… · pp. 35–36

Related papers

Loading 2610.29517v1…