PhysX-3D: Physical-Grounded 3D Asset Generation

1 Introduction

• Existing 3D datasets and generative models largely emphasize geometry and appearance, leaving comprehensive physical properties insufficiently represented for real-world applications.Relevant properties include scale, material, affordance, kinematics, and function descriptions.Source: §1 · p. 2, §2.1 · p. 3
• PhysXNet contains over 26K richly annotated 3D objects with part-level annotations, affordance ranks, and detailed kinematic-constraint parameters, while PhysXNet-XL contains over 6 million procedurally generated annotated objects.Source: §1 · p. 2

Human-in-the-loop annotation pipeline

• PhysXNet uses a scalable human-in-the-loop pipeline that isolates visual components, applies VLM labeling, and refines complex kinematic annotations with expert review.This converts existing geometry-focused datasets into fine-grained physics-annotated assets.Source: §1 · p. 2, Figure 2 · p. 4, Human-in-the-loop annotation pipeline · p. 4

2.3 Articulated and Physical 3D Object Modeling

• PhysXGen is a feed-forward framework that jointly models latent correlations between 3D structures and physical properties while preserving pretrained geometry quality.Its dual-branch design integrates physical knowledge into pretrained 3D generative priors.Source: §1 · p. 2, §2.3 · p. 4

2.1 3D Datasets and Benchmarks

• PhysXNet supplies part-level annotations that support applications such as robotic manipulation and physical simulation more directly than object-level metadata.The comparison specifically contrasts PhysXNet with ABO’s object-level annotation granularity.Source: Table 1 · p. 3, §2.1 · p. 3

3.1 Definition of Physical Properties

• PhysXNet defines properties across identification, function, and operation stages, covering scale, material, affordance, function descriptions, and kinematic parameters.Kinematic annotations include joint types, parent and child parts, movement directions, and movement ranges.Source: §3.1 · pp. 4–5

3.2 Human-in-the-loop Annotation Pipeline

• The annotation pipeline uses GPT-4o for preliminary information gathering, human verification, and human determination of kinematic parameters through contact-region and plane-fitting procedures.For revolute joints, k-means generates candidate rotation-axis locations before final parameter selection.Source: §3.2 · p. 5

3.3 Statistics and Distribution of PhysXNet

• PhysXNet contains over 26K objects averaging around five parts each, with substantial variation in dimensions, kinematic types, materials, and object categories.Its density values follow the metric standardization framework in g/cm3.Source: §3.3 · p. 5
• PhysXNet-XL adds procedurally generated objects through intra-category and cross-category combinations such as cabinets, chairs, drawers, and doors.The reported combinations broaden the procedural dataset beyond the original annotated collection.Source: §3.3 · p. 5

4 PhysXGen Framework

• PhysXGen combines physical properties with a pretrained 3D representation space to generate physically consistent assets while retaining geometry and appearance fidelity.The framework jointly integrates physical information and fine-tunes the structural branch.Source: §4 · pp. 5–6

4.1 Physical 3D VAE Encoding and Decoding

• The physical VAE encodes scaling, affordance, density, kinematic parameters, and function descriptions into a unified physical latent alongside a structural latent.Kinematic parameters include part hierarchies, movement direction and location, movement range, and kinematic type.Source: §4.1 · pp. 6–7

4 PhysXGen Framework

• The architecture contains a physical 3D VAE for compressed latent learning and a physics-aware generative process for structured latent generation.The two stages establish information-rich physical representations and then generate physical latents.Source: Figure 4 · p. 7

4.1 Physical 3D VAE Encoding and Decoding

• A residual connection from the physical decoder to the appearance decoder models how physical properties affect geometry and appearance reconstruction.The VAE loss combines appearance, physical, semantic, KL, and regularization terms.Source: §4.1 · p. 7

4.2 Physical Latent Generation

• A dual-branch transformer diffusion model couples structural and physical generation through learnable residual skip connections and optimizes both branches with Conditional Flow Matching.The final diffusion objective combines geometric and physical losses.Source: §4.2 · p. 7
• Table 2 compares structural and physical-property evaluations for PhysXGen against TRELLIS with a separate PhysPre predictor on PhysXNet test sets.The table’s caption identifies PhysPre as a separate physical property predictor after TRELLIS.Source: Table 2 · p. 8

5.1 Implementation details

• PhysXNet is split into 24K training samples, 1K validation samples, and 1K test cases for evaluating generalizability.Training uses AdamW with an initial learning rate of 1 × 10^-4 and eight NVIDIA A100 GPUs.Source: §5.1 · p. 8

5.2 Evaluation Metrics

• Physical evaluation covers absolute scale, material, affordance, kinematics, and function descriptors using Euclidean distance, PSNR, instantiation distance, and cosine-similarity score maps.Kinematics is evaluated through both structural grouping and kinematic parameters.Source: §5.2 · p. 8
• Geometry and appearance evaluation uses 30 random views for mean PSNR, plus Chamfer Distance and F-score with threshold 0.05.The shape metrics are reported as CD ×10^-3 and FS ×10^-2.Source: §5.2 · p. 8

5.3 Quantitative Results

• PhysXGen improves physical-property generation while also enhancing aesthetic quality compared with the separate TRELLIS+PhysPre baseline.Table 2 evaluates both geometry and appearance and physical properties on PhysXNet test sets.Source: §5.3 · p. 9, Table 3 · p. 9
• Ablations show that geometry and appearance features improve physics generation, while geometry–physics correlation in the VAE improves generated geometry.The full dual architecture with joint training achieves strong performance across physical-property generation.Source: §5.3 · p. 9

5.4 Qualitative Results

• Qualitative results show that PhysXGen distinguishes properties across parts and is more stable for neighboring structures, especially in function description, material, and affordance.Figure 5 presents generated physical-grounded assets, while Figure 6 compares methods qualitatively.Source: Figure 6 · p. 9, §5.4 · p. 10

• PhysX-3D combines a human-in-the-loop physics-enabled dataset pipeline with PhysXGen for physical-grounded 3D asset generation.Source: §6 · p. 10
• PhysXGen integrates physical priors into structural-focused architectures and achieves robust generation performance in experiments on PhysXNet.Source: §6 · p. 10
• The method remains limited in learning fine-grained properties and suffers from artifacts.Future work targets more synthetic-to-real data, additional physical properties, and more kinematic types.Source: §6 · p. 10

• PhysXGen preserves the original geometry-space hyperparameters to reuse pretrained weights while using symmetric physical encoder-decoder pairs.Source: §7 · p. 12
• The physical generator uses 14 transformer blocks instead of 24, reducing computational overhead while maintaining satisfactory performance.Source: §7 · p. 12
• PartNet objects receive compatible UV textures retrieved from ShapeNet, with grey color used when no match exists.Source: §7 · p. 12

Annotation paradigms

• Part-based annotation is more robust for occluded and micro-part annotations, whereas segmentation-based annotation can produce inconsistent structural interpretations.Part-based processing is less scalable for assemblies with many parts because each component requires separate rendering.Source: §8 · p. 13

Annotation preprocessing

• Preprocessing normalizes coordinates, merges insignificant components, and then applies part-based annotation for a more efficient robust pipeline.Merged components include fragments with area ≤0.2 or components meeting face count ≤100 and area ≤0.06.Source: §8 · p. 13

Dataset extension

• The procedural workflow generates over 6 million physically plausible assets in PhysXNet-XL.Source: Figure 8 · p. 14

Annotation strategy

• Global-to-local part annotation improves annotation quality.Source: §8 · p. 14

• Procedural generation combines intra-category variation across structurally diverse object classes with cross-category integration of modular drawers and doors.Source: §9 · p. 14
• New components are connected to base structures and rescaled to align with their geometry, supporting structural and physical consistency.Source: §9 · p. 14
• The resulting PhysXNet-XL contains more than 6 million physical 3D objects.Source: §9 · p. 14

10.1 Comparison with GPT-based baseline

• 24%, 64%, 28%, and 72% relative gains over the GPT-based baseline are reported for absolute scale, material, kinematics, and affordance, respectively.The method performs less robustly on function description than GPT-4o, which the authors attribute primarily to the smaller PhysXNet training dataset.Source: §10.1 · p. 14
• PhysXGen shows robust qualitative physical-asset generation, especially for kinematics and affordance, compared with the GPT-based baseline.Source: §10.1 · p. 15, Figure 9 · p. 15

10.2 Qualitative comparison among different architectures

• Integrating geometry–physics correlations into the VAE and diffusion model progressively improves physical-property generation across architectures.Material, kinematics, and affordance predictions become more stable and accurate with fewer artifacts.Source: §10.2 · p. 15

• Absolute-scale prediction is difficult for objects spanning 1–1000 cm, and PhysXGen struggles to remain robust for extremely large objects.Most samples lie below 300 cm, making both linear and direct logarithmic normalization imperfect for preserving useful distinctions.Source: §11 · p. 15
• Material and affordance predictions exhibit spatial inconsistency artifacts that neighboring-region post-processing may not resolve.The material-density range is narrower than absolute scale, but artifact-induced perturbations remain a critical coherence limitation.Source: §11 · p. 15
• Kinematic generation is constrained by uncertain part counts, regression artifacts in parent–child hierarchies, and missing coordinate-to-structure mappings.Source: §11 · p. 16
• Function descriptions have larger normalized errors than other properties because CLIP embeddings are difficult to generate and cannot be disentangled back into prompts.The framework uses CLIP embeddings compressed through a 3D VAE, which supports joint learning but limits interpretability.Source: §11 · pp. 16–17

• All study annotations were performed by the authors, with no external participants or personal or sensitive data collected.The authors state that the work therefore did not constitute human subjects research and did not require IRB approval.Source: §12 · p. 17
• PhysXNet and PhysXNet-XL derive from PartNet and ShapeNet, so users must comply with ShapeNet license terms.Source: §12 · p. 17
• The dataset may contain representational and VLM-induced biases, and safety-critical use requires rigorous validation because physical-prediction errors could cause adverse outcomes.The annotation pipeline uses GPT-4o with subsequent human verification.Source: §12 · p. 17