PhysChoreo: Physics-Controllable Video Generation with Part-Aware Semantic Grounding

1 Introduction

• Existing video generators often imitate motion patterns without causal reasoning, while physics-guided methods struggle with scene modeling and flexible control.Prior approaches commonly predict coarse properties for single objects or use simulators with limited modes.Source: §1 · pp. 1–2
• PhysChoreo aligns part-level text semantics with detailed 3D structures to estimate material models and continuous physical quantities for each object part.Its part-level supervision couples semantics, geometry, and physics to support physical interpretability and controllability.Source: §1 · pp. 2–3
• Temporally continuous edits to external forces and material attributes let simulated objects undergo counterfactual changes, including liquefying on collision or collapsing on landing.The framework combines physics-based simulation with control injected into the generative process.Source: §1 · pp. 2–3
• The text–part–physics dataset provides global descriptions, part-level prompts, and discrete and continuous physical labels as a benchmark for per-part estimation.The continuous quantities include Young’s modulus, Poisson’s ratio, and density.Source: §1 · p. 3
• PhysChoreo advances state-of-the-art continuous-parameter regression and realizes diverse video instructions while maintaining high visual fidelity and temporal coherence.The video evaluation covers physical realism, instruction following, and visual quality.Source: §1 · p. 3

Physical Property Estimation

• Runtime optimization iteratively tunes physical properties using rendered videos and diffusion models, but does not supervise fine-grained physics and incurs substantial time overhead.The passage says these costs make such methods difficult to use for direct video generation.Source: §2 · p. 3
• NeRF2Physics and PUGS rely on generated 2D features from multi-view images, while Pixie injects 3D features but learns static features rather than distributions.The cited approaches are also described as not directly applicable to general 3D representations.Source: §2 · p. 4

Physics-Based Video Generation and Controllable Generative Video Model

• Physics-based video methods use reconstructed scenes or single-image simulation, while controllable video models use conditions such as optical flow or motion; the approaches retain limits in reconstruction demands, temporal control, or physical plausibility.The passage notes that some methods depend on manual physics settings and produce short, monotonous dynamics.Source: §2 · p. 4

3.1 Part-Aware Physics Reconstruction

• Reconstructed object meshes yield sampled surface points whose positional and segmentation features receive global and part-level text guidance before material classes and continuous parameters are decoded.Hierarchical attention applies global guidance before part-specific refinement, while a set Transformer aggregates point context.Source: §3.1 · pp. 4–7

3.2 Part-Level Physics Supervision

• Per-point regression and classification supervise material predictions, while wave-continuity regularization relates predicted longitudinal and shear speeds to material parameters.The speed calculations use Young’s modulus, Poisson’s ratio, and density.Source: §3.2 · p. 7
• Within-part neighborhood losses smooth predicted fields, while triplet regularization separates parts using elastic-response embeddings built from shear and bulk moduli.The smoothness penalty targets spatial discontinuities within semantic parts; the contrastive loss encourages interpart separability.Source: §3.2 · pp. 7–8
• Prompt-assignment supervision aligns each point’s soft assignment with its semantic part label, coupling language conditioning to geometry for interpretability and editability.Source: §3.2 · p. 8

3.3 Physics-Editable Video Generation

• Time-indexed edits to material properties, forces, and momentum let MPM and rigid simulators change dynamics without restarting, while continuity constraints and bounded stiffness support plausible transitions.The resulting trajectories condition a pretrained video model.Source: §3.3 · pp. 8–9

4 Dataset

• The dataset couples part segmentation and global descriptions with part-level physical properties and textual annotations for training and evaluating the prediction model.It is described as one of the largest point-cloud datasets with this combination of annotations.Source: §4 · p. 10

4.1 Dataset Pipeline

• The annotation pipeline uses GPT-5 and a VLM to generate object- and part-level descriptions and properties, then refines them with material-property constraints and manual verification of 9% of samples.GLM-4.5 calibrates correspondences between material labels and physical properties.Source: §4.1 · p. 10

4.2 Dataset Overview

• The dataset contains 9,580 PartNet samples across 24 semantic categories, with part-level materials and physical properties plus mapped simulator material tags.Five percent of examples carry deliberately counterfactual labels, such as a gelatinous blade or metallic flower.Source: §4.2 · p. 10

5.1 Implementation Details.

• On one RTX 5090, reconstruction and pose estimation take around 120 seconds and simulation around 30 seconds; video generation uses Taichi simulation and Wan2.2-Fun-5B-Control.The implementation also uses GPT-5 for multiple rounds of instruction optimization.Source: §5.1 · p. 11

5.2 Physical Property Prediction

• PhysChoreo achieves the best performance on all physical-property prediction metrics and uses text to control properties of specific parts.The comparison evaluates material-model prediction and physical-property errors; Fig. 4 illustrates part-level control through text.Source: Table 1 · p. 12, Figure 4 · p. 12, §5.2 · p. 12

5.3 Image-to-Video Generation

• PhysChoreo achieves the best physical-commonsense score and a significant semantic-alignment lead; despite lower visual quality than Veo 3.1, it has the best average score.The reported user study contains 642 valid selections; Fig. 5's qualitative comparisons are described as consistent with the evaluation scores.Source: Table 3 · p. 13, Table 2 · p. 13, §5.3 · p. 13

5.4 Ablation Studies

• Text features improve prediction, soft assignment improves efficiency, and segmentation priors yield the best performance while increasing training time.The ablations add cross-attention, soft assignment, and segmentation priors to a transformer baseline.Source: §5.4 · pp. 13–14, Table 4 · p. 14
• Removing Lassign slightly lowers performance but markedly increases iterations; Lsmooth affects within-part high-frequency noise, while Lcon removal lowers material accuracy despite similar error.The loss-function ablations report these effects separately.Source: §5.4 · p. 14, Table 5 · p. 15
• Global-property settings fail to maintain correct part-level dynamics, whereas PhysChoreo's predicted part-level properties produce more distinct part-level responses.The comparison uses oracle global settings with the same simulator and video pipeline.Source: §5.4 · p. 14, Figure 7 · p. 15

5.5 Failure case

• Common failures arise during rendering: video models can mishandle lighting and shadows in dynamic motions, creating fine-grained appearance artifacts despite stable simulated trajectories.The failure case is shown in Fig. 7.Source: §5.5 · p. 14

• PhysChoreo demonstrates strong physical-property reconstruction and rich dynamic-video generation, and its prediction model and solver have broader applications such as robot simulation.Source: §6 · p. 15
• The method focuses on independent objects, leaving its adequacy for large-scale scenes limited.Source: §6 · p. 15
• Internal physical states cannot be precisely predicted despite the use of point-sampling methods.Source: §6 · p. 15