Source-linked AI summary
PhysMAS: Physics-Grounded Multi-Agent Synthesis of Compositional 4D Gaussians
Jiang Qin, Chunji Lv, Yangguang Wei, Yang Gao, Ming Liu, Lizhong Ding, Ye Yuan, Yinjie Lei, Changsheng Li
TL;DR
Existing physics-based 4D Gaussian synthesis struggles with heterogeneous parts, interacting objects, and executable material assignments, while SDS optimization requires repeated per-scene evaluation and backpropagation. PhysMAS uses persistent object–part state, part-wise material reasoning, and solver-aware shared-domain MPM candidate screening. It achieves stronger semantic alignment, physical plausibility, and motion/deformation consistency than compared methods while maintaining short runtime.
Problem
Physics-based 4D Gaussian synthesis lacks reliable object–part binding and executable verification for heterogeneous multi-part and interacting multi-object scenes, while SDS requires repeated per-scene score evaluation and backpropagation.
Method
PhysMAS combines an Object-Part Scene Agent, Material Reasoning Agent, and solver-aware skills to compile persistent identities and part-wise profiles into shared-domain MPM simulations and screen candidates.
Results
PhysMAS obtains the highest observed mean frame-wise CLIPsim and lowest paired-reference trajectory, projected-area, and local-strain errors, with a reported two-minute runtime.
Takeaways & Limitations
Persistent object–part state and executable candidate screening support heterogeneous multi-part and interacting multi-object dynamics without per-scene SDS backpropagation.
Takeaways & Limitations
Future work must strengthen cross-view object–part assignment and expand material, joint, and topology-aware interaction coverage.
Abstract
from arXiv · showhide
Efficient, fully automatic, and physically plausible 4D Gaussian synthesis is an important goal for dynamic scene generation. Recent physics-based methods couple 3D Gaussians with the Material Point Method (MPM) to generate physically driven motion, but extending this paradigm to heterogeneous multi-part objects and interacting multi-object scenes remains challenging. Object-level physical assignment collapses distinct parts into a single material state, while one-shot predictions from large language models, vision-language models, or agents neither reliably bind different materials to identified parts nor verify that the resulting MPM configuration is executable. Score Distillation Sampling (SDS)-based parameter optimization, meanwhile, requires repeated per-scene score evaluations and gradient backpropagation, incurring lengthy optimization and potentially yielding suboptimal or unstable solutions. We therefore present PhysMAS, a physics-grounded multi-agent framework. From a motion prompt and four scene views, an Object-Part Scene Agent establishes persistent identities and calls a Material Reasoning Agent for part-wise profiles. It invokes solver-aware skills to bind these identities and profiles to per-particle MPM fields and execute all objects in a shared domain; the framework then screens candidate forward-simulation results. This supports heterogeneous multi-part and interacting multi-object scenes without per-scene diffusion-score backpropagation. Extensive experiments demonstrate that, compared with recent physics-based 4D Gaussian baselines that rely on SDS, PhysMAS achieves better semantic alignment and perceived physical plausibility while requiring less runtime.
Introduction
PhysMAS addresses compositional 4D Gaussian synthesis by preserving object–part identities, assigning part-wise physical profiles, and screening solver-executable candidates. It supports heterogeneous multi-part and interacting multi-object dynamics while avoiding per-scene SDS backpropagation and reducing runtime.
- Scope: The framework is designed for efficient simulator-driven 4D Gaussian synthesis using explicit 3DGS representations coupled with MPM.Figure 1 depicts vertical impact and gravity-driven toppling from the same registered scene bundle.
- Motivation: Physics-based 4D Gaussian pipelines struggle to preserve distinct material states across heterogeneous parts and interacting objects.Object-wide assignment collapses heterogeneous parts, while missing persistent identities prevents consistent binding across perception and simulation.
- Framework: PhysMAS uses an Object-Part Scene Agent and Material Reasoning Agent to establish persistent identities and part-wise simulator profiles.The framework grounds compound object–part labels and obtains physical parameters for each part.
- Framework: Solver-aware MPM skills bind identities and profiles to particles, execute all objects in a shared domain, and screen execution records.This provides a structured interface between multimodal reasoning and executable simulation.
- Results: PhysMAS obtains the highest observed mean frame-wise CLIPsim and lowest paired-reference trajectory, projected-area, and local-strain errors among compared methods.The reported runtime is two minutes and is substantially faster than evaluated long per-scene SDS optimization baselines.
Related Work
Prior work spans dynamic Gaussian representations, diffusion-guided 4D generation, physics-based property inference, and semantic or segmented Gaussian attributes. PhysMAS distinguishes itself by maintaining an object-scoped part table through profile proposal, particle compilation, and executable candidate screening.
- Dynamic and Generative Methods: Dynamic Gaussian and diffusion-guided methods target reconstruction or plausible animation rather than material-conditioned object–part simulation.The related methods include 4D-GS, Deformable 3D Gaussians, Spacetime Gaussians, L4GM, DreamFusion-derived systems, and CAT4D.
- Physics-Grounded Methods: Physics-grounded methods connect Gaussian appearance to MPM, while other systems automate property inference or address semantic and physical attributes.The surveyed landscape includes PhysGaussian, PhysDreamer, DreamPhysics, MotionPhysics, OmniPhysGS, PhysSplat, and PhysGM.
- PhysMAS Positioning: PhysMAS introduces an object-scoped part table that remains addressable through profile proposal, particle compilation, and executable candidate screening.This connects part structure to downstream simulation and selection rather than stopping at perception or semantic labeling.
- Adjacent Directions: Related work separately covers open-vocabulary segmentation, Gaussian identity or language semantics, physics-scene simulation, and language-agent interaction.These lines of work address complementary components of compositional physical scene synthesis.
Method
PhysMAS converts prompts, calibrated views, and aligned static Gaussians into persistent object–part state, material evidence, compiled candidates, and shared-domain MPM rollouts. Deterministic skills execute, repair, validate, and render accepted candidates while preserving identities and part structure.
- Inputs and State: The input comprises a motion prompt, four calibrated RGB views, aligned static 3DGS, and prescribed output cameras.The scene agent maintains object–part identity and supported plans, while the material agent supplies row-wise profiles.
- Object–Part Representation: Each Gaussian carries an object identity and object-scoped part row, keeping identically named parts of different objects distinct.Semantic and residual rows form the active material-state set, while one-part objects use one semantic row.
- Scene Construction: The scene agent grounds masks and cross-view object indices, fixes compound labels, and constructs a validated plan of roles, targets, directions, magnitudes, and intervals.GroundingDINO and SAM 2 provide masks and confidences before deterministic propagation and compiler validation.
- Material Reasoning: The material agent combines visual material evidence with crop-conditioned E/ν statistics to propose validated profiles from the accepted simulator domain.Candidate construction retains fixed profile branches and parameter variants while preserving scene state, identities, and actions.
- Compilation and Simulation: The compiler forms simulator input and schedules, binding candidates to per-particle identity, parameters, mass, and numerical settings.APIC transfers advance the MPM state, and every particle samples a common grid for inter-object coupling.
- Rendering: PhysMAS transports simulated particle states to dynamic Gaussians for rendering while keeping opacity and appearance fixed and updating viewing direction through deformation.Gaussian-associated particles are rendered, while filled particles support simulation.
- Executable Candidate Selection: The candidate-selection loop executes unseen candidates, screens validity and quality, applies fixed repairs, and returns the highest-quality audited rollout.Validity requires resolved actions, successful termination, stable numerical and runtime status, and fixed geometric criteria.
Experiments
PhysMAS is evaluated on multi-part and multi-object scenes using semantic, physical, runtime, qualitative, and ablation analyses. Across these evaluations, it combines stronger reported alignment and physical plausibility with efficient execution and benefits from part-aware identity and structure-preserving projection.
- Evaluation protocol: PhysMAS evaluates preconstructed 3DGS scenes from PhysGaussian, Objaverse, and PhysAssets, covering six material profiles for multi-part cases and shared-grid interactions for multi-object cases.Multi-profile cases contribute to each applicable profile category, while multi-object results aggregate scenes with at least two identifiable interacting objects.
- Quantitative comparison: PhysMAS obtains the strongest reported multi-part profile results, with the highest CLIPsim and UPR for all six profiles rather than only one favorable material category.The reported macro result is 0.2787/36.8%, and larger UPR differences on Metal and Foam also align with human preference.
- Quantitative comparison: PhysMAS achieves the highest observed mean frame-wise CLIPsim and the lowest paired-reference image-plane trajectory, projected-area, and local-strain errors among compared methods.These comparisons span the multi-part and multi-object evaluations.
- Qualitative comparison: PhysMAS more clearly preserves cake plate/layer separation and sunflower part-dependent bending than the compared methods in synchronized qualitative sequences.The broader gallery also distinguishes material- and part-dependent responses within objects from interactions between distinct objects.
- Ablation study: The object-part ablation reduces macro CLIPsim/UPR from 0.2787/42.5% to 0.2616/25.5%, with the largest profile change on Foam.The ablation replaces agent-owned object-part state and part-aware compilation with an object-level variant.
- Ablation study: Removing structure-preserving projection lowers macro CLIPsim/UPR to 0.2717/32.0%, while the full model gains 17, 14, and 12 UPR points on Jelly, Snow, and Foam.Without projection, the pineapple crown-body support structure shows greater distortion during landing.
Discussion
PhysMAS addresses compositional 4D Gaussian synthesis by automating object–part organization and part-wise material assignment, while compiling and screening executable MPM candidates.
- Persistent object–part identities organize heterogeneous parts and multiple objects for physics-grounded 4D Gaussian synthesis.Material profiles provide the interface between multimodal reasoning and MPM simulation.
Conclusion
PhysMAS combines persistent object–part identities, part-wise material profiles, and solver-aware skills for shared-domain MPM simulation. Compared with the evaluated methods, it improves semantic alignment, perceived physical plausibility, and motion and deformation consistency while retaining short runtime.
- PhysMAS uses two agents and solver-aware skills to carry persistent identities and part-wise profiles into shared-domain MPM simulation without per-scene SDS backpropagation.
- Experiments show superior semantic alignment, perceived physical plausibility, and reference-video motion and deformation consistency among compared methods.
- PhysMAS maintains a short runtime while supporting compositional 4D Gaussian synthesis.
Agent Contracts and Object–Part State
PhysMAS separates agent responsibilities through persistent object–part state and deterministic compiler routines. The scene agent controls identity, planning, tool use, and acceptance, while the material agent supplies row-wise evidence for validated simulation candidates.
- Agent Contracts: The Object-Part Scene Agent alone updates scene identity and plans, invokes tools, and accepts rollouts.Candidate construction, compilation, simulation, repair generation, gating, ranking, and auditing remain fixed routines.
- Agent Contracts: The prompt contract preserves stable object IDs across views and creates part IDs only within parent objects.Identically named parts from different objects remain separate rows, and material evidence is requested for every active row.
- Object–Part State: Scene plans record source and target identities, semantic roles, actions, directions, magnitudes, and active intervals before deterministic validation.Gravity, colliders, cameras, grid resolution, and rendering settings remain in the fixed run specification.
- Agent Contracts: The Material Reasoning Agent is called once per active object–part row and returns confidence-bearing visual-material and simulator-profile evidence with continuous E/ν statistics.It preserves the supplied identity and abstains when required semantic evidence is invalid.
- Object–Part State: The protocol serializes row evidence as ψr and passes normalized records between agents and fixed routines through a backend-independent semantic schema.
Material Profiles and Particle Compilation
PhysMAS maps structured object–part records and material evidence into solver-compatible particle fields and candidate simulations. Fixed mappings, guards, constitutive branches, and repair rules constrain profiles and preserve executable physical configurations.
- Parameter Domains: The implementation combines ordinary part-row domains with enumerated composite-transform domains to form Ksim rather than applying one global clamp.Ordinary rows use local solver ranges, while composite transforms permit distinct whole-object ν and density ranges.
- Structured Records: Structured records link persistent objects, compound object–part rows, scene plans, material evidence, candidate controls, metadata, and execution diagnostics to their owners and consumers.
- Particle Compilation: The compiler converts each particle’s E and ν values into Lamé parameters for solver execution.The transformation uses µ = E/[2(1 + ν)] and λ = Eν/[(1 + ν)(1 − 2ν)].
- Material Profiles: Six simulator profiles select distinct elastic and inelastic constitutive branches, including fixed-corotational metal, Drucker–Prager sand and snow, and viscoplastic foam.
- Evidence Aggregation: Evidence aggregation applies fixed source coefficients, validates VLM records through confidence and probability guards, and selects labels using deterministic tie-breaking.The material mapping includes schema, crop predictor, VLM, role, and whole-object evidence sources.
- Candidate Construction: Candidate construction uses support and soft-token rules, while selected visual-label density priors supply density rather than language-model predictions.
Solver-Aware Skills and Executable Screening
PhysMAS uses deterministic, solver-aware compilation and screening to turn object–part material proposals into executable MPM candidates. Fixed predicates, scoring, repair rounds, and artifact audits enforce valid simulations while preserving distinct object interactions.
- Plan compilation: The compiler canonicalizes scene-plan directions, validates targets, and instantiates actions over explicit spatial regions and intervals.Gravity and collider entries from the fixed run specification are combined with the validated plan.
- Candidate repair: The framework retains at most six initial candidates and allows at most three candidate–execute–observe rounds, with up to three repairs per round.Repairs follow a fixed priority order, and each candidate ID is evaluated once.
- Action implementation: Supported actions are implemented as bounded particle velocity conditions or position blends rather than unconstrained symbolic commands.The serialized torque token is implemented through a tangential velocity field instead of a directly applied mechanical moment.
- Multi-object execution: Multi-object rollouts disable interface bonding but retain object-scoped interaction through the shared MPM grid.This preserves grid-mediated coupling while avoiding support–part bonding in multi-object evaluation.
- Scoring: Solver-aware scoring combines profile, role, density, diversity, video, projection, and bonding terms while separately checking support and contact predicates.The evaluator distinguishes support rows from compliant rows and assigns solver scores using frozen lookup rules.
- Candidate screening: Candidates are admitted only after resolved planning, successful termination, numerical validity, geometric checks, support constraints, and Qexec(k) ≥ 0.72.Admitted candidates are ranked by the same fixed score before mandatory artifact auditing.
Evaluation Protocol
The evaluation compares generated motion against fixed synchronized MPM references using trajectory, projected-area, and local-strain diagnostics, alongside blinded human preference studies. Metrics are aggregated per case, with explicit validity rules preventing missing measurements from becoming artificial successes.
- Reference videos: Each case uses a fixed synthetic MPM reference generated once from a preregistered configuration matched to the compared outputs.References are unavailable to material reasoning, candidate construction, screening, and generation.
- Motion diagnostics: Trajectory nRMSE is computed only over point–frame pairs jointly visible with finite tracks in both videos.Frames with fewer than 16 valid tracked points are excluded.
- Deformation diagnostics: Projected-area log-RMSE measures relative target-mask area change, with an added constant keeping empty associated masks finite.Failed target association produces an invalid record.
- Deformation diagnostics: Local-strain RMSE uses an eight-neighbor least-squares deformation gradient and retains local stretch, compression, and shear while removing rigid translation.Only jointly valid neighborhoods enter the metric, and invalid runs remain failures rather than receiving zero error.
- Aggregation: The three diagnostics quantify image-plane trajectory, projected deformation scale, and local strain relative to synchronized reference videos.Multi-object results average registered objects equally within scenes before averaging across scenes.
- Human preference: The human study uses blinded randomized 5AFC and 3AFC protocols with 20 researchers, selecting the video best matching the prompt and showing plausible natural motion.UPR gives each case equal weight, and multi-object and material-profile sets are aggregated separately.
Limitations and Future Work
The paper identifies three future directions: stronger cross-view grounding, broader constitutive and interaction coverage, and finer spatial or temporal resolution.
- Cross-View Grounding and Scene Inputs: Future cross-view grounding should add occlusion-aware association, confidence-calibrated residual handling, and additional observations.These directions target persistent object–part assignment.
- Constitutive and Interaction Coverage: Broader behavior coverage could come from new material branches, joints, and topology-aware updates beyond the six-profile interface.The stated goal is to support a wider range of part behaviors and interactions.
- Spatial and Temporal Resolution: Adaptive particle or grid resolution and temporal planning could support finer interactions and longer action schedules.These extensions are described as preserving the current agent–compiler interface.
Additional Results
Figures 6–11 provide 42 additional PhysMAS rollouts spanning heterogeneous multi-part objects and multi-object scenes under varied motion configurations.
- Additional Results: Figures 6–11 contain 42 additional PhysMAS rollouts presented as temporally ordered frame sequences.Each row shows five frames.
- Additional Results: The additional rollouts cover both heterogeneous multi-part objects and multi-object scenes.They span varied motion configurations.