Source-linked AI summary
PhyGenHOI: Physically-Aware 4D Generation of Dynamic Human-Object Interactions
Omer Benishu, Gal Fiebelman, Sagie Benaim
TL;DR
PhyGenHOI tackles the challenge of generating 4D human-object interactions that are both visually faithful and physically plausible. It couples motion-diffusion human generation with MPM object simulation under a unified 3DGS representation, using attraction, contact-driven re-simulation, and masked video guidance. The framework eliminates ghosting and interpenetration artifacts while outperforming baselines in text alignment, physical plausibility, and contact quality.
Problem
Existing 4D generation methods often lack physics, while kinematic approaches treat objects as static props and miss coupled human-object dynamics.
Method
The framework represents the human as a motion-diffusion semantic agent and the object as an MPM physical agent under unified 3DGS, coordinating them with attraction, contact re-simulation, and masked Video-SDS.
Results
PhyGenHOI outperforms generative and animation baselines in text alignment, physical plausibility, contact quality, and visual fidelity while eliminating ghosting and interpenetration artifacts.
Takeaways & Limitations
The neuro-physical coupling enables dynamic post-contact object responses while preserving semantically responsive human motion across diverse interaction scenarios.
Takeaways & Limitations
The current formulation targets impulsive contact with discrete momentum transfer; continuous-contact interactions requiring sustained force modeling remain outside its scope.
Abstract
from arXiv · showhide
We address the task of generating physically accurate and visually faithful 4D Human-Object Interaction (HOI). Given a static 3D human and target object represented as 3D Gaussian Splats (3DGS), our goal is to synthesize dynamic scenes where the human actively engages with the object through actions, such as punching or kicking, in accordance with a given input text. To this end, we introduce PhyGenHOI, a novel framework that couples generative human motion with an explicit physical object simulation. We model the human as a semantic agent driven by a Motion Diffusion Model (MDM) and the object as a physical agent simulated via the Material Point Method (MPM), utilizing 3D Gaussians as a unified, differentiable representation. We supervise their interaction through three coupled mechanisms: (1) A Windowed Attraction Loss that temporally synchronizes generative motion to intercept the object; (2) A Contact-Driven Re-simulation step that triggers physically consistent momentum transfer upon impact; and (3) A Masked Video-SDS objective that injects video-based priors to enhance contact fidelity. Experiments show PhyGenHOI generates physically consistent 4D HOI across diverse actions, humans, and objects, outperforming baselines. Project page and videos: https://omerbenishu.github.io/PhyGenHOI/
1 Introduction
PhyGenHOI addresses the gap between semantically coherent but physically implausible 4D generation and kinematic interaction methods that treat objects as static props. It couples generative human motion with physical object simulation through a unified 3DGS representation and targeted interaction mechanisms.
- Motivation: Physically faithful 4D human-object interaction remains challenging because generative methods lack physics, while kinematic methods reduce interaction to geometric constraints.These limitations produce causal anomalies, static-prop behavior, and missing coupled interaction logic.
- Framework: PhyGenHOI uses 3D Gaussian Splatting as a shared substrate for an SMPL-constrained motion-diffusion human agent and an MPM-simulated physical object agent.The human generates diverse text-aligned actions, while object Gaussian kernels map directly to particles for differentiable simulation.
- Interaction Synthesis: A Windowed Attraction Loss guides generated human motion toward the object across space and time to synchronize semantic intent with interception.This mechanism coordinates agents that initially move independently.
- Interaction Synthesis: Contact Detection and MPM Re-simulation update the object after collision to model momentum transfer and material deformation.The method targets impulsive actions such as kicking, punching, and pushing.
- Interaction Synthesis: Temporally-Masked Video-SDS adds visual priors around contact frames to improve interaction fidelity without disrupting physically grounded motion.The framework combines this objective with attraction guidance and contact-driven re-simulation.
- Results: PhyGenHOI outperforms generative and animation baselines in text alignment, physical plausibility, contact quality, and visual fidelity while eliminating ghosting and interpenetration artifacts.The comparison spans diverse dynamic interaction scenarios.
3 Method
PhyGenHOI uses 3D Gaussian Splatting as a shared differentiable representation while synthesizing human motion semantically and object motion physically. It coordinates the agents through motion-prior guidance, contact detection, and contact-driven physical re-simulation.
- 3.1 Scene Representation: 3D Gaussian Splatting provides a shared representation for joint rendering and optimization of the human and object.The human and object are modeled with Gaussian primitives in a unified differentiable pipeline.
- 3.1 Scene Representation: The human is represented by SMPL-bound Gaussians deformed through Linear Blend Skinning, enabling optimization of pose parameters.Each Gaussian is associated with joints using skinning weights.
- 3.1 Scene Representation: The object is modeled as MPM particles whose positions evolve through continuum mechanics, so its motion is determined by physical simulation.An initial forward MPM simulation produces a physically consistent free-motion trajectory before contact.
- 3.2 Agent Motion Synthesis: Human motion is initialized with Human Motion Score Distillation, which pulls text-conditioned motion toward the natural-motion manifold before object coordination.The independently generated motion may not initially produce contact with the object.
- 3.3 Physically-Aware Interaction Synthesis: The Windowed Attraction Loss selects the highest-velocity joint and its peak-velocity frame, then guides that joint toward the object within a temporal window.The Gaussian weighting concentrates guidance near contact while preserving natural wind-up and follow-through.
- 3.3 Physically-Aware Interaction Synthesis: Contact detection uses joint and object bounding-box overlap plus a Gaussian-distance condition, after which momentum transfer updates the object and triggers MPM re-simulation.The object is forward-simulated from the contact frame to produce a physically consistent post-impact trajectory.
4 Experiments
PhyGenHOI is evaluated across diverse human-object interactions against generative and animation baselines using qualitative, automated, and perceptual measures. It achieves physically plausible contact and object responses while avoiding artifacts associated with competing approaches.
- Interaction Generation: PhyGenHOI coordinates dynamic human motion with object trajectories across punching, kicking, and pushing scenarios.The generated object responds according to its material properties across varied initial conditions.
- Evaluation Setup: The evaluation covers 10 human-object interaction scenarios spanning different humans, objects, and interactions.
- Quantitative Evaluation: PhyGenHOI achieves the highest scores on all reported metrics, including VQA Physics of 0.253 versus 0.196 and ViCLIP of 0.295 versus 0.256.The comparison also includes user-study MOS scores across physical plausibility, contact quality, motion naturalness, and photorealism.
- Ablation Study: Ablations associate each component with a distinct failure mode, including missed targets without Attraction and lost material-aware dynamics without MPM.Removing Contact breaks collision causality, removing Video-SDS leaves penetration artifacts, and removing MDM produces anatomically implausible motion.
5 Conclusion
PhyGenHOI couples generative human motion with MPM-based physical simulation under a shared 3DGS representation. The resulting framework eliminates major interaction artifacts, supports post-contact object responses, and outperforms existing baselines on key interaction criteria.
- PhyGenHOI couples generative human motion with MPM-based physical simulation under a shared 3DGS representation.
- The framework eliminates ghosting and interpenetration artifacts while enabling dynamic post-contact object responses.
- PhyGenHOI outperforms existing baselines in text alignment, physical plausibility, and contact quality.
B.1 Implementation Details
The implementation uses a single H200 GPU and combines pretrained human and object Gaussian representations with a multistage processing pipeline. Rendering produces final sequences at 20 FPS.
- Hardware and Runtime: All experiments run on a single NVIDIA H200 GPU, with approximately 74 minutes required per scene.Human motion optimization takes about 10 minutes, MPM simulation 4 minutes, and Video-SDS refinement approximately 1 hour.
- Hardware and Runtime: Final 4D sequence rendering achieves 20 FPS.
- Human Representation: The human is represented by 3D Gaussians bound to the SMPL parametric body model and initialized from a pretrained HUGS model.
- Object Representation: Object 3DGS representations come from the DreamPhysics dataset for the blue ball and Trellis single-image reconstruction for other objects.The representations use standard 3D Gaussian Splatting with default parameters.
B.2 Human Motion Score Distillation Details
Human motion is represented as a sequence of SMPL-space parameters and generated with a pretrained Motion Diffusion Model. Score Distillation uses bounded diffusion timesteps, timestep weighting, and classifier-free guidance.
- Motion Prior: The human motion prior is a pretrained Motion Diffusion Model operating directly in SMPL pose space.
- Motion Representation: Each motion frame contains root translation, global 6D orientation, and per-joint pose parameters for 24 joints.
- Motion Representation: The method generates sequences of 40 frames at 20 FPS.
- Score Distillation: Score Distillation samples diffusion timesteps uniformly from 0 to 100, weights them with w(t) = 1−ᾱ_t, and uses classifier-free guidance scale 7.5.
B.3 Contact Detection and Re-simulation Details
Contact is detected from joint-level spatial overlap and proximity, then used to update the object's velocity and rerun the physical simulation. Video-based optimization is concentrated around contact frames to improve fidelity while preserving motion elsewhere.
- Upon contact, the framework estimates human velocity from joint displacement and defines the contact normal toward the object's center of mass.The normal is computed from contacting object Gaussians toward the object center of mass.
- The object's post-impact velocity adds a restitution-scaled relative normal velocity to its incoming velocity.The update is Vpost = Vobj + (1 + e) · vin · n, with vin defined as relative velocity along the contact normal and e as restitution.
- After contact detection, a forward MPM simulation runs from tc to T using the computed post-impact velocity to produce the final object trajectory.The simulation uses a 64 grid, timestep 4 · 10^-5, 1250 steps per frame, and restitution e = 0.6.
- Temporally masked Video-SDS optimizes only frames within [tc − ∆t, tc + ∆t], with ∆t = 1, focusing visual refinement on contact.The stated goal is to preserve the motion prior's influence outside the contact window.
B.5 Optimization Details
Optimization proceeds in three stages: initialize text-aligned human motion, coordinate it with the object, and refine contact fidelity using temporally masked Video-SDS.
- Stage 1: Motion Initialization: The first stage optimizes human pose parameters with LHMSD alone for 100 iterations using Adam at learning rate 0.005.
- Stage 2: Human-Object Coordination: The second stage runs 200 iterations with a combined objective for human-motion alignment and attraction-based human-object coordination.The attraction weight is λattr = 1.0 and the motion objective weight is λHMSD = 10.0.
- Stage 2: Human-Object Coordination: After coordination, contact detection and MPM re-simulation are performed, and the resulting object trajectory is fixed for later optimization.
- Stage 3: Video-SDS for Contact Fidelity: The third stage optimizes human pose with temporally masked Video-SDS for 3000 iterations at learning rate 0.001.Video diffusion and rendering settings are specified separately in the implementation details.
B.6 Comparisons and Ablations
The comparison setup covers additional evaluations and ablation variants, using author-provided baseline implementations with adaptations for the 3DGS-to-mesh input mismatch.
- Additional Comparisons: Extended comparisons evaluate the framework across diverse actions against generative and animation baselines.
- Baseline Comparisons: The 4D-fy comparison follows the authors' original configuration while increasing motion with system.loss.lambda_sds_video = 0.5.
- Baseline Comparisons: AnimateAnyMesh scenes are converted from 3DGS to GLB meshes, with objects positioned closer because of mesh-input and motion-related setup constraints.
- Ablation Variants: Five ablations remove or replace Video-SDS, attraction, MDM, or contact-driven re-simulation to isolate their effects.The supplied description specifies that w/o Video-SDS skips stage 3 and w/o Attraction sets λattr = 0.
C Additional Qualitative Results
Across the full benchmark, the method coordinates human motion with dynamic objects across diverse action types, while baselines often struggle with causality or contact fidelity.
- PhyGenHOI successfully coordinates the human agent with the dynamic object across diverse action types in the full benchmark comparisons.
D Limitations
PhyGenHOI is scoped to impulsive human-object interactions and has several stated boundaries, including continuous contact, complex object geometries, and one-way physical feedback. The authors also note potential misuse for convincing deepfakes in downstream applications.
- Continuous-contact interactions fall outside the current scope because they require sustained-force modeling rather than instantaneous impact.
- The center-of-mass attraction target may be suboptimal for complex geometries requiring contact at specific surface regions.
- Although MPM models object deformation, the SMPL human remains kinematic and does not respond to reaction forces or secondary collisions.
- Improved realism in punching and kicking could theoretically be misused to generate convincing deepfakes for disinformation or malicious narratives.
- The authors identify gated release and other future mitigation strategies as important for reducing downstream misuse risks.