Source-linked AI summary
ArtHOI: Articulated Human-Object Interaction Synthesis by 4D Reconstruction from Video Priors
Zihao Huang, Tianqi Liu, Zhaoxi Chen, Shaocong Xu, Saining Zhang, Lixing Xiao, Zhiguo Cao, Wei Li, Hao Zhao, Ziwei Liu
TL;DR
Physically plausible articulated human-object interaction remains difficult because zero-shot video-diffusion methods are largely restricted to rigid objects and lack explicit 4D geometric reasoning. ArtHOI reconstructs articulated 4D scenes from monocular diffusion-generated videos using flow-based segmentation and decoupled optimization, and reports stronger contact, penetration, and motion outcomes across articulated scenarios. Its demonstrated scope is single-part articulated objects, with failures on low-texture or reflective surfaces, complex articulations, and long sequences.
Problem
Zero-shot articulated human-object interaction synthesis from monocular video without 3D supervision remains difficult because existing methods largely handle rigid objects and lack explicit 4D geometric reasoning.
Method
ArtHOI reconstructs full 4D articulated scenes from diffusion-generated monocular videos using flow-based part segmentation and a decoupled pipeline that recovers object articulation before conditioning human-motion synthesis.
Results
ArtHOI produces temporally coherent articulated interactions with explicit contact and reports 75.64% contact, 0.08 penetration error, and 0.87 motion smoothness.
Takeaways & Limitations
The reconstruction-based design extends zero-shot interaction synthesis beyond rigid manipulation toward physically grounded articulated human-object scenes.
Takeaways & Limitations
Experiments focus on single-part articulated objects, while performance is constrained by unreliable flow on low-texture or reflective surfaces, complex articulations, and long-term error accumulation.
Abstract
from arXiv · showhide
Synthesizing physically plausible articulated human-object interactions (HOI) without 3D/4D supervision remains a fundamental challenge. While recent zero-shot approaches leverage video diffusion models to synthesize human-object interactions, they are largely confined to rigid-object manipulation and lack explicit 4D geometric reasoning. To bridge this gap, we formulate articulated HOI synthesis as a 4D reconstruction problem from monocular video priors: given only a video generated by a diffusion model, we reconstruct a full 4D articulated scene without any 3D supervision. This reconstruction-based approach treats the generated 2D video as supervision for an inverse rendering problem, recovering geometrically consistent and physically plausible 4D scenes that naturally respect contact, articulation, and temporal coherence. We introduce ArtHOI, the first zero-shot framework for articulated human-object interaction synthesis via 4D reconstruction from video priors. Our key designs are: 1) Flow-based part segmentation: leveraging optical flow as a geometric cue to disentangle dynamic from static regions in monocular video; 2) Decoupled reconstruction pipeline: joint optimization of human motion and object articulation is unstable under monocular ambiguity, so we first recover object articulation, then synthesize human motion conditioned on the reconstructed object states. ArtHOI bridges video-based generation and geometry-aware reconstruction, producing interactions that are both semantically aligned and physically grounded. Across diverse articulated scenes (e.g., opening fridges, cabinets, microwaves), ArtHOI significantly outperforms prior methods in contact accuracy, penetration reduction, and articulation fidelity, extending zero-shot interaction synthesis beyond rigid manipulation through reconstruction-informed synthesis.
1 INTRODUCTION
ArtHOI addresses the difficulty of synthesizing articulated human-object interactions from monocular video without 3D supervision by reconstructing temporally coherent 4D scenes. Its flow-guided, decoupled pipeline improves physical plausibility and articulated motion across diverse scenarios.
- Articulated interactions remain challenging because part-wise kinematic constraints and motion dependencies are difficult to capture from monocular inputs without 3D supervision.
- Existing zero-shot video-diffusion methods are limited to rigid manipulation and lack explicit 4D reconstruction, producing physically implausible or geometrically inconsistent interactions.
- ArtHOI formulates interaction synthesis as 4D reconstruction from diffusion-generated 2D videos, using inverse rendering to recover temporally coherent and geometrically consistent scenes.
- ArtHOI simultaneously supports RGB rendering, articulated object modeling, physical constraints, and zero-shot generalization without 3D supervision.
- The two-stage pipeline first reconstructs articulated object dynamics with optical-flow-based segmentation and kinematic constraints, then synthesizes human motion conditioned on recovered object states.
2 RELATED WORK
Prior HOI methods range from data-driven and supervised interaction synthesis to zero-shot video-diffusion approaches and articulated reconstruction. However, existing methods generally rely on interaction data, known structures, object-centric assumptions, or primarily rigid motion, leaving articulated HOI insufficiently addressed.
- Early HOI methods used motion-capture datasets paired with object trajectories to synthesize physically plausible interactions.
- Language- and waypoint-conditioned methods such as CHOIS, LINGO, and InteractAnything require interaction-specific training data and known object kinematics.
- Zero-shot methods reduce data dependence through foundation-model priors, but approaches such as ZeroHSI assume 6D rigid manipulation and treat dynamic objects as single rigid bodies.
- Template-based articulated reconstruction uses predefined part decompositions or kinematic chains, limiting applicability to novel objects and often requiring multi-view inputs.
- Unsupervised reconstruction methods discover parts from optical flow, scene flow, or point-cloud motion, but operate object-centrically and ignore human-object geometric and physical signals.
- Video-diffusion-based 4D reconstruction and generation methods generally focus on rigid scenes or simple object motions, with performance degrading for part-wise articulated kinematics.
3 METHODOLOGY
ArtHOI formulates articulated human-object interaction synthesis as monocular 4D reconstruction, using flow-based part segmentation and a decoupled pipeline to recover object articulation before human motion.
- 3 METHODOLOGY: ArtHOI reconstructs a full 4D articulated scene from monocular video priors, using the generated 2D video as inverse-rendering supervision.The reconstruction-based formulation introduces geometric and kinematic constraints without 3D supervision.
- 3 METHODOLOGY: Stage II refines human motion while conditioning on the fixed reconstructed object scaffold and recovered contact geometry.Solving object articulation first avoids unstable joint optimization under monocular ambiguity.
- 3 METHODOLOGY: Stage I identifies articulated parts with optical flow, transfers masks to 3D Gaussians, and reconstructs object articulation under kinematic constraints.Flow magnitude separates dynamic from static points; SAM-guided masks, back projection, connectivity refinement, and binding pairs produce coherent partitions.
- 3 METHODOLOGY: The reconstruction objective combines rendering fidelity, kinematic consistency, articulation tracking, and binding constraints to align the 4D object with the video prior.Kinematic regularization prevents implausible configurations such as articulated parts drifting apart.
- 3 METHODOLOGY: Human-motion refinement uses contact, pose-prior, and foot-sliding terms to preserve physical plausibility under reconstructed geometry.Contact targets are derived from Stage I, while the pose prior limits overfitting and foot constraints discourage unrealistic motion during sustained ground contact.
4 EXPERIMENTS
ArtHOI is evaluated against motion-synthesis, contact-aware, zero-shot, and articulated-object baselines on interaction quality, reconstruction, and articulation dynamics. It achieves strong semantic alignment, contact, penetration, and rotation accuracy, while ablations support the two-stage design.
- Experimental Setup: The evaluation covers reconstruction quality, physical plausibility, temporal coherence, interaction quality, and articulated-object dynamics accuracy.Baselines include TRUMANS, LINGO, CHOIS, ZeroHSI, D3D-HOI, and 3DADN across articulated and rigid-object settings.
- Interaction Quality: ArtHOI achieves the highest X-CLIP score of 0.244, lowest foot sliding of 0.31, and highest contact percentage of 75.64%.Smoothness comparisons favor zero-shot methods because non-zero-shot baselines often have minimal object contact.
- Interaction Quality: ArtHOI records the lowest penetration error of 0.08 while maintaining competitive smoothness of 0.87 and higher contact than the cited baselines.The reported contact rates are 75.64% for ArtHOI, 61.95% for ZeroHSI, and 39.72% for CHOIS.
- Articulated Object Dynamics: For articulated-object dynamics, ArtHOI reaches a mean rotation error of 6.71°, reducing error by 73.3% versus D3D-HOI and 68.3% versus 3DADN.Maximum and minimum rotation errors are also lowest for ArtHOI: 21.41° and 0.58°.
- Rigid Object Generalization: On rigid-object interactions, ArtHOI achieves the best foot sliding of 0.28, contact percentage of 76.18%, and penetration percentage of 0.06%.The comparison uses monocular RGB video against SAM3D+FP and ZeroHSI.
- Ablation Studies: Ablations show that replacing two-stage decoupling with joint optimization reduces X-CLIP from 0.244 to 0.187, contact from 75.64% to 61.45%, and increases mean rotation error from 6.71° to 12.34°.Removing the kinematic loss further reduces contact to 59.82% and worsens foot sliding from 0.31 to 0.58.
5 DISCUSSION AND CONCLUSION
ArtHOI extends zero-shot human-object interaction synthesis to articulated objects by reconstructing full 4D scenes from monocular video priors. The framework improves geometric consistency, contact accuracy, and temporal coherence, but remains constrained by object complexity, tracking reliability, sequence length, and camera motion.
- Conclusion: ArtHOI reconstructs full 4D articulated scenes from monocular video priors, addressing zero-shot methods’ restriction to rigid objects.Its reconstruction-based design explicitly models part articulation and human-object contact.
- Conclusion: The method achieves superior geometric consistency, contact accuracy, and temporal coherence compared with prior zero-shot approaches.
- Limitations: Experiments focus on single-part articulated objects, limiting direct evaluation of hierarchical articulated structures.
- Limitations: Optical-flow tracking can fail on low-texture or reflective surfaces, and the resulting distortions propagate into articulation prediction.
- Limitations: The method struggles with multiple degrees of freedom, non-rigid articulations, long sequences, and moving cameras.Longer sequences accumulate articulation-extraction errors, while moving cameras mix ego-motion with object articulation.
- Applications: ArtHOI has practical applications in robotics, virtual and augmented reality, and embodied AI by generating realistic interaction data and content.The paper describes uses including manipulation-policy training, simulation-to-real transfer, virtual environments, telepresence, and scalable embodied-AI data synthesis.