Source-linked AI summary
HSImul3R: Physics-in-the-Loop Reconstruction of Simulation-Ready Human-Scene Interactions
Yukang Cao, Haozhe Xie, Fangzhou Hong, Long Zhuo, Zhaoxi Chen, Liang Pan, Ziwei Liu
TL;DR
HSImul3R addresses the perception–simulation gap in human–scene interaction reconstruction, where visually plausible results can violate physical constraints. It uses a physics simulator as an active supervisor in a bi-directional pipeline to refine human motion and scene geometry, and reports stable simulation-ready reconstructions with humanoid-robot deployment.
Problem
Existing human–scene interaction reconstruction methods can produce visually plausible results that violate physical constraints, limiting their suitability for embodied AI applications.
Method
HSImul3R uses physics-in-the-loop bi-directional optimization, combining scene-targeted reinforcement learning for human motion with DSRO for simulator-guided scene-geometry refinement.
Results
HSImul3R achieves stable, simulation-ready human–scene interaction reconstructions and demonstrates deployment of refined motions on a physical Unitree G1 humanoid robot.
Takeaways & Limitations
The framework provides a simulation-ready reconstruction pipeline and a benchmark with diverse objects and interaction scenarios for embodied AI research.
Takeaways & Limitations
Success rates remain limited for complex interactions or scenes with more than three objects, and dataset and source-model biases may constrain out-of-domain generalization.
Abstract
from arXiv · showhide
We present HSImul3R, a unified framework for simulation-ready 3D reconstruction of human-scene interactions (HSI) from casual captures, including sparse-view images and monocular videos. Existing methods suffer from a perception-simulation gap: visually plausible reconstructions often violate physical constraints, leading to instability in physics engines and failure in embodied AI applications. To bridge this gap, we introduce a physically-grounded bi-directional optimization pipeline that treats the physics simulator as an active supervisor to jointly refine human dynamics and scene geometry. In the forward direction, we employ Scene-targeted Reinforcement Learning to optimize human motion under dual supervision of motion fidelity and contact stability. In the reverse direction, we propose Direct Simulation Reward Optimization, which leverages simulation feedback on gravitational stability and interaction success to refine scene geometry. We further present HSIBench, a new benchmark with diverse objects and interaction scenarios. Extensive experiments demonstrate that HSImul3R produces the first stable, simulation-ready HSI reconstructions and can be directly deployed to real-world humanoid robots.
1 Introduction
HSImul3R addresses the perception–simulation gap in human–scene interaction reconstruction by using physics-in-the-loop optimization to refine both human motion and scene geometry. It also introduces HSIBench and demonstrates stable simulations and real-world humanoid deployment.
- Motivation: Embodied AI requires physically grounded modeling of human motion, spatial layouts, and interaction stability for real-world robotic deployment.Reconstructing human–scene interactions from images or videos provides supervision for scalable simulation-ready datasets.
- Motivation: Existing methods often produce visually plausible reconstructions that violate physical constraints because humans and environments are modeled without explicit physical coupling.The paper identifies separate scene reconstruction, human motion estimation, and interaction-modeling directions as sources of this gap.
- Approach: HSImul3R uses a bi-directional physics-aware optimization pipeline in which the simulator supervises refinement of human motion and scene geometry.The framework combines forward motion optimization with reverse geometry refinement using simulation feedback.
- Benchmark: HSIBench contains 19 objects, over 50 motion sequences, and 300 unique interaction instances across three participants.The dataset supports training and benchmarking across diverse human–scene interactions.
- Results: Experiments report stable, simulation-ready reconstructions across diverse scenarios and demonstrate retargeting refined motions to a Unitree humanoid robot.The evaluation covers simulation stability, post-simulation human motion quality, and image-to-3D generation improvements.
2 Related Works
The related work spans geometric and learned 3D reconstruction, physically grounded modeling, and physics-based humanoid simulation. HSImul3R is presented through a physics-in-the-loop pipeline that combines structural priors, motion optimization, and scene refinement.
- 3D Scene Reconstruction: 3D scene reconstruction progresses from structure-from-motion and multi-view stereo to learned depth, explicit 3D representations, and differentiable neural formulations.The surveyed representations include voxels, point clouds, meshes, signed distance functions, occupancy fields, neural radiance fields, and Gaussian splatting.
- Physically-sounded Modeling: Physically grounded modeling includes physics-constrained generation and physics-integrated methods that use simulation-derived losses or physical priors.Examples address stable reconstruction, self-supporting structures, and alignment between generators and simulation.
- Human Simulation Imitating: Physics-based humanoid simulation includes robust motion imitation, adaptive whole-body imitation, and generalizable control methods.The surveyed approaches use reinforcement learning, differentiable physics, human demonstrations, and compact latent skill spaces.
- HSImul3R: HSImul3R combines explicit 3D structural priors with forward scene-targeted reinforcement learning and reverse direct simulation reward optimization.Its pipeline refines human motion for stable interaction and scene geometry using simulator feedback.
3 Our Approach
HSImul3R reconstructs humans and scenes, aligns them with explicit 3D structural constraints, and uses physics-based feedback to refine motion and object geometry for stable simulation.
- 3.1 Reconstruction and alignment: The pipeline independently reconstructs static scene geometry and dynamic human motion before aligning them with explicit 3D structural priors.DUSt3R reconstructs the environment, while SAM2, 4DHumans, and ViTPose provide human masks, tracks, motion, and keypoints.
- 3.2 Human-scene alignment: 3D generative priors address disconnected geometry, missing surfaces, non-watertight topologies, and alignment errors caused by purely 2D supervision.These defects can produce physical instability and drifting in the simulator.
- 3.2 Human-scene alignment: The alignment stage optimizes human and object positions differently for non-contact and contact cases, using vertex distances or signed penetration depth.The non-contact objective brings nearby human and object vertices together, while the contact objective penalizes penetration.
- 3.3 Forward-pass optimization: Scene-targeted reinforcement learning refines humanoid motion by enforcing proximity between human contact keypoints and corresponding object-surface points.This supervision encourages physically plausible contact during simulation.
- 3.4 Reverse-pass optimization: Direct Simulation Reward Optimization refines generated object geometry using simulator feedback rather than human annotations or 3D ground truth.Its stability feedback evaluates gravitational stability and interaction stability, while generated-object defects can otherwise prevent stable contact.
- 3.5 Extension to monocular videos: The framework extends to monocular video by combining scene reconstruction, human motion estimation, and SAM2-based dynamic 3D alignment under a static-scene assumption.The only moving entity is currently assumed to be the human subject.
4 Experiments
Experiments evaluate HSImul3R’s reconstruction fidelity, simulation stability, motion preservation, scene-object geometry, and real-world deployment across benchmarks, comparisons, ablations, and varying input views. The method consistently improves simulation-ready HSI reconstruction and supports deployment on Unitree G1 robots, while performance remains limited in complex multi-object interactions.
- Quantitative Evaluations: HSImul3R consistently outperforms HSfM and variants on post-simulation interaction stability, human–scene penetration, and motion changes.Table 1 evaluates these three dimensions of reconstruction and simulation quality.
- Quantitative Evaluations: DSRO improves physical plausibility, interaction stability, and geometric accuracy in image-to-3D generation.The comparison includes MIDI and DSO, with DSO fine-tuned on the pre-trained MIDI model for fairness.
- Qualitative Evaluations: Qualitative comparisons show that HSImul3R produces more structurally faithful reconstructions and more stable simulated interactions than HSfM, DSO, and MIDI.HSfM often displaces objects, while the DSRO fine-tuned model reduces geometric distortions associated with simulation instability.
- Input-View Analysis: Additional input views slightly improve human motion quality but have little effect on interaction stability and increase scene penetration.The number of views therefore does not substantially improve the main simulation-stability measures.
- Real-world Robotics Deployment: Refined motions are retargeted and deployed on Unitree G1 humanoid robots for real-world human–scene interaction scenarios.The deployment uses GMR, diffusion-guided reinforcement learning, IsaacGym training, and the Unitree SDK.
- Ablation Studies: Removing scene-targeted distance minimization destabilizes the humanoid, causing exaggerated motions and frequent object displacement.The ablation study links the distance-minimization term to stable interaction behavior.
- Limitations and Failure Cases: The successful ratio remains low in complex interactions or scenarios involving more than three objects, and failures may leave humans and objects standing independently.The fine-tuned image-to-3D model also inherits biases from MIDI and HSIBench, constraining out-of-domain generalizability.
5 Conclusion
HSImul3R reconstructs simulation-ready human–scene interactions from uncalibrated sparse views through physics-aware optimization and dedicated interaction modeling. The framework combines stable simulations, high-quality reconstructions, and improved simulation success rates.
- HSImul3R is presented as the first framework for reconstructing simulation-ready human–scene interactions from uncalibrated sparse views.
- A contact-aware interaction model mitigates human–scene penetration issues in 3D reconstruction.
- Scene-targeted reinforcement learning promotes stable interactions within the simulator.
- Direct simulation reward optimization uses simulation feedback to fine-tune the image-to-3D generative model and improve simulation success rates.
- HSIBench supports training and evaluation, while experiments report stable simulations, high-quality image-to-3D reconstructions, and superiority over existing state-of-the-art methods.