PaperlayerView exact revision on arXiv

Source-linked AI summary

CRISP: Contact-Guided Real2Sim from Monocular Video with Planar Scene Primitives

Zihan Wang, Jiashun Wang, Jeff Tan, Yiwen Zhao, Jessica Hodgins, Shubham Tulsiani, Deva Ramanan

arXiv:2512.14696v3cs.CVcs.GRcs.RO

TL;DR

CRISP addresses the difficulty of reconstructing human–scene interactions and simulation-ready geometry from casual monocular video. It fits planar primitives to scene point clouds, uses contact modeling to recover occluded geometry, and applies RL to validate interactions. The authors report a 93.1% real-to-sim success rate and faster simulation throughput.

  • Problem

    Human–scene interaction reconstruction remains less studied, and existing reconstructions can contain artifacts or noisy geometry that complicate physical simulation.

    Source: §1 · p. 2

  • Method

    CRISP fits compact convex planar primitives to scene point clouds, uses human–scene contact to recover occluded geometry, and drives a humanoid controller with RL.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 4

  • Results

    93.1% real-to-sim success and 43% faster simulation throughput are reported for CRISP.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments report improved world-grounded human-motion accuracy, interaction fidelity, and downstream RL training performance.

    Source: §5 · p. 9

    Planar decomposition can leave small gaps that make reconstructions appear visually incomplete, although the authors report no effect on simulation quality.

    Source: §4.2 · p. 9

Abstract

from arXiv · show

We introduce CRISP, a method that recovers simulatable human motion and scene geometry from monocular video. Prior work on joint human-scene reconstruction relies on data-driven priors and joint optimization with no physics in the loop, or recovers noisy geometry with artifacts that cause motion tracking policies with scene interactions to fail. In contrast, our key insight is to recover convex, clean, and simulation-ready geometry by fitting planar primitives to a point cloud reconstruction of the scene, via a simple clustering pipeline over depth, normals, and flow. To reconstruct scene geometry that might be occluded during interactions, we make use of human-scene contact modeling (e.g., we use human posture to reconstruct the occluded seat of a chair). Finally, we ensure that human and scene reconstructions are physically-plausible by using them to drive a humanoid controller via reinforcement learning. Our approach reduces motion tracking failure rates from 55.2\% to 6.9\% on human-centric video benchmarks (EMDB, PROX), while delivering a 43\% faster RL simulation throughput. We further validate it on in-the-wild videos including casually-captured videos, Internet videos, and even Sora-generated videos. This demonstrates CRISP's ability to generate physically-valid human motion and interaction environments at scale, greatly advancing real-to-sim applications for robotics and AR/VR.

1 INTRODUCTION

CRISP targets physically faithful human–scene reconstruction from casual monocular video, where interaction reconstruction and simulation-ready geometry remain challenging. It combines contact-guided geometry recovery with RL-based validation and reports improved real-to-sim performance and simulation throughput.

Motivation

  • The target is a physically plausible reconstruction of the human, environment, and their interactions from casual monocular video.The introduction frames physical understanding around contact, such as a foot contacting a ground surface for support.Source: §1 · p. 2

Problem

  • Prior interaction reconstructions struggle with parallax and body occlusions, while small ground-plane noise can disrupt physical humanoid simulation.The resulting geometry can contain duplicate structures or missing regions, and earlier video-driven simulation work often used environments with limited scene interaction.Source: §1 · p. 2

Contributions

  • CRISP fits clean, convex, watertight planar primitives to point clouds and uses inferred body shape to recover occluded interaction surfaces.Vision–language models detect common interactions, and RL drives a simulated humanoid through the reconstructed motion and scene.Source: §1 · p. 2

Results

  • 93.1% real-to-sim success and 43% faster simulation throughput demonstrate gains on human-centric benchmarks and in RL training.The reported throughput comparison is against dense-mesh approaches.Source: §1 · p. 2

2 RELATED WORK

The related work traces monocular human-motion recovery from parametric-model fitting to feed-forward methods and world-frame motion, scene, and contact estimation. It also reviews contact prediction and physics-based interaction control, contrasting CRISP with concurrent approaches.

2.1 MONOCULAR HUMAN MOTION ESTIMATION.

  • Human-motion recovery has moved from fitting parametric body models to feed-forward regression and joint world-frame estimation of trajectories, scene geometry, and contacts.The passage cites classic optimization, feed-forward methods, and world-frame work [Ye et al., 2023; Wang et al., 2024a].Source: §2.1 · pp. 2–3
  • CRISP’s pipeline recovers camera parameters and a global point cloud, fits about 50 planar primitives, infers occluded contact surfaces, and drives a humanoid with RL.The caption gives examples of occluded stair and platform surfaces recovered through contact modeling.Source: Figure 2 · p. 3

2.2 HUMAN-SCENE INTERACTION

  • Interaction work spans contact prediction and physics-based controllers, while concurrent VideoMimic jointly reconstructs humans and environments for humanoid control.CRISP is reported to provide more accurate human, scene, and contact modeling than VideoMimic, with improved simulation stability, efficiency, and RL success rates.Source: §2.2 · p. 3

3 METHOD

CRISP converts monocular human-scene videos into simulatable assets by fitting planar primitives, using contact to complete occluded geometry, and evaluating reconstructions through RL-based motion tracking.

3.1 OVERVIEW

  • From casual monocular video, CRISP estimates camera parameters, scene point clouds, and world-frame human motion in a shared coordinate system.It uses human scale to calibrate the scale-ambiguous scene point cloud before recovering simulatable motion and geometry.Source: §3.1 · pp. 3–4

3.2 NORMAL–BASED PLANAR PRIMITIVE FITTING

  • Clustering point-cloud geometry into about 50 convex primitives yields compact, noise-regularized, simulation-ready scene reconstructions.The pipeline segments normals, spatially splits segments, merges consistent planes across frames using optical flow, and fits each merged region with RANSAC.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3 CONTACT AS A CUE FOR SCENE COMPLETION

  • Contact predictions help complete interaction surfaces that are occluded by the person, while temporal filtering retains consistently confident predictions at low-motion frames.The method uses InteractVLM vertex-contact predictions and selects a frame with minimal human motion across a window of consistent predictions.Source: §3.3 · pp. 4–5

3.4 PHYSICS-BASED MOTION TRACKING

  • A humanoid policy tracks extracted motion from its current state and future target poses, and qualitative rollouts compare CRISP with VideoMimic across six sequences.CRISP produces stable rollouts, whereas VideoMimic's scene artifacts and geometry errors are associated with simulation failures.Source: §3.4 · p. 5, Figure 4 · p. 6

4 EXPERIMENTS

The experiments assess whether CRISP’s human and scene reconstructions support accurate motion recovery, faithful human–scene interactions, and downstream reinforcement-learning simulation. Across benchmark comparisons and ablations, they evaluate both reconstruction quality and whether the resulting assets support stable, video-faithful rollouts.

4.1 EXPERIMENT SETUP

  • Evaluation covers EMDB motion sequences and PROX indoor scenes, compares geometry and HMR baselines, and tests whether reconstructed assets support RL tracking.EMDB-2 contains 21 sequences; PROX has 12 indoor settings and is used primarily for interaction fidelity because its pseudo-ground-truth motion annotations are noisy.Source: §4 · p. 6, §4.1 · pp. 7–8

4.2 OVERALL REAL-TO-SIM PERFORMANCE

  • 93.1% vs 44.8% success and 23K vs 16K FPS: CRISP’s planar primitives outperform VideoMimic on RL reliability and throughput.Table 1 compares success, throughput, reconstruction quality, non-penetration, and HMR quality across VideoMimic and geometry variants.Source: Table 1 · p. 7, §4.2 · p. 8

4.1 EXPERIMENT SETUP

  • The metrics span world-grounded joint errors, trajectory error and smoothness, scene Chamfer alignment, non-penetration, RL success, and training throughput.Success rate counts episodes whose joints remain within 0.5 meters of reference motion throughout the trajectory; FPS includes simulation and policy-learning steps.Source: §4.1 · pp. 7–8

4.2 OVERALL REAL-TO-SIM PERFORMANCE

  • Planar primitives achieve the lowest one-sided Chamfer error and highest non-penetration score, despite slightly worse bidirectional Chamfer than NKSR.The authors attribute the bidirectional gap to missing fine, typically non-contact details; extra noisy geometry can create contact artifacts that destabilize rollouts.Source: Table 1 · p. 7, §4.2 · pp. 8–9

4.3 GLOBAL HUMAN MOTION ESTIMATION

  • 70.60 mm WA-MPJPE100 and 175.93 mm W-MPJPE100 after RL motion tracking yield the best overall EMDB accuracy among methods.Without RL refinement, CRISP is competitive with GVHMR and better than WHAM, TRAM, and VideoMimic on the reported joint-accuracy measures.Source: §4.3 · p. 9, Table 2 · p. 9
  • CRISP has the lowest root translational error and temporal jitter of 8.14, while VideoMimic shows joint errors above 500 mm and high drift.The authors report that RL refinement reduces pose drift and stabilizes temporal dynamics.Source: §4.3 · p. 9

4.4 ABLATION STUDY ON CONTACT CUES

  • Contact-guided reconstruction improves scene alignment in both Chamfer directions, and added occluded support surfaces produce stable rollouts that better track reference motion.The figure contrasts reconstructions without contact, which omit supports and can cause falling or unnatural movement, with reconstructions that include them.Source: Figure 5 · p. 8, §4.4 · p. 9

5 CONCLUSION

CRISP converts unconstrained monocular human videos into simulation-ready assets for physically valid, natural human–scene interactions. The conclusion reports gains across motion estimation, interaction fidelity, and downstream RL training; the appendix notes LLM-assisted writing and planned open-sourcing.

5 CONCLUSION

  • CRISP combines compact convex planar scene primitives, contact-guided recovery of occluded surfaces, and RL-based validation and refinement of reconstructed interactions.The conclusion reports improvements in world-grounded HMR accuracy, human–scene interaction fidelity, and downstream RL training performance.Source: §5 · p. 9

A APPENDIX

  • The appendix says LLMs were used to polish abstract and introduction writing and some captions, and that code and data will be open-sourced upon acceptance.It also reports testing on publicly available online videos and experiments on standard benchmarks and additional public internet videos.Source: A APPENDIX · p. 13

B DETAILS OF PLANAR FITTING.

Planar fitting converts per-frame scene points into globally associated planar groups and compact primitive parameters. The pipeline segments points by normals and space, links segments across frames using flow-warped overlap and normal agreement, then fits and optionally augments planes.

Planar fitting

  • The pipeline outputs M primitive rotations, translations, sizes, and point-to-primitive assignments from per-frame point clouds and optical flow.Its input also includes covisibility masks.Source: B DETAILS OF PLANAR FITTING · p. 13
  • Per-frame segmentation estimates point normals, clusters them into K groups, then spatially clusters points within each normal group.This produces frame-specific segments for later cross-frame association.Source: B DETAILS OF PLANAR FITTING · p. 13
  • Cross-frame association warps segments with optical flow and links them when overlap and normal-cosine scores exceed fixed thresholds.Accepted warped overlaps are removed when combining segments, and associations across pairs are unified into M global planar groups.Source: B DETAILS OF PLANAR FITTING · p. 13
  • RANSAC fits each global group to a plane; projected inliers determine its in-plane axes, minimum-area rectangle, size, and center.Nonrectangular footprints can optionally be split along a principal in-plane axis and refit.Source: B DETAILS OF PLANAR FITTING · p. 13
  • Optional contact-guided fitting repeats the plane fit on predicted contact points to add planes, with the normal-direction size clamped to at least 0.05 m.The algorithm returns the primitive rotations, translations, sizes, and point assignments.Source: B DETAILS OF PLANAR FITTING · p. 13

C CONTACT-GUIDED SCENE COMPLETION: RELIABILITY ANALYSIS

Contact-guided completion is filtered for temporal consistency and is most useful where interactions occlude support surfaces. The authors assess its reliability through geometry and physical-plausibility measures, while noting that it can add inaccurate primitives when pose or contact estimates drift.

Contact-guided scene completion: reliability analysis

  • Contact completion is most beneficial for occluded supports such as chair seats, stair treads, and platforms, where a small completed region can prevent falls and raise rollout success.Contact predictions are filtered to windows with consistently high confidence and low body speed, using the most stationary frame to reduce premature false positives.Source: C CONTACT-GUIDED SCENE COMPLETION:… · p. 13
  • On PROX, contact modeling does not increase success because most sequences involve sitting and missing seats lead to squatting, but it improves physical plausibility quantitatively.The ablation is reported without RL refinement.Source: Table 3 · p. 14
  • Contact-augmented primitives can be inaccurate when HMR or contact estimates drift, producing slightly tilted planes with small translation offsets.RL tests whether reconstructed assets support stable execution using them as collision geometry; it does not directly modify the scene primitives.Source: C CONTACT-GUIDED SCENE COMPLETION:… · p. 14

D LIMITATIONS AND FUTURE WORK

CRISP’s limitations include imperfect detail for highly curved shapes, inability to represent fluids and deformables, and lack of support for dynamic interactions. The authors point to more expressive primitives and dynamic interaction handling as future directions.

D LIMITATIONS AND FUTURE WORK

  • Highly curved or organic objects may appear faceted or under-fitted, although this limitation does not significantly reduce locomotion success rates.The authors suggest more expressive convex primitives such as superquadrics to improve fine-grained curved-surface detail.Source: D LIMITATIONS AND FUTURE WORK · p. 14
  • The method assumes static, rigid scene geometry and cannot faithfully model fluids or deformable objects, including soft cushions under large deformation.The codebase could be extended to dynamic rigid objects or scenes using time-varying primitives.Source: D LIMITATIONS AND FUTURE WORK · p. 14
  • CRISP currently handles static human-scene interaction, while its RL training supports locomotion but not loco-manipulation.Future work includes dynamic human-object interaction handling and more expressive primitives for complex scenes.Source: D LIMITATIONS AND FUTURE WORK · p. 14

E RUNTIME ANALYSIS

The runtime analysis identifies visual SLAM and feed-forward depth and flow estimation as the main computational bottlenecks, while planar fitting is lightweight. A separate scene-aware policy adds nearby primitive points as input and improves success and obstacle avoidance in some sequences.

E RUNTIME ANALYSIS

  • Visual SLAM and feed-forward depth and flow estimation are the main runtime bottlenecks, whereas planar fitting from points to primitives can run in real time.The reported test uses a 300-frame, 10-second video at 1440 × 1920 resolution on one RTX A6000; VideoMimic takes 1282.94 seconds to prepare geometry and reference motion on the same machine.Source: E RUNTIME ANALYSIS · pp. 14–15, Table 4 · p. 15
  • Real-time operation is expected when the pipeline is paired with real-time RGB-D SLAM and detection, using sensor depth and flows from relative camera motion.Source: E RUNTIME ANALYSIS · p. 15

F SCENE-AWARE POLICY.

  • The scene-aware policy adds PointNet features from 2.8k nearby primitive points and achieves higher success and better obstacle avoidance in some sequences.The authors frame this variant as a comparison, not the main test of whether reconstructions directly support simulation.Source: F SCENE-AWARE POLICY · p. 15

A scene-aware policy

CRISP's scene-aware policy is optional for substantiating the paper's central claims, while potentially adding deployment performance. The authors also describe CRISP as a drop-in replacement for the first two stages of VideoMimic's sim-to-real pipeline.

A scene-aware policy

  • A blind policy can avoid obstacles and route around reconstruction errors, but this reduces its sensitivity to reconstruction quality.Scene-aware inputs are an optional extra performance gain for deployment, for example when targeting sim-to-real.Source: A scene-aware policy · p. 15

G SEAMLESS TRANSFER TO CURRENT SIM-TO-REAL PIPELINE.

  • CRISP is described as a drop-in replacement for stages 1 and 2 of VideoMimic's four-stage pipeline.Those stages are MoCap pretraining and tracking reconstructed video motions.Source: G SEAMLESS TRANSFER TO CURRENT… · p. 15

H RL TRAINING DETAILS

CRISP's reconstructed assets yield higher rewards more quickly and require lower torques than VideoMimic for the same motions. The section also specifies the tracking reward, RL training settings, and policy and critic architectures.

H RL TRAINING DETAILS

  • RL training uses γ = 0.99, GAE with τ = 0.95, a 2×10−5 learning rate, 2048 parallel environments, and batch size 8192.Source: H RL TRAINING DETAILS · p. 15
  • The tracking reward is defined by weights wp = 2.5, wr = 1.5, wv = 0.5, wω = 0.5, wh = 1, and we = 0.001.The associated α values are αp = 1.5, αr = 0.3, αv = 0.12, αω = 0.05, and αh = 20.Source: H RL TRAINING DETAILS · p. 15
  • The policy uses a two-layer transformer encoder with latent dimension 256, feed-forward size 512, and two attention heads; the critic is an MLP with hidden sizes [1024, 512].Source: H RL TRAINING DETAILS · p. 15
  • CRISP's reconstructed assets let the agent achieve higher rewards more quickly than VideoMimic on each motion clip.The authors interpret this as faster convergence and better overall tracking quality.Source: H RL TRAINING DETAILS · p. 15, Figure 6 · p. 16
  • VideoMimic consistently requires higher torques than CRISP to complete the same motions.The authors attribute this to lower-quality assets with geometry artifacts and unstable supports, which force excessive control effort.Source: H RL TRAINING DETAILS · p. 16, Figure 7 · p. 17

Related papers

Loading 2512.14696v3…