PaperlayerView exact revision on arXiv

Source-linked AI summary

EmbodMocap: In-the-Wild 4D Human-Scene Reconstruction for Embodied Agents

Wenjia Wang, Liang Pan, Huaijin Pi, Yuke Lou, Xuqian Ren, Yifan Wu, Zhouyingcheng Liao, Lei Yang, Rishabh Dabral, Christian Theobalt, Taku Komura

arXiv:2602.23205v2cs.CV

TL;DR

Embodied AI needs scalable human–scene data, but existing capture systems are costly and studio-bound. EmbodMocap uses two moving iPhones to jointly calibrate dual RGB-D sequences and reconstruct humans and scenes in a metric world frame, supporting three embodied-AI applications with reported effectiveness and scalability. Its scope is constrained by iPhone LiDAR range, moving-object scenes, and extreme lighting.

  • Problem

    Existing systems for high-quality human–scene capture rely on costly hardware and controlled environments, limiting scalable data collection from natural human behavior.

  • Method

    EmbodMocap jointly calibrates dual RGB-D inputs from two moving iPhones to reconstruct metrically accurate, world-anchored humans and scenes.

  • Results

    EmbodMocap supports monocular human–scene reconstruction, physics-based character animation, and humanoid robot motion control, demonstrating utility across embodied-AI tasks.

  • Takeaways & Limitations

    The pipeline lowers the barrier to scalable, physically grounded human–scene data acquisition for embodied AI research.

  • Takeaways & Limitations

    Capture fails beyond approximately 5 meters and degrades in scenes dominated by moving objects or under extremely bright lighting.

Abstract

from arXiv · show

Human behaviors in the real world naturally encode rich, long-term contextual information that can be leveraged to train embodied agents for perception, understanding, and acting. However, existing capture systems typically rely on costly studio setups and wearable devices, limiting the large-scale collection of scene-conditioned human motion data in the wild. To address this, we propose EmbodMocap, a portable and affordable data collection pipeline using two moving iPhones. Our key idea is to jointly calibrate dual RGB-D sequences to reconstruct both humans and scenes within a unified metric world coordinate frame. The proposed method allows metric-scale and scene-consistent capture in everyday environments without static cameras or markers, bridging human motion and scene geometry seamlessly. Compared with optical capture ground truth, we demonstrate that the dual-view setting exhibits a remarkable ability to mitigate depth ambiguity, achieving superior alignment and reconstruction performance over single iphone or monocular models. Based on the collected data, we empower three embodied AI tasks: monocular human-scene-reconstruction, where we fine-tune on feedforward models that output metric-scale, world-space aligned humans and scenes; physics-based character animation, where we prove our data could be used to scale human-object interaction skills and scene-aware motion tracking; and robot motion control, where we train a humanoid robot via sim-to-real RL to replicate human motions depicted in videos. Experimental results validate the effectiveness of our pipeline and its contributions towards advancing embodied AI research.

1. Introduction

EmbodMocap addresses the difficulty of collecting scalable, scene-aware human-motion data by using two moving iPhones to reconstruct metrically accurate humans and scenes in a shared world frame. The resulting data supports reconstruction, physics-based animation, and humanoid-control applications for embodied AI.

  • Motivation: Existing high-quality human–scene capture systems rely on costly multi-view rigs, wearable sensors, or LiDAR and are largely restricted to controlled studios.These constraints make scalable acquisition in diverse indoor and outdoor environments difficult.
  • Method: EmbodMocap jointly calibrates dual RGB-D inputs to reconstruct metrically accurate humans and scenes in a unified world coordinate frame using only two moving iPhones.The pipeline preserves human motion and scene context during in-the-wild capture.
  • Applications: The collected data supports monocular human–scene reconstruction, physics-based character skills and scene-aware tracking, and sim-to-real humanoid motion control.These applications evaluate geometric consistency, physical realism, and embodied transferability, respectively.
  • Related Work: Recent 4D datasets add scene context to human motion but commonly depend on multi-view cameras, IMUs, electromagnetic sensors, or other specialized capture setups.Earlier motion datasets such as AMASS primarily lack the 3D scene context needed for human–scene interaction modeling.

3. Proposed Capture System

The capture system first establishes a metric scene world, then processes synchronized dual-view human sequences and calibrates both camera trajectories into that frame. Finally, it triangulates world-space keypoints and optimizes human parameters for temporally consistent motion reconstruction.

  • Stage I: Scene Reconstruction: A single iPhone RGB-D sequence reconstructs a metrically accurate static scene that defines the Z-up world coordinate system.SpectacularAI provides metric camera trajectories, while refined LiDAR depths are fused into a global mesh.
  • Stage II: Sequence Processing: Two synchronized iPhones capture dual-view RGB-D videos, producing per-view calibrated cameras and human observations for later alignment.YOLO, ViTPose, SAM2, PromptDA, and VIMO provide detection, keypoints, masks, depth refinement, and camera-space SMPL parameters; a laser cue synchronizes streams.
  • Stage III: Sequence Calibration: Joint calibration refines the two rigid transformations with point-tracking, Chamfer-distance, and bundle-adjustment losses while constraining rotations to preserve gravity alignment.The aligned extrinsics unify both camera coordinate systems with the metric scene frame.
  • Stage III: Sequence Calibration: COLMAP supplies an initial rigid alignment between each dual-view trajectory and the scene coordinate system using known intrinsics and background-only SIFT features.The offset transformation is estimated by minimizing trajectory discrepancies and solving the centered problem with SVD.
  • Stage IV: Motion Optimization: Motion optimization triangulates dual-view 2D keypoints into world-space 3D positions and fits SMPL shape, pose, and root translation under fixed cameras and scene geometry.Weighted reprojection minimization uses the view projection matrices, followed by a two-stage optimization for body shape, transition, and full parameters.

4. Evaluation

The evaluation ablates optimization losses and compares monocular, single-view, and dual-view capture against optical motion-capture ground truth. Dual-view optimization addresses occlusion and depth ambiguity while improving alignment to scene coordinates.

  • 4.1. Ablation Study on Loss Functions: The evaluation ablates tracking, Chamfer, reprojection, smoothness, and kp3d losses under multiple optimization settings.The ablation evaluates mask overlap, reprojection, and related reconstruction metrics.
  • 4.1. Ablation Study on Loss Functions: Ltrack stitches the two views together and significantly improves overall reconstruction performance.The ablation identifies Ltrack as highly impactful on final results.
  • 4.1. Ablation Study on Loss Functions: Lkp3d provides 3D human-joint positions and eliminates depth ambiguity compared with reprojection loss.This makes Lkp3d critical to the overall reconstruction performance.
  • 4.2. Comparison on Capture Methods: The capture-method comparison evaluates monocular GVHMR, single-view optimization, dual-view optimization, and optical mocap ground truth.The optical-studio experiment records five sequences totaling 9420 frames from one participant.
  • 4.2. Comparison on Capture Methods: Dual-view capture handles body-joint occlusion and scene-coordinate alignment more effectively than single-view capture.Two iPhones provide dense correspondence that resolves per-view depth ambiguity; dual-view scene calibration is about 5cm versus over 30cm for single view.

5. Downstream Tasks

The paper evaluates EmbodMocap across monocular reconstruction, physics-based interaction skills, scene-aware motion tracking, and humanoid robot control. Across these applications, captured scene-aware motion supports metric world-space reconstruction, scalable skill training, and embodied transfer.

  • 5.1. Monocular Human & Scene Reconstruction: The authors fine-tune π3 and VIMO with captured RGB-D, camera, and SMPL annotations for monocular human-and-scene reconstruction.π3 processes camera and point-map outputs, while VIMO is fine-tuned for metric-scale human motion reconstruction.
  • 5.1. Monocular Human & Scene Reconstruction: The EMDB evaluation reports W-MPJPE, WA-MPJPE, and Root Translation Error on global coordinates, with fine-tuning improving VIMO and π3 world-coordinate prediction.The evaluation uses 100-frame chunks and measures joint errors in millimeters and root translation error as a normalized percentage.
  • 5.2.1. Human Object Interaction Skill Training: The framework trains follow, climb, sit, lie, Prone, and Support interaction skills from reconstructed motion using goal-conditioned reinforcement learning and PPO.The four common skills are compared across optical-capture, EmbodMocap, and monocular-estimated data; Prone and Support extend prior skill coverage.
  • 5.2.1. Human Object Interaction Skill Training: The novel Support skill requires hands to bear body weight while feet remain close together, and monocular-estimated motions achieve only 20% success.The paper reports that monocular-based policies could not perform standard Support skill, while EmbodMocap supports the additional interaction task.
  • 5.2.2. Scene-aware Motion Tracking: Scene-aware motion tracking incorporates height maps into policy observations and trains one policy per scene across four 3D scenes.The dataset supports tracking long-horizon motion in diverse scene contexts, while humanoid control uses real-to-sim reconstruction and sim-to-real reinforcement learning.

6. Conclusion

EmbodMocap is a portable, affordable two-iPhone framework for metrically accurate 4D human-and-scene capture in diverse real-world environments. The paper evaluates it through reconstruction, physics-based character animation, and humanoid robot motion control.

  • 6. Conclusion: EmbodMocap captures high-quality 4D human-and-scene data using two iPhones and jointly reconstructs metrically accurate human motion and scene meshes.The framework is designed for diverse real-world environments without multi-camera setups, mocap suits, or controlled environments.
  • 6. Conclusion: Dual-view capture addresses body occlusion and sequence coordinate alignment more effectively than the paper’s single-view design.The conclusion bases this comparison on direct optical-capture-studio evaluation.
  • 6. Conclusion: Downstream experiments demonstrate utility for monocular human-scene reconstruction, physics-based character animation, and humanoid robot motion control.The conclusion presents these applications as evidence of the approach’s effectiveness and scalability.

7. Limitations and Future Work.

The pipeline has operating boundaries involving sensing range, scene dynamics, and lighting. The paper identifies these conditions as causes of depth-recording or registration failures and suggests more robust reconstruction and synchronization tools as future work.

  • 7. Limitations and Future Work: Depth recording fails beyond the iPhone LiDAR sensor’s approximately 5-meter range.This limits capture in scenes where relevant geometry lies farther from the phone.
  • 7. Limitations and Future Work: Scenes dominated by moving objects degrade the SLAM SDK results, constraining capture in highly dynamic environments.The limitation concerns the scene-motion assumptions used by the capture pipeline.
  • 7. Limitations and Future Work: Extremely bright lighting can cause COLMAP failures and incorrect registration.The authors suggest robust structure-from-motion tools such as H-Loc as a possible remedy.
  • 7. Limitations and Future Work: Automatic iPhone synchronization applications could reduce the human effort required by the capture process.This is proposed as future work rather than a capability demonstrated in the current pipeline.

9. More Details of EmbodMocap

EmbodMocap uses two moving iPhones to capture performers from complementary viewpoints, then calibrates the sequences into a shared metric world frame. Optional marker-based refinement addresses residual alignment errors.

  • Capture technique: Two photographers follow the performer with vertically oriented iPhones, maintaining a 60–120° angle for effective triangulation.Camera motion also provides diverse viewpoints and helps reduce occlusion from environmental objects.
  • Synchronization and alignment: Temporal synchronization requires operators to record frame indices, typically taking about 1 minute per sequence.Contact-frame annotation for optional marker-based alignment costs 1–2 minutes per sequence.
  • Synchronization and alignment: Marker-based refinement aligns the performer’s feet at specified frames with fixed 3D scene markers using an optimized rigid transformation.The method targets residual errors caused primarily by sparse-keypoint depth inaccuracies and iPhone sensor depth errors.
  • World-frame calibration: The world-frame transformation is restricted to an xy-plane rotation ϕc about the z-axis plus translation Tc.The resulting transformation updates global SMPL orientation, pelvis world position, and camera pose for consistent scene representation.

10. More Details of Monocular Human-Scene Reconstruction Pipeline

The monocular baseline combines camera-and-scene reconstruction with SMPL pose estimation, aligns temporal chunks globally, and resolves scale by matching human and scene depths. The resulting representations are transformed into world coordinates.

  • Baseline architecture: The baseline combines π3 for camera trajectories and scene point clouds with VIMO for SMPL-based human pose estimation.π3 processes long videos in overlapping chunks, while VIMO predicts pose, translation, and body shape parameters.
  • Chunk alignment: Overlapping point-cloud chunks are aligned with Procrustes analysis by estimating scale, rotation, and translation between adjacent regions.The optimal alignment parameters are computed using singular value decomposition.
  • Metric-scale recovery: Metric scale is estimated by matching SMPL vertex depths in meters with π3 point-cloud depths in arbitrary units.This step resolves the scale ambiguity in the monocular reconstruction pipeline.
  • World-space output: The reconstructed point clouds and SMPL global orientation and translation are transformed into the world coordinate system using the estimated rotation and translation.This applies the same transformation formula used for the world-frame alignment procedure.

11. More Details of Human-Object Interaction Skills

The interaction-skill suite defines task-specific observations, targets, and rewards for navigation and object interactions. It covers follow, sitting, climbing, lying, prone positioning, and support, with support evaluated through hand and foot stability.

  • Follow skill: The follow task requires the character to track a predefined 2D trajectory over a 10-second episode using 10 future waypoints observed over the next 1.0 second.Waypoints are sampled every 0.1 seconds by linear interpolation, and the reward depends on root-position distance.
  • Sit skill: The sitting task places the root at a target 3D location 10 cm above a chair seat and rewards minimizing distance to that target.Its observation includes target and root state, facing direction, and the object bounding box.
  • Climb skill: The climbing task targets a root position 94 cm above an object surface and adds a foot-lifting reward to support successful climbing.Its observation includes the target root position and the object’s bounding-box corners.
  • Lying and prone skills: The lying task combines approaching a designated standing point with accurate target positioning, head-height alignment, and body alignment.The prone task similarly combines target accuracy, surface alignment, and maintaining a facedown orientation.
  • Support skill: The support task requires hands on an object’s top surface while keeping feet close together for stable posture and interaction.Success is measured by combined hand-to-surface XY distance and Z-axis deviation, together with foot proximity.

12. More Details of Scene-Aware Imitation Policy

The scene-aware imitation policy combines character proprioception with surrounding scene geometry and uses a reward that balances reference-motion tracking against control jitter. Penalizing action changes encourages continuous, stable trajectories.

  • Policy inputs: Character proprioception represents link-relative positions, quaternion rotations, and linear and angular velocities in a root-centered local frame.The root is the pelvis, with the local x-axis aligned to its facing direction.
  • Reward design: The policy’s reward combines a tracking term for kinematic agreement with reference motion and a jitter penalty for action changes.The tracking term compares translation, rotation, linear velocity, and angular velocity across humanoid links.
  • Reward design: Minimizing the difference between consecutive actions incentivizes continuous and stable control trajectories, reducing jittery behavior.The action-difference term uses the current and previous actions.

13. More Details of Captured Dataset Used in Main Paper

The captured dataset spans 23 scenes and 104 sequences, with high-precision meshes and multimodal annotations. Its recordings cover varied camera and human trajectories, scene sizes, and sequence durations.

  • The dataset contains 23 scenes, 104 sequences, and approximately 200,000 video frames, each with a high-precision mesh and multimodal annotations.Annotations include depth maps, segmentation masks, camera trajectories, bounding boxes, 2D keypoints, and SMPL parameters.
  • Camera trajectories range from 4 meters to over 30 meters, while human trajectories span 5 meters to over 30 meters.
  • Indoor scene meshes range from 20 to 90 square meters, whereas outdoor scenes can reach 200 square meters.
  • Most sequences last between 30 and 60 seconds, with additional camera-space and world-space results reported in dedicated sections.
  • A 3D demonstration showcases the captured dataset.
Loading 2602.23205v2…