Source-linked AI summary
Visual Imitation Enables Contextual Humanoid Control
Arthur Allshire, Hongsuk Choi, Junyi Zhang, David McAllister, Anthony Zhang, Chung Min Kim, Trevor Darrell, Pieter Abbeel, Jitendra Malik, Angjoo Kanazawa
TL;DR
Humanoid robots need a way to acquire environment-appropriate whole-body skills from ordinary human videos rather than behavior-specific rewards or motion-capture data. VIDEOMIMIC jointly reconstructs humans and scenes, retargets the motion, and trains a unified environment-conditioned policy. On a real Unitree G1, the policy performs robust contextual locomotion and chair sit-stand behaviors across environments, including unseen ones.
Problem
Humanoids need to learn environment-appropriate skills such as sitting and stair climbing from everyday videos without relying on hand-tuned rewards or motion-capture data for each behavior and environment.
Method
VIDEOMIMIC jointly reconstructs 4D humans and scenes from monocular videos, retargets motion to a humanoid, and trains a single policy through simulation reinforcement learning and distillation.
Results
The resulting policy robustly performs stair ascents and descents, chair and bench sit-stand behaviors, and other whole-body skills on a real Unitree G1 across indoor, outdoor, and unseen environments.
Takeaways & Limitations
VIDEOMIMIC provides a scalable path for teaching humanoids contextual skills directly from everyday videos using environment geometry and root-direction commands.
Takeaways & Limitations
Monocular reconstruction remains brittle in the wild, while the 11×11 LiDAR heightmap lacks resolution for precise contacts, manipulation, and overhanging obstacles.
Abstract
from arXiv · showhide
How can we teach humanoids to climb staircases and sit on chairs using the surrounding environment context? Arguably, the simplest way is to just show them-casually capture a human motion video and feed it to humanoids. We introduce VIDEOMIMIC, a real-to-sim-to-real pipeline that mines everyday videos, jointly reconstructs the humans and the environment, and produces whole-body control policies for humanoid robots that perform the corresponding skills. We demonstrate the results of our pipeline on real humanoid robots, showing robust, repeatable contextual control such as staircase ascents and descents, sitting and standing from chairs and benches, as well as other dynamic whole-body skills-all from a single policy, conditioned on the environment and global root commands. VIDEOMIMIC offers a scalable path towards teaching humanoids to operate in diverse real-world environments.
1 Introduction
VIDEOMIMIC addresses contextual humanoid control by converting everyday monocular videos into transferable, environment-conditioned skills. It jointly reconstructs human motion and scenes, retargets motion to a humanoid, and trains a unified policy validated on real-robot deployment.
- Contextual control lets humanoids execute environment-appropriate actions such as sitting on chairs or climbing staircases from observed everyday videos.
- VIDEOMIMIC converts monocular videos into transferable humanoid skills through joint 4D human-scene reconstruction, motion retargeting, and reinforcement-learning policy training.
- The unified policy observes proprioception, a local height-map, and desired root direction, then outputs low-level motor actions without explicit task labels or skill selection.
- Training uses 123 monocular RGB videos and produces a real Unitree G1 deployment that generalizes humanoid motor skills to unseen environments.
2 Related Work
Prior work separates reward-engineered control from motion imitation and often reconstructs humans or cameras without jointly modeling scene context. VIDEOMIMIC combines human motion and scene geometry transfer with context-aware simulation and real-world deployment.
- Learning Skills on Legged Robots: Reward-based legged-robot methods use handcrafted objectives and environment scripting, requiring new reward tuning for each behavior.
- VIDEOMIMIC learns context-aware control in simulation and deploys the resulting policy in real-world environments.
- Learning Skills on Legged Robots: Data-driven methods imitate reference motions from MoCap or monocular video, while VIDEOMIMIC transfers both human motion and scene geometry into simulation.
- Human and Scene Reconstruction from Images and Videos: Earlier monocular-video reconstruction methods often model the person or camera alone, leaving global trajectory and context-aware dynamics undefined.
3 Real-to-Sim Data Acquisition
The Real-to-Sim stage turns a monocular video into metrically aligned human trajectories and a simulator-ready scene mesh. It jointly optimizes human motion and scene scale, then gravity-aligns, meshes, and retargets the result for policy learning.
- The pipeline extracts human poses and a raw scene point cloud, jointly optimizes metrically aligned trajectories and geometry, then prepares motion-mesh pairs for policy learning.
- Per-frame human pose, shape, joints, foot contacts, depth, camera poses, and scene points are estimated from monocular RGB videos using pretrained perception and SfM methods.
- Joint Human–Scene Reconstruction: A similarity initialization and pixel-depth unprojection provide coarse global trajectories and lifted 3D joints for subsequent human-scene optimization.
- Joint Human–Scene Reconstruction: Joint optimization solves for global translations, orientations, local poses, and scene scale, using metric human height as the reference for scale recovery.
- Generating Simulation-Ready Data: Gravity alignment and NKSR meshification convert noisy point clouds into lightweight geometric constraints suitable for physics simulation and parallel training.
- Generating Simulation-Ready Data: Policy-training inputs combine Motion Capture trajectories, video reference trajectories, and heightmap observations across reconstructed environments.
4 Policy Learning
Policy learning progressively bridges noisy video references and deployable contextual control. The pipeline pretrains on MoCap, adds scene-conditioned tracking, distills away target-joint inputs, and finetunes with reduced observations.
- Observations: The policy uses proprioception and target-related observations, including joint histories, body orientation cues, and desired root direction.
- Rewards: Data-driven tracking rewards use link and joint positions, joint velocities, and foot contacts to reduce handcrafted reward priors while promoting physical feasibility.
- Stage 1: MoCap Pre-Training: MoCap pretraining initializes reinforcement-learning policies on retargeted human motion before fine-tuning them on reconstructed video data.
- Stage 2: Scene-Conditioned Tracking: Scene-conditioned tracking introduces environment heightmaps while policies continue tracking motion-specific reference targets across reconstructed terrains.
- Stage 3: Distillation: DAgger distillation removes target joint-angle and root roll/pitch observations, leaving desired root directions as control signals.
- Stage 4: Under-conditioned RL Finetuning: Under-conditioned RL finetuning adapts the distilled policy to reduced observations and can improve performance over the distilled policy.
5 Results
The evaluation combines reconstruction benchmarks, dataset curation, ablations, and real-robot deployment to assess VIDEOMIMIC’s robustness and versatility. A single context-conditioned policy performs diverse whole-body behaviors on the Unitree G1.
- 5.1 Reconstruction and Data: Our method consistently achieves the best reconstruction performance against baselines for human trajectory accuracy and scene geometry.The evaluation uses WA/W-MPJPE for trajectory accuracy and Chamfer Distance for scene geometry.
- 5.1 Reconstruction and Data: The reconstruction pipeline handles dynamic human-scene interaction, multiple people, and ego-view RGB-D rendering from dense point clouds.The RGB-D rendering capability is presented as a future direction rather than being used in the current policy.
- 5.1 Reconstruction and Data: The curated dataset contains 123 casually recorded smartphone videos spanning everyday activities in diverse indoor and outdoor settings.Activities include sitting, standing from furniture, stair climbing and descent, and stepping onto blocks.
- 5.1 Reconstruction and Data: Removing motion-capture pretraining significantly hinders learning effective behaviors from noisy video references.MPT also helps stabilize early learning when robot poses are unstable or interpenetrate the scene.
- 5.2 Real-world Deployment: Without task-specific tuning, the same network climbs and descends stairs, traverses steep and rough terrain, and sits or stands from chairs and benches.The controller uses proprioception and a noisy 360° LiDAR height-map, with recovery from unexpected stair-foot slides via a brief single-leg hop.
- 5.2 Real-world Deployment: The real-world demonstrations constitute, to the authors’ knowledge, the first deployment of a context-aware humanoid policy learned from monocular human videos.The deployment jointly demonstrates perceptive locomotion and environment-prompted sitting, standing, and stair climbing.
6 Conclusion
VIDEOMIMIC converts everyday human videos into environment-conditioned humanoid control policies through reconstruction, retargeting, and dynamics-aware reinforcement learning. The resulting single policy performs robust contextual skills, while future work targets richer interactions, multimodal context, and multi-agent behavior.
- 6 Conclusion: VIDEOMIMIC converts everyday human videos into environment-conditioned humanoid control policies through a real-to-sim-to-real pipeline.The system reconstructs humans and surrounding geometry, retargets motion to a feasible humanoid, and trains with the recovered scene as terrain.
- 6 Conclusion: A single policy delivers robust, repeatable stair ascents and descents and chair sit-stand behaviors using environment geometry and a root direction command.These behaviors are driven without task-specific policy selection in the described deployment.
- 6 Conclusion: The authors position VIDEOMIMIC as a scalable path for teaching humanoids contextual skills directly from videos.
- 6 Conclusion: Future work includes richer human–environment interactions, multimodal sensor-based context learning, and multi-agent behavior modeling.
7 Limitations
The pipeline remains constrained by brittle monocular reconstruction, limited sensing, rigid-scene assumptions, and a small motion dataset. These constraints affect geometric fidelity, retargeting, perception, simulation scope, and motion smoothness.
- Reconstruction: Monocular 4D human–scene recovery is brittle in the wild because camera drift, dynamic-point errors, low texture, holes, oversmoothing, and stray points can degrade meshes.Aggressive filtering may leave holes, while oversmoothing can remove narrow stair treads needed for robot control.
- Retargeting: Retargeting assumes every reference pose becomes feasible after scaling, but clutter and competing foot-contact and collision costs can trap optimization in poor local minima.The RL controller must subsequently clean up these retargeting errors.
- Sensing and policy input: The 11×11 LiDAR height-map supports terrain and chairs but lacks resolution for precise contacts, manipulation, and overhanging obstacles.Richer RGB-D inputs or learned occupancy grids are proposed to broaden applicability and improve semantic understanding.
- Simulation fidelity: The simulator represents scenes as single rigid meshes, leaving articulated or deformable objects outside the current modeling scope.Handling them requires more expressive simulators and object-level reconstruction pipelines.
- Data scale and motion quality: Training uses only 123 video clips and occasionally relies on recovery behaviors, which can produce jerky motions.Larger, more diverse video corpora and iterative real-world fine-tuning are identified as ways to improve smoothness and robustness.
- Assumptions: The method assumes a monocular video clearly captures both person and scene, the environment remains static, and robot kinematics and joint limits are known.These assumptions define the supported training setting for treating human motion and terrain as rigid.
A.3 Generating Simulation Ready Data
The simulation-ready data pipeline gravity-aligns, filters, and meshes reconstructed scene geometry, then retargets human motion to the G1 using embodiment, contact, and collision constraints.
- Scene processing: Gravity alignment reorients the reconstruction so +z points upward, making the scene compatible with physics engines.GeoCalib supplies roll–pitch estimates used to construct the gravity transformation.
- Scene processing: Point-cloud filtering removes noisy points, applies gravity alignment and scale, crops around SMPL joints, and reduces each cloud to about 5% of its original size.A 0.1 m voxel grid retains at most 20 samples per occupied cell without losing surface detail.
- Scene processing: Meshification uses NKSR, fills large holes through top-down ray casting and inverse-distance interpolation, then reruns NKSR to produce the final mesh.Processing a 300-frame sequence takes roughly 60 s, including gravity alignment, filtering, and meshing.
- Humanoid motion retargeting: Motion retargeting formulates transfer to the G1 as a nonlinear optimization over joint angles, root poses, and per-link scale factors.A Levenberg–Marquardt solver handles the human-in-scene optimization landscape.
- Humanoid motion retargeting: The retargeting objective combines kinematic-tree motion-transfer costs with contact matching, foot-skating penalties, and self- and world-collision avoidance.The robot is modeled with capsules, the world with a heightmap, and learned foot-contact estimates provide contact signals.
- Humanoid motion retargeting: A 300-frame clip takes around 10 seconds to retarget on a single NVIDIA A100 using PyRoki.The optimization also includes joint limits, temporal smoothness, and knee-yaw regularization for stable leg poses.
A.4 Evaluation Details
The evaluation details describe quantitative assessment of reconstructed humans and scenes, alongside ego-view RGB-D rendering as a future perception capability. Rendered views can contain holes in occluded regions, motivating novel-view synthesis.
- Quantitative evaluation: Human trajectory reconstruction is evaluated on SLOPER4D using W-MPJPE and WA-MPJPE, while scene geometry is evaluated with Chamfer Distance.Sequences are divided into 100-frame segments; W-MPJPE aligns only the first two frames to emphasize global consistency.
- Ego-view rendering: The metrically scaled point cloud is projected into a virtual camera at the Unitree G1 head sensor to render ego-view RGB-D frames.These visual observations are not consumed by the current policy but could support future perception-conditioned policies, active vision, and semantic scene understanding.
- Ego-view rendering: Occluded regions appear as holes or gaps in rasterized ego-view images, limiting the completeness of the rendered observations.The paper identifies data-driven novel-view synthesis as a direction for hallucinating plausible geometry and appearance for invisible surfaces.
B.1 RL Setup and Training
The RL setup uses terrain-aligned global root references and predominantly data-driven rewards to train physically feasible humanoid motion-tracking policies. Training combines simulator-specific controls, action regularization, and domain randomization for deployment-oriented robustness.
- World-frame conditioning: Global root references are expressed in the terrain-aligned world frame, enabling long-horizon imitation of motions tied to terrain.This alignment lets policies use global root references while retaining observations available on the real robot.
- Action constraints: Actions are clipped to a chosen range and regularized with a bounds loss to discourage bang-bang control during early training.For the Unitree G1, the action magnitude is limited to 8.0 and the bounds-loss coefficient is 0.0005.
- Simulation setup: Isaac Gym runs at 200Hz with control decimation 4, producing an effective policy timestep of 0.02s.The simulator setup is paired with terrain duplication and spatially distributed robot spawns when training on few clips to reduce inter-robot collisions.
- Reward design: Rewards primarily track link and joint positions and velocities, foot contacts, and physical feasibility while minimizing manually designed priors.The reward terms remain consistent across training phases, with annealing for action-rate and ankle-action penalties.
B.6 Termination Criteria
Termination thresholds vary across motion-pretraining, terrain tracking, and RL fine-tuning, with looser tolerances during later stages. The stated reward design balances preserving reference-motion character with recovery behavior while limiting manually injected priors.
- Thresholds by stage: Cartesian tracking-error thresholds are 0.3 for MPT, 0.5 for terrain tracking, and 1.2 for RL fine-tuning.The terrain-tracking threshold is higher because vision-derived references can be noisy, while fine-tuning prioritizes recovery behavior.
- Generalization constraint: More manually added human priors make the motion-tracking system less general.The paper contrasts this objective with classical sim-to-real RL pipelines that rely heavily on practitioner-designed reward shaping.
- Fine-tuning rationale: A loose termination tolerance during fine-tuning helps preserve the essence of data motions while maintaining strong recovery behaviors.The stated rationale is to avoid terminating episodes too readily during RL fine-tuning.
- Robustness: The training pipeline adds domain randomization to simulate unmodeled physical effects and improve policy robustness.These settings are specified through the paper’s domain-randomization and noise configuration.
B.8 Training Data Distribution
The training distribution consists mainly of 123 clips collected through the authors’ reconstruction pipeline, supplemented by 10 flat-terrain walking clips from LaFan. Evaluation proceeds progressively through real-robot stages before final deployment testing.
- Training distribution: 123 self-collected human-motion clips provide 90% of training, while 10 LaFan flat-terrain walking clips provide the remaining 10%.Sampling is class-balanced per clip, producing a 90%/10% split between the authors’ data and LaFan data.
- Training distribution: The clip collection is categorized and its distribution is reported in Table 6.The table summarizes the categories and distribution of the 123 self-collected training clips.
- Real-robot deployment: Deployment code runs onboard the Jetson Orin NX at 50Hz using ROS and the Unitree SDK 2.The implementation uses joint gains of Kp = 75 and Kd = 2 except at the ankle.
- Evaluation stages: Policies are evaluated progressively, beginning with motion-capture tracking and continuing through distillation and heightmap-conditioned policies from the full pipeline.This staged process was used to debug capabilities in the real world before final deployment.