PaperlayerView exact revision on arXiv

Source-linked AI summary

VICAR: Video Imitation with Contact-Aware Retargeting for Humanoid Interactive Skills

Dvij Kalaria, Bike Zhang, S. Shankar Sastry

arXiv:2610.13079v1cs.RO

TL;DR

Humanoids must transfer third-person demonstrations across different morphology and contact geometry, while preserving the world-frame contacts that define contact-rich tasks. VICAR reconstructs task-relevant obstacles, retargets motion with collision and global contact-pose penalties while preserving speed, and trains a generator conditioned on movable contact points. Success rose from 76.2% to 99.8% for tabletop pickup and from 56.0% to 95.5% for ladder climbing relative to OmniRetarget.

  • Problem

    Humanoids must transfer third-person demonstrations across different morphology and contact geometry, while preserving the world-frame contacts that define contact-rich tasks.

    Source: §I · p. 1

  • Method

    VICAR reconstructs task-relevant obstacles, retargets motion with collision and global contact-pose penalties while preserving speed, and trains a generator conditioned on movable contact points.

    Source: §I · pp. 1–2

  • Results

    Success rose from 76.2% to 99.8% for tabletop pickup and from 56.0% to 95.5% for ladder climbing relative to OmniRetarget.

    Source: C. Retargeting Results · p. 6

  • Takeaways & Limitations

    Across ten contact-rich motions, VICAR adapts video-derived skills to new object or ball positions before a tracking policy executes them on hardware.

    Source: §VI · p. 8

    Ladder hardware deployment failed because the same blind tracking policy stumbled, despite a smooth retargeted trajectory.

    Source: §V · p. 8

Abstract

from arXiv · show

Third-person human videos offer a scalable alternative to task-specific reward engineering, teleoperation, and manually scripted demonstrations for humanoid loco-manipulation. Recent video-to-humanoid systems have shown that noisy human motion, object trajectories, and contacts can be extracted from unstructured videos and refined into deployable skills. However, a key gap remains for contact-rich imitation: the task is often defined by world-frame contacts whose poses, timing, and speed must survive embodiment mismatch and runtime scene changes. Towards this, we present VICAR (Video Imitation with Contact-Aware Retargeting), a pipeline for learning humanoid interactive skills from third-person human videos. The system first recovers human motion, reconstructs task-relevant obstacles from monocular video, and then retargets the motion to a humanoid while preserving global contact poses, motion speed, and collision clearance. Unlike retargeting methods that only remove penetration or preserve local interaction geometry, our formulation represents user-specified contact poses and collision clearance as high-priority Lagrangian penalty terms and optimizes the full motion to remain smooth through contact-rich transitions. The resulting motions are used to train a contact-conditioned motion generator whose inputs include movable contact points enabling runtime adaptation to new contact positions. We demonstrate the framework on ten diverse third-person video skills, including six table-tennis serves, three object-pickup tasks, and ladder climbing, and deploy the learned skills on hardware. These results suggest that contact-preserving retargeting is a practical step toward humanoids that learn useful interactive behavior by watching humans.

I. INTRODUCTION

VICAR addresses the difficulty of transferring contact-rich human demonstrations to humanoids with different morphology and contact geometry. It preserves task-defining global contacts during retargeting and uses movable contact points to adapt generated skills at runtime.

I. INTRODUCTION

  • Third-person video imitation is difficult because embodiment mismatch can shift task-defining world-frame contacts even when the retargeted motion resembles the human motion.For the shorter G1, a table near an adult’s waist is near chest height; matching human joints or keypoints can therefore miss the object.Source: §I · p. 1, Figure 2 · p. 2
  • VICAR targets a different emphasis from prior video-to-humanoid systems: preserving specific global contact poses that may move at runtime.The cited systems reconstruct human-scene geometry, co-track robot and object states, or preserve local interaction relations.Source: §I · p. 1
  • VICAR reconstructs task-relevant obstacles and retargets motion with high-priority collision and global contact-pose penalties while preserving motion speed.An initial feasible reference is refined to keep the full motion smooth through contact-rich transitions.Source: §I · pp. 1–2
  • The retargeted motions seed a contact-conditioned generator with movable contact points that can adapt serves to ball positions and pickups to object placements.A downstream policy tracks the generated reference using proprioception and gravity observations.Source: §I · pp. 1–2
  • The framework is evaluated on ten video skills, including six table-tennis serves, object pickups, and ladder climbing; all but ladder climbing are evaluated on hardware.Figure 1 shows temporally aligned human and robot frames for representative serving, bimanual and under-table pickup, and ladder-climbing tasks.Source: §I · pp. 2–3, Figure 1 · p. 2

II. RELATED WORK

Related work spans video-derived humanoid skills, motion retargeting, scene reconstruction, and contact-conditioned control. VICAR focuses on embedding task-specified world-frame contacts into retargeted references, particularly when contacts are externally determined or movable at runtime.

a) Humanoid learning from human videos

  • Video-learning systems broaden reference acquisition and policy learning, while VICAR focuses on geometric adaptation that keeps sparse task-defining contacts valid after embodiment transfer.Prior examples include VideoMimic [1], HDMI [2], HumanX, and SUGAR [15], [16]; the passage says they do not explicitly preserve runtime-changing global contact poses.Source: §II · p. 3

b) Human-to-humanoid retargeting

  • VICAR represents selected contacts as world-frame pose objectives and preserves source speed, unlike methods emphasizing keypoint matching or local interaction relations.The comparison table defines “global contact poses” as task-specified world-frame pose objectives; the cited tabletop example describes discontinuous references despite local clearance.Source: §II · p. 3, Table I · p. 3

c) Scene reconstruction for robot learning

  • VICAR reconstructs semantic regions likely to constrain the robot rather than requiring a complete scene model.Depth is unprojected and scale is resolved from human-scene geometry, retaining surfaces for collision and contact reasoning.Source: §II · p. 3

d) Contact-conditioned motion generation

  • VICAR places a contact-conditioned generator between retargeting and policy learning when the world specifies an exact pose that must be embedded in the reference.This separates geometric adaptation from low-level control; policy-level contact commands alone are described as sufficient when the controller can resolve the interaction from motion and contact mode.Source: §II · p. 3

III. METHOD

VICAR takes monocular third-person interaction video and produces a policy that tracks a humanoid reference, with selected contact poses variable at inference. It recovers human motion and contact annotations, reconstructs task-relevant obstacles, and converts selected scene regions into scaled voxel geometry.

A. Human Motion and Contact Annotation

  • Contact annotations specify a frame, robot body, and desired global pose; in the prototype, contact frames, body parts, and orientations are labeled manually.A contact is a pose objective at a timestep, not a model of force, contact mode, or sliding.Source: A. Human Motion and Contact Annotation · p. 3

B. Obstacle Reconstruction from Video

  • Obstacle reconstruction uses monocular depth and semantic segmentation to select task-relevant surfaces, such as a tabletop, floor patch, or support surface.GPT-4o mini selects likely contact regions using the segmented frame and task description.Source: B. Obstacle Reconstruction from Video · p. 3
  • Selected depth pixels are transformed into task-world points, scaled using human-height estimates, filtered, and converted into voxelized mesh obstacles.The voxel grid uses 0.05 m cuboids, with at least five points required for an obstacle voxel.Source: B. Obstacle Reconstruction from Video · pp. 3–4

C. Contact- and Speed-Preserving Retargeting

Retargeting first fits an initial robot trajectory using an interaction-mesh keypoint formulation, then refines it with contact, collision, and speed objectives. The initial pass omits obstacle penalties, providing a motion prior before scene geometry is enforced.

C. Contact- and Speed-Preserving Retargeting

  • Retargeting uses two stages: an initial feasible robot trajectory, followed by optimization with contact and collision penalties while preserving end-effector speed.The refinement operates on the full trajectory and includes temporal regularization and joint feasibility in the initial fitting problem.Source: C. Contact- and Speed-Preserving… · p. 4

1) Stage 1: Interaction-Mesh Initialization:

  • The initial stage uses an OmniRetarget-style interaction mesh to match local keypoint geometry through a Laplacian-coordinate deformation loss.It fits active joints with temporal regularization and joint feasibility.Source: 1) Stage 1: Interaction-Mesh… · p. 4
  • The initial pass omits obstacle-avoidance and penetration penalties, so its trajectory prior is not forced around potentially mismatched scene geometry.Implementation uses the batched joint-angle solver with an empty obstacle list.Source: 1) Stage 1: Interaction-Mesh… · p. 4

2) Stage 2: Contact- and Speed-Preserving Refinement:

The refinement optimizes the robot trajectory using key-link motion, anchor, and collision terms. High-priority penalties favor clearance and desired global contact poses, while smoothing and joint clamps support feasible motion.

2) Stage 2: Contact- and Speed-Preserving Refinement:

  • The trajectory objective uses robot key-link positions and orientations, source human positions and orientations, and semantic correspondence between human and robot keypoints.The robot configuration includes its floating base and joints; orientation distance is geodesic on SO(3).Source: 2) Stage 2: Contact- and… · pp. 4–5
  • High-priority collision and anchor penalties prioritize obstacle clearance and desired global contact poses, but gradient descent does not guarantee exact equality.When an annotation conflicts with feasible geometry, the optimizer returns the nearest feasible pose and reports its residual.Source: 2) Stage 2: Contact- and… · p. 5
  • The full trajectory is optimized over each task’s active joints, with collision penalties against cuboid or mesh obstacles, anchor costs over a small time window, and joint clamps.Source: 2) Stage 2: Contact- and… · p. 5

D. Contact-Conditioned Motion Generator

The motion generator uses style, movable contact conditions, and temporal context to predict reference robot and object motion. Contact-point augmentation creates optimized variants for training across movable contact poses.

D. Contact-Conditioned Motion Generator

  • The generator receives a motion-style vector, movable contact conditions, and temporal context, then predicts reference robot and object motion.The style vector identifies a source video or motion family; contact conditions encode task-specific positions or poses.Source: D. Contact-Conditioned Motion Generator · p. 5
  • Contact-point augmentation varies serve hit points, pickup object poses, and ladder support anchors to produce optimized variants for training.Serve variants preserve toss and style; bimanual pickup variants synchronize both hands with the box.Source: D. Contact-Conditioned Motion Generator · p. 5

E. RL Environment and Training

The G1 tracking policy is trained in Isaac Lab with joint-position actions and parallel simulated environments. Training uses PPO and randomizes robot and contact parameters, with occasional random pushes.

E. RL Environment and Training

  • Training uses 4096 parallel environments, a nominal 50 Hz policy rate, and a 25 Hz low-frequency variant.The physics step is 0.005 s with action decimation of 4; episodes follow the motion length.Source: E. RL Environment and Training · p. 5
  • PPO trains actor and critic MLPs with hidden dimensions [512, 256, 128] for up to 30,000 iterations.The actor and critic use ELU activations, and checkpoints are saved every 500 iterations.Source: E. RL Environment and Training · p. 5
  • Domain randomization varies friction, restitution, joint offsets, torso center of mass, and periodic linear and angular pushes.The randomization follows a minimal strategy without task-specific curricula.Source: E. RL Environment and Training · p. 5

F. Rewards

The reward combines motion tracking with penalties for undesirable actions and contacts. The evaluation covers contact-aware retargeting questions across ten video skills, including movable-contact tasks.

F. Rewards

  • The reward combines anchor, body-pose, body-velocity, and optional object tracking with action-rate, soft joint-limit, and undesired-contact penalties.Orientation errors use geodesic quaternion distance, and Table II gives the G1 policy weights.Source: F. Rewards · pp. 5–6

IV. EXPERIMENTS

  • The evaluation asks whether contact-aware retargeting improves contact error, collision, and smoothness, and whether generated references adapt and transfer to hardware.It also compares against OmniRetarget-style retargeting where global contact pose matters.Source: §IV · p. 6

A. Tasks

  • The ten video skills comprise six distinct table-tennis serves, three object-pick tasks, and ladder climbing with sequential hand and foot supports.Serve skills vary toss and ball-hit points, while pickup skills vary object poses.Source: A. Tasks · p. 6

B. Baselines and Metrics

The comparison controls contact penalties, augmentations, and policy training to focus on retargeting quality. It reports penetration, residual, smoothness, speed-preservation, and policy-success measures, alongside figures on augmentation.

B. Baselines and Metrics

  • All methods receive identical post-hoc contact-pose penalties and anchor-pose augmentations, with matching policy training settings to isolate retargeting quality.Compared methods include OmniRetarget, GMR, PHC, VideoMimic, and an ablation without speed preservation.Source: B. Baselines and Metrics · p. 6
  • Penetration metrics include mean signed depth, mean position residual at annotated anchors, and the fraction of frames exceeding 1 cm penetration.The latter is defined as the infeasibility rate.Source: B. Baselines and Metrics · p. 6
  • Smoothness uses mean joint jerk and maximum joint velocity, while speed preservation measures end-effector displacement-magnitude error against the source.The speed metric covers wrists and feet.Source: B. Baselines and Metrics · p. 6
  • Figure 5 presents motion augmentation by contact points, and Figure 6 plots held-out in-distribution success rate against augmented-trajectory count.The captions identify the figure scopes but do not state comparative outcomes.Source: Figure 5 · p. 6, Figure 6 · p. 6
  • Downstream reinforcement-learning policy success rate is measured in simulation over 1000 trials.Source: B. Baselines and Metrics · p. 6

C. Retargeting Results

VICAR’s strongest reported retargeting gains are on obstacle-critical pickup and ladder tasks; for tabletop pickup, it preserves contact and speed along a smooth path where post-hoc correction is discontinuous.

C. Retargeting Results

  • For tabletop pickup, OmniRetarget’s platform-face transition makes later non-penetration correction discontinuous, while VICAR preserves contact and speed along a smooth path.Source: C. Retargeting Results · p. 6
  • The evaluation task suite includes ladder climbing with hand and foot rung supports.Source: C. Retargeting Results · p. 7, Table III · p. 7
  • 99.8% vs 76.2% tabletop pickup and 95.5% vs 56.0% ladder climbing: VICAR exceeds OmniRetarget on both obstacle-critical tasks.Source: C. Retargeting Results · p. 6, Table IV · p. 7
  • Table IV reports retargeting and downstream tracking results, including VICAR anchor residual and infeasibility, with speed ablation on two obstacle-critical tasks.Source: Table IV · p. 7

D. Contact-Point Generalization

The section describes held-out contact-point evaluation, hardware deployment, and the framework’s limitations. VICAR’s conclusion reports results across ten contact-rich motions, from table tennis to pickup tasks and ladder climbing.

D. Contact-Point Generalization

E. Hardware Deployment

  • Deployment used no real-robot training data, so it is zero-shot sim-to-real in that specific sense, not visual sim-to-real.Source: E. Hardware Deployment · p. 7
  • Hardware success was 21/30 (70.0%) for table-tennis serves, 10/10 (100%) for tabletop pickup, 7/10 (70.0%) for under-table pickup, and 6/10 (60.0%) for bimanual box pickup.Source: E. Hardware Deployment · p. 7

V. LIMITATIONS

  • Contact frames, body parts, and SE(3) poses are manually annotated, taking approximately 20–30 minutes per skill; new task families still need per-task annotation and scene reconstruction.Source: §V · p. 7
  • Unreachable contacts, unsupported balance, or dexterous manipulation beyond the hand model can fail; ladder deployment also failed when the blind tracking policy stumbled despite a smooth retargeted trajectory.Source: §V · p. 8

VI. CONCLUSION

  • VICAR’s reported scope spans ten contact-rich motions, from table-tennis serves to pickup tasks and ladder climbing.Source: §VI · p. 8
Loading 2610.13079v1…