PaperlayerView exact revision on arXiv

Source-linked AI summary

Counterfactual Video Generation Enables Scalable Humanoid Loco-Manipulation

Zihan Wang, Zhen Wu, Pieter Abbeel, Rocky Duan, Jitendra Malik, Carmelo Sferrazza, C. Karen Liu, Guanya Shi, Angjoo Kanazawa

arXiv:2609.38172v1cs.ROcs.CVcs.GR

TL;DR

Collecting diverse, high-quality human–object interaction videos is a practical bottleneck for visual imitation. PRISM expands a few real videos through V2V generation, then uses contact-anchored reconstruction and retargeting to train a depth-based humanoid policy that performs object interactions zero-shot in the real world.

  • Problem

    Finding diverse, high-quality interaction videos with clear full-body views and visible object interactions is costly and impractical at scale.

    Source: §1 · p. 2

  • Method

    PRISM uses V2V to expand a few real videos into counterfactual interactions, then contact-anchored reconstruction and retargeting to form physically plausible demonstrations for policy training.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    PRISM trains a unified whole-body policy for joystick-controlled pick-up, carry, and drop from onboard depth, including zero-shot real-world generalization to unseen object instances and categories absent from generated videos.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The results support video generation as a practical data source for generalizable whole-body interaction.

    Source: §6 · p. 9

    Four seed videos yield 256 generated clips, 137 feasible trajectories, and 129 successful teacher rollouts, a scale insufficient to characterize scaling behavior; reconstruction and retargeting remain practical bottlenecks.

    Source: §7 · p. 9

Abstract

from arXiv · show

Teaching humanoids loco-manipulation skills, such as carrying diverse objects, via visual imitation is a promising path toward generalist robots. However, collecting diverse, high-quality interaction videos, such as clips that clearly show a person's full body and unoccluded interactions with objects, poses a practical barrier to scaling this approach. We propose PRISM, a real-to-sim-to-real framework that overcomes this limitation by amplifying a handful of real videos into a large, diverse training set. PRISM first generates hundreds of diverse "counterfactual" human-object interaction videos via video-to-video (V2V) generation from a few exemplar real videos. Our contact-anchored real-to-sim pipeline then reconstructs both human and object motions, retargeting this imperfect video data into physically plausible trajectories. The intra-class variability across these counterfactual videos lets us train a single policy that generalizes to unseen objects within each category. We demonstrate the full pipeline by deploying this policy on a real robot without any real-world fine-tuning. Using only onboard depth observations, our humanoid picks up, carries, and drops objects, including boxes, barrels, bins, and balls, across novel instances, sizes, and initial configurations.

1 Introduction

Visual imitation could help humanoids acquire coordinated whole-body skills for diverse object interactions, but finding high-quality videos at scale is costly. PRISM expands a few real videos into counterfactual interactions and uses contact-anchored reconstruction and retargeting to train a policy for real-world object handling.

1 Introduction

  • High-quality interaction videos are costly to curate because online clips often lack clear full-body views and visible interactions, despite the abundance of internet video.Human demonstrations provide examples of coordinated motions for approaching, lifting, carrying, and dropping objects.Source: §1 · p. 2
  • Four real-world videos of people carrying boxes are expanded into 256 counterfactual videos featuring boxes, balls, bins, and barrels.The generated clips vary object geometry and initial configurations, and the pipeline retargets imperfect reconstructions into physically plausible trajectories.Source: §1 · p. 2
  • Contact serves as a shared constraint across reconstruction, retargeting, and policy learning, helping turn imperfect video data into physically plausible robot demonstrations.Human motion guides object trajectories, object contact corrects reconstructed human pose, and contact anchors guide robot end-effectors and teacher-policy rewards.Source: §1 · p. 2
  • PRISM trains a unified policy for joystick-controlled pick-up, carry, and drop from onboard depth, generalizing to unseen object instances and categories absent from generated videos.The policy is trained from physically plausible robot–object trajectories derived from counterfactual videos.Source: §1 · pp. 2–3

2 Related Work

Prior work studies whole-body humanoid interaction and learning skills from human video, while diverse, high-quality interaction footage remains difficult to collect at scale. Video models have been used for both inference-time planning and offline data synthesis; PRISM uses grounded video-to-video generation to expand human demonstrations.

Humanoid Loco-Manipulation

  • Some humanoid systems track human references [5], but require dense motions or object trajectories at inference, while newer policies use sparse goals or interaction-centric representations [8] [9] [10].PRISM instead supports joystick-controlled pick-up, carry, and drop from onboard depth without reference or motion-capture systems during deployment.Source: §2 · p. 3

Learning from Human Video

  • Human-video methods learn locomotion and dynamic object interactions [1] [5] [11] [12], but collecting diverse, high-quality interaction clips at scale remains challenging.PRISM addresses this data challenge by expanding a few real videos into interactions used to train a single real-world policy.Source: §2 · p. 3

Video Model as Data++

  • Video models support inference-time planning [13] [14] [2] and offline demonstration synthesis [15] [16]; PRISM uses offline grounded V2V to generate counterfactual human interactions.Conditioning on real demonstrations preserves realistic motion, lighting, and affordance-aware contacts while enabling variation across object categories, poses, and instances.Source: §2 · p. 3

3 Real-to-Sim Data Acquisition

PRISM expands real interaction videos and reconstructs human, scene, and object motion in a shared world frame before retargeting them into humanoid–object trajectories. Contact anchors connect object reconstruction and robot retargeting, helping convert imperfect monocular data into physically plausible demonstrations.

3 Real-to-Sim Data Acquisition

  • The data-acquisition task is to recover camera parameters, a static scene mesh, dynamic-object geometry and poses, and 4D human motion in a unified world frame, then retarget them for policy learning.The source is a monocular video of a person interacting with a dynamic object in a static scene.Source: §3 · p. 3

3.1 Counterfactual Interaction Video Generation

  • V2V generation uses a seed video and category-level prompt to replace the object while retaining the scene, lighting, viewpoint, and coarse task structure.Four seed recordings and 16 samples per category produced 256 counterfactual videos across boxes, bins, barrels, and balls.Source: §3.1 · pp. 3–4
  • Generated clips vary human behavior as well as object appearance, with people adapting posture, hand spacing, contact strategy, or carrying behavior to object affordances.The resulting interactions provide behavior-level variation beyond geometry-only augmentation.Source: §3.1 · p. 4

3.2 Contact-Anchored Real-to-Sim

  • The reconstruction backend is extended to recover dynamic-object geometry and motion; human–object contact couples the two trajectories and helps correct monocular motion errors.During contact, the object is anchored to the SMPL-X palm using a fixed palm-relative transform; outside contact, it remains supported by the scene.Source: §3.2 · pp. 4–5
  • Contact-anchored retargeting treats reconstructed human motion as a kinematic reference and object-frame anchors as the interaction interface for correcting noisy motions.The anchors specify robot end-effector targets on the object, enabling physically plausible robot–object trajectories despite imperfect reconstruction.Source: §3.2 · p. 5
  • Monocular reconstruction remains imperfect; PRISM optimizes sufficiently close reconstructions into physically plausible robot demonstrations.The authors state that existing pipelines cannot recover physically plausible human–object motions directly from monocular video.Source: §3.2 · p. 5

4 Learning a Unified Visuomotor Interaction Policy

Cross-domain evaluation reports that PRISM-trained policies transfer to OMOMO and unseen PRISM-OOD interactions, while OMOMO-trained policies transfer poorly to PRISM. The approach trains a unified policy by distilling a simulation teacher into a depth-based student.

4 Learning a Unified Visuomotor Interaction Policy

  • The teacher formulates interaction as co-tracking reconstructed humanoid motion and dynamic object trajectories, then distills its privileged simulation policy into a unified depth-based student.The student uses onboard depth and joystick commands for zero-shot sim-to-real deployment.Source: §4 · p. 5, §4.1 · p. 5

4.1 Training a Co-Tracking Teacher Policy

  • The real-robot policy handles object-pose, within-category topology and appearance, scale, and approach-distance variations, including approaching up to 4.3 feet before grasping.When an object is within reach, it can grasp directly using onboard depth.Source: Figure 3 · p. 6
  • The teacher observes reference motion, robot state, and current and target object states; its contact-aware reward improves grasping and carrying under noisy monocular reconstructions.The reward encourages reaching desired contact positions and exceeding a predefined contact-force threshold.Source: §4.1 · p. 6
  • PRISM-trained policies transfer to OMOMO and unseen PRISM-OOD interactions, while OMOMO-trained policies transfer poorly to PRISM.PRISM-ID evaluates the 80 training interactions.Source: Table 1 · p. 7

4.2 Distilling a Unified Depth-Based Student Policy

  • The student combines DAgger and PPO objectives, using proprioception, joystick commands, and onboard depth without MOCAP or reference motion at deployment.Training gradually increases λ and keeps it at 0.9 for the final 20K iterations.Source: §4.2 · p. 7
  • Student training randomizes camera pose and depth artifacts; box-only actor warm-start accelerates convergence, and the student trains for 28K iterations on 8 NVIDIA L40S GPUs.The training setup also uses 4096 environments per GPU.Source: §4.2 · p. 7

5 Results

The evaluation tests PRISM data and policies through cross-domain transfer, real-world deployment, and component ablations. PRISM-trained policies generalize across OMOMO and PRISM interactions, and contact-aware components improve downstream task success.

5.1 Reconstruction Quality from Real-to-Sim

  • 126 OMOMO sequences and an earlier PRISM student evaluated on 80 PRISM-ID and 48 PRISM-OOD interactions; success required completing pick–carry–drop at the target.The OMOMO set comprised 90 training and 36 held-out test sequences.Source: §5.1 · p. 7

5.3 Ablation Study

  • Anchored object-pose reconstruction, contact-aware retargeting, and contact reward progressively improve downstream task success on PRISM-ID and PRISM-OOD.The baseline initializes object geometry and poses with SAM3D, tracks with FoundationPose, and retargets with OmniRetarget.Source: Table 3 · p. 8, §5.3 · p. 8

5.1 Reconstruction Quality from Real-to-Sim

  • PRISM-trained policies generalize across OMOMO and PRISM interactions, whereas OMOMO-trained policies transfer poorly to PRISM, especially for out-of-domain objects.The reported OMOMO evaluation used 126 sequences, while the PRISM comparison used an earlier student on 80 PRISM-ID and 48 PRISM-OOD interactions.Source: §5.1 · pp. 7–8

5.2 Real-World Deployment

  • Real-world trials tested all test objects zero-shot across five varied placements, requiring a grasp and stable 3 m carry without drops or falls.The test objects were not used for training through scans or reconstructions.Source: §5.2 · p. 8

5.3 Ablation Study

  • Figure 4 compares planar positions and pooled yaw-bin counts for 137 generated clips and four upright seeds, distinguishing upright from lying-down objects.The yaw counts use six 30° bins modulo 180°.Source: §5.3 · p. 8, Figure 4 · p. 9
  • 35° ramp and 0.43 m elevated-support pickups succeed without additional tuning, while the policy fails at 45° and 0.45 m.The policy was trained only on flat terrain; elevated V2V training data may extend the demonstrated range.Source: Figure 5 · p. 9, §5.3 · p. 9

6 Conclusion

PRISM expands a few real videos into diverse humanoid demonstrations using contact-anchored reconstruction and retargeting. The resulting depth-based policy handles unseen objects zero-shot in the real world.

6 Conclusion

  • PRISM expands a few real videos into diverse humanoid demonstrations.Contact-anchored reconstruction and retargeting convert imperfect counterfactual videos into physically plausible robot–object trajectories.Source: §6 · p. 9
  • The resulting unified depth-based policy picks up, carries, and drops unseen objects zero-shot in the real world.The conclusion identifies video generation as a practical data source for generalizable whole-body interaction.Source: §6 · p. 9

7 Limitations and Future Work

PRISM’s practical limitations arise in reconstruction and retargeting, training-data scale, object-physics modeling, and the simulated object types it can handle.

Reconstruction and Retargeting

  • Monocular 4D recovery struggles to preserve consistent contacts and interaction dynamics, and its errors can propagate through retargeting to compromise robot demonstrations.The authors suggest that contact-aware retargeting and simulation-based validation may improve reliability.Source: §7 · p. 9

Data Scale

  • Four seed videos yield 256 clips, 137 feasible trajectories, and 129 successful teacher rollouts, insufficient to characterize scaling behavior.Reconstruction and retargeting remain practical bottlenecks.Source: §7 · p. 9

Object Physics

  • Category-specific nominal masses and mesh-derived inertias do not capture instance-specific surface properties or internal mass distributions, which can cause sim-to-real contact mismatch.The setup uses coupled mass–inertia scaling and randomized friction.Source: §7 · p. 9

Simulation Limitations

  • Rigid-body simulation may limit performance on articulated or deformable objects whose contact geometry changes during interaction.The authors give foldable chairs as an example, where shifting joints may cause unstable grasps or loss of control.Source: §7 · p. 9

A Counterfactual Video Generation Details

The generation pipeline uses four real carrying videos as category-conditioned templates for counterfactual clips, then evaluates the policy on unseen in-domain and out-of-domain objects.

A Counterfactual Video Generation Details

  • Four seed videos of people carrying boxes condition V2V generation, which replaces the object while preserving the scene and coarse task structure.The generated interactions allow object properties and human behavior to vary.Source: A Counterfactual Video Generation… · p. 12
  • Sixteen samples per category across four categories and four seeds produce 256 counterfactual videos for the real-to-sim pipeline.Each seed contributes 64 clips spanning boxes, bins, barrels, and balls.Source: A Counterfactual Video Generation… · p. 12
  • The category-level prompt leaves individual geometry, size, and pose unspecified, allowing within-category object variation and corresponding human-interaction adaptation.Figure 6 depicts the shared template, with the object category substituted for <CLS>.Source: A Counterfactual Video Generation… · p. 12, Figure 6 · p. 12

B Real-World Test Objects

  • Real-world tests use unseen instances from the four generated categories and objects from categories absent from generated training videos, without training scans or reconstructions.Both groups vary in appearance, geometry, weight, and scale.Source: B Real-World Test Objects · p. 12, Figure 7 · p. 12
  • Each test object is evaluated in five trials with varied initial poses and distances; success requires grasping and carrying stably for at least 3 m without dropping or falling.Table 2 reports the success rates.Source: B Real-World Test Objects · p. 12

C Training Data and Simulation Evaluation

The section reports trajectory recovery and teacher-rollout counts, then specifies the simulation data and success criterion, while noting that the main-paper cross-domain results use an earlier student.

Reconstruction and retargeting

  • 137 of 256 generated videos yield feasible robot–object trajectories; the others fail the constrained solver, mainly because of severe collisions.Some generated objects are too large for the Unitree G1 to carry without object–body collisions.Source: C Training Data and Simulation… · p. 13

Training data

  • Training the teacher on the 137 trajectories yields 129 successful rollouts, which supply simulation motion references for student distillation.The unified depth-based student is trained for 28K iterations on those 129 demonstrations.Source: C Training Data and Simulation… · p. 13

Reported evaluation setting

  • The reported main-paper cross-domain results use an earlier student trained from 80 successful rollouts of a 20K-iteration teacher checkpoint, not the updated 129-demonstration student.Its PRISM-ID evaluation set comprises those 80 interactions.Source: C Training Data and Simulation… · p. 13

Simulation evaluation data

  • The OMOMO comparison uses 126 sequences split into 90 training and 36 held-out test sequences, alongside 48 reconstructed out-of-domain interactions from four absent categories.The out-of-domain set includes 12 instances each from tables, chairs, lamps, and monitors.Source: C Training Data and Simulation… · p. 13

Simulation evaluation protocol

  • Simulation success requires completing pick–carry–drop and placing the object at the target, whereas the real-world metric measures grasping and stable carrying for at least 3 m.The two evaluation metrics therefore use different task endpoints.Source: C Training Data and Simulation… · p. 13

D Training and Distillation Hyperparameters

The section describes large-scale teacher and student training, depth-based deployment, contact rewards, and the documented observation, reward, and optimization specifications.

D Training and Distillation Hyperparameters

  • Teacher and student use 3-layer MLPs trained for 40K and 28K iterations, respectively, with 32,768 environments across eight NVIDIA L40S GPUs.Object physics and payload sensitivity are detailed in Appendix E.Source: D Training and Distillation… · p. 13
  • At deployment, the student uses proprioception, joystick commands, and onboard depth; stereo depth is estimated externally, and the policy runs at 50 Hz without reference motion or motion capture.The depth camera operates at 30 Hz, and the authors report that stereo depth substantially reduces the sim-to-real gap in depth observations.Source: D Training and Distillation… · p. 13
  • The student combines DAgger imitation with PPO, is warm-started from box-only actor weights, and is trained on four object categories with randomized camera pose and depth corruptions.Training from scratch also succeeds; warm starting accelerates convergence.Source: D Training and Distillation… · p. 13

D.1 Observation and Reward Specifications

D.2 Optimization and Distillation Settings

  • Table 4 defines T/S as teacher/student, A/C as actor/critic, and marks privileged student-critic inputs that are not used by the deployment policy.Table 6 documents hyperparameters for the teacher and 129-rollout student and distinguishes the earlier 80-rollout evaluation.Source: Table 4 · p. 14, Table 6 · p. 15

E Object Physics and Payload Sensitivity

The supplementary section examines object-physics and payload settings, compares V2V with geometric augmentation, and reports contact-anchor reliability, pipeline runtime, and additional interaction reconstructions.

E Object Physics and Payload Sensitivity

  • Category-specific masses and mesh-derived inertias are jointly scaled by s∼U(0.33, 3.0), while shared friction is randomized and instance-specific surface properties are not identified.Dynamic friction is set to a sampled fraction of static friction; the physics model does not identify instance-specific mass distributions either.Source: E Object Physics and Payload Sensitivity · p. 15, Table 7 · p. 15
  • Payload sensitivity evaluates simulation success across five payload ranges for objects weighing 0.11–4.99 kg, using an earlier 80-rollout student.The tested real-world objects span a box at 0.11 kg to a chair at 4.99 kg.Source: E Object Physics and Payload Sensitivity · p. 16, Table 8 · p. 16

F V2V versus Geometric Augmentation

G.1 Contact-Anchor Reliability

  • Contact anchors use 2D overlap for timing and mesh intersections with the palm-center segment for object-frame placement, but retained solutions lack formal contact or feasibility guarantees.Pose and mesh errors can cause penetration and inverse-kinematics failure; the pipeline retains non-penetrating IK solutions and full-trajectory teacher rollouts.Source: §G.1 · p. 16

G.2 Pipeline Runtime

  • Table 10 reports mean processing times per 8-second clip, with reconstruction and retargeting on one NVIDIA L40S GPU and V2V latency recorded separately.The supplied text does not state the table's runtime values.Source: §G.2 · p. 16, Table 10 · p. 16

G.3 Additional Interaction Examples

  • Reconstructions show ball rolling and door opening with suitable contact points and object representations, but these examples are not policy or real-world evaluations.Tool use, regrasping, broader deformable-object manipulation, and extreme dynamics remain untested.Source: §G.3 · p. 16, Figure 8 · p. 16
Loading 2609.38172v1…