PaperlayerView exact revision on arXiv

Source-linked AI summary

Dexterous Manipulation Policies from RGB Human Videos via 3D Hand-Object Trajectory Reconstruction

Hongyi Chen, Tony Dong, Tiancheng Wu, Liquan Wang, Yash Jangir, Yaru Niu, Yufei Ye, Homanga Bharadhwaj, Zackory Erickson, Jeffrey Ichnowski

arXiv:2602.09013v2cs.ROcs.CV

TL;DR

Dexterous manipulation learning needs scalable supervision beyond specialized hardware, but RGB human videos lack robot actions and precise 3D interaction information. VIDEOMANIP reconstructs 3D hand-object trajectories from RGB videos and uses optimized grasp modeling plus synthesized demonstrations; policies trained from these data achieve 62.86% average success across seven real-world tasks.

  • Problem

    Existing manipulation data collection often requires specialized hardware or controlled capture, while RGB videos lack robot actions and precise 3D information needed for supervision.

    Source: §I · p. 1

  • Method

    VIDEOMANIP reconstructs 3D hand-object trajectories from monocular RGB videos, retargets human motion to robot hands, optimizes contacts, and synthesizes diverse demonstrations.

    Source: §I · p. 2

  • Results

    62.86% average success rate was achieved across all real-world manipulation tasks using reconstructed and synthesized trajectories.

    Source: B. Manipulation Experiments · p. 7

  • Takeaways & Limitations

    The reconstructed trajectories provide physically grounded supervision for training dexterous grasping and manipulation policies without specialized sensing devices.

    Source: §V · p. 7

    The framework depends on multiple 3D vision models that can introduce compounding errors and currently assumes approximately static camera setups.

    Source: §V · p. 7

Abstract

from arXiv · show

Multi-finger robotic hand manipulation and grasping are challenging due to the high-dimensional action space and the difficulty of acquiring large-scale training data. Existing approaches largely rely on human teleoperation with wearable devices or specialized sensing equipment to capture hand-object interactions, which limits scalability. In this work, we propose VIDEOMANIP, a device-free framework that learns dexterous manipulation directly from RGB human videos. Leveraging recent advances in computer vision, VIDEOMANIP reconstructs explicit 3D robot-object trajectories from monocular videos by estimating human hand poses, object meshes, and retargets the reconstructed human motions to robotic hands for manipulation learning. To make the reconstructed robot data suitable for dexterous manipulation training, we introduce hand-object contact optimization with interaction-centric grasp modeling, as well as a demonstration synthesis strategy that generates diverse training trajectories from a single video, enabling generalizable policy learning without additional robot demonstrations. In simulation, the learned grasping model achieves a 70.25% success rate across 20 diverse objects using the Inspire Hand. In the real world, manipulation policies trained from RGB videos achieve an average 62.86% success rate across seven tasks using the LEAP Hand, outperforming retargeting-based methods by 15.87%. Project videos are available at videomanip.github.io.

I. INTRODUCTION

VIDEOMANIP addresses the scalability limits of specialized hardware and robot demonstrations by learning dexterous manipulation from RGB human videos alone. It reconstructs 3D hand–object trajectories, optimizes contacts, and synthesizes demonstrations for policy learning.

Motivation

  • Specialized hardware, controlled environments, and direct human involvement limit the scalability of prior human-data approaches.RGB videos are more widely available, but they lack robot actions and precise 3D information.Source: §I · pp. 1–2

Evaluation

  • VIDEOMANIP evaluates learning from progressively relaxed data conditions, including in-scene and in-the-wild videos.In-the-wild videos require gravity-based alignment because they lack robot-camera extrinsic calibration.Source: §I · pp. 1–2

Related Work

  • Prior approaches often depend on specialized sensing, robot demonstrations, deployment-specific videos, or pre-scanned object models, and have limited multi-fingered-hand deployment.These dependencies restrict scalability or data diversity.Source: §I · pp. 1–2

Contribution

  • VIDEOMANIP learns dexterous grasping and manipulation entirely from RGB human videos without wearables, external sensors, robot demonstrations, or pre-scanned object meshes.It estimates human hand poses, object meshes, and object scales, then retargets reconstructed motions to robot hands.Source: §I · p. 2
  • The framework reconstructs explicit 3D trajectories and uses contact optimization, interaction-centric grasp modeling, and trajectory synthesis to support dexterous policy training.DemoGen generates diverse trajectories from a single reconstructed video trajectory.Source: §I · p. 2

A. Manipulation Learning from Human Videos

Human videos offer abundant data for manipulation learning but do not directly provide robot-executable actions or unambiguous hand–object interactions. Existing approaches therefore commonly retain robot demonstrations, specialized hardware, or deployment-specific collection requirements.

Motivation

  • Human videos are abundant, but they lack robot-executable actions and often contain ambiguous hand–object interactions.This makes learning dexterous manipulation policies from them challenging.Source: A. Manipulation Learning from Human… · p. 2

Prior Approaches

  • Several video-based methods still require robot demonstrations to fine-tune models pretrained on in-the-wild datasets.This limits generalization from human data alone.Source: A. Manipulation Learning from Human… · p. 2
  • Video-based alternatives also extract intermediate representations such as object affordances and point flows, or predict future object trajectories with world and generative video models.The passage presents these as parallel approaches to manipulation learning.Source: A. Manipulation Learning from Human… · p. 2
  • Other approaches use specialized sensing hardware, including smart glasses, wearable headsets, or hand-motion capture systems.These systems introduce additional hardware dependencies.Source: A. Manipulation Learning from Human… · p. 2

B. Human-Object Interaction Reconstruction from Videos

Human–object reconstruction methods increasingly infer hand poses, object representations, and contact from visual data, enabling reconstructed interactions to support robotic manipulation. However, existing approaches still face reconstruction inaccuracies, long runtimes, or dependence on additional robot demonstrations.

Reconstruction Methods

  • Modern reconstruction methods infer low-dimensional hand pose and shape representations and support diverse object meshes, point clouds, and primitives.Transformer-based models and strong 3D supervision facilitate shared modeling across object categories.Source: B. Human-Object Interaction… · p. 2

Interaction Modeling

  • Joint hand–object reasoning increasingly uses diffusion models, implicit signed distance fields, and differentiable pose estimation to optimize 3D contact.These techniques make interaction reconstruction from images and videos increasingly feasible.Source: B. Human-Object Interaction… · p. 2

Robotic Applications

  • Reconstructed hand–object interactions have been used to train dexterous robot grasping and manipulation policies, often by retargeting human motions to robot actions.This connects visual reconstruction with robotic control.Source: B. Human-Object Interaction… · p. 2

Limitations

  • Existing reconstruction-based methods may require extensive filtering for inaccurate reconstructions, incur long reconstruction times, or rely on additional robot demonstrations.These limitations constrain direct use of reconstructed human interactions for policy learning.Source: B. Human-Object Interaction… · p. 2

III. METHOD

VIDEOMANIP reconstructs robot-relevant hand–object trajectories from RGB human videos and uses them to learn dexterous grasping and manipulation policies. Its pipeline combines 3D reconstruction, trajectory generation, and policy learning from reconstructed actions and object point clouds.

3D Reconstruction

  • VIDEOMANIP reconstructs hand mesh H and object mesh O from RGB human videos to produce explicit 3D robot hand trajectories and object pose changes.The reconstruction supports finger-level manipulations and resulting object motion.Source: §III · pp. 2–3

Pipeline

  • The framework contains a reconstruction stage followed by grasping and manipulation policy learning from reconstructed hand actions and object point clouds.The overview organizes these stages as Sec. III-A and Sec. III-B.Source: §III · p. 2, Figure 2 · p. 3

Assumptions

  • The data assumptions include static-camera egocentric videos and one video per task or object, covering both in-scene and in-the-wild recordings.The method is designed around limited video availability for each task or object.Source: §III · p. 3

A. 3D Hand-Object Trajectory Reconstruction from Video

VIDEOMANIP reconstructs metric 3D hand-object trajectories from monocular RGB videos by estimating object geometry, pose, hand motion, and robot-retargeted actions.

Object Mesh Reconstruction and Pose Estimation

  • Object meshes are reconstructed from masked RGB regions, while object scale is refined by evaluating candidate scalings through pose-estimation rendering error.The initial scale comes from a coarse physical-dimension estimate, followed by fine-grained verification.Source: A. 3D Hand-Object Trajectory… · p. 3

Human Hand Mesh Estimation and Robot Hand Retargeting

In-the-Wild Videos Calibration

  • In-the-wild trajectories require gravity-alignment calibration because unknown camera orientation leaves reconstructed actions misaligned with the world frame.Source: A. 3D Hand-Object Trajectory… · p. 4

B. Dexterous Grasp and Manipulation Learning

The framework converts reconstructed trajectories into physically plausible, diverse robot demonstrations through contact optimization, interaction-centric grasp modeling, and DemoGen synthesis.

Contact Optimization and Interaction-Centric Grasp Modeling

Manipulation Demonstration Synthesis and Training

Contact Optimization and Interaction-Centric Grasp Modeling

Manipulation Demonstration Synthesis and Training

IV. EXPERIMENTS

The experiments evaluate VIDEOMANIP on both object grasping and multi-object manipulation, with quantitative results summarized in Fig. 3.

  • The study reports separate experiments for grasping objects and manipulating different objects, presenting the main quantitative results in Fig. 3.Source: §IV · p. 5

A. Grasping Experiments

Grasping experiments evaluate video-reconstructed grasps across 20 objects in IsaacGym and test whether contact optimization and additional viewpoints improve robustness.

  • Each grasp is evaluated in IsaacGym with an 18-DoF Inspire Hand under six-direction force disturbances, using displacement within 3 cm as the success criterion.Source: A. Grasping Experiments · p. 5
  • The evaluation asks whether video-reconstructed grasp data generalizes across object categories and whether diverse additional viewpoints help initially difficult objects.Source: A. Grasping Experiments · p. 5
  • 63.75% success across all 20 objects is achieved with optimized grasps, while unoptimized grasps reach 30.7%.The optimized model reaches 82.13% when averaged over the 15 successfully grasped objects.Source: A. Grasping Experiments · p. 5, Figure 4 · p. 6
  • 70.25% overall success follows augmentation with additional videos for initially failed objects, compared with 63.75% using the original 20 videos.On the five initially failed objects, augmentation raises success from 8.6% to 40.8%.Source: A. Grasping Experiments · p. 5, Table I · p. 6

B. Manipulation Experiments

Manipulation experiments evaluate policies trained from single reconstructed human-video trajectories, with DemoGen augmentation, across seven tasks and multiple baselines. The approach achieves the strongest reported overall performance while revealing calibration and baseline-generation effects.

  • Evaluation used a four-finger LEAP Hand on a 7-DoF xArm across three in-scene and four in-the-wild manipulation tasks.A single human demonstration video was used per task, and a shared DRO grasp model was trained across grasped objects.Source: B. Manipulation Experiments · p. 6
  • The execution pipeline predicts grasps from object point clouds and rolls out DP3 closed-loop, while assuming the hand-object relative pose remains fixed during rollout.This approximation addresses occlusion of the grasped object by the LEAP Hand; Pick&Place Can release is handled implicitly.Source: B. Manipulation Experiments · pp. 6–7
  • 62.86% overall success was achieved across seven tasks with DP3 trained on 1000 DemoGen trajectories synthesized from one reconstructed source trajectory.The evaluation uses randomized target object locations over 10 trials.Source: Table II · p. 6, B. Manipulation Experiments · p. 7
  • LVP baselines failed to consistently generate feasible grasps and exhibited task-specific motion failures, including missing rotational motion for Screw Bulb.Reported failures also include incorrect object placement for Move Jenga Box.Source: B. Manipulation Experiments · p. 7
  • Increasing synthesized trajectories improved Pour Tea generalization from 1/15 to 13/15 target locations within the perturbed region.The target Bowl was evaluated at 15 locations, with between 1 and 1000 trajectories synthesized.Source: B. Manipulation Experiments · p. 7
  • Without gravity-alignment calibration, in-the-wild Pour Tea success dropped to 0%, whereas the two video sources showed no significant difference when calibrated.The comparison uses worldTcam for in-scene videos and graRcam for in-the-wild videos.Source: B. Manipulation Experiments · p. 7

V. CONCLUSION, LIMITATIONS, AND FUTURE WORK

VIDEOMANIP reconstructs 3D hand-object trajectories from RGB human videos without specialized sensing and uses them to train dexterous manipulation policies. The framework is effective in the reported real-world executions, but its current pipeline remains bounded by reconstruction, camera, and occlusion-related assumptions.

Conclusion

  • VIDEOMANIP recovers 3D hand-object trajectories from RGB-only in-the-wild videos without specialized sensing devices.The framework is presented as a low-cost reconstruction-based alternative for dexterous manipulation learning.Source: §V · p. 7
  • The reconstructed trajectories provide physically grounded supervision for training dexterous grasping and manipulation policies.The conclusion contrasts this supervision with retargeting-based approaches relying on generated videos.Source: §V · p. 7

Limitations and Future Work

  • The framework may accumulate errors across multiple 3D vision stages and currently favors approximately egocentric, static-camera recordings for reliable reconstruction.Future work proposes trajectory verification and extension to dynamic camera settings.Source: §V · p. 7
  • Occlusion of object point clouds by the robot hand remains a real-world constraint, motivating the current assumption of fixed hand-object relative contact during rollout.This limitation affects point-cloud tracking during manipulation execution.Source: §V · p. 7

Conclusion

  • Figure 6 illustrates the pipeline from RGB human videos to reconstructed 3D trajectories and execution on a real-world LEAP Hand.The figure contrasts in-scene and in-the-wild video sources.Source: Figure 6 · p. 8

Related papers

Loading 2602.09013v2…