PaperlayerView exact revision on arXiv

Source-linked AI summary

Dexterous Manipulation Policies from RGB Human Videos via 3D Hand-Object Trajectory Reconstruction

Hongyi Chen, Tony Dong, Tiancheng Wu, Liquan Wang, Yash Jangir, Yaru Niu, Yufei Ye, Homanga Bharadhwaj, Zackory Erickson, Jeffrey Ichnowski

arXiv:2602.09013v2cs.ROcs.CV

TL;DR

Dexterous manipulation learning needs scalable supervision beyond specialized hardware, but RGB human videos lack robot actions and precise 3D interaction information. VIDEOMANIP reconstructs 3D hand-object trajectories from RGB videos and uses optimized grasp modeling plus synthesized demonstrations; policies trained from these data achieve 62.86% average success across seven real-world tasks.

  • Problem

    Existing manipulation data collection often requires specialized hardware or controlled capture, while RGB videos lack robot actions and precise 3D information needed for supervision.

  • Method

    VIDEOMANIP reconstructs 3D hand-object trajectories from monocular RGB videos, retargets human motion to robot hands, optimizes contacts, and synthesizes diverse demonstrations.

  • Results

    62.86% average success rate was achieved across all real-world manipulation tasks using reconstructed and synthesized trajectories.

  • Takeaways & Limitations

    The reconstructed trajectories provide physically grounded supervision for training dexterous grasping and manipulation policies without specialized sensing devices.

  • Takeaways & Limitations

    The framework depends on multiple 3D vision models that can introduce compounding errors and currently assumes approximately static camera setups.

Abstract

from arXiv · show

Multi-finger robotic hand manipulation and grasping are challenging due to the high-dimensional action space and the difficulty of acquiring large-scale training data. Existing approaches largely rely on human teleoperation with wearable devices or specialized sensing equipment to capture hand-object interactions, which limits scalability. In this work, we propose VIDEOMANIP, a device-free framework that learns dexterous manipulation directly from RGB human videos. Leveraging recent advances in computer vision, VIDEOMANIP reconstructs explicit 3D robot-object trajectories from monocular videos by estimating human hand poses, object meshes, and retargets the reconstructed human motions to robotic hands for manipulation learning. To make the reconstructed robot data suitable for dexterous manipulation training, we introduce hand-object contact optimization with interaction-centric grasp modeling, as well as a demonstration synthesis strategy that generates diverse training trajectories from a single video, enabling generalizable policy learning without additional robot demonstrations. In simulation, the learned grasping model achieves a 70.25% success rate across 20 diverse objects using the Inspire Hand. In the real world, manipulation policies trained from RGB videos achieve an average 62.86% success rate across seven tasks using the LEAP Hand, outperforming retargeting-based methods by 15.87%. Project videos are available at videomanip.github.io.

I. INTRODUCTION

VIDEOMANIP addresses the scalability limits of specialized hardware and robot demonstrations by learning dexterous manipulation from RGB human videos alone. It reconstructs 3D hand–object trajectories, optimizes contacts, and synthesizes demonstrations for policy learning.

  • Motivation: Specialized hardware, controlled environments, and direct human involvement limit the scalability of prior human-data approaches.RGB videos are more widely available, but they lack robot actions and precise 3D information.
  • Evaluation: VIDEOMANIP evaluates learning from progressively relaxed data conditions, including in-scene and in-the-wild videos.In-the-wild videos require gravity-based alignment because they lack robot-camera extrinsic calibration.
  • Related Work: Prior approaches often depend on specialized sensing, robot demonstrations, deployment-specific videos, or pre-scanned object models, and have limited multi-fingered-hand deployment.These dependencies restrict scalability or data diversity.
  • Contribution: VIDEOMANIP learns dexterous grasping and manipulation entirely from RGB human videos without wearables, external sensors, robot demonstrations, or pre-scanned object meshes.It estimates human hand poses, object meshes, and object scales, then retargets reconstructed motions to robot hands.
  • Contribution: The framework reconstructs explicit 3D trajectories and uses contact optimization, interaction-centric grasp modeling, and trajectory synthesis to support dexterous policy training.DemoGen generates diverse trajectories from a single reconstructed video trajectory.

A. Manipulation Learning from Human Videos

Human videos offer abundant data for manipulation learning but do not directly provide robot-executable actions or unambiguous hand–object interactions. Existing approaches therefore commonly retain robot demonstrations, specialized hardware, or deployment-specific collection requirements.

  • Motivation: Human videos are abundant, but they lack robot-executable actions and often contain ambiguous hand–object interactions.This makes learning dexterous manipulation policies from them challenging.
  • Prior Approaches: Several video-based methods still require robot demonstrations to fine-tune models pretrained on in-the-wild datasets.This limits generalization from human data alone.
  • Prior Approaches: Video-based alternatives also extract intermediate representations such as object affordances and point flows, or predict future object trajectories with world and generative video models.The passage presents these as parallel approaches to manipulation learning.
  • Prior Approaches: Other approaches use specialized sensing hardware, including smart glasses, wearable headsets, or hand-motion capture systems.These systems introduce additional hardware dependencies.

B. Human-Object Interaction Reconstruction from Videos

Human–object reconstruction methods increasingly infer hand poses, object representations, and contact from visual data, enabling reconstructed interactions to support robotic manipulation. However, existing approaches still face reconstruction inaccuracies, long runtimes, or dependence on additional robot demonstrations.

  • Reconstruction Methods: Modern reconstruction methods infer low-dimensional hand pose and shape representations and support diverse object meshes, point clouds, and primitives.Transformer-based models and strong 3D supervision facilitate shared modeling across object categories.
  • Interaction Modeling: Joint hand–object reasoning increasingly uses diffusion models, implicit signed distance fields, and differentiable pose estimation to optimize 3D contact.These techniques make interaction reconstruction from images and videos increasingly feasible.
  • Robotic Applications: Reconstructed hand–object interactions have been used to train dexterous robot grasping and manipulation policies, often by retargeting human motions to robot actions.This connects visual reconstruction with robotic control.
  • Limitations: Existing reconstruction-based methods may require extensive filtering for inaccurate reconstructions, incur long reconstruction times, or rely on additional robot demonstrations.These limitations constrain direct use of reconstructed human interactions for policy learning.

III. METHOD

VIDEOMANIP reconstructs robot-relevant hand–object trajectories from RGB human videos and uses them to learn dexterous grasping and manipulation policies. Its pipeline combines 3D reconstruction, trajectory generation, and policy learning from reconstructed actions and object point clouds.

  • 3D Reconstruction: VIDEOMANIP reconstructs hand mesh H and object mesh O from RGB human videos to produce explicit 3D robot hand trajectories and object pose changes.The reconstruction supports finger-level manipulations and resulting object motion.
  • Pipeline: The framework contains a reconstruction stage followed by grasping and manipulation policy learning from reconstructed hand actions and object point clouds.The overview organizes these stages as Sec. III-A and Sec. III-B.
  • Assumptions: The data assumptions include static-camera egocentric videos and one video per task or object, covering both in-scene and in-the-wild recordings.The method is designed around limited video availability for each task or object.

A. 3D Hand-Object Trajectory Reconstruction from Video

VIDEOMANIP reconstructs metric 3D hand-object trajectories from monocular RGB videos by estimating object geometry, pose, hand motion, and robot-retargeted actions.

  • Metric depth maps and camera intrinsics establish a shared 3D coordinate frame for subsequent hand-object reconstruction.
  • Object Mesh Reconstruction and Pose Estimation: Object meshes are reconstructed from masked RGB regions, while object scale is refined by evaluating candidate scalings through pose-estimation rendering error.The initial scale comes from a coarse physical-dimension estimate, followed by fine-grained verification.
  • Human Hand Mesh Estimation and Robot Hand Retargeting: HaMeR recovers human hand meshes, and metric depth corrections align them with reconstructed objects despite weak-perspective depth ambiguity.
  • Human Hand Mesh Estimation and Robot Hand Retargeting: Human hand poses are optimized against corresponding robot-hand keypoints to produce wrist poses and finger configurations for each frame.
  • In-the-Wild Videos Calibration: In-the-wild trajectories require gravity-alignment calibration because unknown camera orientation leaves reconstructed actions misaligned with the world frame.

B. Dexterous Grasp and Manipulation Learning

The framework converts reconstructed trajectories into physically plausible, diverse robot demonstrations through contact optimization, interaction-centric grasp modeling, and DemoGen synthesis.

  • Contact Optimization and Interaction-Centric Grasp Modeling: Differentiable hand-object contact optimization corrects interpenetration and invalid contacts caused by reconstruction errors before grasp learning.
  • Manipulation Demonstration Synthesis and Training: DemoGen synthesizes spatially randomized demonstrations from a single reconstructed trajectory, addressing the insufficiency of one trajectory for robust policy learning.
  • Manipulation Demonstration Synthesis and Training: The pipeline splits demonstrations into grasping and manipulation stages, each learned by a dedicated policy.
  • Contact Optimization and Interaction-Centric Grasp Modeling: DRO learns dense robot-object point distances from optimized interactions and uses them to model grasps relative to object point clouds.
  • Manipulation Demonstration Synthesis and Training: The DP3 policy receives robot-hand point clouds, proprioceptive state, and object point clouds, then predicts closed-loop configuration changes.

IV. EXPERIMENTS

The experiments evaluate VIDEOMANIP on both object grasping and multi-object manipulation, with quantitative results summarized in Fig. 3.

  • The study reports separate experiments for grasping objects and manipulating different objects, presenting the main quantitative results in Fig. 3.

A. Grasping Experiments

Grasping experiments evaluate video-reconstructed grasps across 20 objects in IsaacGym and test whether contact optimization and additional viewpoints improve robustness.

  • Each grasp is evaluated in IsaacGym with an 18-DoF Inspire Hand under six-direction force disturbances, using displacement within 3 cm as the success criterion.
  • The evaluation asks whether video-reconstructed grasp data generalizes across object categories and whether diverse additional viewpoints help initially difficult objects.
  • 63.75% success across all 20 objects is achieved with optimized grasps, while unoptimized grasps reach 30.7%.The optimized model reaches 82.13% when averaged over the 15 successfully grasped objects.
  • 70.25% overall success follows augmentation with additional videos for initially failed objects, compared with 63.75% using the original 20 videos.On the five initially failed objects, augmentation raises success from 8.6% to 40.8%.

B. Manipulation Experiments

Manipulation experiments evaluate policies trained from single reconstructed human-video trajectories, with DemoGen augmentation, across seven tasks and multiple baselines. The approach achieves the strongest reported overall performance while revealing calibration and baseline-generation effects.

  • Evaluation used a four-finger LEAP Hand on a 7-DoF xArm across three in-scene and four in-the-wild manipulation tasks.A single human demonstration video was used per task, and a shared DRO grasp model was trained across grasped objects.
  • The execution pipeline predicts grasps from object point clouds and rolls out DP3 closed-loop, while assuming the hand-object relative pose remains fixed during rollout.This approximation addresses occlusion of the grasped object by the LEAP Hand; Pick&Place Can release is handled implicitly.
  • 62.86% overall success was achieved across seven tasks with DP3 trained on 1000 DemoGen trajectories synthesized from one reconstructed source trajectory.The evaluation uses randomized target object locations over 10 trials.
  • LVP baselines failed to consistently generate feasible grasps and exhibited task-specific motion failures, including missing rotational motion for Screw Bulb.Reported failures also include incorrect object placement for Move Jenga Box.
  • Increasing synthesized trajectories improved Pour Tea generalization from 1/15 to 13/15 target locations within the perturbed region.The target Bowl was evaluated at 15 locations, with between 1 and 1000 trajectories synthesized.
  • Without gravity-alignment calibration, in-the-wild Pour Tea success dropped to 0%, whereas the two video sources showed no significant difference when calibrated.The comparison uses worldTcam for in-scene videos and graRcam for in-the-wild videos.

V. CONCLUSION, LIMITATIONS, AND FUTURE WORK

VIDEOMANIP reconstructs 3D hand-object trajectories from RGB human videos without specialized sensing and uses them to train dexterous manipulation policies. The framework is effective in the reported real-world executions, but its current pipeline remains bounded by reconstruction, camera, and occlusion-related assumptions.

  • Conclusion: VIDEOMANIP recovers 3D hand-object trajectories from RGB-only in-the-wild videos without specialized sensing devices.The framework is presented as a low-cost reconstruction-based alternative for dexterous manipulation learning.
  • Conclusion: The reconstructed trajectories provide physically grounded supervision for training dexterous grasping and manipulation policies.The conclusion contrasts this supervision with retargeting-based approaches relying on generated videos.
  • Limitations and Future Work: The framework may accumulate errors across multiple 3D vision stages and currently favors approximately egocentric, static-camera recordings for reliable reconstruction.Future work proposes trajectory verification and extension to dynamic camera settings.
  • Limitations and Future Work: Occlusion of object point clouds by the robot hand remains a real-world constraint, motivating the current assumption of fixed hand-object relative contact during rollout.This limitation affects point-cloud tracking during manipulation execution.
  • Conclusion: Figure 6 illustrates the pipeline from RGB human videos to reconstructed 3D trajectories and execution on a real-world LEAP Hand.The figure contrasts in-scene and in-the-wild video sources.
Loading 2602.09013v2…