PaperlayerView exact revision on arXiv

Source-linked AI summary

Do as I Do: Dexterous Manipulation Data from Everyday Human Videos

Bhawna Paliwal, Haritheja Etukuru, William Liang, Pieter Abbeel, Nur Muhammad Mahi Shafiullah, Jitendra Malik

arXiv:2606.19333v1cs.ROcs.CV

TL;DR

Monocular RGB human videos are abundant, but reconstructing hand-object interactions and bridging the human-to-robot embodiment gap have limited their use for robot manipulation data. DO AS I DO reconstructs and retargets those videos for dexterous hands, reporting strong reconstruction and retargeting results and robot-playable trajectories.

  • Problem

    Monocular RGB human videos are abundant, but hand-object estimation and crossing the human-to-robot embodiment gap remain challenges to using them as robot manipulation data.

    Source: §1 · p. 2

  • Method

    DO AS I DO reconstructs hand-object interactions from monocular RGB videos and retargets them into actions for multi-fingered dexterous robot hands.

    Source: §1 · p. 2

  • Results

    DO AS I DO outperforms state of the art on reconstruction benchmarks and improves scalable dynamics-aware retargeting; raters preferred its tracking over FPose 67% of the time.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    The method provides a path toward scaling robot data by observing humans across egocentric, exocentric, and online video sources.

    Source: §5 · p. 9

    The approach assumes rigid objects and semi-accurate metric depth, may fail without either, and reconstructs only the hand and object rather than the full scene.

    Source: §5 · p. 9

Abstract

from arXiv · show

How can we scalably generate data for robotic manipulation, especially on human-like platforms such as dexterous multi-fingered hands? Learning from human videos has recently emerged as a likely answer to this question. However, difficulties in estimating hand-object interaction and crossing the human-to-robot embodiment gap have hindered the adoption of abundant monocular RGB-only human videos as the primary source of robot manipulation data. In this work, we present DO AS I DO, an algorithm to reconstruct and retarget monocular RGB human videos to multi-fingered dexterous robotic hands. DO AS I DO reconstructs hand-object interactions from various egocentric and exocentric in-the-wild video sources. The algorithm then retargets these hand-object interaction estimates into a sequence of actions executable in the real world, yielding robot-complete manipulation data from disparate human videos. Overall, DO AS I DO outperforms previous state of the art in estimating hand-object interactions and extracting dexterous manipulation trajectories from RGB videos, as we show in experiments on datasets with ground truths and on a dataset of video clips collected online. Our experiments enable us to propose an efficacy playbook for practitioners collecting human data for manipulation.

1 Introduction

Monocular RGB human videos offer observational data, but reconstructing hand-object interactions and retargeting them to robots remain key challenges. DO AS I DO addresses this gap by reconstructing and retargeting videos into dexterous robot actions.

1 Introduction

  • DO AS I DO is a two-step algorithm that reconstructs behaviors from monocular RGB videos and retargets them to multi-fingered dexterous hands.It produces robot hand and arm actions performing the same task on the environment.Source: §1 · p. 2
  • The hand-object reconstruction process outperforms state of the art on relevant metrics across diverse egocentric and exocentric videos, including in-the-wild and generated clips.The method is designed without limiting assumptions on displayed behavior or object classes, supporting arbitrary rigid bodies.Source: §1 · p. 2
  • Retargeting improves on existing scalable dynamics-aware techniques by adding components that robustify noisy reconstructed reference trajectories.Source: §1 · p. 2
  • The resulting robot data is playable on a dexterous hand and arm, completing a pipeline from internet video to real dexterous-hand rollouts.Source: §1 · p. 2

2 Related Work

Prior work uses human videos for robot learning through learned priors, structured representations, hand-object reconstruction, and human-to-robot retargeting. DO AS I DO builds on reconstruction approaches while addressing diverse in-the-wild data and noisy references.

2 Related Work

  • Table 1 summarizes prior methods and data sources in rough order of difficulty, with “Self” marking data collected by the authors.Source: Table 1 · p. 3
  • Prior work extracts visual, policy, and dynamics priors, or structured affordances, flows, and 3D reconstructions; DO AS I DO focuses on the reconstruction category and diverse in-the-wild sources.Source: §2 · pp. 2–3
  • Hand-object reconstruction spans hand pose, object shape and pose, and joint modeling; DO AS I DO instead modularly combines HaWoR, SAM 3D meshing, and SAM3D-based object tracking.Source: §2 · p. 3
  • Dynamics-aware retargeting addresses physical interaction, but most prior approaches assume clean ground-truth references, unlike DO AS I DO’s noisy, potentially discontinuous reconstructions.Source: §2 · p. 3

3 Method

DO AS I DO first builds temporally tracked 3D hand-object references from video, then optimizes physically feasible robot trajectories despite morphology and contact-information gaps.

3 Method

  • The pipeline reconstructs and tracks the 3D hand and object, then retargets them into dynamically feasible robot trajectories.The reference signal lacks contact information and forces, while human and robot morphologies differ.Source: §3 · p. 3, §3.2 · p. 5

3.1 Reconstruction

  • SAM 3D tracking fixes an anchor-frame object shape and updates pose over time, guiding sampling toward the previous-frame pose for temporal consistency.Independent framewise generation would produce different meshes and a pose sequence with no temporal coherence.Source: §3.1 · p. 4
  • Adaptive pose guidance derives object rotation guidance from rotational velocity estimated using 2D point tracks, at the cost of one offline tracking pass per video.The approach avoids fixed pose guidance that may cause over-rigidity or spurious flips.Source: §3.1 · pp. 4–5
  • Pose candidates are sampled and clustered under weighted SE(3) distance; consensus filtering with mask-IoU selects a mode-best pose without rerunning the diffusion backbone.This avoids the much higher compute cost of ranking candidates by conditional log-density.Source: §3.1 · p. 5
  • Hand-object alignment treats the hand scale as ground truth, estimates object translation scale from centroid depth, solves per-frame translation by least squares, and aligns gravity.The method first compensates for independently reconstructed hand and object scales.Source: §3.1 · p. 5

3.2 Retargeting

  • Dynamics-aware MPPI-style retargeting follows reconstructed references in simulation, using annealed sampling from broad exploration toward local refinement.The method targets physically plausible trajectories where kinematic solvers or heuristics may not ensure plausibility or general-purpose expressiveness.Source: §3.2 · p. 5
  • Warmup steps hold the object while the robot hand adjusts before reference tracking, helping avoid unrecoverable initial states without grasp-sampling heuristics.The added warmup spans H steps before the rollout horizon.Source: §3.2 · pp. 5–6
  • Random rollout forces encourage controls robust to minor disturbances, while a transition penalty rewards floor contact during rest and hand-object contact during in-hand phases.Transition stages are defined by reference hand-object distance under threshold ϵ.Source: §3.2 · p. 6

4 Experiments

Experiments assess reconstruction and retargeting on annotated datasets, in-the-wild video, and real-world deployment. Results show stronger tracking and retargeting, while filtering finds that few sampled online clips survive quality checks.

4.1 Experimental Setup

  • The evaluation covers annotated reconstruction benchmarks, 150 in-the-wild videos, 655 reconstructed references, and 1,352 OakInk2 trajectories, with retargeting success defined by position and rotation error thresholds.Success requires mean position error below 0.1 m and mean rotation error below 0.5 rad.Source: §4.1 · pp. 6–7

4.2 Reconstruction Results

  • On 150 in-the-wild videos, raters preferred our tracker to FoundationPose 67% of the time, and it established state of the art on DexYCB and HOI4D.The reconstruction table reports F-5, F-10, and Chamfer distance on the two datasets.Source: §4.2 · p. 7, Table 2 · p. 7
  • Adaptive pose guidance improves reconstruction quality, while clustering-based selection performs on par with pose-likelihood selection and is up to 30× faster.The adaptive guidance uses point tracking.Source: §4.2 · p. 7

4.3 Retargeting Results

  • On reconstructed in-the-wild data, DO AS I DO achieves 71% retargeting success versus the 25% baseline; on OakInk2, success improves from a 72% baseline to 81%.Source: §4.3 · pp. 7–8

4.4 Real-World Deployment

  • Real-world deployments execute representative trajectories for 10 tasks on a bimanual robot setup after manually aligning each initial pose to the workspace.The 10 motions include varied object geometries and grasp classes.Source: §4.4 · p. 8, Figure 6 · p. 8

4.5 Human Data Filtering Playbook

  • Of 2,000 sampled 100DOH clips, 83 (4%) survive the reconstruction quality check; about 107 clips (5%) are projected to be directly relevant for dexterous manipulation.The filtering analysis identifies failures including boundary truncation, absent activity, camera motion, and reconstruction issues.Source: §4.5 · pp. 8–9

5 Conclusion

DO AS I DO reconstructs and retargets everyday human videos onto dexterous robot hands, with results across egocentric, exocentric, and online sources. Its scope is bounded by assumptions about object rigidity and monocular depth, incomplete scene reconstruction, and approximate simulation dynamics.

5 Conclusion

  • The framework reconstructs and retargets everyday human videos onto dexterous robot hands, spanning egocentric, exocentric, and online sources.The authors describe this as a path toward scaling robot data by observing humans.Source: §5 · p. 9
  • The authors position DO AS I DO as a step toward making human videos first-class data sources for robot learning.They hope the framework helps move human videos into the robot-learning data landscape.Source: §5 · p. 9
  • The method assumes rigid objects and semi-accurate monocular metric depth, and may fail when either assumption does not hold.It reconstructs only the hand and one object, so it cannot reason about obstacles or articulations; approximate simulator dynamics also bound achievable real-world performance.Source: §5 · p. 9

A Reconstruction

The reconstruction pipeline combines monocular hand and object estimates into a temporally consistent, aligned 4D hand-object trajectory. It uses guided object tracking, pose-sample selection, and depth-based alignment, with adaptive guidance improving over fixed guidance in the reported benchmarks.

A Reconstruction

  • SAM 3D supplies an object mesh from one frame, guided diffusion tracks its pose across frames, and HaWoR tracks the hand to form a consistent trajectory.Each frame samples candidate poses anchored to the predicted shape and previous frame’s pose before selecting one by clustering.Source: Figure 7 · p. 15
  • Adaptive pose guidance derives its strength from tracked object rotation and consistently improves over fixed guidance on DexYCB and HOI4D.For consecutive frames, 20 points inside the object mask are tracked, filtered by the next mask, and fit with a 2D rigid transform.Source: A Reconstruction · p. 15
  • The clustering heuristic matches pose-likelihood ranking while adding essentially no compute beyond generation, using consensus among 25 samples and mask-IoU ranking of surviving clusters.Exact pose-density scoring costs about 8.7k forward and backward passes per frame, roughly two orders of magnitude more than generation.Source: A Reconstruction · pp. 15–16
  • Depth-based alignment rescales the hand-to-object offset into HaWoR units, then adjusts object translation along its viewing ray while preserving recovered orientation.The per-frame target uses the hand centroid and the visible mesh centroid; repeating the alignment yields a metric 4D trajectory.Source: A Reconstruction · p. 16, Figure 8 · p. 16

B Retargeting

Retargeting combines a kinematic reference with sampling-based, dynamics-aware optimization in MuJoCo Warp. Additional procedures address simulation setup, reference continuity, kinematic local minima, and objects that begin in non-flat poses.

B Retargeting

  • MuJoCo Warp runs at 200 Hz, and object meshes are convex-decomposed and thickened or dilated by 2 mm to stabilize contact-rich interactions.The simulator timestep is 0.005 seconds.Source: B Retargeting · p. 17
  • Dynamics-aware retargeting plans every 0.5 seconds over a 3-second horizon, evaluating 1,024 samples for 32 iterations to track hand and object while penalizing penetration.The method first computes a kinematic reference by matching fingertip positions; rollouts also use a transition reward.Source: B Retargeting · p. 17
  • Interpolating optimized controls into the reference avoids the sharp transitions and jerky motions caused by simply appending reference steps.Multiple kinematic retargeting runs from random initial poses can improve reference quality and help avoid local minima.Source: B Retargeting · p. 17
  • A flat base plate stabilizes objects that begin upright or otherwise off a surface, contacting only the floor and requiring no task-specific assumptions or extra scene reconstruction.The plate is added to the bottom of the object mesh and does not contact the robot.Source: B Retargeting · pp. 17–18

C Experimental Setup and Results

The experiments compare hand-object reconstruction and object-tracking baselines, and evaluate in-the-wild tracking through human preference because ground-truth object poses are unavailable.

C Experimental Setup and Results

  • Joint hand-object baselines included HO, IHOI, HORSE, MCC-HO, and G-HOP; object-tracking baselines were FoundationPose and Any6D.For HOI4D, the authors evaluated released per-clip checkpoints; for DexYCB, they ran test-time optimization on the 160-clip test set using the released mix-data prior.Source: C Experimental Setup and Results · p. 18
  • Object-tracker baselines were compared in a controlled setup by replacing the tracking module while holding every other component fixed.The comparison used FoundationPose and Any6D as object trackers within the authors’ framework.Source: C Experimental Setup and Results · p. 18
  • Raters preferred our method on 67% of in-the-wild videos, versus 18% for FoundationPose and 15% ties; its non-tie win rate was 79%.The evaluation covered 150 videos with three ratings per video; 75% received unanimous agreement, and inter-rater agreement was substantial (Fleiss’ κ = 0.65).Source: C Experimental Setup and Results · p. 18
  • Table 5 ablates two design axes of the object-tracking method.The supplied caption identifies the ablation’s scope but does not specify the axes or results.Source: Table 5 · p. 19
  • Figure 10’s evaluator interface shows two projected object poses; in the illustrated example, option 1 aligns with the true object position.Evaluators compared reprojections with the original interaction video and judged which track better matched the object’s true motion.Source: C Experimental Setup and Results · p. 18, Figure 10 · p. 19

D Robot Deployment

The retargeted trajectories are first replayed in simulation on a dual-arm setup, then rolled out on the real-world robot setup for six tasks.

D Robot Deployment

  • The simulated digital twin represents dual UR3e arms with Sharpa Wave hands in MuJoCo, visualized with Viser.The retargeted trajectories cannot be executed directly on the robot and are first mapped onto the arm-and-hand setup.Source: Figure 11 · p. 19, D Robot Deployment · p. 19
  • Real-world rollouts ran at roughly half speed, with both arms and hands commanded at 50 Hz.The reported rollout examples cover spreading, whisking, dusting, pouring, erasing, and picking.Source: D Robot Deployment · p. 20, Figure 12 · p. 20
  • Figure 12 presents real-world rollout frames for six tasks: spreading, whisking, dusting, pouring, erasing, and picking.The paper notes that additional tasks and videos are available on its webpage.Source: Figure 12 · p. 20
Loading 2606.19333v1…