OmniHOI: Dexterous Hand-Object Interaction from Monocular Human Video

1 Introduction

• Monocular depth ambiguity can confuse occlusion with contact, while geometric errors and morphology differences can make visually plausible human motion infeasible for robot hands.Small errors can turn contact into penetration or separation.Source: §1 · p. 2
• Prior video pipelines train task-specific policies to absorb reconstruction errors, while motion-capture transfer methods assume clean trajectories and do not handle noisy video reconstruction.Do as I Do starts from video without training but mainly aligns hand and object centroids, handling contact explicitly only in simulation.Source: §1 · p. 2
• OmniHOI refines hand-object geometry for occlusion and contact, preserves contact while retargeting to robot morphology, then optimizes keyframes in simulation.The simulation stage uses CMA-ES to improve object tracking, contact, and grasp stability.Source: §1 · p. 2
• 39–89% success across five dexterous hands, versus at most 31% for prior transfer methods; video clips yield 53%, versus 28% for VideoManip and 25% for Do as I Do.The resulting trajectories also execute on a real bimanual robot.Source: §1 · p. 2
• The pipeline combines monocular video reconstruction, interaction-preserving transfer, and contact- and dynamics-aware refinement to produce task-effective motion across dexterous hands.Its reconstruction method addresses occlusion, contact, penetration, and finger-crossing errors.Source: §1 · p. 2

Dexterous manipulation from human videos

• Human-video manipulation priors often require robot data or target grippers, while precise dexterous demonstrations still rely on gloves, AR glasses, or headsets.Video-to-policy pipelines reconstruct interactions and train a policy on them, including Qin et al. (2022).Source: §2 · p. 2

Monocular hand-object reconstruction

• Monocular reconstruction methods predict, lift, or fit object geometry, with fitting-based approaches optimizing hand and object together under contact, collision, and occlusion constraints.VideoManip and Do as I Do instead relate hand and object through keypoint depth or centroids, leaving contact–occlusion ambiguity open; OmniHOI uses pixel-aligned registration and per-finger corrections.Source: §2 · pp. 2–3

Human-to-robot motion transfer

• Kinematic and interaction-preserving retargeting map human motion to robot hands, while dynamics-aware transfer uses simulation but generally assumes clean motion-capture references.OmniHOI resolves contact geometrically before simulation, then refines keyframe offsets with CMA-ES in rolling horizons.Source: §2 · p. 3

3.1 Keyframe reconstruction

• Keyframes are placed at equal increments of cumulative second-order hand motion, concentrating them at reversals and transients and sparsifying steady stretches.Figure 3 contrasts this adaptive timing with uniform sampling of the same budget.Source: §3.1 · p. 4, Figure 3 · p. 4
• Image alignment adjusts articulation and shape to fix occlusion order, while contact-penetration optimization adjusts articulation and wrist pose to restore contact and repel the hand from object interiors.Exponential weighting by enclosed object area helps prevent deeply wrapped fingers from sinking into the object.Source: §3.1 · pp. 4–5, Figure 4 · p. 5
• Parity checks along each finger’s keypoint polyline detect crossings through thin objects, then minimum-pose repair removes crossing residuals.Source: §3.1 · p. 5

3.2 Interaction-faithful human-to-robot transfer

• Retargeting assigns contact regions and robot-surface anchors, then repairs crossings and reduces penetration, enclosure, and contact residuals while minimizing pose change.The anchors are constrained to a narrow band around the demonstrated contact region.Source: §3.2 · p. 5, Figure 5 · p. 6
• PCHIP-interpolated keyframes are corrected with CMA-ES rollouts scored for object tracking, penetration, contact, antagonistic force, and end-window stability.The rolling search freezes each best offset before advancing to the next keyframe.Source: §3.2 · pp. 5–6

4.1 Hand-object reconstruction

• OmniHOI beats all baselines on object reconstruction and hand-pose errors; intersection volume is comparable to iHOI (2.75 vs. 2.90 cm3).Source: Table 1 · p. 7, §4.1 · p. 7

4.2 Human-to-robot motion transfer

• 85–89% success on the four higher-DoF hands, versus at most 31% for baselines; OmniHOI leads on all five hands, with 39% on the 6-DoF Inspire.Transfer efficiency is 19 min per trajectory, compared with 0.3 min for OmniRetarget, 37 min for SPIDER, and 385 min for ManipTrans.Source: §4.2 · pp. 7–8, Table 2 · p. 8

4.3 End-to-end video-to-robot transfer

• OmniHOI fails on 28 selected video clips, most often because the object is reconstructed at the wrong size (12) or rotates in the hand (7).The 60 clips were selected for relatively low occlusion and large hand-object pixel coverage; none came from ARCTIC.Source: §4.3 · p. 8

4.4 Real-world deployment

• Seven representative trajectories were executed on two XHands mounted on UR5e arms after workspace alignment and simulated arm-motion planning.The tasks include bimanual coordination, precise insertion, and long-horizon grasp-and-place.Source: §4.4 · pp. 8–9, Figure 8 · p. 9

4.5 Ablations

• Intersection volume fell from 4.27 to 2.75 cm3 as reconstruction refinements were added, while PA-MPVPE and PA-MPJPE rose by only 0.67 mm.The authors accept this trade-off because a robot hand initialized inside an object cannot separate from it in simulation.Source: Table 4 · p. 9, §4.5 · p. 9
• Physics-in-the-loop refinement raised success from 9–29% to 85–89% on four higher-DoF hands and from 21–25% to 36–39% on the Inspire Hand.Source: Table 5 · p. 9, §4.5 · p. 9

• OmniHOI converts monocular RGB hand-object video into an interaction-faithful dexterous-hand trajectory without relying on a learned policy to absorb errors.Source: §5 · p. 10
• Experiments show lower reconstruction hand-pose error and intersection, higher transfer success across five hands with 6 to 22 DoF, and end-to-end video success where prior pipelines fail.Source: §5 · p. 10
• Monocular perception accuracy, especially metric scale, estimated object geometry and physical properties, and open-loop real-world execution limit the current pipeline.Source: §5 · p. 10

• For the single-image benchmark, depth uses MoGe-3’s own metric scale because one frame permits neither frame alignment nor a hand-defined unit.Source: A Reconstruction details · p. 15
• The cited scale choice is specific to the single-image benchmark, whose camera intrinsics come from the dataset.Source: A Reconstruction details · p. 15
• Table 6 lists the key reconstruction settings for Section 3.1.Source: Table 6 · p. 15

B Adaptive keyframe sampling

• Accumulated second-order hand motion drives keyframe allocation toward dynamic periods and away from smooth or stationary stretches.Each hand accumulates its own motion measure, so a stationary hand does not use the other hand’s keyframes.Source: B Adaptive keyframe sampling · p. 15
• At lower sampling ratios, adaptive sampling matches or exceeds uniform sampling on all three metrics, with comparable rollout counts.At r = 1 the sampling rules coincide; sparse keyframes require 30–50% more rollouts per keyframe than at r = 1.Source: B Adaptive keyframe sampling · p. 15

C Algorithms

• Algorithm 1 covers finger-crossing repair, while Algorithm 2 covers contact-aware retargeting refinement.Source: C Algorithms · p. 15

D Simulation settings

• Every method is replayed in the same MuJoCo scene against the same ground truth; only the hand trajectory differs.Commands are projected onto each underactuated hand’s joint coupling before replay.Source: D Simulation settings · pp. 15–16
• OmniHOI and OmniRetarget keyframes are interpolated with PCHIP, using an SO(3) spline for rotations, on a 60 Hz grid.SPIDER controls are resampled from 100 Hz to 60 Hz, and ManipTrans uses its executed states frame for frame.Source: D Simulation settings · pp. 15–16
• At each sampling ratio, both keyframe rules select the same number of keyframes and retain the first and last frame.Rollouts count the candidate actions tried in physics simulations, with the reported measure being the median over tasks.Source: Table 7 · p. 16

E Transfer ablation details

• Contact refinement lowers hand-object contact frames from 41.4% to 35.1% after physics-in-the-loop refinement.Object errors fall on Inspire Hand and SharpaWave, and change by at most 0.1 cm and 1° on the other three hands.Source: E Transfer ablation details · p. 16
• Most open-loop losses in Table 5 are false successes: interlocked objects follow fingers passing through them and meet the ATE criterion without being held.The remaining losses come from contact being lost between interpolated keyframes.Source: E Transfer ablation details · p. 16
• The contacts removed by refinement are largely penetration-driven interlocks, rather than contacts that grasp or support the object.This explains why fewer contact frames can coincide with lower or nearly unchanged object errors.Source: E Transfer ablation details · p. 16

F End-to-end failure analysis

• MoGe camera estimates put TACO scenes at about half their true distance, producing 0.73–0.93 m wrist errors; objects move by more than 1 cm in only 2 of 101 replays.TACO’s 24° telephoto lens is read as 42–52° by MoGe, making upstream image-based geometry the largest bottleneck for both baselines.Source: F End-to-end failure analysis · p. 16
• On OakInk2, the 70° head-mounted camera yields 37–59 mm wrist error, yet the hand pushes the object aside and reproduces at most 3% of ground-truth contacts.The passage identifies grasping as a second bottleneck, independent of depth.Source: F End-to-end failure analysis · p. 16
• OmniHOI fails on 28 clips: 14 in perception, 13 in execution, and 1 in motion planning.Perception failures mostly involve object size from masks that miss or overrun it (12 clips); execution failures include object rotation (7), failure to close (5), and path drift (1).Source: F End-to-end failure analysis · p. 16
• Contact-aware refinement iteratively reduces crossing and penetration violations, ranking fewer crossing fingers ahead of the summed residual.It starts from a solver pose and uses object samples, hand vertices, an SDF, anchors, and contact regions; it returns an update only if the lexicographic score improves.Source: F End-to-end failure analysis · p. 17

G Real-world deployment

• The seven real-world tasks are shown as paired film strips, with human video above and robot execution below.The task set includes insertion, stacking, sweeping, scanning, scrubbing, and two pouring tasks.Source: G Real-world deployment · p. 17, Figure 9 · p. 18, Figure 10 · p. 19
• Each row contains seven frames evenly spaced over the motion, and the two videos for a task are sampled independently.The paired videos can differ in length.Source: G Real-world deployment · p. 17