RLHND: Video Foundation Models as Physically Grounded Hand Trackers for Robot Learning

1 INTRODUCTION

• Robot training from human video requires bridging action-space differences and missing physical information, including contact and force cues.Tactile cues can distinguish a successful grasp from hovering, but human datasets generally lack them without specialized sensors.Source: §1 · pp. 1–2
• Existing reconstructions can jitter in depth and vary hand size across frames, making their 3D trajectories physically implausible for robot action labeling.Perspective scale-depth ambiguity lets depth errors appear as hand-shape variation; small motion errors can yield substantially different or infeasible actions.Source: §1 · p. 2
• RLHND uses Cosmos 3 Nano through its clean-context-frame conditioning interface as a video feature encoder, retaining a fine-tuning setup aligned with pretraining.Its design also caches one shape per video, constrains pose to 29 anatomical rotational degrees of freedom, and trains a separate tactile expert with the pose stream frozen.Source: §1 · p. 2
• RLHND consistently outperforms baselines on HOT3D, ARCTIC, and held-out EgoDex for motion reconstruction, and achieves the best contact and force estimation among the evaluated benchmarks.Its outputs also produce more accurate, less jittery retargeted actions, and RLHND keypoints notably improve Dexterous Point Policy training.Source: §1 · p. 2
• RLHND extends video-foundation-model hand tracking to jointly estimate metrically accurate bimanual motion and dense surface contact and force from monocular video.Its contribution list highlights cached hand shape, anatomically constrained pose, and a decoupled tactile expert with LBS-based feature spreading.Source: §1 · p. 3
• Retargeting RLHND demonstrations to five dexterous hands yields accurate, smooth commands and supports training Dexterous Point Policy with automatically labeled contact.The policy trained with RLHND keypoints improves notably over the original tracker.Source: §1 · p. 3

Hand motion reconstruction

• Image-based trackers estimate MANO parameters from cropped frames independently, while video methods add temporal context but remain vulnerable to detector-box jitter and weak temporal consistency.Video-foundation-model approaches infer both hands over a clip, but per-window shape regression still causes hand-size variation; RLHND fixes one shape per video and constrains pose anatomically.Source: §2 · p. 3

Hand contact and force estimation

• Pressure boards and tactile gloves supply force labels, but dataset sensor calibration and hand-surface correspondence make their labels difficult to pool directly.PressureVisionDB focuses on palm-pad contact, while OpenTouch provides a palmar taxel grid during natural manipulation.Source: §2 · p. 3
• Geometric hand-object distances can supervise contact, whereas force estimation requires physical measurements; RLHND follows HOPE’s unified MANO-surface label format.The paper reports improved tactile estimation with its decoupled tactile expert.Source: §2 · p. 3

3 METHOD

• RLHND converts clips into deterministic video features for predicting MANO pose, metric translation, hand presence and visibility, and per-vertex contact and force.The architecture uses separate pose and tactile expert streams.Source: §3 · pp. 3–4

3.1 CLEAN-LATENT VIDEO DIFFUSION ENCODER

• Cosmos 3 encodes each clip into clean latents; fine-tuned patch embeddings and a LoRA-adapted generation pathway yield spatiotemporal features through the conditioning interface with an empty prompt.The conditioning interface keeps clean-frame inputs aligned with pre-training.Source: §3.1 · p. 4

3.2 POSE EXPERT STREAM

• Cached MANO shape conditions hand tokens; articulation constrains feasible axes to 29 DoF, leaves the thumb unconstrained, and fits 45-DoF labels to the feasible set via inverse kinematics.When no pre-calibrated shape is available, the shape estimated from the first sufficiently visible clip is reused throughout the video.Source: §3.2 · pp. 4–5

3.3 TACTILE EXPERT STREAM

• Because tactile supervision is scarcer, tactile learning is decoupled from pose; 16 bone tokens are expanded with MANO LBS weights for vertex-level contact and force readout.The bone-level representation avoids attention over all hand vertices at every frame.Source: §3.3 · p. 5

3.4 TRAINING

• Training is staged: stage 1 trains only the pose stream; stage 2 freezes it and trains the tactile stream using reweighted contact BCE and force-total and distribution supervision.The contact loss reweights positive examples because contact vertices are rarer than non-contact vertices.Source: §3.4 · p. 6

4.1 HAND POSE ESTIMATION

• RLHND outperforms baselines on nearly every motion-reconstruction metric across benchmarks, including zero-shot EgoDex, and β-caching removes shape variation but not jitter.The comparison includes per-window β estimation and β-caching; the table marks best and second-best values.Source: Table 1 · p. 7, §4.1 · p. 7

4.2 TACTILE ESTIMATION

• RLHND achieves the best contact F1 and AUROC across OpenTouch, HOT3D, DexYCB, and ARCTIC, with competitive force estimation on OpenTouch and PVDB.The PressureVision family predicts pressure on the image plane, so its contact metrics are not reported.Source: §4.2 · p. 8, Table 2 · p. 8

4.3 EFFECTIVENESS ON ROBOT LEARNING

• DPP + RLHND improves real-robot performance across evaluated tasks, with larger gains on complex bimanual tasks.Both compared rows use RLHND contact labels and differ only in their hand keypoints.Source: §4.3 · p. 8, Table 4 · p. 9
• With the same DexPilot solver and wrist-relative targets, RLHND with or without β-cache has the lowest mean Q-err across five hands and among the smoothest commands.Q-err and joint jerk are averaged over HOT3D, ARCTIC ego, and EgoDex.Source: §4.3 · p. 8, Table 3 · p. 8

4.4 ABLATION STUDIES

• The Cosmos 3 encoder is the dominant ablated factor; removing it degrades performance across metrics and test sets, while removing anatomical constraints or shape caching also worsens ARCTIC MPJPE.Using ground-truth β improves Q-err but not joint-level metrics; removing contact-only data causes a much larger contact drop without a notable force change.Source: §4.4 · p. 9

• RLHND achieves state-of-the-art motion reconstruction and tactile estimation, produces smooth retargeted commands across five dexterous hands, and lets DPP outperform its original tracker without manual annotation.Source: §5 · p. 9

• The work uses public datasets under their licenses and consent procedures, collects no new human data, and releases no raw video.Source: ETHICS STATEMENT · p. 10
• Because contact and force are inferred rather than measured, the authors recommend torque or current limits and human supervision for retargeted-command deployment.Source: ETHICS STATEMENT · p. 10
• Limited diversity in public datasets may cause performance variation across subjects, hand sizes, and skin tones not captured by the evaluation sets.Source: ETHICS STATEMENT · p. 10

AI USE STATEMENT

• AI-assisted figure and table edits, routine coding, text, and figures were reviewed, and the authors independently produced and verified the reported results.Source: AI USE STATEMENT · p. 10

A IMPLEMENTATION DETAILS

• β-cache uses the first sufficiently visible clip’s shape estimate for every clip, re-decodes earlier held-back clips, and falls back to per-clip estimates if no clip qualifies.With externally calibrated shape, inference starts with the cache initialized and decodes each clip once.Source: A IMPLEMENTATION DETAILS · p. 16, Figure 5 · p. 17

A.2 PSEUDO-LABEL GENERATION

• Damped Gauss–Newton is selected for pose-label conversion after more accurate alternatives prove too costly and undamped solving has unacceptable worst-case errors.Three iterations at λ=10^-3 leave 0.5–1 mm residuals across the corpus; retaining fingertips in the residual reduces DexYCB error from 0.66 to 0.47 mm.Source: §A.2 · p. 16, Table 6 · p. 18
• The constrained-label conversion changes MANO parameters but leaves the datasets’ 2D and 3D keypoint labels unchanged, preserving the released evaluation targets.The conversion is applied to every stage-1 training source.Source: §A.2 · p. 17
• Signed-distance contact labels retain penetrating vertices as contact, recovering complete regions that unsigned-distance labeling can miss.The continuous signed distances are stored so the contact threshold can be tuned during training.Source: §A.2 · p. 18
• OpenTouch glove readings are lifted onto covered MANO vertices, while unsensed vertices are masked from contact and force losses.The glove’s 169 active taxels cover 205 of 778 vertices.Source: §A.2 · p. 18

A.3 RAY-BASED CAMERA SOLVER

• The ray head predicts clip-level viewing rays from tapped features; their Fourier encoding supplies the ray positional embedding, while calibrated intrinsics are used by reported models.Source: §A.3 · p. 19

Training objectives

• Training masks let heterogeneous supervision sources contribute jointly, while pose losses supervise articulation, placement, image projections, presence, visibility, smoothness, and calibrated pixel rays.The shape loss applies only to hands with ground-truth shape that are not teacher-forced, matching the unconditioned test setting.Source: §A.4 · pp. 19–20

Pose metrics

• Detection-gated pose metrics penalize missed on-screen hands with canonical-pose or image-diagonal errors, preventing apparent gains from dropping difficult frames.MPJPE-p and PA-p use wrist-aligned 3D joints, while EPE2D-p measures on-screen 2D joint error.Source: §A.4 · p. 20
• MPJPE+OOS bypasses the detection gate and includes out-of-sight ground-truth hands, evaluating temporal extrapolation across missing observations.The all-frame measure includes wrist-aligned error whenever a ground-truth 3D hand is present.Source: §A.4 · p. 21

Temporal metrics

• Jitter measures temporal consistency—not accuracy—as the joint-averaged second temporal difference of camera-frame joints over qualifying runs.Runs require at least three consecutive frames with the hand predicted present and on-screen in ground truth; evaluation also checks that the ranking is unchanged on common detected frames.Source: §A.4 · p. 21

Shape metric

• Shape jitter measures within-recording hand-size drift; a single shape per recording yields zero, while per-frame shape estimates can fluctuate.The measure uses middle-finger bone-chain length at a fixed pose and averages over recordings and hands with at least 30 eligible frames.Source: §A.4 · p. 21

Tactile metrics

• Contact F1 uses a validation-selected 0.8 threshold while AUROC evaluates ranking; force MAE and RMSE use per-vertex pressure in kPa, with contact labels varying by source.OpenTouch labels pressure above 1 kPa; DexYCB, HOT3D, and ARCTIC use geometric proximity within 5 mm, and ARCTIC reports validation because its test split lacks ground truth.Source: §A.4 · pp. 21–22

Retargeting metrics

• Retargeting metrics compare tracker-derived joint commands with ground-truth-derived commands over contiguous labeled runs, and report Q-err and median second-difference jerk.The median is used for jerk because sparse ground-truth steps dominate the mean; DexPilot's low-pass filter is disabled so jerk reflects tracker output.Source: §A.4 · p. 22

Evaluation and optimization

• Baselines are evaluated with official code and released weights on the scored frames; the two RLHND rows share a checkpoint and differ only in per-window shape prediction versus β-caching.The deployed DPP policies are fine-tuned for 60k steps, with the 60k snapshot used for deployment.Source: §A.5 · pp. 23–24

Data

• The 200 human demonstrations span six tasks, alongside 200 teleoperated robot episodes; the 100:100 mixture uses all episodes, while 100:10 and 100:50 mixtures sample fewer robot episodes.Human demonstrations are divided across four pick-and-place objects, plastic-bag flips, and tissue assemblies; Figure 7 shows examples from both data sources.Source: §A.5 · p. 23, Figure 7 · p. 24

Packing and objective

• DPP observations combine stereo object points, language embeddings, 42 hand keypoints, and camera pose; its action is a normalized 16-step, 0.8-second keypoint-displacement chunk with a masked flow objective and detached-feature contact loss.Episodes are resampled to 20 fps; the contact loss does not backpropagate into the trunk.Source: §A.5 · p. 23

Real-robot evaluation

• Robot evaluation uses an RB-Y1 with two 20-DoF WUJI hands, a head-mounted stereo camera, and 16 trials per condition across four single-hand and two bimanual tasks.Human demonstrations use an identical camera rig to match the robot episodes' image format, intrinsics, and viewpoint.Source: §A.5 · pp. 24–26

Human hand labels

• The DPP comparison changes only human hand keypoint labels: HaWoR for the original policy and RLHND for DPP + RLHND; both use RLHND contact labels and identical robot data and training settings.RLHND and HaWoR labels receive the same stereo-depth alignment and cleaning before packing.Source: §A.5 · pp. 23, 26

Real-robot evaluation

• At inference, object points are extracted at 5 Hz, and policy outputs are retargeted by inverse kinematics; keypoints above a 0.05 contact probability receive a 15 mm outward-normal offset.The policy predicts a 16-step chunk of keypoint displacements and contact probabilities for both hands.Source: §A.5 · p. 26

B.1 REAL-ROBOT EXPERIMENTS WITH DPP

• 2.3 cm median right-wrist depth deviation for RLHND across 200 demonstrations is 17.8% lower than HaWoR's; after shared per-frame alignment, residual error is 0.1 cm versus 0.2 cm.The passage attributes much of the remaining offset to monocular scale-depth ambiguity and notes that a pre-defined β-cache can reduce it further.Source: §B.1 · p. 27, Figure 12 · p. 28
• 0.55, 0.53, and 0.54 cm versus 0.63, 0.59, and 0.58 cm: DPP + RLHND's sampled keypoint displacement errors are lower at 100:10, 100:50, and 100:100, and its deployed policy has higher rollout success.The flow-matching loss is also consistently lower across the three human:robot ratios.Source: §B.1 · pp. 27–28
• The β-cache removes within-recording shape drift but leaves joint jitter essentially unchanged; RLHND's residual jitter is predominantly image-plane and wrist-level.The reported decomposition separates depth and image-plane jitter; the cache's benefit is confined to size consistency, while temporal stability comes from clip-level decoding.Source: §B.1 · pp. 28–29, Table 8 · p. 28
• On ARCTIC, crop-based trackers have 18–47 mm/frame2 depth jitter versus 1.5 for RLHND, a 12–32× gap; their per-frame distance estimates directly affect the robot action space.Their image-plane jitter is only 2–5× higher.Source: §B.1 · p. 29

B.3 INFERENCE COST

• Despite its 7.8B parameters, Cosmos 3 Nano gives RLHND per-frame cost comparable to one ViT-H crop and lower than crop-based trackers including a second hand and detector; ACE-Ego-Hand is 4× faster but has nearly twice RLHND's HOT3D error.The comparison uses synchronized model-forward time per frame for both hands on one H100, with shared detector cost included for crop-based trackers.Source: §B.3 · p. 29, Table 9 · p. 29

B.4 PER-DATASET RETARGETING RESULTS

• On HOT3D, RLHND has the lowest or second-lowest Q-err on every embodiment; on ARCTIC ego, it leads on Sharpa, Shadow, and ALLEX.HaPTIC and WiLoR are closer on WUJI and Inspire; on EgoDex, all trackers are within 3° on WUJI and Sharpa, likely due to ground-truth joint noise.Source: §B.4 · p. 29
• Oracle jerk is 0.13-0.27 deg/frame2 on HOT3D and 0.17-0.43 deg/frame2 on EgoDex, showing that the reference commands used for Q-err contain frame-to-frame noise.Source: §B.4 · p. 29
• Across all three test sets, crop-based per-frame trackers exhibit substantially higher jerk than clip-level trackers HandFlow, ACE-Ego-Hand, and RLHND.The jerk differences are of the same order of magnitude.Source: §B.4 · p. 29
• RLHND remains within 2° of the best method on every EgoDex embodiment, has the lowest Q-err on Shadow, and ties ACE-Ego-Hand for lowest jerk.Each method achieves the lowest jerk on some of the five hands.Source: §B.4 · p. 29

B.5 ADDITIONAL QUALITATIVE RESULTS

• On unseen EgoDex windows, crop-based trackers have jagged wrist trajectories, ACE-Ego-Hand misses hands in several windows, and RLHND tracks both hands more consistently near ground truth.The wrist jaggedness is consistent with depth instability and the jerk metric.Source: §B.5 · p. 30
• Across additional OpenTouch and PVDB scenes, RLHND places contact and pressure in regions corresponding to glove or pressure-pad measurements, including single-fingertip presses.HACO predicts broader contact regions, HOPE has fragmented contact and lower pressure estimates, and PressureVision++ produces little output outside its planar-sensor representation.Source: §B.5 · p. 30

Per-vertex image sampling

• Vertex-level contact F1 was 0.709/0.543/0.570 on OpenTouch/DexYCB/HOT3D and OpenTouch force MAE was 1.256 kPa, versus 0.696/0.572/0.589 and 0.489 kPa for the default readout.The image-sampling branch was omitted from the final design because per-vertex image evidence added little information beyond pose-conditioned features and label supervision.Source: §B.6 · p. 30

Conditioning the expert on the pose stream

• AUROC was nearly unchanged across OpenTouch, DexYCB, and HOT3D (0.979/0.910/0.955 vs. 0.980/0.915/0.959), indicating little change in vertex ranking.The variant mainly shifts the operating point without a consistent direction.Source: §B.6 · p. 33
• The existing bone tokens provide sufficient pose information for the tactile expert, so the additional attention pathway is omitted while the frozen pose predictions remain bit-identical.Source: §B.6 · p. 33
• One-way pose-token attention lowered contact F1 to 0.635 vs. 0.696 on OpenTouch and 0.565 vs. 0.572 on DexYCB, while OpenTouch force MAE rose to 0.573 vs. 0.489 kPa.Source: Table 11 · p. 33, §B.6 · p. 33

• External shape calibration is demonstrated only with ground-truth β available in ARCTIC; its benefit in unconstrained settings and cache robustness to a poor initial view remain unevaluated.Source: C LIMITATIONS · p. 34
• Tactile training uses one glove and one planar pressure pad, limiting the diversity of observed pressure magnitudes and contact surfaces.Scaling force supervision across heterogeneous sensors remains challenging until surface representations and pressure units are more consistent.Source: C LIMITATIONS · p. 34
• Robot experiments use fixed inverse-kinematics retargeting with locked abduction joints and a heuristic contact offset; retargeting fidelity is measured against a solver, not task success.Mapping across differing hand kinematics, contact geometry, and compliance, using predicted force beyond binary contact, and real-time processing remain outside the work’s scope.Source: C LIMITATIONS · p. 34