PaperlayerView exact revision on arXiv

Source-linked AI summary

DewHand: Direct Metric World-Space Hand Motion Reconstruction from Egocentric Videos

Naru Suzuki, Satoshi Ikehata, Takehiko Ohkawa, Yoichi Sato, Shuhei Kurita

arXiv:2610.29936v1cs.CV

TL;DR

Egocentric video makes metric world-space hand reconstruction difficult because camera motion, monocular scale ambiguity, and incomplete observations complicate trajectory recovery. DewHand jointly generates hand and camera motion with geometric and hand-specific features and flow matching. It improves rigid-aligned world error over StableHand on HOT3D and ARCTIC and recovers camera scale within 10% on two thirds of HOT3D clips.

  • Problem

    Egocentric video does not directly provide metric scale, and camera motion and incomplete hand observations make recovery of a consistent world-space hand trajectory challenging.

    Source: §1 · p. 2

  • Method

    DewHand jointly generates metric world-space hand and camera motion using geometric and hand-specific features with conditional flow matching.

    Source: §1 · p. 2

  • Results

    32.9 to 28.4 mm on HOT3D and 76.2 to 36.2 mm on ARCTIC: DewHand reduces rigid-aligned world error relative to StableHand.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Metric, stable-world hand trajectories support human-activity analysis, manipulation-skill capture, and transfer of demonstrations to embodied agents.

    Source: §1 · p. 2

Abstract

from arXiv · show

Recovering metric world-space hand motion from egocentric video supports understanding human interactions and transferring them to embodied agents. Three challenges complicate this task: multi-stage pipelines propagate errors from independently estimated hand and camera motion; monocular observations leave physical scale ambiguous; and occlusion and out-of-view hands admit multiple plausible motion trajectories. We present DewHand, a unified model for direct joint reconstruction of world-space hand and camera motion. To avoid compounding errors from multi-stage estimation, DewHand jointly predicts hand and camera motion in a shared world coordinate system within a single-stage pipeline. To resolve metric-scale ambiguity, it combines scene-level geometric features from VGGT-$\Omega$ with metric-space supervision, while hand-specific features preserve fine-grained articulation. To handle occlusion and out-of-view hands, we formulate reconstruction as conditional motion generation with flow matching. Under the same architecture, this generative formulation outperforms direct regression in both accuracy and temporal smoothness with only two sampling steps. Experiments on HOT3D and ARCTIC demonstrate improved world-space accuracy and temporal consistency over state-of-the-art methods. DewHand reduces rigid-aligned world-space error relative to StableHand from 32.9 to 28.4 mm on HOT3D and from 76.2 to 36.2 mm on ARCTIC. It also recovers camera scale within 10% on two thirds of HOT3D clips, versus about a quarter for SLAM. Moreover, in zero-shot evaluations on HoloAssist, H2O, and TACO, captured with different devices, DewHand achieves the lowest rigid-aligned world-space error (WA-MPJPE) on each dataset.

1 INTRODUCTION

DewHand targets metric, stable-world hand trajectories from egocentric video, where camera motion, scale ambiguity, and incomplete observations complicate reconstruction. It jointly generates hand and camera motion, combining geometric and hand-specific features with flow matching.

1 INTRODUCTION

  • Separate hand and camera estimates can propagate drift and mismatches; monocular scale ambiguity and occlusion further leave multiple trajectories consistent with the video.Hands may also exit the field of view, making observations incomplete.Source: §1 · p. 2
  • DewHand directly generates metric world-space motion for both hands and the camera, jointly predicting wrist poses, local articulation, and camera motion in a shared frame.It uses geometric features from VGGT-Ω with metric-space supervision, hand-specific features, and conditional flow matching.Source: §1 · p. 2
  • Supervising camera motion reduces rigid-aligned hand error by 6.4 mm.The formulation generates hand and camera motion jointly rather than composing separately estimated trajectories.Source: §1 · p. 2
  • Replacing geometric features with DINOv2 increases hand error by 13 mm and camera error by 57 mm.The architecture combines geometry-aware and hand-specific video features to model wrist placement, articulation, and camera motion.Source: §1 · p. 2
  • 32.9 to 28.4 mm on HOT3D and 76.2 to 36.2 mm on ARCTIC: DewHand reduces rigid-aligned world error relative to StableHand.Camera scale is within 10% on two thirds of HOT3D clips, versus about a quarter for SLAM.Source: §1 · p. 3

2 RELATED WORK

Prior work improves articulated hand pose, localization, and interaction reconstruction, while world-space methods differ in how they handle camera motion and metric coordinates.

2 RELATED WORK

  • Large-scale hand models recover articulated pose or localization, while interaction priors and spatio-temporal context address occlusion and motion ambiguity.Other strategies include contrastive pre-training, controllable synthetic data, exemplar-based reconstruction, and kinematics-aware architectures.Source: §2 · p. 3
  • Egocentric datasets capture hand-object interactions, and recent methods estimate hand-object pose or camera-space hand pose.These advances improve pose and shape, but a shared metric world trajectory also requires camera motion.Source: §2 · p. 3
  • HaWoR and Dyn-HaMR construct global hand motion from camera-space estimates and separately recovered camera trajectories; HandFlow lifts its camera-space predictions using SLAM.StableHand refines world-space observations assembled from separate hand and camera estimates.Source: §2 · p. 3
  • UniHand generates hand motion in the first camera frame without generating camera trajectory, whereas ReViV jointly models camera trajectories and camera-space hand kinematics before metric alignment.DewHand jointly generates metric hand and camera trajectories in one stage, conditioned on geometric and hand-specific features.Source: §2 · p. 3

3 METHOD

DewHand jointly generates metric world-space hand and camera motion from egocentric video, conditioning on scene geometry and hand-specific features. It uses conditional flow matching to model plausible motion when hand observations are missing or ambiguous.

3 METHOD

  • DewHand generates hand and camera trajectories together in a metric world frame, without an initial reconstruction, using geometric conditioning for scale and conditional flow for missing observations.Source: §3 · p. 3

3.1 PROBLEM FORMULATION AND STATE REPRESENTATION

  • The first camera coordinate system serves as the world frame, and global motion and local hand parameters form coupled streams with distinct geometry and conditioning cues.Source: §3.1 · p. 3

3.2 VISUAL CONDITIONING AND MISSING OBSERVATIONS

  • VGGT-Ω supplies scene geometry and camera cues, while HaMeR provides separate hand-specific features; missing hand features are replaced by side-specific mask tokens during training.Source: §3.2 · p. 4

3.4 FLOW MATCHING AND TRAINING OBJECTIVES

  • Flow matching transports noise toward trajectories compatible with visual features, while metric-space hand and camera losses and wrist-motion terms supervise the generated state; inference uses two Euler steps.Source: §3.3 · p. 5, §3.4 · p. 5

3.3 JOINT HAND-CAMERA FLOW MODEL

  • The model uses global-motion, local-hand, and condition streams, with within-stream spatio-temporal attention followed by joint attention across streams at each frame.Source: §3.3 · p. 5

4 EXPERIMENTS

DewHand improves world-space hand reconstruction on HOT3D and ARCTIC, improves camera estimates, and transfers zero-shot across three datasets. Ablations support the roles of camera supervision, geometric and hand-specific features, and flow matching.

4.1 WORLD-SPACE HAND MOTION

  • Experiments use 150-frame, 30-fps clips, training on HOT3D and fine-tuning on ARCTIC; default inference uses two Euler steps and five samples.Comparisons include per-frame estimators lifted with ViPE and video-based methods; all but HandFlow are trained on ARCTIC.Source: §4.1 · p. 6
  • Against StableHand with predicted shape, DewHand reduces SE(3)-aligned WA-MPJPE from 43.74 to 28.56 mm on HOT3D StableHand and from 78.63 to 36.35 mm on ARCTIC.It also improves PA-MPJPE; the small Sim(3)-to-SE(3) error gap indicates retained metric scale.Source: §4.1 · pp. 6–7

4.2 CAMERA MOTION

  • DewHand improves camera scale and trajectory accuracy over frozen VGGT-Ω features, with gains after similarity alignment indicating improvements beyond scale correction.Table 2 reports HOT3D camera scale accuracy, raw and aligned ATE, rotation error, and one-frame RPE; camera trajectories are relative to frame 0.Source: §4.2 · pp. 7–8, Table 2 · p. 7

4.3 CROSS-DEVICE ZERO-SHOT GENERALIZATION

  • Without fine-tuning, DewHand has the lowest reported errors across HoloAssist metrics and the lowest SE(3)-aligned joint and wrist WA-MPJPE and AccEr on H2O and TACO.Other metrics, particularly local pose and inter-hand placement, are mixed on H2O and TACO.Source: §4.3 · p. 8, Table 3 · p. 8

4.4 ABLATION STUDY

  • Removing camera supervision raises SE(3)-aligned WA-MPJPE from 35.94 to 42.36 mm; replacing VGGT-Ω with DINOv2 raises it from 35.94 to 49.15 mm and camera ATE from 31.5 to 88.9 mm.DA3 gives comparable hand and camera accuracy, while removing hand-specific features worsens local pose without changing camera accuracy; the ablations change one named component at a time.Source: §4.4 · pp. 8–9, Table 4 · p. 9
  • Compared with L2 regression, flow matching improves PA-MPJPE from 7.04 to 6.76 mm, SE(3)-aligned WA-MPJPE from 45.13 to 35.94 mm, and AccEr from 3.34 to 1.89 m/s2.Camera accuracy remains comparable; the architecture is shared, and one Euler step performs worse than two-step sampling on hand error and AccEr.Source: §4.4 · p. 9, Table 4 · p. 9

5 CONCLUSION

DewHand jointly generates hands and camera motion in a shared world frame, using complementary visual features and flow matching. It achieves strong world-space results on the evaluated datasets, while extension beyond fixed offline windows remains future work.

5 CONCLUSION

  • DewHand achieves the lowest rigid-aligned world-space hand error on HOT3D and ARCTIC, and on HoloAssist, H2O, and TACO in zero-shot evaluation.Other metrics do not always lead on the zero-shot datasets.Source: §5 · p. 9
  • Geometric and hand-specific features provide complementary cues for global placement and articulation, while flow matching supports motion reconstruction under incomplete observations.The model jointly generates global hand and camera motion, local articulation, and hand shape in a shared world frame.Source: §5 · p. 9
  • Future work could extend DewHand beyond fixed offline windows and incorporate manipulated objects and hand-object contact.Source: §5 · p. 9

A IMPLEMENTATION DETAILS

Training randomly masks hand observations at several granularities and uses a six-layer, 16-head model with width 512.

A IMPLEMENTATION DETAILS

B.1 HAND METRICS

The hand metrics differ in how they align predicted joints to ground truth and which frames determine the alignment. Their alignment choices determine whether scale errors and later trajectory drift remain in the score.

B.1 HAND METRICS

  • PA-MPJPE uses per-frame Sim(3) alignment to measure local articulation; W-MPJPE fits Sim(3) or SE(3) on the first two frames, leaving later drift visible.Source: §B.1 · p. 13
  • WA-MPJPE fits Sim(3) or SE(3) across all frames to measure global placement; SE(3) preserves predicted scale, so scale errors remain.wWA applies WA-MPJPE to the wrist alone, separating wrist placement from finger articulation.Source: §B.1 · p. 13
  • MRRPE-A measures inter-wrist displacement after shared rigid alignment, with shared translation canceling and no scale applied.AccEr measures joint-acceleration error without alignment.Source: §B.1 · p. 13

B.2 CAMERA METRICS

The camera metrics measure trajectory position, scale, rotation, and one-frame translation, with position error reported under raw or aligned coordinates.

B.2 CAMERA METRICS

  • ATE is the root-mean-square distance between predicted and ground-truth camera centers, measured raw or after clip-level SE(3) or Sim(3) alignment.Camera trajectories are expressed relative to frame 0.Source: §B.2 · p. 13
  • Scale accuracy δ < κ is the percentage of clips whose predicted-to-ground-truth trajectory scale ratio satisfies max(s, 1/s) < κ.Source: §B.2 · p. 13
  • Rotation error is the mean geodesic distance between corresponding camera rotations, while RPEt@1 is the root-mean-square error of one-frame relative camera translations.Source: §B.2 · p. 13

B.3 EVALUATION SETS

Evaluation uses 150-frame, 30-fps windows and common annotated frames, with dataset-specific handling for baselines and predictions.

B.3 EVALUATION SETS

  • All methods are scored on the same ground-truth-annotated frames without visibility gating, using the same metric definitions.Missing baseline predictions are linearly interpolated, clips without a method output are excluded from the common set, and MRRPE-A uses clips with both hands annotated and predicted.Source: §B.3 · pp. 13–14
  • Evaluation windows contain 150 frames at 30 fps, and camera-space baselines are lifted with ViPE on full HOT3D and ARCTIC sequences but per window with ground-truth intrinsics on zero-shot datasets.Source: §B.3 · p. 13, Table 5 · p. 14
  • HandFlow left-hand predictions are obtained by mirroring video, HoloAssist's 26-joint tracks are mapped to 21 joints, and ReViV's two-second outputs are rigidly stitched into 150-frame sequences.StableHand uses its released enhanced checkpoint and a separate DewHand model trained on HOT3D windows excluding overlap with its 244 test clips; other segments of those sequences remain in training, following StableHand's protocol.Source: §B.3 · p. 14

C ADDITIONAL RESULTS

The additional results examine inference-step count, incomplete detections, detection-failure rates, runtime, and qualitative examples across five datasets.

C.1 INFERENCE STEPS

  • Two Euler steps give the lowest world-aligned hand and camera trajectory errors; more steps reduce AccEr but do not improve world-space placement.The authors therefore use two steps by default, with five-sample velocity averaging in the evaluated full model.Source: Table 6 · p. 14, §C.1 · p. 14

C.2 SAMPLING WITH MISSING OBSERVATIONS

  • Without WiLoR detections, sample spread increases and tracks error (Spearman ρ = 0.58), while DewHand has lower WA-MPJPE and AccEr than L2 regression in both detection groups.Sample spread is the disagreement among five independent samples and uses no ground truth; the table also reports a five-prediction mean and a best-of-five result that is not realizable.Source: §C.2 · p. 14, Table 7 · p. 15

C.3 DETECTION FAILURES

  • DewHand has the lowest world-space hand error in every WiLoR detection-failure group, including clips with the most frames undetected.The comparison covers 606 clips with valid hand ground truth and reports clip-level WA-MPJPE after rigid SE(3) alignment in millimeters.Source: Table 8 · p. 15, §C.3 · p. 15

C.4 INFERENCE COST

  • DewHand is 2.6–3.7× faster than SLAM-based pipelines, and its 12.0-second runtime includes 1.7 seconds for VGGT-Ω and the flow head.The remaining time is rectification, hand detection, and HaMeR feature extraction; StableHand is omitted because its full preprocessing pipeline is unavailable.Source: Table 9 · p. 15, §C.4 · p. 15

D ADDITIONAL QUALITATIVE COMPARISONS

  • Figures 5–9 show qualitative sequences from HOT3D, ARCTIC, HoloAssist, H2O, and TACO, with shared viewpoint and display scale within each sequence.Green and blue curves trace right and left wrists, purple frustums show the camera, and one SE(3) transform fitted to both hands is also applied to the camera without scale correction.Source: D ADDITIONAL QUALITATIVE COMPARISONS · p. 16, Figure 5 · p. 17, Figure 6 · p. 18

Related papers

Loading 2610.29936v1…