EgoHand: Monocular 3D Hand Reconstruction from Egocentric Video

Motivation

• Reconstructed hand trajectories provide action supervision connecting visual observations to manipulation behavior in pipelines for learning from human demonstrations.The passage describes this capability as important for scaling robot learning from human demonstrations.Source: §1 · p. 2
• Reliable annotations remain challenging because egocentric recordings vary in hands, backgrounds, objects, and cameras, while heterogeneous datasets differ in joint definitions and annotation reliability.Inference must also preserve image alignment, recover camera-relative placement from crops, and remain efficient for large video collections.Source: §1 · p. 2

Approach

• EgoHand combines bounding-box conditioning and ray positional encoding with a separate 2D joint branch to refine lateral wrist translation while retaining predicted depth.Joint training couples image alignment with reconstructed geometry and camera-relative placement.Source: §1 · p. 2
• The 14-million-image corpus combines synthetic hands, in-the-wild egocentric video, and multi-camera motion capture, with annotations expressed in a common SOMA-Hand representation.The sources combine controlled geometric labels, varied scenes and objects, and precise supervision on real imagery.Source: §1 · p. 2

Results

• 77.0% of Apple EgoDex clips and 95.5% of Ropedia Xperience-10M clips received majority preference for EgoHand’s projected poses in a blinded study.Graders reported comparable visual quality with Lightwheel EgoStandard on 91.0% of clips.Source: §1 · p. 3
• On H2O, EgoHand achieves the lowest mean and p90 joint-position, root-translation, and acceleration errors among the evaluated methods.On HOT3D and ARCTIC, it outperforms HaMeR and the Dyn-HaMR reproduction across all five evaluated metrics in zero-shot.Source: §1 · p. 3

3D Hand Representations

• SOMA supports different skeletons and shape backends, and EgoHand uses native SOMA-Hand with 25 joints and the MHR backend’s explicit skeletal scale controls.MANO’s learned shape space couples skeletal proportions and surface geometry, limiting direct control of individual bone lengths.Source: §2 · p. 3

Hand Reconstruction Models

• EgoHand combines camera conditioning and a separate 2D joint branch, improving joint-position and root-translation accuracy over evaluated baselines while retaining efficient single-frame inference.Related models include HaMeR, WiLoR (Potamias et al., 2025), Dyn-HaMR (Yu et al., 2025), HaWoR (Zhang et al., 2025), and ViDiHand (Wang et al., 2026).Source: §2 · p. 3

Hand Reconstruction Datasets

• EgoHand’s roughly 14 million training images combine synthetic data, in-the-wild video, and multi-camera motion capture, with real recordings fitted to 3D landmarks in SOMA-Hand.Existing resources include real captures such as ARCTIC and HOT3D and synthetic renderings such as RHD and BEDLAM.Source: §2 · p. 3

3.1. Network Architecture

• A DINOv3 encoder feeds a 3D reconstruction branch and a 2D joint branch, whose image-space correspondences refine lateral wrist translation while preserving predicted depth.The resulting hand geometry is expressed in camera coordinates.Source: Figure 2 · p. 4, §3.1 · p. 4

3.2. 3D Hand Reconstruction

• Bounding-box conditioning retains crop location and scale, while ray positional encoding associates image features with viewing directions for 3D hand prediction.The 3D decoder predicts SOMA-Hand pose and shape, and a camera head predicts translation in camera coordinates.Source: §3.2 · p. 4

3.3. 2D Joint Prediction and Translation Refinement

• Predicted 2D joints correspond to reconstructed 3D joints; minimizing reprojection error estimates lateral translation while retaining the 3D branch’s wrist depth.When the solve is invalid, the model retains its direct translation estimate; uncertainty and presence outputs help select inference correspondences.Source: §3.3 · p. 5

3.4. Training Objective

• Joint training combines hand-reconstruction, direct 2D-prediction, and translation losses to supervise geometry, image alignment, and translation.Backpropagation through a valid correspondence solve couples predicted 2D coordinates, reconstructed 3D geometry, and depth; invalid solves are excluded from translation loss.Source: §3.4 · p. 5

4.1. Data Sources

• EgoHand’s 14 million training images combine synthetic data, in-the-wild video, and multi-view motion capture to balance scene and camera diversity with annotation accuracy.The sources provide controlled synthetic variation, diverse real scenes and interactions, and precise motion-capture supervision, including during egocentric occlusion.Source: §4 · p. 6, §4.1 · p. 6

4.2. Data Annotation Pipeline

• The annotation pipeline fits varied real-world observations into SOMA-Hand, using calibrated video, 3D landmarks, segmentation masks, and 2D–3D consistency checks.A shared shape per clip and per-frame poses account for differences in anatomical joint definitions.Source: §4.2 · p. 6
• Fitting targets come from projected 3D landmarks and hand segmentation masks, with forearms and ambiguous hand overlaps excluded from silhouette comparison.For shape estimation, selected frames share shape parameters while poses and shape are alternately refined using landmark, silhouette, and dimension constraints.Source: §4.2 · p. 6
• The inference workflow prepares crops, batches EgoHand predictions, smooths hand parameters, and uses synchronized camera trajectories to place hands in a shared world frame.Fisheye regions are resampled into virtual pinhole views, while pinhole images use standard crops.Source: Figure 3 · p. 7
• With shape fixed, per-frame hand poses are fitted to 3D landmarks, 2D correspondences, and masks, while temporal constraints reduce jitter.Poorly aligned or unstable intervals receive additional refinement; accepted updates pass alignment and motion checks, and exported annotations retain one shape per clip.Source: §4.2 · p. 7

5. End-to-End Inference Pipeline

• Video processing detects hands, prepares crops, reconstructs geometry, smooths hand sequences, and can transform predictions into a shared world frame using aligned camera poses.The example pipeline includes an RT-DETRv2-S detector and uses virtual pinhole views to handle fisheye imagery.Source: §5 · pp. 7–8
• The stages exchange hand boxes, camera parameters, and reconstructed sequences, allowing detector, camera-pose source, postprocessing, and execution infrastructure to change independently of the reconstruction model.Source: §5 · p. 7
• Adaptive smoothing filters SOMA-Hand parameters with local quadratic fits, narrowing windows when broader fits miss local motion to reduce jitter while preserving motion changes.Shape and scale are smoothed too, and joints and meshes are regenerated from the filtered parameters.Source: §5 · p. 8

5.1. Hand Reconstruction at Production-scale

• The production pipeline targets collections exceeding 10,000 hours across tens of thousands of clips, where predictable resource use and scheduling matter.Source: §5.1 · p. 8
• Fixed-size chunks, bounded SLAM keyframe storage, local postprocessing windows, and per-job caps keep memory and resource use bounded.Older keyframes move to a compact CPU-side record, while cleanup allows workers to share a machine.Source: §5.1 · p. 8
• Fixed-size crop batches and batched downstream work reduce per-item overhead; processing multiple videos per job reduced fixed overhead by roughly an order of magnitude in early runs.Downloads and uploads run concurrently with computation, and results stream as processing units finish.Source: §5.1 · p. 8
• Immutable SHA-256 build digests support tracing outputs to software versions, while distributed GPU and CPU workflows assemble LeRobot datasets for downstream policy-learning workflows.Model checkpoints are included at build time, with a public default as fallback, to avoid repeated runtime downloads.Source: §5.1 · p. 8

6.1. Experimental Setup

• The evaluation uses held-out HOT3D, H2O, and ARCTIC data, ground-truth hand boxes, and dataset-specific crop protocols; HaWoR and Dyn-HaMR receive ground-truth camera poses.The reported metrics include mean and p90 wrist-relative MPJPE, PA-MPJPE, root translation error, 2D EPE, and acceleration error.Source: §6.1 · pp. 8–9

6.2. Evaluation Results

• On H2O, EgoHand has the lowest mean and p90 MPJPE, RTE, and acceleration error, while WiLoR leads on PA-MPJPE and 2D EPE.Source: §6.2 · p. 9, Table 1 · p. 9
• On HOT3D, EgoHand has lower mean and p90 errors than HaMeR and Dyn-HaMR across all five metrics.Source: §6.2 · p. 9, Table 2 · p. 9
• On ARCTIC, EgoHand beats HaMeR and Dyn-HaMR across all five metrics with or without smoothing; smoothing lowers mean acceleration error from 7.64 to 2.64 mm/frame2.With smoothing, mean MPJPE, RTE, and acceleration error are 34.9%, 51.4%, and 67.5% lower than HaMeR, respectively; ARCTIC's full-resolution 2D EPE coordinates contribute to its larger values.Source: §6.2 · pp. 9–10, Table 3 · p. 10

6.3. Ablation Study

• Ablations show RayPE and bounding-box conditioning reduce MPJPE, while the 2D branch improves image-alignment metrics but can raise RTE and acceleration error.Adding the 2D branch reduces H2O MPJPE from 15.85 to 13.31 mm and 2D EPE from 12.67 to 9.51 px, while increasing RTE and acceleration error on both datasets.Source: §6.3 · p. 10, Table 4 · p. 10

6.4. Inference Latency

• Per-frame latency is measured on indicated NVIDIA GPUs, with EgoHand timing including batch preparation, inference, projection, and mirroring but excluding detection, cropping, and rendering.Post-processing is timed separately, and Dyn-HaMR's total wall-clock time includes its HaMeR initialization stage.Source: §6.4 · p. 11, Table 5 · p. 11

6.5. Human Preference Study

• In a blinded study, EgoHand received majority preference on 77.0% of EgoDex clips and 95.5% of Xperience-10M clips; EgoStandard was tied or inconclusive on 93.5% of clips.Six hundred clip pairs were rated by three graders each using projected 3D-joint overlays.Source: §6.5 · p. 11, Table 6 · p. 11, Table 7 · p. 11

• EgoHand is a monocular 3D hand reconstruction system trained on 14 million images spanning synthetic scenes, in-the-wild recordings, and motion capture unified through SOMA-Hand.It combines camera geometry cues, auxiliary 2D joint prediction, and translation refinement.Source: §7 · p. 12
• Zero-shot evaluations on HOT3D, H2O, and ARCTIC show strong reconstruction accuracy, and adaptive smoothing further reduces acceleration error.Source: §7 · p. 12
• A blinded human study favors EgoHand's projected poses over supplied EgoDex and Xperience-10M annotations, with predominantly no meaningful difference on EgoStandard.Source: §7 · p. 12
• Monocular depth-scale ambiguity, severe occlusion, and sensitivity to hand detection and camera calibration remain challenges.Future work will explore stereo and wrist-mounted views, tighter detection-reconstruction integration, and annotation quality's impact on downstream robot learning.Source: §7 · p. 12

A. Contributors and Acknowledgments

• Core contributors had sustained, significant impact across multiple subsystems and accountability for key design and implementation decisions; they are ordered by merit.Source: Core Leadership · p. 16
• Contributors made important project contributions and are listed alphabetically.Source: Core Leadership · p. 16
• Leadership members contributed to organizational and technical direction related to Isaac Lab and are listed alphabetically.Source: Core Leadership · p. 16
• Core leadership had primary responsibility for Isaac Lab’s organizational and technical direction and are listed alphabetically.Source: Core Leadership · p. 16

B. Metric Definitions

• MPJPE averages distances between corresponding wrist-relative joints in millimeters, retaining orientation, scale, and articulation errors but removing global translation.The wrist has zero error but remains in the 21-joint average.Source: B. Metric Definitions · pp. 16–17
• RTE measures the camera-coordinate distance between predicted and ground-truth wrists, assessing absolute hand localization in millimeters.Source: B. Metric Definitions · p. 17
• Acceleration error compares wrist-relative hand motion after transforming joints into a common world frame, and is reported in mm/frame^2.It excludes global wrist-trajectory acceleration; comparisons require consistent frame rates and sampling intervals.Source: B. Metric Definitions · p. 17
• PA-MPJPE reports residual joint-configuration error after per-hand, per-frame similarity alignment, so it does not retain absolute localization or global scale error.The alignment minimizes squared distances, while the score averages Euclidean distances in millimeters.Source: B. Metric Definitions · p. 17
• 2D EPE measures pixel error between projections of the evaluated 3D joints using the calibrated image projection; comparisons require matching projection and resolution.It does not measure outputs of the separate 2D prediction branch.Source: B. Metric Definitions · p. 17