PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoForce: Forearm-Guided Camera-Space 3D Hand Pose from a Monocular Egocentric Camera

Christen Millerdurai, Shaoxiang Wang, Yaxu Xie, Vladislav Golyanik, Didier Stricker, Alain Pagani

arXiv:2605.12498v1cs.CVcs.GR

TL;DR

Absolute camera-space hand pose from a single egocentric RGB image remains difficult because of depth ambiguity and varied camera optics. EgoForce combines forearm-guided hand–arm modeling with ray-space lifting, and across three benchmarks reports higher camera-space accuracy with stable performance across perspective and fisheye cameras.

  • Problem

    Single-camera monocular methods struggle with depth–scale ambiguity and diverse head-mounted optics, while root-relative predictions cannot provide absolute hand position for interaction-centric tasks.

  • Method

    EgoForce jointly models hand and forearm geometry with a unified architecture, uses a differentiable forearm representation, and applies a cross-camera ray space solver for absolute camera-space lifting.

  • Results

    Across three egocentric benchmarks, EgoForce delivers higher camera-space accuracy, stable temporal predictions under occlusions, and effective performance across perspective and fisheye optics.

  • Takeaways & Limitations

    Forearm context and ray-based geometric constraints support monocular camera-space hand tracking across compact, diverse AR/VR camera configurations.

  • Takeaways & Limitations

    EgoForce relies on calibrated 3D training datasets, remains sensitive to camera intrinsics, and precise metric hand–arm scaling remains underconstrained without user-specific scale cues.

Abstract

from arXiv · show

Reconstructing the absolute 3D pose and shape of the hands from the user's viewpoint using a single head-mounted camera is crucial for practical egocentric interaction in AR/VR, telepresence, and hand-centric manipulation tasks, where sensing must remain compact and unobtrusive. While monocular RGB methods have made progress, they remain constrained by depth-scale ambiguity and struggle to generalize across the diverse optical configurations of head-mounted devices. As a result, models typically require extensive training on device-specific datasets, which are costly and laborious to acquire. This paper addresses these challenges by introducing EgoForce, a monocular 3D hand reconstruction framework that recovers robust, absolute 3D hand pose and its position from the user's (camera-space) viewpoint. EgoForce operates across fisheye, perspective, and distorted wide-FOV camera models using a single unified network. Our approach combines a differentiable forearm representation that stabilizes hand pose, a unified arm-hand transformer that predicts both hand and forearm geometry from a single egocentric view, mitigating depth-scale ambiguity, and a ray space closed-form solver that enables absolute 3D pose recovery across diverse head-mounted camera models. Experiments on three egocentric benchmarks show that EgoForce achieves state-of-the-art 3D accuracy, reducing camera-space MPJPE by up to 28% on the HOT3D dataset compared to prior methods and maintaining consistent performance across camera configurations. For more details, visit the project page at https://dfki-av.github.io/EgoForce.

1 Introduction

EgoForce targets absolute camera-space 3D hand pose from a single egocentric RGB camera, using forearm context and camera-model-agnostic geometric lifting to handle depth ambiguity and diverse optics.

  • Motivation: Single-camera egocentric hand tracking is important for compact AR/VR systems and applications requiring metric 3D hand motion.The paper specifically motivates onsite teleoperation and surgical training.
  • Problem: Root-relative methods recover pose only up to unknown translation and scale, preventing absolute hand positioning for interaction-centric tasks.The target is full 6-DoF pose in the headset’s metric coordinate frame.
  • Approach: EgoForce jointly leverages hand and forearm imagery to recover absolute 3D hand pose from a single egocentric camera.Forearm–hand coupling supplies metric cues that help resolve monocular depth–scale ambiguity.
  • Approach: HALO jointly regresses hand and forearm pose with shape proxies from monocular images.The architecture is designed for unified hand–arm reasoning.
  • Approach: A differentiable forearm representation provides metric cues and improves absolute 3D hand-pose estimation through contextual arm–hand reasoning.The representation is one of the paper’s stated contributions.
  • Approach: The cross-camera ray space solver recovers absolute hand–forearm placement across fisheye, perspective, and distorted wide-FOV optics.This supports deployment across diverse head-mounted camera configurations.

2 Related Works

Prior work often predicts root-relative pose or depends on camera- and scale-specific assumptions. EgoForce instead combines crop-based hand–forearm processing with cross-attention and a generative prior for missing forearms.

  • Prior limitations: Most monocular methods predict root-relative pose under weak perspective, discarding absolute hand position and often ignoring crop geometry effects.This limits direct camera-space reconstruction from a single image.
  • Prior approaches: Existing camera-space approaches include single-stage regressors, root-depth methods, known-intrinsics methods, and implicit neural formulations, each retaining different camera or scale dependencies.The passage contrasts several categories rather than identifying one universally dominant approach.
  • EgoForce: The complete pipeline predicts root-relative hand–arm geometry and lifts it to camera space with the Ray Space Solver.The method is organized around hand and forearm models, HALO, ray-space lifting, and supervision.
  • EgoForce: EgoForce processes hand and forearm regions separately, then fuses them through cross-attention to capture kinematic structure.Separate crops preserve fine-grained hand geometry while retaining forearm information.
  • EgoForce: When the forearm is not visible, a generative forearm prior infers a plausible arm orientation to maintain continuity and realism in 3D hand–forearm motion.The workflow uses missing-arm tokens and a hand-conditioned variational prior.

3 The EgoForce Framework

EgoForce combines a unified hand–arm prediction pipeline with a differentiable forearm representation and a ray-space solver to recover camera-space hand and forearm geometry from one egocentric RGB frame.

  • EgoForce predicts camera-space 3D hand and forearm meshes from a single egocentric RGB frame.The framework defines hand and forearm models, predicts root-relative geometry, and lifts it to camera space.
  • The differentiable forearm representation supplies metric cues and supports contextual arm–hand reasoning for absolute 3D hand-pose estimation.FARM models forearm shape, rotation, and translation, while the hand–forearm mesh is connected at the wrist with an elbow-direction offset.
  • HALO jointly predicts 2D joints, confidence scores, MANO parameters, and FARM parameters from hand and arm image crops.Its encoder undistorts crops, forms patch tokens, and uses hand- and arm-specific queries to decode geometry and pose.
  • The Ray Space Solver estimates one shared camera-space translation by minimizing confidence-weighted perpendicular distances from predicted joints to calibrated camera rays.It removes unknown depths through plane projection, uses a closed-form least-squares solution, and applies a Kalman filter to stabilize per-frame estimates.
  • Because ray directions can be computed for any camera projection model, the solver supports arbitrary calibrated cameras and different optical configurations.The framework includes a cross-camera calibration-conditioned module for deployment across devices with different optics.
  • Training combines heatmap, root-relative 3D-joint, MANO, FARM, relative-orientation, forearm-prior, and camera-space 3D-joint losses.When FARM parameters are unavailable, FARM and relative-orientation losses are set to zero, and out-of-frame keypoints are masked.

4 Experimental Evaluation

EgoForce is evaluated across multiple egocentric datasets, camera configurations, and ablations, achieving strong camera-space accuracy, articulation, and temporal stability. Results show that explicit camera geometry and forearm context are especially valuable under fisheye distortion, occlusion, and near-field depth ambiguity.

  • Datasets and metrics: 3.67M RGB images with MANO parameters and 3D joints support training across six datasets, with evaluation mainly on egocentric H2O, HOT3D, and ARCTIC.
  • Benchmark results: EgoForce achieves state-of-the-art articulation and camera-space accuracy on ARCTIC, where strong hand–object occlusions challenge monocular depth estimation and localization.
  • Benchmark results: 22% lower CS-ACC and 17% lower RS-ACC show improved temporal stability, with the largest gains occurring at 25-55% hand-joint visibility.This visibility range corresponds to approximately 5-12 visible joints and commonly arises during hand–object manipulation.
  • Benchmark results: 28% CS-MJE reduction on HOT3D relative to HandDGP demonstrates robust camera-space reconstruction under severe fisheye distortion and wide-FOV motion.
  • Camera geometry ablation: 60.5% lower CS-MJE and 62.9% lower RS-MJE result from undistortion alone on HOT3D, while combining undistortion with crop-specific intrinsic conditioning yields the best reported errors.The combined configuration reaches 45.8 mm CS-MJE and 18.9 mm RS-MJE; full-frame rectification degrades performance.
  • Arm-context ablation: Arm context improves hand performance when visible, while a hand-conditioned variational prior substantially improves forearm recovery when the arm is occluded.With invisible arms, RS-MJE drops from 28.7 to 12.8 mm and CS-ACC from 20.6 to 18.4 m/s2.
  • Depth-scale stability: Arm input reduces near-field hand-scale error from 4.7 to 2.7 mm at 200-300 mm camera distance, while frame-wise scale variation remains 4 mm on HOT3D and 2 mm on ARCTIC.
  • Calibration robustness: Performance improves from 43.9 to 39.3 mm at 50% intrinsic noise and degrades gracefully only under large camera-geometry mismatches above 150%.

5 Limitations

EgoForce depends on calibrated 3D training data and remains sensitive to camera intrinsics, limiting its generalization scope.

  • Calibrated 3D datasets are required for training, preventing use of large 2D hand datasets and limiting generalization to in-the-wild imagery.
  • EgoForce remains sensitive to camera intrinsics.

6 Conclusion

EgoForce combines forearm context with camera-model-aware ray-space lifting for absolute camera-space hand pose. Across three egocentric benchmarks, it provides accurate and temporally stable predictions under occlusion and perspective or fisheye optics.

  • EgoForce recovers absolute camera-space 3D hand pose using forearm context and camera-model-aware ray-space lifting.
  • Across three egocentric benchmarks, EgoForce delivers higher camera-space accuracy and stable temporal predictions under hand–hand and hand–object occlusions.
  • Explicit distortion modeling and crop-aware intrinsic conditioning strongly benefit wide-FOV tracking across perspective and fisheye optics.

Supplementary Material

Supplementary figures examine camera-space projections, undistortion, and crop-intrinsics conditioning across diverse egocentric camera settings.

  • Predicted hand-arm meshes maintain accurate image projections across fisheye, perspective, and distorted-perspective cameras.The evaluated datasets are HOT3D, H2O, and ARCTIC.
  • Lens-model undistortion preserves pixel-to-ray geometry and yields the most accurate camera-space reconstruction, especially near image peripheries.Raw fisheye crops suffer nonlinear-ray errors, while full-frame perspective rectification introduces peripheral warping and resampling artifacts.
  • Crop Intrinsics Tokens encode crop-specific camera geometry, enabling explicit local reasoning and reducing camera-space mesh error.The tokens improve alignment between predictions and ground truth.

7 Additional Details about our Framework

The framework combines a differentiable forearm mesh, learned shape regularization, camera-geometry tokens, distortion correction, and unified hand-arm modeling to support camera-general reconstruction.

  • ForeArm Representation Model: FARM represents the forearm as a lightweight, differentiable truncated-cone mesh with elbow, mid-forearm, and wrist joints.Its parameterized generator produces a watertight triangular mesh suitable for downstream pose and shape reasoning.
  • ForeArm Representation Model: FARM discards axial twist because its radially symmetric truncated-cone geometry cannot determine forearm pronation.The representation retains the swirl component while omitting twist around the longitudinal axis.
  • ForeArm Representation Model: A PCA prior maps a low-dimensional latent code to forearm parameters and regularizes shapes toward plausible adult geometries learned from AMASS.The model uses d=5 and is trained from 2806 SMPL meshes covering 344 subjects.
  • Discussion: FARM’s single-segment adult-shape assumptions limit fidelity for axial rotation and anatomy outside the adult SMPL distribution.The paper identifies pronation and supination of up to ±90° and atypical anatomy as boundaries, suggesting off-axis markers, IMUs, or EMG as mitigations.
  • Crop Intrinsics & Distortion Correction: Crop Intrinsics Tokens concatenate local ray angles and normalized crop-intrinsic quantities after lens-model distortion correction.This conditioning encodes crop geometry relative to the camera for cross-camera 3D hand-pose estimation.

7.3 Ray Space Solver (RSS)

The Ray Space Solver recovers camera-space translation by fitting predicted 3D joints to viewing rays, using calibrated camera models and confidence-weighted closed-form least squares.

  • Solver formulation: RSS takes 3D joints, estimated 2D keypoints, and confidence weights as inputs to compute camera-space translation.It enforces that each translated joint lies along its corresponding viewing ray.
  • Ray construction: Ray directions are obtained by unprojecting keypoints with the applicable camera model and then L2-normalizing them.The formulation supports pinhole, equidistant fisheye, rational-polynomial, and Kannala–Brandt models.
  • Solver formulation: A weighted least-squares objective eliminates per-joint depth in closed form and solves for the shared translation.The projector formulation removes along-ray depth components and measures shortest point-to-ray residuals.
  • Robustness: RSS adds a small Tikhonov term and condition-number checks to stabilize translation estimation.When κ(M) ≥ 10^6, the method halves the weights and resolves the system.
  • Temporal filtering: A constant-velocity Kalman filter smooths predicted camera-space translations using offline-tuned noise parameters.Its tuning trades visible-frame accuracy against temporal smoothness by penalizing acceleration.

8 Experimental Evaluation

The evaluation uses three egocentric benchmarks and reports camera-space, root-relative, and Procrustes-aligned joint errors, with dataset-specific split and annotation constraints.

  • Datasets: ARCTIC evaluation uses 1.7M training images and 23K egocentric validation images from recordings involving 11 objects and 10 subjects.Training combines official-split exocentric and egocentric frames, while evaluation uses the egocentric validation subset.
  • Datasets: HOT3D’s official 20% test split is excluded because camera-space joint-error annotations and an official evaluation server are unavailable.The remaining data are divided into 60% training and 20% validation.
  • Metrics: The reported metrics include CS-MJE, RS-MJE, and PS-MJE for camera-space, root-relative, and Procrustes-aligned joint errors.CS-MJE captures translation in addition to pose, scale, and rotation, whereas RS-MJE ignores translation.

9 Additional Experiments

Additional experiments evaluate EgoForce against competing methods, camera configurations, calibration mismatch, depth–scale stabilization, and camera-space lifting. Results show strong performance across datasets and robustness to moderate calibration errors, while calibration and out-of-domain limitations remain.

  • Comparison with UmeTrack: EgoForce reduces CS-MJE by 68.1% on ARCTIC, 83.2% on HOT3D, and 79.9% on H2O versus UmeTrack* under realistic 2D-box crops.It also improves PS-MJE by 67.3%, 77.2%, and 74.1%, respectively.
  • Comparison with HaWoR: EgoForce achieves 49.5 mm CS-MJE on ARCTIC and 25.0 mm on H2O, outperforming HaWoR’s 319.9 mm and 72.5 mm, respectively.HaWoR depends on SLAM, which is sensitive to tracking failures in close-range interaction scenes.
  • Depth–scale stabilization: Arm input improves hand scale estimation, especially in the near field, supporting its role in reducing depth–scale ambiguity.The analysis reports stable frame-wise hand-scale variation of around 4 mm on HOT3D and 2 mm on ARCTIC without per-user calibration.
  • Camera-space lifting: 43.9 mm CS-MJE and 14.0 m/s2 CS-ACC are achieved by RSS with Kalman filtering on the evaluated datasets, giving the best lifting performance.On HOT3D, RSS without filtering reaches 45.8 mm CS-MJE and 23.5 m/s2 CS-ACC, while DGP reaches 115.6 mm and 25.0 m/s2.
  • CIT ablation: CIT consistently reduces hand CS-MJE across radial image regions even when undistortion is fixed, including peripheral areas with strongest fisheye distortion.This indicates benefits beyond preprocessing and supports robustness throughout the image.

10 Implementation Details

The implementation combines joint hand–arm visual encoding, plausible arm completion, and camera-space translation recovery from calibrated image rays. A Kalman filter then temporally smooths the estimated translation.

  • Arm–Hand Crop Encoder: Hand and arm crops produce 196 hand tokens and 49 arm tokens, concatenated into 245 visual tokens using a ViT-H/16 backbone.The hand crop is 224×224, the arm crop is 112×112, and the patch size is 16.
  • Camera-space lifting: The model estimates camera-space translation from 24 2D–3D correspondences comprising 21 hand and 3 forearm joints.Crop coordinates are mapped back to full-image pixels before ray construction.
  • Camera-space lifting: Predicted full-image joints are normalized by calibrated intrinsics, unprojected into unit bearing rays, and solved for translation using weighted point-to-ray least squares.A constant-velocity Kalman filter smooths translation at 30 Hz.

11 Limitations

EgoForce remains limited by monocular metric-scale underconstraint and by difficult visual conditions such as severe occlusion, limited field of view, and fast motion. Its forearm meshes are plausible but not yet optimized for precise hand–forearm localization.

  • Metric scale: Precise metric scaling of a MANO hand–arm configuration remains underconstrained without user-specific cues such as hand size or limb length.The forearm prior mitigates, but does not eliminate, monocular depth–scale ambiguity.
  • Arm modeling: Predicted arm meshes are often plausible but are not yet optimized for tasks requiring precise hand–forearm localization.Arm modeling primarily functions as a geometric prior for stabilizing hand pose estimation.
  • Challenging conditions: Inferring individual limb geometry remains difficult under severe occlusion, limited field of view, and fast motion.The paper suggests broader limb or upper-body reasoning as a possible way to provide stronger kinematic context.
Loading 2605.12498v1…