PaperlayerView exact revision on arXiv

Source-linked AI summary

HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videos

Jinglei Zhang, Jiankang Deng, Chao Ma, Rolandos Alexandros Potamias

arXiv:2501.02973v1cs.CV

TL;DR

Existing methods mainly reconstruct hands in camera coordinates, leaving world-space motion difficult to recover in egocentric videos with moving hands and cameras. HaWoR separates hand reconstruction from camera-trajectory estimation, adds motion infilling for missing views, and reports state-of-the-art performance across benchmark datasets. Its scope is limited by reliance on off-the-shelf hand tracking and independent per-hand modeling.

  • Problem

    Existing 3D hand-reconstruction methods predominantly estimate camera-space poses, overlooking world-space hand motion in egocentric videos where hands and cameras move continuously.

  • Method

    HaWoR decouples camera-space hand-motion reconstruction from world-space camera estimation and adds adaptive egocentric SLAM plus motion infilling for missing or occluded hands.

  • Results

    HaWoR achieves state-of-the-art performance for hand-motion reconstruction and world-frame camera-trajectory estimation across egocentric benchmark datasets.

  • Takeaways & Limitations

    HaWoR provides a unified approach for reconstructing high-fidelity hand motion in world coordinates from single, in-the-wild egocentric videos.

  • Takeaways & Limitations

    HaWoR relies on off-the-shelf hand tracking, so identity failures can propagate erroneous detections into reconstruction; independently modeled hands can also self-penetrate during interaction.

Abstract

from arXiv · show

Despite the advent in 3D hand pose estimation, current methods predominantly focus on single-image 3D hand reconstruction in the camera frame, overlooking the world-space motion of the hands. Such limitation prohibits their direct use in egocentric video settings, where hands and camera are continuously in motion. In this work, we propose HaWoR, a high-fidelity method for hand motion reconstruction in world coordinates from egocentric videos. We propose to decouple the task by reconstructing the hand motion in the camera space and estimating the camera trajectory in the world coordinate system. To achieve precise camera trajectory estimation, we propose an adaptive egocentric SLAM framework that addresses the shortcomings of traditional SLAM methods, providing robust performance under challenging camera dynamics. To ensure robust hand motion trajectories, even when the hands move out of view frustum, we devise a novel motion infiller network that effectively completes the missing frames of the sequence. Through extensive quantitative and qualitative evaluations, we demonstrate that HaWoR achieves state-of-the-art performance on both hand motion reconstruction and world-frame camera trajectory estimation under different egocentric benchmark datasets. Code and models are available on https://hawor-project.github.io/ .

1. Introduction

HaWoR addresses the gap between camera-space hand reconstruction and world-space hand trajectories in egocentric videos, where moving cameras, occlusions, and rapid motions complicate estimation. It decomposes the task into camera-space hand reconstruction, world-space camera estimation, and motion infilling.

  • Existing monocular hand-reconstruction methods primarily estimate camera-space poses, overlooking world-space hand trajectories needed to interpret human movement.
  • Egocentric reconstruction is difficult because hands and cameras move simultaneously, while occlusions, rapid motion, and interactions obscure true world-coordinate trajectories.
  • Earlier world-space methods rely on multi-view capture, visual odometry, or heavy optimization, limiting robustness beyond controlled environments or under complex human motion.
  • HaWoR decomposes the problem into camera-space hand reconstruction and world-space camera-trajectory estimation, augmenting hand reconstruction with a motion-infilling module.
  • HaWoR presents a world-coordinate hand-motion method with high-fidelity infilling and a robust egocentric camera-trajectory pipeline.

2. Related Work

Prior work spans monocular hand-pose reconstruction and global human-motion estimation, while HaWoR combines camera-frame hand modeling with world-space camera alignment for egocentric videos.

  • Hand-pose estimation progressed from depth-camera methods to monocular MANO regression and direct vertex prediction, with recent work emphasizing larger data and models.
  • HaWoR’s method overview combines a transformer-based camera-frame hand-motion module, a motion infiller, and adaptive egocentric SLAM aligned to world coordinates.
  • Global human-motion methods have used multi-camera setups, wearable sensors, trajectory regressors, or joint optimization of SLAM and human motion.

3. Method

HaWoR reconstructs hand motion in camera coordinates, estimates a robust world-space camera trajectory, and infills hands missing because of truncation or occlusion. Its modules use temporal motion priors, hand-aware SLAM processing, canonicalization, and transformer-based completion.

  • Method overview: The framework represents complete MANO hand motion in world coordinates using hand-motion estimation, camera-trajectory estimation, and motion-infilling modules.
  • Hand motion estimation: Egocentric hand estimation suffers from jitter, boundary truncation, occlusion, motion blur, and missing motion priors, motivating a spatio-temporal reconstruction network.
  • Hand motion estimation: The hand network extracts tracked image features and applies temporal image attention to improve feature robustness around truncated and occluded regions.
  • Hand motion estimation: Pose attention applies temporal self-attention to MANO parameters, learning motion priors that improve temporal coherence and constrain 3D reconstruction.
  • Camera trajectory estimation: Egocentric SLAM is challenged by hand-dominated feature matching and scale ambiguity, so HaWoR adapts SLAM to exclude hand regions from camera estimation.
  • Camera trajectory estimation: Adaptive sampling restricts scale estimation to reliable intermediate-depth, non-hand points, and robustly aligns SLAM with metric depth using German-McClure loss.
  • Hand motion infilling: The infiller completes out-of-view or occluded hand sequences by transforming camera-space motion into canonical space and predicting missing poses with a positional transformer encoder.

4. Experiments

HaWoR is evaluated on camera-frame and world-space hand reconstruction, camera trajectory estimation, qualitative trajectories, runtime, and component ablations. Across these evaluations, it remains robust under occlusion, challenging motion, and truncated visibility while outperforming the cited baselines.

  • Datasets and metrics: HOT3D provides egocentric videos with dynamic cameras, ground-truth camera trajectories, and world-coordinate MANO annotations, while DexYCB evaluates camera-frame reconstruction under occlusion.The evaluation uses PA-MPJPE, AUC, W-MPJPE, WA-MPJPE, RTE, and acceleration error.
  • Camera-frame 3D hand motion: HaWoR maintains robust camera-frame reconstruction across occlusion rates, with PA-MPJPE 5.07 versus WiLoR’s 5.68 at 75%-100% occlusion.The cited comparison reports stronger performance under severe occlusion than the referenced state-of-the-art methods.
  • Qualitative trajectories: HaWoR follows ground-truth hand trajectories during large, truncated, complex, and long-range movements where baseline trajectories fail.Figures 3 and 5 provide qualitative comparisons on challenging HOT3D examples and against HMP-SLAM.
  • Global camera trajectory: Adaptive SLAM produces camera trajectories matching ground-truth motion and improves world-scale estimation over DROID-SLAM in egocentric views.The cited evaluation discusses both aligned-scale ATE and estimated-scale ATE-S, with scale errors worsening for direct DROID-SLAM-based estimation.
  • World-space 3D hand motion: HaWoR significantly outperforms regression- and optimization-based baselines on both PA-MPJPE and W-MPJPE, while achieving lower acceleration error and stable reconstructions under occlusion.Lower acceleration error is reported as evidence of smoother reconstructed motions across frames.
  • Efficiency: HaWoR requires 40 ms per frame in one forward pass, reducing inference runtime by 75% relative to HMP-SLAM’s 160 ms per frame.The comparison is reported alongside superior camera trajectory estimation.
  • Ablation: The ablations show that image and pose motion priors improve reconstruction robustness, while the motion infiller significantly outperforms linear interpolation and last-visible-pose replication.The infiller evaluation uses invisible HOT3D validation sequences.

5. Conclusion and Limitations

HaWoR reconstructs 3D hand motion in world space using camera-frame hand estimation, motion infilling, and adaptive egocentric SLAM. It achieves state-of-the-art performance, but remains far from real-time.

  • HaWoR combines a camera-frame transformer, a motion infiller, and adaptive egocentric SLAM for world-space hand motion reconstruction.The infiller estimates motion between missing frames, while SLAM estimates the global camera trajectory under challenging views.
  • HaWoR achieves state-of-the-art performance across different benchmark datasets for hand motion reconstruction and camera trajectory estimation.
  • HaWoR accelerates hand motion reconstruction compared with previous approaches but remains far from real-time.The paper suggests future work on foundational models for direct world-space camera trajectory estimation.

6. Generalization on In-the-Wild Videos

On unseen EPIC-KITCHENS videos, HaWoR recovers camera and hand reconstructions consistent with the input, including sequences with hands outside the view frustum.

  • HaWoR generalizes to EPIC-KITCHENS videos despite not being trained on those in-the-wild data.The reconstructed hands and cameras remain consistent with the input videos.
  • HaWoR accurately reconstructs sequences containing frames where the hands are outside the camera view frustum.The method leverages motion priors and an infilling network for these cases.

7. Comparison on In-the-Wild Videos

On in-the-wild EPIC-KITCHENS sequences, HaWoR is qualitatively compared with HaMeR and WiLoR in the camera view. It performs better when hand truncation occurs at the image boundary.

  • HaWoR achieves significantly better results than HaMeR and WiLoR when hand truncation occurs at the boundary.

8. Implementation Details

The implementation uses multiple annotated hand-video datasets, canonical-space normalization, motion infilling, and world-space evaluation with hand, trajectory, smoothness, and motion-quality metrics.

  • Hand Motion Estimation Network: The hand motion estimator is trained on four datasets totaling 1M frames with 3D and 2D hand annotations.
  • Hand Motion Estimation Network: Training data include HOT3D, ARCTIC, DexYCB, and HO3D, covering egocentric activity, manipulation, grasping, and hand-object interaction.The datasets contribute 573K, 165K, 169K, and 66K frames, respectively.
  • Hand Motion Estimation Network: The estimator is trained with AdamW for 250K iterations while freezing the pretrained WiLoR ViT backbone.Training uses four NVIDIA A800 GPUs and a total batch size of 32.
  • Hand Motion Infiller: Canonical-space transformation aligns each sequence’s first frame to zero translation and zero rotation.This standardizes the input sequence for training.
  • Hand Motion Infiller: The canonical transformation uses camera-to-world rotation and translation together with the first frame’s hand rotation and translation.
  • Hand Motion Infiller: The infiller is trained on HOT3D sequences by masking continuous middle frames while retaining the start and end frames as context.Non-visible hand segments and randomly masked sequences provide training examples.
  • Evaluation Details: HOT3D provides egocentric videos with moving camera trajectories and world-coordinate MANO annotations, including frames where hands leave the camera frustum.The evaluation uses 110 training videos and 27 validation videos.
  • Evaluation Details: Evaluation uses camera-frame pose metrics, world-frame MPJPE metrics, trajectory error, acceleration error, FID, and camera trajectory errors.These include PA-MPJPE, AUC, W-MPJPE, WA-MPJPE, RTE, Accel, FID, ATE, and ATE-S.

9. Limitations

HaWoR has two stated limitations: dependence on off-the-shelf hand tracking and independent hand modeling without inter-penetration constraints.

  • Off-the-shelf hand-tracking errors, especially identity failures, can propagate into HaWoR and cause reconstruction errors.The paper illustrates incorrect left/right hand tracking as one such failure case.
  • Independent modeling of each hand without inter-penetration constraints can cause self-penetrations during two-hand interactions.The authors identify hand interaction modeling and penetration constraints as future directions.
Loading 2501.02973v1…