PaperlayerView exact revision on arXiv

Source-linked AI summary

WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild

Rolandos Alexandros Potamias, Jinglei Zhang, Jiankang Deng, Stefanos Zafeiriou

arXiv:2409.12259v2cs.CV

TL;DR

WiLoR addresses the limited hand-detection support for real-world multi-hand 3D reconstruction. It combines a real-time detector trained on a large in-the-wild dataset with transformer-based high-fidelity reconstruction, and outperforms prior methods across detection and reconstruction benchmarks. The method also achieves robust temporal coherence without a temporal module, while remaining limited on extreme poses and crowded scenes.

  • Problem

    Existing 3D hand pose methods and datasets provide limited support for detecting and reconstructing multiple hands in diverse in-the-wild conditions.

  • Method

    WiLoR combines a lightweight real-time hand detector trained on a large-scale in-the-wild dataset with a transformer-based 3D reconstruction model using a refinement module.

  • Results

    WiLoR outperforms previous state-of-the-art methods on hand detection and the FreiHAND and HO3D reconstruction benchmarks, with the detector reaching over 130 FPS and improving average mAP by 26%.

  • Takeaways & Limitations

    WiLoR provides a comprehensive solution for multi-hand detection, localization, and 3D reconstruction, including temporally coherent monocular-video tracking without a temporal module.

  • Takeaways & Limitations

    WiLoR can fail on extreme finger poses and crowded environments, and bottom-up reconstruction may not adequately capture interactions and contacts between hands.

Abstract

from arXiv · show

In recent years, 3D hand pose estimation methods have garnered significant attention due to their extensive applications in human-computer interaction, virtual reality, and robotics. In contrast, there has been a notable gap in hand detection pipelines, posing significant challenges in constructing effective real-world multi-hand reconstruction systems. In this work, we present a data-driven pipeline for efficient multi-hand reconstruction in the wild. The proposed pipeline is composed of two components: a real-time fully convolutional hand localization and a high-fidelity transformer-based 3D hand reconstruction model. To tackle the limitations of previous methods and build a robust and stable detection network, we introduce a large-scale dataset with over than 2M in-the-wild hand images with diverse lighting, illumination, and occlusion conditions. Our approach outperforms previous methods in both efficiency and accuracy on popular 2D and 3D benchmarks. Finally, we showcase the effectiveness of our pipeline to achieve smooth 3D hand tracking from monocular videos, without utilizing any temporal components. Code, models, and dataset are available https://rolpotamias.github.io/WiLoR.

1. Introduction

WiLoR addresses the gap between increasingly capable 3D hand reconstruction and limited hand detection for diverse, multi-hand in-the-wild images. It combines real-time detection with high-fidelity reconstruction and introduces a large-scale dataset to support this setting.

  • Hand pose estimation is difficult because hand appearance and articulation vary widely, with frequent occlusions and motion blur.
  • Existing 3D pose methods primarily process images containing a fixed number of hands, limiting generalization to in-the-wild images.
  • Hand detectors are essential because 3D hand frameworks operate on tight hand crops, yet existing methods struggle with multiple hands, challenging poses, or real-time operation.
  • WiLoR proposes a robust single-stage anchor-free detector operating at over 100 fps.
  • The reconstruction method predicts MANO parameters rather than directly regressing vertices, supporting explainable and plausible hand poses.
  • The pipeline combines a large-scale in-the-wild dataset, a real-time detector, and a transformer-based reconstruction method with a novel refinement module.

3. WHIM Dataset

WHIM addresses limited multi-hand in-the-wild annotations by collecting diverse YouTube videos and automatically annotating hands. The pipeline combines detector-based localization, landmark fitting, and biomechanical and pose priors for hand reconstruction.

  • WHIM contains millions of in-the-wild hands spanning diverse poses, illuminations, occlusions, and skin tones.
  • The collection pipeline processes more than 1,400 YouTube videos covering sign language, cooking, everyday activities, sports, and games.The videos include ego- and exo-centric viewpoints, motion blur, varying hand scales, and interactions.
  • Hand localization combines bounding-box predictions from three detectors using confidence-weighted averaging.The estimated bounding box is denoted by b_i.
  • The reconstruction fits shape and pose parameters of a 3D parametric hand model by minimizing landmark re-projection loss.The projection uses a weak-perspective transform and an estimated intrinsic camera matrix.
  • Biomechanical losses constrain bone lengths and joint angles, while a PCA prior models feasible hand-pose distributions.The biomechanical loss sums bone-length and joint-angle terms; the PCA prior penalizes mesh reconstruction error from the learned pose space.

4. Method

The method combines real-time fully convolutional hand localization with transformer-based 3D reconstruction. Detection uses multi-scale features and multiple prediction heads, while reconstruction refines a coarse MANO estimate using image-aligned features.

  • Hand Detection and Localization: The detector uses a one-stage fully convolutional architecture to localize hands and predict their side labels, bounding boxes, and joints.It extracts multi-scale backbone features and processes them with a PANet neck.
  • Hand Detection and Localization: The detector is trained with a multi-task loss combining classification, distributional focal, bounding-box overlap, and keypoint terms.The loss weights balance these four components.
  • Hand Reconstruction: The reconstruction model takes a tightly cropped hand image and estimates MANO pose, shape, and camera parameters.The predicted MANO parameters and camera parameters define the reconstructed 3D hand.
  • Hand Reconstruction: A ViT encoder uses image tokens plus learnable pose, shape, and camera tokens to produce a coarse MANO and camera estimate.The coarse estimate provides a prior for the subsequent refinement network.
  • Hand Reconstruction: The refinement module projects the coarse hand onto multi-resolution feature maps, samples image-aligned vertex features, and predicts pose and shape residuals.Vertex features are aggregated with functions such as mean, max, or sum before residual prediction.
  • Hand Reconstruction: The reconstruction model is supervised using 3D vertices, 2D joints, MANO parameters, and an adversarial loss when the corresponding annotations are available.The losses constrain 3D reconstruction, 2D reprojection, MANO parameters, and parameter plausibility.

5. Experiments

Experiments evaluate WiLoR's hand detector, 3D pose estimator, and frame-wise dynamic reconstruction. Across detection and reconstruction benchmarks, the method combines high speed, accuracy, robustness, and temporal coherence without temporal modules.

  • Evaluation scope: The experiments assess hand detection, 3D pose estimation, and dynamic reconstruction using established benchmarks and qualitative analyses.The evaluation includes COCO-WholeBody, OxfordHands, FreiHAND, HO3D, and frame-wise reconstruction tests.
  • Hand detection: More than 130 FPS and a 26% average mAP improvement over prior state-of-the-art models characterize the medium detector's benchmark performance.The small version reaches up to 175 FPS; the detector is also 45× faster and 32× smaller than ContactHands.
  • Hand detection: Data augmentation improves cross-dataset detection mAP by 14%, while landmark regression loss produces more precise and robust detections.Reducing WHIM training data causes a significant performance drop, and WHIM training outperforms OxfordHands training in the reported comparison.
  • 3D hand pose estimation: WiLoR outperforms previous methods under all reported metrics on FreiHAND and HO3Dv2, including aligned joint and mesh accuracy measures.The reconstruction model uses image-aligned refinement features and remains effective under challenging articulations.
  • 3D hand pose estimation: Multi-scale refinement and large-scale training are supported by ablations showing that single-scale refinement and reduced training sets deteriorate reconstruction performance.The compared variants include training only on FreiHAND or on the datasets used by prior work.
  • Dynamic reconstruction: WiLoR outperforms baseline methods in temporal coherence on HO3D without a temporal module, using frame-wise reconstruction evaluated by joint jerk and wrist root translation error.The reported evaluation measures jitter in 3D joint motion and displacement of the wrist across frames.

6. Conclusion

WiLoR is presented as a full-stack framework for multi-hand detection, localization, and 3D reconstruction. It combines a lightweight detector trained on more than 2M in-the-wild images with a high-fidelity pose estimator and achieves robust benchmark performance and over 130 FPS detection.

  • Conclusion: WiLoR is a full-stack hand detection and 3D pose estimation framework for comprehensive multi-hand detection, localization, and reconstruction.The framework is described as the first full-stack system of this kind.
  • Conclusion: The lightweight detector reaches over 130 FPS while robustly detecting hands under varied occlusions and illuminations.It is trained using a large-scale in-the-wild dataset.
  • Conclusion: The high-fidelity 3D pose estimator uses a refinement module to mitigate alignment issues reported for previous methods.The conclusion reports robust performance on challenging cases.
  • Conclusion: WiLoR outperforms previous state-of-the-art methods on two benchmark datasets and delivers robust performance on challenging cases.The conclusion presents this as evidence for the framework's comprehensive reconstruction capability.

7. Implementation Details

Implementation uses large-scale detector training and a transformer-based hand pose estimator. The detector is trained on WHIM, while the pose model uses a ViT-Large backbone with dedicated pose, shape, and camera tokens.

  • Detector training: The hand detector is trained on WHIM, which contains over 2M in-the-wild images from daily activities.Training uses Adam for 200 epochs with early stopping, a batch size of 256, and two NVIDIA RTX 4090 GPUs for three weeks.
  • Detector training: Detector optimization starts at a learning rate of 0.01 and decreases linearly to 1e−6 during the final 30 training epochs.The training schedule includes early stopping after more than 30 epochs without loss decrease.
  • Pose estimation model: The pose estimator uses a ViT-Large backbone initialized from pre-trained ViTPose weights with a hidden dimension of 1280.Three learnable tokens represent hand pose, shape, and camera translation and scale.
  • Pose estimation model: The model maps output tokens through fully connected layers toward MANO pose parameters representing joint rotations.The implementation initializes the tokens from mean pose, shape, and camera parameters in the training set.

8. Comparison with existing datasets

WHIM is introduced as a substantially larger multi-hand dataset for in-the-wild detection. It contains diverse hand appearances and conditions and is reported as 100× larger than previous in-the-wild multi-hand datasets.

  • WHIM dataset: WHIM contains images with multiple hands under varied occlusions, views, illuminations, and skin tones.These properties address requirements for training a powerful hand detector rather than a tightly cropped hand regressor.
  • Dataset scale: WHIM is 100× larger than previous in-the-wild multi-hand datasets.The comparison is presented as part of the existing-dataset analysis.

9. Limitations

WiLoR achieves state-of-the-art performance but remains vulnerable to challenging poses, crowded scenes, hand interactions, and camera-space limitations.

  • WiLoR still fails to recover challenging cases despite state-of-the-art performance on 3D hand pose estimation and hand detection.
  • Training data remain limited to common hand poses and appearances, reducing generalization to samples far from the training distribution.
  • WiLoR can fail under extreme finger poses and may miss hands in crowded environments.
  • Bottom-up reconstruction may inadequately capture interactions and contacts between hands in 3D space.
  • Camera-space pose estimation may produce inaccurate assumptions about the overall 3D scene.

10. Training Datasets

WiLoR trains its hand-pose estimator on 14 datasets totaling 4.2M images, combining controlled, interaction, in-the-wild, and synthetic data. These sources provide 2D and 3D annotations and include conditions intended to improve generalization and occlusion handling.

  • WiLoR can fail on complex finger poses or when detecting small hands in crowded environments.
  • The training mixture contains 14 datasets with 2D and 3D annotations, totaling 4.2M images across three major categories.The categories include controlled-environment hand images, hand-object interaction, and in-the-wild and synthetic datasets.
  • Interaction datasets include HO3D, H2O3D, and DEX YCB, with H2O3D introducing increased hand occlusion through two-hand object interactions.These datasets provide multi-view imagery and 2D and 3D hand annotations where specified.
  • Synthetic datasets provide accurate 2D and 3D ground truth under varied illumination and environments.RHD, Re:InterHand, and BEDLAM are included among the synthetic sources described.

11. Temporal Coherence

WiLoR produces temporally coherent reconstructions from monocular videos despite being trained on single images. Demonstrations include challenging activities such as kneading dough and playing guitar.

  • WiLoR provides smooth reconstructions from monocular videos without temporal components.
  • The demonstrated scenarios include kneading dough and playing guitar.
  • The reported temporal coherence is attributed to stable and robust detections from the single-image model.
Loading 2409.12259v2…