Source-linked AI summary
Reconstructing Hands in 3D with Transformers
Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, Jitendra Malik
TL;DR
HaMeR addresses the need to perceive hands in 3D from monocular visual input by using a fully transformer-based hand mesh recovery approach. Scaling training data and model capacity yields improvements over previous work on standard benchmarks and challenging in-the-wild images, supported by the HInt annotation dataset.
Problem
Perceiving hands in 3D from visual input remains necessary for interpreting hand interactions occurring in 3D, while benchmark progress is often constrained by controlled-condition datasets.
Method
HaMeR combines large-scale training data with a high-capacity fully transformer-based architecture using a ViT-H backbone and transformer head for hand and camera parameter regression.
Results
HaMeR achieves state-of-the-art results, including a 2-3× improvement in PCK@0.05 on in-the-wild datasets compared to previous works, and outperforms previous baselines on HInt.
Takeaways & Limitations
HInt complements traditional 3D hand pose benchmarks with diverse in-the-wild 2D keypoint and visibility annotations, enabling a more holistic evaluation of hand reconstruction approaches.
Abstract
from arXiv · showhide
We present an approach that can reconstruct hands in 3D from monocular input. Our approach for Hand Mesh Recovery, HaMeR, follows a fully transformer-based architecture and can analyze hands with significantly increased accuracy and robustness compared to previous work. The key to HaMeR's success lies in scaling up both the data used for training and the capacity of the deep network for hand reconstruction. For training data, we combine multiple datasets that contain 2D or 3D hand annotations. For the deep model, we use a large scale Vision Transformer architecture. Our final model consistently outperforms the previous baselines on popular 3D hand pose benchmarks. To further evaluate the effect of our design in non-controlled settings, we annotate existing in-the-wild datasets with 2D hand keypoint annotations. On this newly collected dataset of annotations, HInt, we demonstrate significant improvements over existing baselines. We make our code, data and models available on the project website: https://geopavlakos.github.io/hamer/.
1. Introduction
HaMeR applies scaling in training data and model capacity to monocular 3D hand reconstruction, with strong performance on controlled benchmarks and challenging in-the-wild images. The authors also introduce HInt to complement controlled evaluation with diverse 2D hand annotations.
- HaMeR reconstructs 3D hand meshes from images and video frames using scaled training data and a high-capacity deep architecture.The approach targets varied poses, viewpoints, and visual conditions.
- HaMeR improves over existing baselines on standard 3D hand pose benchmarks and performs especially strongly on challenging in-the-wild images.
- HInt provides 2D keypoint and visibility annotations for 40.4K hands, with 86.7% showing natural contact.The dataset draws from diverse sources, including YouTube and egocentric video.
- HInt complements existing benchmarks because it evaluates the 2D aspect of 3D reconstruction on more diverse data.
- 2-3× improvement in PCK@0.05 is reported on in-the-wild datasets compared to previous works.
2. Related work
Prior work estimates 3D hand pose and shape with parametric or non-parametric methods, while evaluation has largely relied on controlled datasets. HInt extends this landscape with diverse in-the-wild 2D annotations, including egocentric and third-person video sources.
- Earlier methods regress MANO pose and shape parameters, while later non-parametric methods directly regress MANO mesh vertices.
- Direct vertex regression can align better with image evidence but is more prone to failure under occlusions and truncations.
- Related work also addresses joint hand-pose and object reconstruction, two interacting hands, and high-speed inference.
- Existing 3D datasets support training and evaluation, but the authors argue that in-the-wild images are needed for a more holistic assessment of accuracy and robustness.
- HInt sources egocentric and third-person video imagery to broaden 2D hand-keypoint evaluation beyond datasets with few egocentric or transitionary moments.
3. Technical approach
HaMeR uses a fully transformer-based architecture to regress MANO hand and camera parameters from RGB images, combining 2D and 3D supervision. Training consolidates large multi-source annotation sets, including 2D in-the-wild data.
- Architecture and data: HaMeR follows a simple, fully transformerized design that scales both training data and deep model architecture.
- Hand representation: MANO maps pose parameters θ and shape parameters β to a hand mesh with 778 vertices and 21 joints.
- Hand representation: The regressor estimates MANO pose, shape, and camera parameters, enabling projection of reconstructed 3D joints into the image.
- Architecture and data: A ViT-H backbone processes image patches, and a transformer decoder cross-attends to backbone tokens to regress hand and camera parameters.
- Training losses: 2D reprojection loss aligns projected 3D joints with ground-truth 2D keypoints and is also applied when 3D ground truth is available.The authors state that this promotes consistency in image space.
- Training losses: Adversarial losses on hand shape, pose, and individual joint angles encourage natural reconstructions when only 2D keypoints are available.
- Architecture and data: The consolidated training data contains 2.7M examples, four times the training set of FrankMocap, with 5% from in-the-wild datasets.
4. HInt: Hand Interactions in the wild
HInt complements existing hand datasets with in-the-wild 2D keypoint and occlusion annotations drawn from video sources and natural interactions. It contains 40.4K annotated hands, with documented annotation consistency.
- Dataset construction: HInt annotates 2D hand keypoints and occlusion labels using Hands23, Epic-Kitchens, and Ego4D video datasets.Its sources include New Days, VISOR, and Ego4D critical frames.
- Dataset construction: Video sourcing provides transitional moments and natural poses, while training samples emphasize hand-object or hand-hand interactions in New Days and VISOR.Ego4D samples are randomly selected because its critical frames already typically focus on interactions.
- Annotation protocol: HInt annotations include keypoint existence and occlusion labels, enabling more fine-grained analysis of pose estimation systems.Existence indicates whether a keypoint lies within the image; occlusion indicates whether it is occluded.
- Dataset scale: 40.4K hands are annotated: 12.0K from New Days, 5.3K from VISOR, and 23.2K from Ego4D.The Ego4D subset includes 9.3K hands from sequences, supporting future temporal-task evaluation.
- Annotation quality: 90.5% of occlusion labels and 100% of existence labels are consistent across duplicated annotations.94.6% of visible keypoints have offset distance within 0.25× of palm length.
5. Experiments
HaMeR is evaluated on controlled 3D benchmarks and the in-the-wild HInt benchmark, alongside ablations and qualitative comparisons. It achieves state-of-the-art performance, with larger gains on challenging in-the-wild data and improvements associated with scaling data and model capacity.
- 3D pose accuracy: HaMeR achieves state-of-the-art results and consistently outperforms prior methods across most metrics on FreiHAND and HO3Dv2.These controlled multi-camera benchmarks provide 3D mesh ground truth and evaluate 3D joint and mesh accuracy.
- 2D pose accuracy: HaMeR outperforms previous baselines on HInt, with much larger improvements than on FreiHAND and HO3D.HInt evaluates PCK at multiple thresholds across New Days, VISOR, and Ego4D, including visible and occluded joints.
- 2D pose accuracy: In-the-wild evaluation better represents robustness across visual conditions, viewpoints, and interactions such as contacts with surrounding objects.Controlled benchmarks are more saturated and therefore show smaller improvement margins.
- Ablation analysis: Combining large-scale training data with a high-capacity architecture outperforms versions that increase either data volume or model scale alone.The ablation compares models on HInt and identifies the combined design as the strongest version.
- Ablation analysis: Training with HInt improves performance particularly on the egocentric VISOR and Ego4D datasets, while gains on third-person New Days are smaller but consistent.The comparison uses PCK scores on the HInt test set.
- Qualitative results: Qualitative results show robustness to viewpoints, skin tones, gloves, and varying occlusion from interacting objects.Comparisons with METRO, MeshGraphormer, and FrankMocap indicate greater robustness and precision for HaMeR.
6. Conclusion
HaMeR scales both training data and model architecture for monocular 3D hand mesh reconstruction, outperforming previous work on benchmarks and challenging in-the-wild images. Its high-capacity design improves robustness across occlusion, interactions, viewpoints, and varied hand appearances.
- Combining 4× more training data with ViT-H obtains the best HInt results by large margins.The ablation compares a ResNet50 base model, increased data, ViT-H, and their combination.
- Qualitative comparisons show HaMeR consistently outperforming METRO, Mesh Graphormer, and FrankMocap, especially in motion blur and hand-interaction cases.METRO and Mesh Graphormer regress MANO vertices, whereas FrankMocap and HaMeR regress MANO parameters.
- HaMeR is particularly robust to heavy occlusion and interactions with objects or other hands in diverse in-the-wild images.Qualitative HInt results include New Days, VISOR, Ego4D, and Internet images.
- HaMeR outperforms previous work on traditional 3D hand pose benchmarks and achieves even bigger improvements on HInt.The HInt evaluation uses diverse 2D keypoint annotations, including egocentric and YouTube imagery.