Source-linked AI summary
Hand Visibility Detector: Per-Keypoint Visibility Estimation for Hands
Ryosei Hara, Masashi Hatano, Rintaro Yanagi, Atsushi Hashimoto, Takuma Yagi, Mariko Isogawa
TL;DR
Most hand pose estimators do not explicitly assess whether individual joints are visible, although visibility matters under occlusion and has mainly served as an auxiliary signal. Hand Visibility Detector makes per-joint visibility a standalone task using a lightweight head on a frozen pretrained HPE backbone, achieving stronger visibility estimation and reducing reprojection error through visibility-weighted triangulation.
Problem
Most HPE methods output joint positions without explicitly indicating joint visibility, and prior visibility estimates have mainly been evaluated indirectly through pose accuracy.
Method
Hand Visibility Detector attaches a lightweight visibility head to a frozen backbone of a large-scale pretrained HPE model and predicts per-joint visibility.
Results
The method outperforms baselines by 3.4 points in mAP and reduces mean reprojection error by up to 10.1% through visibility-weighted triangulation.
Takeaways & Limitations
Per-joint visibility estimation can be evaluated independently and used to improve multi-view 3D hand pose annotation.
Takeaways & Limitations
The method relies on pretrained HPE models to provide hand-structure prior knowledge and currently targets image input rather than temporally consistent video visibility.
Abstract
from arXiv · showhide
Hand Pose Estimation (HPE) is a fundamental technology for various applications such as AR/VR and robotics. In these applications, the visibility of each hand joint in the image is crucial for assessing the reliability of estimation results under occlusion. However, most existing HPE methods output joint positions without explicitly indicating their visibility. Although some methods account for occlusion or visibility, visibility estimation has mainly been used as an auxiliary signal for improving pose estimation. To our knowledge, per-joint hand visibility estimation has not been systematically studied as a standalone task. In this work, we propose Hand Visibility Detector, a model for estimating the visibility of individual hand joints, and present the first systematic investigation of visibility estimation as an independent task. We show that leveraging the prior knowledge of HPE models pretrained on large-scale data as a backbone yields high performance in this task. We further demonstrate the utility of Hand Visibility Detector on a downstream task of 3D hand pose annotation via multi-view triangulation of 2D keypoints, showing that visibility-weighted triangulation reduces reprojection error. Our method is released as a ready-to-use package, and the code and demo are available at https://github.com/ryhara/hand_visibility_detector .
1. Introduction
Hand Visibility Detector treats per-joint hand visibility as a standalone estimation task rather than merely an auxiliary signal for pose estimation. It uses a pretrained HPE backbone and shows benefits for visibility estimation and visibility-weighted multi-view triangulation.
- Most HPE methods output joint positions without explicitly indicating whether each joint is directly observable or occluded.
- Visibility has typically been evaluated indirectly through downstream pose accuracy rather than as an independent estimation target.
- Hand Visibility Detector attaches a lightweight visibility head to a frozen, large-scale pretrained HPE backbone.
- Visibility-weighted triangulation reduces reprojection errors across the DexYCB, HO3D, and H2O multi-view datasets.
- 3.4 points in mAP and up to 10.1% lower mean reprojection error are reported for visibility estimation and downstream triangulation, respectively.
2. Related Work
Prior work estimates visibility mainly within hand pose estimation or related occlusion-aware systems, while independent visibility performance and generalization remain underexamined. Multi-view triangulation also remains sensitive to unreliable 2D keypoints.
- Earlier hand pose methods modeled per-joint visibility as part of pose estimation, using it to guide heatmap enhancement or represent occluded-joint uncertainty.
- Other approaches estimate finger-level occlusion or recover occluded appearances rather than performing per-joint visibility estimation.
- Prior visibility estimators were trained on controlled datasets, leaving their generalization as visibility estimators unvalidated.
- Standard multi-view triangulation treats views equally, so remaining low-confidence 2D keypoints can affect 3D annotation accuracy.
3. Method
The method predicts visibility for the 21 MANO hand joints using a frozen pretrained HPE encoder and a lightweight visibility head. The head converts spatial features into per-joint visibility probabilities and is trained with binary cross-entropy.
- Given a cropped hand image, the model outputs 21 visibility probabilities corresponding to the MANO hand keypoints.
- Hand Encoder: A frozen pretrained ViT backbone extracts spatial features, preserving learned hand-structure knowledge while avoiding overfitting to limited visibility labels.
- Visibility Head: The lightweight visibility head compresses feature maps, models global spatial dependencies with a Gated Attention Unit, and produces per-joint logits.
- Visibility Head: A sigmoid converts the per-joint logits into visibility probabilities.
- Visibility Head: Binary cross-entropy trains the head against joint visibility labels, with out-of-frame joints treated as invisible.
4. Experiments
Experiments evaluate visibility estimation on diverse in-the-wild HInt data and compare the proposed model with existing visibility estimators and alternative backbones. The supplied passages establish the evaluation dataset and shared experimental setup, while reported comparisons show pretrained hand-specific backbones are effective.
- Dataset: HInt provides manual per-joint visibility labels across diverse web images and egocentric video frames, with 25,273 training and 5,374 evaluation frames.
- Baselines: The evaluation compares against visibility estimators based on methods by Kim et al. and Contact4D.
- Experimental Setup: Only the visibility head is trained, while the backbone remains frozen across the backbone ablation experiments.
- Experimental Setup: The visibility head contains 0.83M parameters, representing 0.131% of the 631M-parameter model.
5. Results
The proposed visibility estimator outperforms baselines using pretrained hand-specific HPE features, while visibility-weighted triangulation reduces reprojection errors across three datasets.
- Visibility estimation: 0.931 mAP and 0.896 F1 substantially outperform Kim et al. and Contact4D on both visibility metrics.Both baselines use CNN features pretrained on ImageNet-1k, whereas the proposed method leverages pretrained hand priors.
- Backbone choice: Fine-tuning the WiLoR backbone degrades mAP from 0.931 to 0.622, supporting a frozen-backbone design.The authors attribute this degradation to corruption of pretrained feature representations.
- Visibility head: Removing the GAU degrades mAP from 0.931 to 0.887, while a linear visibility head also underperforms the proposed head.The GAU models global dependencies among spatial positions.
- Threshold sensitivity: F1 peaks around a binarization threshold of 0.5 and remains above 0.88 from 0.3 to 0.7.Performance is relatively insensitive to threshold selection across this range.
- Qualitative evaluation: Qualitative results cover self-occlusion, image truncation, and object occlusion, with the proposed method achieving the highest-confidence visibility estimates in all cases.Green marks predicted-visible joints and red marks predicted-invisible joints at ground-truth joint positions.
- Downstream triangulation: Visibility-weighted triangulation achieves lower median, mean, and interquartile-range reprojection errors than both comparison schemes on all three datasets.The largest improvement occurs on HO3D, where mean reprojection error decreases by 10.1%.
6. Conclusion
The paper introduces Hand Visibility Detector as a dedicated per-joint visibility model using pretrained HPE knowledge. It also shows that visibility weighting improves multi-view triangulation across datasets.
- Contribution: Hand Visibility Detector is presented as the first model dedicated to hand joint visibility estimation.The work frames visibility estimation as an independent task rather than only an auxiliary pose-estimation signal.
- Visibility estimation: The method substantially outperforms existing methods and general-purpose backbones on the HInt dataset.Its design leverages the prior knowledge of large-scale pretrained HPE models.
- Downstream utility: Visibility-weighted triangulation reduces reprojection errors on multiple datasets.The downstream experiment demonstrates utility for multi-view 3D hand pose annotation.
- Future work: Future work includes extending the method to video input for temporally consistent visibility estimation.The authors also release the code as a ready-to-use package.