Source-linked AI summary
HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Network
JoonKyu Park, Yeonguk Oh, Gyeongsik Moon, Hongsuk Choi, Kyoung Mu Lee
TL;DR
Severe hand-object occlusions and depth ambiguity make complete 3D hand mesh estimation difficult, while prior spatial-attention approaches can disregard occluded-region information. HandOccNet injects correlated hand features into those regions with FIT and refines them using SET, achieving state-of-the-art performance on severely occluded 3D hand mesh benchmarks. The method is evaluated on realistic hand-object interaction datasets, although the supplied limitation evidence notes that related augmentation approaches use synthetic occlusions.
Problem
Severe object occlusions and depth ambiguity make robust 3D hand mesh estimation difficult, and spatial attention can disregard information from occluded regions.
Method
HandOccNet injects correlated primary hand features into occluded regions with FIT and refines the resulting feature map using SET.
Results
HandOccNet achieves state-of-the-art performance on 3D hand mesh benchmarks containing severe occlusions and significantly outperforms prior methods on hand-object interaction datasets.
Takeaways & Limitations
Using occluded regions as secondary information provides a richer feature representation for occlusion-robust 3D hand mesh estimation.
Takeaways & Limitations
Related occlusion-aware augmentation methods rely on synthetic occlusions, which limits how directly their training conditions reflect realistic occlusions.
Abstract
from arXiv · showhide
Hands are often severely occluded by objects, which makes 3D hand mesh estimation challenging. Previous works often have disregarded information at occluded regions. However, we argue that occluded regions have strong correlations with hands so that they can provide highly beneficial information for complete 3D hand mesh estimation. Thus, in this work, we propose a novel 3D hand mesh estimation network HandOccNet, that can fully exploits the information at occluded regions as a secondary means to enhance image features and make it much richer. To this end, we design two successive Transformer-based modules, called feature injecting transformer (FIT) and self- enhancing transformer (SET). FIT injects hand information into occluded region by considering their correlation. SET refines the output of FIT by using a self-attention mechanism. By injecting the hand information to the occluded region, our HandOccNet reaches the state-of-the-art performance on 3D hand mesh benchmarks that contain challenging hand-object occlusions. The codes are available in: https://github.com/namepllet/HandOccNet.
1. Introduction
Occlusion-robust 3D hand mesh estimation remains challenging because spatial attention can suppress occluded-region information, while hand-object occlusions and depth ambiguity make complete 3D recovery difficult. HandOccNet addresses this by injecting correlated hand features into occluded regions and refining them with Transformer modules, achieving significantly better accuracy than prior methods.
- Motivation: Spatial attention can attenuate features in occluded regions, leaving sparse information for 3D hand mesh estimation.Its validity for occlusion-robust 3D hand mesh estimation is not established, especially under severe object occlusions and depth ambiguity.
- Proposed approach: HandOccNet treats occluded-region features as secondary information and injects correlated primary hand features into those locations to obtain richer representations.Primary features correspond to high-attention regions, while secondary features correspond to low-attention regions.
- Proposed approach: FIT combines sigmoid-based and softmax-based attention maps and removes the query residual connection so secondary features can be replaced by primary-feature values.The sigmoid component helps avoid undesirable high correlations when all key-query products are small.
- Results: HandOccNet achieves significantly better 3D hand mesh estimation accuracy than previous state-of-the-art methods on challenging hand-object interaction datasets.The experiments use datasets including HO-3D and FPHA, which contain realistic hand-region occlusions.
- Proposed approach: FIT uses correlations between primary and secondary features to perform feature injection, while SET refines the resulting feature map with self-attention.The modules are successive Transformer-based components designed for feature injection and refinement.
2. Related works
Prior occlusion-robust human pose methods use augmentation, temporal information, or spatial attention, but these approaches have limitations for severely occluded 3D hand mesh estimation. HandOccNet instead introduces feature injection to produce richer features in hand-object interaction scenarios.
- Occlusion-robust human pose estimation: Occlusion-robust human pose estimation commonly uses occlusion-aware data augmentation, temporal information, or spatial attention.These approaches address missing visual information through synthetic masking, video context, or spatial feature reweighting.
- Occlusion-robust human pose estimation: Occlusion-aware augmentation methods can rely on synthetic occlusions created by masking or replacing image regions during training.Such synthetic occlusions may not match realistic occlusions encountered in hand-object interaction.
- Spatial attention: Spatial-attention methods developed mainly for 2D human pose estimation filter or reweight occluded-region features.Their applicability is less direct to 3D hand mesh estimation, which involves depth ambiguity and severe object occlusions.
- Hand-object interaction: HandOccNet differs by introducing feature injection for occlusion-robust 3D hand mesh estimation under hand-object interaction.The mechanism is intended to produce highly rich features even when hands are severely occluded.
- Transformers: Transformers have been applied to modeling global or spatial dependencies in 3D human pose, shape, and mesh estimation.HandOccNet builds on this broader use of Transformers while targeting occluded 3D hand mesh estimation.
3. HandOccNet
HandOccNet uses occluded-region features as a secondary source of hand information, then enhances the resulting representation with two Transformer-based modules before regressing MANO parameters and producing a 3D mesh.
- Overall architecture: The backbone extracts primary hand features FP and secondary occluded-region features FS, which are processed by FIT, SET, and a regressor.The final mesh is obtained by forwarding predicted MANO parameters through the MANO layer.
- Feature injecting transformer (FIT): FIT injects primary-feature information into secondary features by modeling correlations between queries from FS and keys from FP.This uses secondary features to identify where primary hand information should be inserted.
- Feature injecting transformer (FIT): FIT combines softmax-based and sigmoid-based attention to identify relevant correlations while filtering undesired high scores caused by weak overall key relationships.The sigmoid branch uses pooled global key correlations, which the authors report makes the map robust to noisy correlations.
- Feature injecting transformer (FIT): The injected residual feature is passed through a feed-forward module, then combined with the primary feature through residual connections to form FIT output FFIT.The output is reshaped to R32×32×256.
- Self-enhancing transformer (SET): SET applies self-attention to FFIT, using queries, keys, and values from the same feature to refine distant information before regression.Because each query can correlate with itself, SET does not face the case where all key information is unrelated to a query pixel.
- Regression and training: The regressor converts the enhanced feature into MANO pose and shape parameters, from which the final 3D hand mesh is generated.Its pipeline includes an hourglass block for joint heatmaps and residual blocks operating on the enhanced feature and heatmaps.
4. Experiments
Experiments evaluate HandOccNet on challenging hand-object interaction datasets, compare FIT and SET variants, and show that the proposed method outperforms prior state-of-the-art approaches.
- Datasets and evaluation metrics: HO-3D and FPHA provide challenging hand-object interaction data with severe or diverse occlusions for evaluation.HO-3D includes RGB images with MANO-based annotations, while FPHA contains egocentric RGB-D videos and 3D hand-pose annotations.
- Datasets and evaluation metrics: The evaluation reports mean joint error, mesh error, and F-scores on HO-3D, plus mean joint error on FPHA, after Procrustes alignment.Joint error before Procrustes alignment is additionally reported for HO-3D in supplementary material.
- Ablation study: FIT and SET consistently improve all metrics and improve 3D hand-mesh accuracy when severe occlusions appear in the input.The ablation compares variants sharing the same backbone and regressor but differing in components between them.
- Ablation study: The FIT combining softmax-based and sigmoid-based attention achieves the best results across metrics, while query residual connections are removed by design.The sigmoid-based component filters undesirable high correlations, and the ablation reports that removing both query residual connections performs best.
- Comparisons with the state-of-the-art methods: HandOccNet achieves the best results on HO-3D and FPHA and produces better qualitative estimates than state-of-the-art methods on HO-3D.The method estimates global hand rotation accurately under severe occlusion, and the results are consistent with the feature-injection ablation.
5. Conclusion
The conclusion presents HandOccNet as an occlusion-robust 3D hand-mesh framework that injects primary-feature information into secondary-feature locations using FIT and SET.
- Conclusion: HandOccNet uses feature injection to make feature maps robust to occlusion by placing primary-feature information at secondary-feature locations.The framework uses two successive Transformers: FIT and SET.
- Conclusion: Experiments show state-of-the-art performance on 3D hand-mesh benchmarks containing severe occlusions.
Supplementary Material for “HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Network”
The supplementary material details FIT and SET, adds evaluations before Procrustes alignment, extends experiments to Dex-YCB, and provides additional visual comparisons.
- Supplementary overview: Section S1 describes the specifications of the FIT and SET Transformer-based modules.
- Supplementary overview: Section S2 reports quantitative results before Procrustes alignment to further evaluate HandOccNet.
- Supplementary overview: Section S3 evaluates HandOccNet on Dex-YCB, which includes severe hand-object occlusion and more data.
- Supplementary overview: Section S4 provides additional visual comparisons with other state-of-the-art methods.
S1. Specifications of FIT and SET
The supplementary specifications describe FIT as a correlation-based feature-injection module and SET as a self-attention refinement module, with pseudocode defining their processing steps.
- FIT and SET overview: FIT injects primary-feature information into correlated secondary-feature regions, while SET refines FIT output using distant information.
- FIT: FIT computes queries and keys from secondary and primary features, respectively, then uses their correlations to guide feature injection.The pseudocode projects features with 1×1 convolutions and reshapes them before attention computation.
- FIT: FIT combines softmax-based and sigmoid-based attention, with average pooling used to make sigmoid correlations robust to noisy correlations.The supplementary text states that removing this pooling causes training divergence.
- SET: SET applies self-attention to FIT output using projected queries, keys, and values, followed by residual and feed-forward processing.Its pseudocode computes a residual feature from attention and query terms before producing the final SET feature.
- FIT: FIT forms a residual feature from the correlation map and value features, then applies feed-forward processing to produce its output.
S2. Evaluation: Before Procrustes Alignment
HandOccNet is compared with state-of-the-art methods on HO-3D using MPJPE before Procrustes alignment, and it achieves better performance in this evaluation.
- HandOccNet achieves better MPJPE before Procrustes alignment than state-of-the-art methods on HO-3D.This comparison evaluates hand mesh estimation without Procrustes alignment.
S3. Results on the larger dataset, Dex-YCB [1]
HandOccNet is further evaluated on Dex-YCB, a larger dataset containing extensive RGB-D hand-object interaction sequences and severe occlusions.
- Dex-YCB contains 582K RGB-D frames across 1,000 sequences, 10 subjects, 20 objects, and 8 independent views.The dataset supports evaluation across varied hand-object interaction conditions.
- Evaluation on Dex-YCB is intended to further justify HandOccNet’s robustness when hands are severely occluded.
S4. Qualitative comparisons
Qualitative comparisons on HO3D and FPHA show that HandOccNet estimates 3D hand meshes robustly under severe object occlusion and improves reconstruction of occluded parts such as thumbs.
- HandOccNet robustly estimates 3D hand meshes in severely occluded HO3D images.
- Feature injection improves reconstruction of occluded regions by injecting relevant hand information into them.The first and fourth rows of Figure S2 show better thumb reconstruction under this mechanism.
- On FPHA, HandOccNet produces better qualitative results than Hasson et al..
- The supplementary comparisons include HO3D images with severe occlusions, standard HO3D examples, and FPHA examples.