Source-linked AI summary
Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time
Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, Xiaolong Wang
TL;DR
The paper tackles difficult 3D hand-object pose estimation with scarce annotations by combining joint Transformer-based interaction reasoning and semi-supervised video training. Spatial-temporally filtered pseudo-labels improve hand and object pose performance and cross-domain generalization, including object gains despite using pseudo-labels only for hands.
Problem
3D hand and object pose annotations are scarce and difficult to obtain, limiting supervised estimation and generalization to novel scenes and out-of-domain environments.
Method
A unified model uses Transformer-based hand-object contextual reasoning, then filters video-generated hand pseudo-labels with spatial-temporal consistency for self-training.
Results
The method substantially improves hand and object pose performance and cross-domain generalization; object pose improves by more than 10% for some objects despite hand-only pseudo-labels.
Takeaways & Limitations
Large-scale hand-object videos can provide useful pseudo-labels that improve both hand estimation and jointly learned object pose.
Takeaways & Limitations
The approach assumes that hand and object poses change smoothly and continuously in videos for selecting coherent pseudo-labels.
Abstract
from arXiv · showhide
Estimating 3D hand and object pose from a single image is an extremely challenging problem: hands and objects are often self-occluded during interactions, and the 3D annotations are scarce as even humans cannot directly label the ground-truths from a single image perfectly. To tackle these challenges, we propose a unified framework for estimating the 3D hand and object poses with semi-supervised learning. We build a joint learning framework where we perform explicit contextual reasoning between hand and object representations by a Transformer. Going beyond limited 3D annotations in a single image, we leverage the spatial-temporal consistency in large-scale hand-object videos as a constraint for generating pseudo labels in semi-supervised learning. Our method not only improves hand pose estimation in challenging real-world dataset, but also substantially improve the object pose which has fewer ground-truths per instance. By training with large-scale diverse videos, our model also generalizes better across multiple out-of-domain datasets. Project page and code: https://stevenlsw.github.io/Semi-Hand-Object
1 UC San Diego 2 NVIDIA
Figure 1 depicts a semi-supervised pipeline that filters video pseudo-labels using spatial-temporal consistency before self-training.
- An initial model trained on annotated data generates pseudo-labels from a large-scale video dataset.
- Spatial-temporal consistency selects pseudo-labels for self-training.
- After semi-supervised learning, hand-object pose estimation and hand-pose generalization improve.
1. Introduction
The paper addresses scarce and difficult 3D annotations by combining hand-object interaction reasoning with temporally consistent pseudo-labels from videos. This improves pose estimation and cross-domain generalization while also benefiting object pose estimation.
- 3D hand and object pose estimation supports action understanding and applications such as augmented reality, imitation learning, and human-machine interaction.
- Human 3D pose annotations are difficult to obtain, limiting supervised learning and generalization to novel scenes and out-of-domain environments.
- Hand and object poses provide complementary interaction cues: grasping hand pose indicates object orientation, while object pose constrains hand approach.
- The method trains a joint model on fully annotated data, generates video pseudo-labels, filters them using spatial-temporal consistency, and self-trains on the retained labels.
- More than 10% improvement is achieved for object pose estimation on some objects, while hand pose also improves on HO-3D and out-of-domain datasets.
- The contributions are a unified joint estimator, a video-based semi-supervised pipeline, and improved pose performance with out-of-domain generalization.
2. Related Work
Related work covers RGB hand pose, 6-Dof object pose, hand-object interaction, semi-supervised learning, and interaction reasoning. The proposed framework combines joint pose estimation with Transformer-based contextual reasoning and video pseudo-labeling.
- Hand pose estimation: RGB hand pose methods use either model-free joint or heatmap prediction or model-based estimation with the MANO model.
- Object pose estimation: Object pose methods either directly regress 6-Dof pose or regress projected 3D control points and recover pose through 2D-to-3D correspondence.
- Hand-object interaction: Hand-object interaction estimation is difficult because of self-occlusion; this work uses a Transformer to model spatial dependencies explicitly between hand and object representations.
- Semi-supervised Learning: Semi-supervised approaches use models trained on human-annotated data to generate pseudo-labels on unlabeled data for further training.
- Interaction Reasoning: The framework overview combines a shared Res-50-FPN encoder, Transformer contextual reasoning, and separate hand and object decoders.
3. Overview
The method has two components: a joint hand-object estimator with contextual reasoning and a semi-supervised pipeline that uses spatial-temporally consistent video pseudo-labels.
- The joint learning framework uses a shared encoder, separate hand and object decoders, and a Transformer-based contextual reasoning module.
- The semi-supervised pipeline filters hand pseudo-labels from large-scale videos using spatial-temporal consistency before retraining on annotated and confident pseudo-labeled data.
4. Hand-Object Joint Learning Framework
The framework jointly estimates 3D hand mesh and 6-Dof object pose by exchanging hand-object interaction context and decoding both outputs from enhanced features.
- Joint framework: The joint framework extracts hand and object RoI features, applies contextual reasoning, and sends enhanced maps to separate hand and object decoders.The total loss combines the hand and object decoder losses.
- Contextual reasoning: The contextual reasoning module uses object features as queries and intersecting hand-object features as keys and values for Transformer cross-attention.Attention output is fused with the original query, followed by a feed-forward module with residual connections.
- Hand decoder: The hand decoder combines 2D joint localization with mesh regression to predict MANO hand pose and shape parameters.The MANO model maps these parameters to hand mesh vertices and 3D joints.
- Object decoder: The object decoder predicts 2D projections of predefined 3D control points and confidence scores, then recovers object 6-Dof pose with PnP.A grid-based prediction scheme gives every feature-map grid predictions for each control point to better handle self-occlusion.
5. Semi-Supervised Learning
The method generates hand pseudo-labels from unlabeled video by combining ensemble predictions with spatial and temporal consistency checks, then retrains on confident labels alongside annotated data. Spatial filtering uses camera reprojection, bounding-box overlap, plausible hand geometry, and temporal smoothness to retain reliable frames.
- Pseudo-Label Generation: The model is deployed on unlabeled video, where eight augmented predictions are averaged to generate hand pseudo-label candidates.Each frame produces 2D and 3D joints, hand-mesh vertices, and MANO parameters.
- Spatial Consistency Constraints: Camera projection parameters are optimized by matching estimated 3D joints projected under a weak-perspective model to observed 2D joints.SMPLify is used for this optimization because direct camera inference is infeasible across the video's varied viewpoints.
- Spatial Consistency Constraints: Spatial filtering retains predictions whose reprojected mesh and annotated bounding box have IoU at least 0.6, while rejecting pose reprojection errors above tp = 0.65.The pipeline also checks normalized bone length and joint-angle plausibility to remove unnatural hand poses.
- Temporal Consistency Constraints: Only frames satisfying all three spatial constraints proceed to temporal filtering based on smooth 2D joints and MANO pose parameters across consecutive frames.The stated thresholds are tj = 0.5 for 2D pose distance and tθ = 0.01 for MANO pose distance.
- Temporal Consistency Constraints: Within each video, frames with hand-shape deviations exceeding twice the confident subset's standard deviation are removed.The confident subset's mean hand shape is used as the sequence-level reference.
- Re-training with Pseudo-Labels: Self-training combines annotated data with filtered hand pseudo-labels, applying a binary mask so pseudo-labeled samples contribute only hand loss because object pseudo-labels are unavailable.The total retraining loss is L = Lhand + B · Lobject, with B set to 1 for fully annotated data and 0 otherwise.
6. Experiment
Experiments evaluate joint hand-object pose estimation, contextual reasoning, semi-supervised learning, and cross-dataset generalization. The method improves hand and object estimation, benefits from contact-aware reasoning and filtered pseudo-labels, and generalizes better across domains.
- Evaluation setup: The evaluation covers HO-3D performance, CR-module ablations, semi-supervised learning, and generalization on FPHA and Freihand.HO-3D provides annotated hand and object poses, while Something-Something supplies unlabeled interaction videos for semi-supervised learning.
- Qualitative results: The CR module uses contact regions for relational reasoning and helps recover accurate 3D hand meshes and object 6-Dof poses under occlusion.Cross-attention visualizations show high responses near hand-object contact regions.
- Pose estimation performance: 80.9% mesh-AUC is achieved on HO-3D, with a 9.5mm hand mesh error and simultaneous object estimation.The mesh-AUC is 1.9% and 3.6% higher than the cited competing methods and [24], respectively.
- CR-module ablation: 5.5% average object ADD-0.1D improvement is obtained by enhancing object representation with object features as the CR query.Using hand features as the query does not improve hand pose estimation and can degrade performance.
- Semi-supervised learning: 8.4% average object ADD-0.1D improvement occurs with semi-supervised learning, despite collecting pseudo-labels only for hands.Improvements for cleanser, bottle, and can are 4.1%, 18.5%, and 2.7%, respectively.
- Pseudo-label filtering: Both spatial and temporal filtering constraints are critical because removing either degrades hand pose accuracy.The constraints select coherent pseudo-labels that change continuously over time and remain spatially consistent.
- Cross-domain generalization: Semi-supervised training improves cross-dataset hand generalization on FPHA and Freihand by using diverse hand-object videos from the wild.The reported evaluation uses joint AUC, mesh AUC, and F-scores after Procrustes alignment.
7. Conclusion
The paper concludes with a semi-supervised framework that jointly estimates 3D hand and 6-Dof object poses using interaction context and filtered video pseudo-labels. Experiments show improved pose performance and cross-domain generalization.
- Conclusion: The framework jointly estimates 3D hand pose and 6-Dof object pose while using hand-object interaction regions as Transformer context for object reasoning.The model is retrained using real labels together with high-quality pseudo-labels.
- Conclusion: Spatial-temporal filtering selects high-quality pseudo hand labels from large-scale videos before semi-supervised retraining.The final training set is the union of real- and pseudo-labeled data.
- Conclusion: The experiments report substantially improved hand and object pose performance together with better cross-domain generalization.The conclusion states these gains as the overall experimental outcome.
Appendix A: Network architecture
The network extracts hand and object features with ROIAlign, decodes hand joints and MANO parameters, and predicts object control points with confidence values.
- Feature extraction: ROIAlign crops 256-channel hand and object features at 32-pixel spatial resolution from the FPN P2 level.The CR module enhances object query features while preserving feature size.
- Hand Decoder: The hand decoder produces 21-channel 2D joint heatmaps and converts them into 2D joint positions using weighted heatmap coordinates.The mesh branch regresses 48 pose parameters and 10 shape parameters for the MANO model.
- Hand Decoder: The mesh regression branch predicts MANO pose parameters θ ∈R48 and shape parameters β ∈R10 from downsampled hand features.Residual blocks, max-pooling layers, and fully connected layers perform the regression.
- Object Decoder: The object decoder predicts 2D control-point coordinates and confidence values across grids for 21 object control points.Grid-specific offsets relate each grid location to the target control-point position.
Appendix B: Performance on FPHA Dataset with Fully Supervision
Under fully supervised training, the method performs strongly on FPHA, achieving the best overall reported performance while estimating both hand and object poses. The appendix also documents the network configuration and comparison tables.
- FPHA performance: The supervised model achieves the best overall performance on FPHA, with the lowest object estimation error and strong hand pose accuracy.It outperforms the compared approaches by a large margin for object estimation and is comparable to [72] for hand pose estimation.
- Evaluation setting: The FPHA evaluation excludes FPHA training data because visible annotation markers create significant appearance differences from other datasets.The comparison is therefore conducted using the supervised model trained without FPHA semi-supervised data.
- Appendix tables: Table 6 records the proposed network architecture and configurations, while Table 8 compares cross-domain generalization against other semi-supervised approaches.The provided table descriptions identify their respective organizational roles.
Appendix C: Cross-domain Semi-supervised Learning Results on FPHA Dataset
On FPHA, the method outperforms prior semi-supervised approaches in a cross-domain setting, using real-world hand-object videos to improve generalization across datasets.
- The method performs significantly better than previous semi-supervised approaches on FPHA in a cross-domain evaluation.None of the compared methods are trained with FPHA data.
- Training on real-world hand-object videos covers domains represented across different test datasets, improving cross-domain generalization.