Source-linked AI summary
HOPE-Net: A Graph-based Model for Hand-Object Pose Estimation
Bardia Doosti, Shujon Naha, Majid Mirbagheri, David Crandall
TL;DR
Hand-object pose estimation is important for understanding human activity and interaction, but hand and object poses are interdependent and jointly estimating them can help address occlusion. HOPE-Net uses graph-based joint modeling with a 2D-to-3D estimation strategy, and the authors report improved performance over state-of-the-art approaches while running in real time.
Problem
Hand-object pose estimation supports applications such as augmented reality, action recognition, robotics, and telepresence, while hand and object poses constrain each other and jointly modeling them can help overcome occlusion.
Method
HOPE-Net represents the hand and object as a single graph and first estimates 2D keypoints before converting them to 3D coordinates with graph convolutional networks.
Results
HOPE-Net outperforms state-of-the-art models for joint hand-object 3D pose estimation while running in real time, including better 2D object pose estimates than Tekin et al.
Takeaways & Limitations
The results support graph convolution as a lightweight real-time approach for joint hand-object pose estimation from single RGB images.
Takeaways & Limitations
The model may not generalize well to object shapes unlike those seen during training, particularly non-convex objects lacking a tight 3D bounding box.
Abstract
from arXiv · showhide
Hand-object pose estimation (HOPE) aims to jointly detect the poses of both a hand and of a held object. In this paper, we propose a lightweight model called HOPE-Net which jointly estimates hand and object pose in 2D and 3D in real-time. Our network uses a cascade of two adaptive graph convolutional neural networks, one to estimate 2D coordinates of the hand joints and object corners, followed by another to convert 2D coordinates to 3D. Our experiments show that through end-to-end training of the full network, we achieve better accuracy for both the 2D and 3D coordinate estimation problems. The proposed 2D to 3D graph convolution-based model could be applied to other 3D landmark detection problems, where it is possible to first predict the 2D keypoints and then transform them to 3D.
1. Introduction
HOPE jointly estimates hand and handled-object poses despite occlusion and camera-motion challenges. HOPE-Net addresses this with a lightweight graph-based cascade that predicts 2D keypoints and converts them to 3D in real time.
- HOPE is important for understanding human activity and applications including augmented reality, action recognition, robotics, and telepresence.
- Hand-object pose estimation is challenging because interaction creates occlusions, while first-person cameras introduce unpredictable motion.
- Jointly modeling hand and object poses exploits their correlation and can help overcome occlusion through physical and anatomical constraints.
- HOPE-Net represents the hand and object as one graph, predicts 2D hand-joint and object-boundary keypoints, then hierarchically recovers depth for 3D poses.
- The model uses a lightweight framework and Adaptive Graph U-Net to estimate 2D and 3D coordinates from single RGB images in real time.
- Extensive experiments report state-of-the-art performance for joint hand-object 3D pose estimation while retaining real-time operation.
2. Related Work
Prior work jointly models hand and object pose using complementary networks, interaction context, depth, and graph convolutions. HOPE-Net extends this direction with an adaptive Graph U-Net designed for sparse hand and object graphs.
- Hand-Object Pose Estimation: Prior hand-object methods use multiview interaction context or separate object-centered and hand-centered networks to improve pose representations.
- Hand-Object Pose Estimation: Other approaches generate hand pose and unknown-object models from depth or iteratively focus on hand and object regions before fusing depth images.
- Graph Convolution Networks: Graph convolutional networks learn relationships among graph nodes and have been applied to 2D and 3D human pose through semantic, multiscale, and spatial-temporal representations.
- Graph Convolution Networks: Graph U-Net introduced pooling and unpooling, but its pooling was reported to work poorly on graphs with few edges such as skeletons and object meshes.
- Graph Convolution Networks: HOPE-Net proposes adaptive graph convolution with new trainable pooling and unpooling layers for its Graph U-Net architecture.
3. Methodology
HOPE-Net combines image encoding, adaptive graph convolution, and an Adaptive Graph U-Net to estimate hand-object pose from initial 2D keypoints through 3D coordinates.
- Architecture: HOPE-Net uses an image encoder to predict initial 2D locations for hand joints and object bounding-box corners.The encoder is ResNet10, whose features are combined with the initial keypoint estimates for graph processing.
- Architecture: A 3-layer adaptive graph convolution refines the initial 2D predictions using image features and neighboring node information.The graph represents hand and object keypoints jointly, with adaptive connectivity learned during training.
- Adaptive Graph U-Net: The Adaptive Graph U-Net converts refined 2D hand-object keypoints into 3D coordinates through graph convolutions, pooling, unpooling, and skip connections.Pooling reduces graph nodes, unpooling restores them, and skip features preserve local information during decoding.
- Adaptive Graph U-Net: Adaptive adjacency and trainable pooling and unpooling layers address connectivity and gradient problems in sparse hand and object graphs.The model learns node relationships after pooling to avoid fragmenting sparse graphs, while its pooling module updates during training more effectively than gPool.
- Loss Function and Training: The training objective combines initial 2D, refined 2D, and 3D losses, with α and β set to 0.1 to align pixel- and millimeter-scale errors.Each loss uses Mean Squared Error.
4. Results
HOPE-Net was evaluated across egocentric, third-person, and synthetic hand-object datasets, with experiments covering robustness, pose accuracy, ablations, and runtime. It outperformed comparison methods on reported 2D and 3D tasks, while remaining lightweight and real-time.
- Datasets: The experiments used First-Person Hand Action and HO-3D for evaluation, with synthetic ObMan images for pre-training.The datasets span egocentric and third-person views; ObMan did not generalize well alone but helped when followed by fine-tuning on real images.
- Datasets: 21,501 First-Person Hand Action frames had 6D object-pose annotations, with 11,019 for training and 10,482 for evaluation.Object poses were converted into eight 3D corners of a tight oriented bounding box for the graph model.
- Implementation Details: Roughly half of First-Person Hand Action frames contained keypoints outside the image, motivating regression-based initial 2D coordinate prediction.A lightweight ResNet was used to reduce overfitting and support near-real-time inference.
- Hand-Object Pose Estimation Results: HOPE-Net outperformed Tekin et al. [26] on 2D object pose estimation without an object locator or temporal constraints.The architecture was also described as lightweight and faster to run.
- Hand-Object Pose Estimation Results: HOPE-Net outperformed both Tekin et al.’s RGB-based model [26] and Herando et al.’s depth-based model [6] in 3D pose estimation.The graph model also effectively removed Gaussian noise from keypoint coordinates in tests using noisy 2D inputs.
- Adaptive Graph U-Net Ablation Study: The adaptive Graph U-Net performed better than fully connected and non-pooling graph-convolution alternatives by a large margin.The authors associate this margin with the U-Net structure and its pooling and unpooling layers; their pooling method also outperformed gPool, and identity adjacency initialization performed best.
- Runtime: A complete 2D-and-3D inference pass required 0.005 seconds on an Nvidia Titan Xp.The image encoder and two graph convolutional networks were described as lightweight and suitable for real-time inference.
5. Conclusion and Future Work
The proposed model estimates hand-object 2D and 3D poses from a single image using an image encoder and cascaded graph convolutional networks. It achieves real-time performance and state-of-the-art results, but generalization is limited for unfamiliar object shapes.
- Conclusion: The model estimates hand-object 2D and 3D poses from a single image with an image encoder followed by two graph convolutional neural networks.The authors describe this as the paper’s introduced model.
- Conclusion: The approach beats the state-of-the-art while running in real-time.This result is reported for the proposed hand-object pose estimation model.
- Limitations: Training on FPHA and HO-3D suits objects resembling the training data but may not generalize well across object-shape categories.Non-convex objects lacking a tight 3D bounding box are identified as a challenge.
- Future Work: Future work includes adding temporal information, graph classification, and applying the Adaptive Graph U-Net to other problems.The proposed extensions target improved pose estimation, action detection, and categorical semantic understanding.