Source-linked AI summary
Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimation
Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, Vincent Lepetit
TL;DR
Two-hand 3D pose estimation from a single color image is challenging because occlusions and similar joints make localization and recognition ambiguous. The paper separates these tasks with a Keypoint Transformer and reports state-of-the-art performance with roughly half the parameters on InterHand2.6M, while extending the approach to hand-object pose estimation and introducing a new annotated dataset.
Problem
Two-hand pose estimation is difficult because similar joint appearances and severe occlusions make joint localization and identification ambiguous.
Method
The Keypoint Transformer first detects unlabeled 2D keypoints, then uses self-attention to associate them with joint identities and predict 3D poses.
Results
The method achieves state-of-the-art performance on existing hand-interaction datasets, with roughly half the model parameters on InterHand2.6M.
Takeaways & Limitations
The approach supports accurate hand and object pose estimation in challenging images and may generalize to other articulated-object pose problems.
Takeaways & Limitations
The method does not model contacts or interpenetration and can fail on highly complex, severely occluded interactions.
Abstract
from arXiv · showhide
We propose a robust and accurate method for estimating the 3D poses of two hands in close interaction from a single color image. This is a very challenging problem, as large occlusions and many confusions between the joints may happen. State-of-the-art methods solve this problem by regressing a heatmap for each joint, which requires solving two problems simultaneously: localizing the joints and recognizing them. In this work, we propose to separate these tasks by relying on a CNN to first localize joints as 2D keypoints, and on self-attention between the CNN features at these keypoints to associate them with the corresponding hand joint. The resulting architecture, which we call "Keypoint Transformer", is highly efficient as it achieves state-of-the-art performance with roughly half the number of model parameters on the InterHand2.6M dataset. We also show it can be easily extended to estimate the 3D pose of an object manipulated by one or two hands with high performance. Moreover, we created a new dataset of more than 75,000 images of two hands manipulating an object fully annotated in 3D and will make it publicly available.
1. Introduction
Two-hand 3D pose estimation from a single RGB image is difficult because similar joints and severe occlusions make localization and identification ambiguous. The Keypoint Transformer separates keypoint localization from joint association and achieves strong performance with an efficient architecture.
- Challenge: Two-hand pose estimation is harder than single-hand estimation because similar joint appearances and close-interaction occlusions impede identification.Detecting hands separately and predicting their poses independently performs poorly in close interactions.
- Challenge: Previous bottom-up methods use one heatmap per joint, but appearance similarity and occlusion can produce ambiguous heatmaps and inaccurate localization.These failures can cause incorrect hand poses, especially for occluded hands.
- Approach: The method first detects unlabeled keypoints, then associates them with joints using self-attention between keypoint features.Keypoints may include false positives or miss some joints while still supporting correct 3D pose prediction.
- Approach: The Keypoint Transformer explicitly disambiguates keypoint identities and performs well on complex hand configurations.Its design uses keypoint-focused processing rather than the full-image, low-resolution feature strategy of DETR.
- Results: The method achieves state-of-the-art performance on existing hand-interaction datasets and roughly halves model parameters on InterHand2.6M.The paper also introduces H2O-3D, a challenging dataset of two hands manipulating objects with complete and accurate 3D annotations.
2. Related Work
Prior work spans generative, discriminative, and hybrid approaches for hand and object pose estimation, but close interactions remain difficult. This paper presents a unified discriminative approach that models relationships among detected joints and introduces a two-hand object-interaction dataset.
- Existing approaches: Generative methods fit parametric hand models under constraints, while discriminative methods directly predict poses from individual frames.Generative approaches often rely on tracking and can drift; discriminative approaches depend strongly on training-data variability.
- Existing approaches: Generative methods can lose tracking during close interactions or occlusions, motivating combinations with discriminative guidance.Prior systems also use physical constraints, collision terms, or hand-model fitting to refine estimates.
- This work: The paper proposes a unified discriminative approach for hand-hand and hand-object interaction scenarios.It models relationships between all detected joints while keeping model complexity low.
- This work: The method addresses close-interaction ambiguity by modeling relationships among detected joints rather than treating similar-looking joints independently.This distinguishes the approach from methods designed around hand centers or less ambiguous body joints.
- Dataset: The authors introduce a two-hand-and-object dataset containing videos of different subjects manipulating YCB objects with 3D hand and object pose annotations.Their architecture performs well on the dataset and provides a strong baseline.
3. Method
The Keypoint Transformer separates keypoint localization from joint identity association, then uses attention to estimate hand poses and extend prediction to manipulated objects. It detects flexible keypoints, encodes their appearance and spatial context, and uses learned queries to predict pose representations for hands and objects.
- 3.1. Keypoint Detection and Encoding: The method detects up to 64 unlabeled 2D keypoints from a single-channel heatmap, without initially recognizing their corresponding joints.A U-Net predicts the heatmap, and local maxima are retained after non-maximum suppression; there are 42 hand joints across two hands.
- 3.1. Keypoint Detection and Encoding: Each keypoint combines sampled multi-layer CNN appearance features with spatial encoding before entering the Transformer.The concatenated 3968-D appearance vector is reduced to 224 dimensions and combined with a 32-D sine positional encoding to form a 256-D representation.
- 3.2. Keypoint-Joint Association: Self-attention models relationships among keypoints and produces context-aware features that encode joint identity alongside localized image information.The keypoint-joint association loss trains features to distinguish hand identity, joint index, and background.
- 3.3. Pose Estimation from Identity-Aware Keypoints: The association stage permits duplicate assignments and background detections, while cross-attention with learned joint queries selects relevant keypoints for pose prediction.Background-associated keypoints are ignored, and multiple keypoints assigned to one joint can all contribute to its estimated pose.
- 3.4. Hand Pose Representations and Losses: The architecture supports 3D joint locations, 2.5D locations, and MANO joint angles as hand-pose representations.MANO angles provide access to a complete hand mesh for modeling contacts and interpenetration, while direct joint-location regression can be more accurate for joint error.
- 3.5. Object Pose Estimation: For hand-object interaction, an additional object segmentation head supplies 20 randomly selected object keypoints that are encoded and processed jointly with hand keypoints.Object keypoints receive an object identity, and dedicated cross-attention queries attend to them when estimating object pose.
4. Evaluation
The evaluation covers hand-hand, hand-object, and two-hand object interaction datasets, comparing accuracy with baselines and prior methods. Results indicate strong performance under complex interactions and occlusions, alongside a newly introduced H2O-3D benchmark.
- Datasets: The evaluation uses InterHand2.6M, HO-3D, and the newly introduced H2O-3D dataset.H2O-3D contains two hands manipulating objects with complete 3D annotations.
- InterHand2.6M: On InterHand2.6M, the method achieves 10% higher accuracy than InterNet and 16% and 12% higher accuracy than two Transformer baselines.The comparisons use a 2.5D pose representation.
- InterHand2.6M: Keypoint Transformer outperforms prior hand-interaction methods while maintaining the same model complexity as InterNet.The compared methods use per-joint heatmaps with visibility and segmentation-guided features, resulting in higher complexity.
- HO-3D: On HO-3D, Keypoint Transformer performs significantly better than previous methods for hand-object pose estimation.Evaluation includes hand pose metrics and symmetry-aware object pose using MSSD.
- H2O-3D: H2O-3D contains 60,998 training images and 15,342 test images, including seven seen objects and one unseen object.The dataset uses five subjects, ten YCB objects, and a five-camera RGBD setup.
- H2O-3D: On H2O-3D, the method obtains 3.09 cm MPJPE, 8.28 cm MRRPE, and 7.96 cm object MSSD.The MRRPE is about 2.5 times worse than on InterHand2.6M because object occlusions make hand translation more challenging.
5. Discussion
The discussion examines how cross-attention handles occlusion and noisy keypoints, and how pose representation, attention depth, and backbone size affect performance. The method remains accurate with relatively compact configurations.
- Cross-attention: When joints are occluded, cross-attention selects nearby visible keypoints to estimate their poses.For visible joints, queries attend to keypoints at corresponding image locations; for occluded joints, nearby evidence supports estimation.
- Robustness: Noisy keypoints are largely classified as background, while joint-associated noisy keypoints receive higher cross-attention weights.This allows the method to select the relevant noisy keypoints for final pose estimation.
- Pose representation: The 3D and 2.5D pose representations achieve similar accuracy, whereas joint-angle representation performs worse.This result is consistent with observations from previous works cited by the authors.
- Attention depth: Adding more cross-attention layers has little effect on MPJPE, while the method outperforms InterNet even without self-attention layers.These comparisons are reported across different self-attention and cross-attention layer combinations.
- Model size: A ResNet-18 backbone still outperforms InterNet with approximately half the total number of parameters.Larger backbones improve accuracy, but the method retains an advantage with the smaller backbone.
6. Conclusion
The method integrates keypoint detection with a Transformer to estimate 3D hand and object poses in challenging images. It supports multiple pose representations, including parent-relative 3D joints, 2.5D joints, and MANO joint angles.
- Conclusion: Integrating a keypoint detector into a Transformer enables accurate 3D pose prediction for hands and objects in challenging images.The authors report that this approach is more accurate than a standard Transformer architecture.
- 3D representation: The parent-relative 3D representation predicts one vector per skeletal bone and reconstructs root-relative joint locations by accumulation.It does not require camera intrinsics.
- 2.5D representation: The 2.5D representation combines each joint’s 2D location with parent-relative depth differences to reconstruct camera-coordinate 3D pose.Reconstruction requires camera intrinsics and the root joint’s absolute depth or hand scale.
- Query design: The 2.5D formulation uses 21 joint queries per hand, while the parent-relative 3D formulation uses 20 bone queries per hand.Root depth is predicted separately with RootNet in the 2.5D setup.
- MANO representation: The MANO representation estimates 16 3D joint angles per hand, plus shape and relative hand translation through an additional query.The predicted angles and shape parameters determine root-relative 3D joint locations.
B. Comparison with state-of-the-art on HO3D Dataset
On HO-3D, the method is competitive with leading approaches but does not consistently achieve the best accuracy across alignment metrics and dataset versions.
- HO-3D(V2): On HO-3D(V2), the method is close to HandOccNet on scale-translation aligned MPJPE but trails it on Procrustes-aligned MPJPE.HandOccNet achieves the highest accuracy for the Procrustes-aligned metric.
- HO-3D(V3): On HO-3D(V3), the method performs worse than Artiboost.The comparison is reported for the HO-3D(V3) dataset.
C. Method Limitations
The method can produce accurate interaction poses but remains limited by missing physical modeling and severe occlusion.
- Physical plausibility: Without modeling hand-object contacts and interpenetration, some predicted interaction poses are not physically plausible.The limitation concerns both hands and objects during interaction.
- Severe occlusion: Under highly complex and severely occluded hand interactions, the method can fail to obtain reasonable hand poses.The authors suggest temporal information and physical modeling as future directions.
E. Implementation details
The implementation uses a ResNet-50 U-Net with 256×256 inputs and 128×128 heatmaps, trained with Adam and online augmentation.
- Network: The CNN backbone is ResNet-50, and the decoder uses upsampling, convolutional layers, and skip connections.The architecture is implemented as a U-Net.
- Resolution: The input resolution is 256×256 pixels, while the heatmap resolution is 128×128 pixels.The training Gaussian kernel variance is σ = 1.25.
- Evaluation: Tables 7 and 8 report comparisons with state-of-the-art methods on HO-3D(V2) and HO-3D(V3), respectively.The HO-3D(V3) caption notes that ArtiBoost uses additional training data not used by this method.
- Optimization: The model is trained for 50 epochs on 3 Titan V GPUs with a total batch size of 78.Training uses Adam with learning rates of 10^-4 for attention modules and 10^-5 for the CNN backbone.
- Augmentation: Online augmentation includes rotation, scale, and mirroring during training.The image patch is loosely cropped around the hand and object.
F. Baseline Architectures
The CNN+SA baseline converts spatial CNN features into keypoint representations, augments them with positional embeddings, and processes them through self-attention. The CNN+SA+CA baseline adds cross-attention to this architecture.
- CNN feature extraction: 256 × 256 cropped images produce an 8 × 8 feature map with 2048 channels in the baseline architecture.The spatial feature map is flattened into 64 features.
- Feature projection: Three MLP layers convert the 64 spatial features into 224-dimensional representations.These representations are then used as the baseline token features.
- Positional encoding: The baseline concatenates the projected features with 32-D positional embeddings to form 256-D features.The resulting representations are used by the attention-based baseline architectures.
G. Robustness to Noisy Keypoints
The experiments evaluate robustness to noisy, missing, and object-related keypoints while describing the full hand-object pose-estimation pipeline. Qualitative results indicate accurate estimation despite substantial keypoint perturbations, although contact-related implausible poses can occur.
- Robustness tests: The robustness experiments add 30 noisy keypoints to 48 detected keypoints or retain only 30 detected keypoints.These scenarios test sensitivity to false-positive and missing keypoints.
- Noisy keypoints: The method estimates accurate poses when additional noisy keypoints are introduced.The experiment explicitly evaluates robustness to noisy detections.
- H2O-3D dataset: The H2O-3D dataset contains 60’998 training images and 15’342 test images from 17 multi-view sequences.It uses five RGBD cameras, six subjects, and ten YCB objects; only 9’098 test images are used for object-pose evaluation because the remaining objects are barely visible.
- Pose-estimation pipeline: The pipeline samples object keypoints from a regressed object segmentation map and hand keypoints from a single-channel keypoint heatmap.CNN image features and spatial embeddings encode the detected keypoints before joint association.
- Qualitative limitation: The method can produce implausible hand-object poses because it does not model contacts and interpenetration.This limitation is illustrated in qualitative results on HO-3D and H2O-3D.
I.2. InterHand2.6M Qualitative Results
Qualitative InterHand2.6M results show that the method handles complex interactions and strong occlusions in many cases. Attention visualizations indicate that joint queries select corresponding visible keypoints and nearby visible features when joints are occluded, while severe occlusion can still cause failures.
- Qualitative comparison: The method performs well during complex interactions and strong occlusions compared with the InterNet method.The comparison highlights the difficulty of joint confusion in purely CNN-based approaches.
- Pose quality: Retrieved MANO-angle poses are very similar to ground-truth poses in most qualitative examples.A failure occurs when the hand is severely occluded during complex interaction.
- Attention behavior: Cross-attention selects keypoints at the corresponding joint location when the joint is visible.For occluded joints, it selects features from nearby visible joints.
- Failure case: The qualitative results include a failure case where complex pose and severe occlusion prevent accurate pose recovery.This establishes a boundary on the method’s qualitative robustness.