Source-linked AI summary
TransHands: Repurposing Human Pose Encoders as Hand Pose Encoders
Milo Piccioli, Gianluca Amprimo, Claudia Ferraris, Gabriella Olmo
TL;DR
Limited large-scale, diverse 3D hand annotations make monocular 2D-to-3D hand-pose lifting difficult, while body-motion data provide abundant learned motion representations. TransHands adapts pre-trained human motion encoders with lightweight topological alignment and staged backbone tuning, achieving consistent transfer gains across architectures and strong cross-domain generalization. Its evaluation remains bounded by reliance on mostly ground-truth 2D keypoints.
Problem
3D hand-pose lifting lacks the diverse, large-scale annotated hand data available for human motion, limiting evidence for learning robust hand kinematic priors.
Method
TransHands uses a learnable topological adapter, pre-trained body-motion encoder, task-specific decoder, and partial unfreezing to adapt body representations to hand kinematics.
Results
Motion priors transfer consistently across four motion-encoding architectures, yielding accuracy gains and strong cross-domain generalization, particularly in challenging egocentric settings.
Takeaways & Limitations
Body-derived motion priors provide strong inductive biases for data-efficient and robust hand-pose lifting with minimal architectural change across articulated domains.
Takeaways & Limitations
Evaluation relies mostly on ground-truth 2D keypoints, so real-world accuracy remains bounded by the 2D detector; integration of 2D estimation and 3D lifting remains future work.
Abstract
from arXiv · showhide
Lifting 3D hand poses from 2D monocular representations remains challenging due to the limited availability of large-scale, diverse 3D-annotated hand datasets, in contrast to the abundance of human body motion data. We address this limitation by transferring motion representations learned from large body pose corpora to the hand domain. We introduce TransHands, a backbone-agnostic transfer learning framework that enables pre-trained human motion encoders to be effectively adapted for 3D hand pose estimation from 2D pose inputs. Rather than training hand-specific biomechanical models from scratch, TransHands combines a two-stage training and fine-tuning strategy with a lightweight hand-specific input adaptation module that aligns hand kinematics with the representation space learned for full-body motion. We evaluate TransHands across four state-of-the-art motion modeling architectures, including transformer-based, graph-based, and frequency- domain models. Results demonstrate that motion priors learned from body pose data transfer consistently across architectures, yielding consistent accuracy gains, strong cross-domain generalization, particularly in challenging egocentric settings, and applicability for downstream tasks in real-world contexts.
1 Introduction
TransHands addresses limited hand-pose data by transferring motion priors learned from large body datasets through lightweight topological adaptation. Across diverse motion architectures, the framework provides consistent gains and cross-domain generalization.
- Motivation: 3D hand-pose lifting is constrained by the scarcity of diverse, large-scale annotated hand datasets despite abundant body-motion data.Hands also involve complex biomechanical constraints and high-frequency articulations that are difficult to capture and annotate at scale.
- Motivation: TransHands asks whether motion priors learned from large-scale human body datasets can transfer to hand biomechanical modeling despite topological differences.The paper identifies a lack of systematic study on transferring body motion priors to the hand domain.
- Approach: TransHands repurposes pre-trained human motion encoders for 3D hand-pose estimation through a topological adapter, body-motion backbone, and task-specific decoder.The approach preserves body-model temporal dynamics while aligning hand joint hierarchies with the body-centric latent space.
- Evaluation: The framework is evaluated across transformer-based, frequency-domain, and graph-based motion encoders.This evaluation tests transfer across four diverse motion-encoding architectures.
- Results: Motion priors transfer consistently across architectural families, yielding accuracy gains, strong cross-domain generalization, and particularly strong performance in challenging egocentric settings.The framework also requires minimal modification to existing backbones, supporting integration of future motion-modeling advances.
- Contributions: The paper presents a systematic study of body-motion-prior transfer and a modular framework that decouples topological adaptation from temporal modeling.The contributions also include demonstrating that biomechanical knowledge from body datasets provides inductive biases for hand kinematics and data-efficient hand-pose estimation.
2 Related Works
Prior work established data-driven kinematic priors from large human-motion datasets, but body encoders remain difficult to reuse for hands because their skeletal topologies differ. TransHands addresses this mismatch with lightweight alignment modules and a modular framework.
- Human motion priors: Large-scale datasets such as AMASS enabled pre-training motion encoders that capture biomechanical properties, joint coordination, and long-range temporal dependencies.Earlier approaches also used direct regression with paired 2D–3D data and exploited geometric consistency and temporal smoothness.
- Cross-topology transfer: Body-pose motion encoders cannot be directly reused for hand pose because they are optimized for full-body skeletal structures with different topologies.This mismatch leaves open whether body encoders can be systematically repurposed for hand estimation.
- Cross-topology transfer: TransHands reframes hand pose estimation as biomechanical lifting and uses lightweight, modular alignment modules to adapt body motion encoders to hand kinematics.The framework is designed to bridge topology differences without redesigning the temporal modeling backbone.
3 Method
TransHands adapts pretrained body-motion encoders to 3D hand-pose lifting through topological alignment, selective fine-tuning, and projection modules. The framework supports transformer, frequency-domain, and graph-based backbones while preserving their motion representations.
- Problem Formulation: The task is formulated as sequence-to-sequence lifting from 2D hand coordinates to trajectories of 3D coordinates for 21 MANO joints.Inputs contain sequences of hand-joint coordinates with backbone-dependent channel dimensionality.
- Framework: TransHands maps hand inputs into a body encoder’s latent kinematic space and decodes the adapted representations into 3D hand poses.The pipeline uses a topological adapter, frozen or partially trainable body encoder, projection module, and task-specific decoder.
- Topological Adapter: The topological adapter resolves mismatches between hand and body skeletons by transforming hand coordinates into backbone-compatible channels and latent representations.It can synthesize a confidence channel for MotionBERT or preserve two-dimensional coordinates for MixSTE and PoseFormerV2.
- Topological Adapter: A Neural ODE provides a continuous nonlinear transformation from flattened hand coordinates to the body encoder’s required compatibility vector.The hidden state is integrated with a fixed-step fourth-order Runge–Kutta solver over t ∈ [0, 1].
- Motion Encoder Backbones: Partial unfreezing updates only the backbone’s deepest layers, embedding, and output layers while retaining learned body dynamics and adapting to hand micro-articulations.The strategy is used with four distinct state-of-the-art motion-encoder architectures.
- Motion Encoder Backbones: The framework is evaluated with spatio-temporal transformers, a frequency-domain transformer, and a graph-based convolutional architecture.The evaluated backbones are MotionBERT, MixSTE, PoseFormerV2, and ST-GCN; ST-GCN transfers only its temporal convolutions while training its graph branch from scratch.
- Latent Space Alignment: A multi-scale retention projection aligns heterogeneous encoder outputs before decoding and standardizes them to a common 256-dimensional latent representation.Residual retention layers process the projected features, after which a Layer-Normalized MLP reconstructs the 3D pose.
4 Experiments
TransHands is evaluated through staged transfer, refinement, cross-domain testing, transfer-learning comparisons, and architectural ablations across four motion encoders. The experiments show consistent benefits from transferred body-motion priors, with performance depending on backbone and adaptation design.
- Experimental setup: The evaluation uses Re:InterHand for primary adaptation, AssemblyHands for egocentric generalization, and GigaHands for bimanual activities using high-fidelity 2D and 3D annotations.These datasets isolate 2D-to-3D lifting from 2D-detection errors.
- Transfer protocol: The three-stage protocol progresses from frozen geometric alignment to in-domain partial unfreezing and multi-dataset adaptation.Stage 1 trains adapters while preserving the body encoder; later stages selectively refine the backbone or adapt across target domains.
- In-domain results: MixSTE achieves the lowest refined Re:InterHand error at 7.16 mm, improving 21.34% over its frozen baseline, while MotionBERT reaches 7.47 mm with a 21.45% gain.Transformer-based models outperform graph convolutional networks in the final refinement stage.
- In-domain results: PoseFormerV2 records the largest relative refinement gain, +36.53%, reducing error from 12.62 mm to 8.01 mm despite fewer parameters.ST-GCN reaches 11.56 mm and the lowest overall gain, +10.73%, indicating weaker partial-refinement performance.
- Cross-domain generalization: Multi-dataset adaptation reduces GigaHands errors from above 115 mm initially to a few millimetres, including 2.21 mm for MotionBERT, while MixSTE reaches 10.92 mm on AssemblyHands.The initial GigaHands error is attributed to global rigid misalignment, with competitive PA-MPJPE of approximately 24–30 mm.
- Transfer-learning impact: Transfer learning lets PoseFormerV2 match the full-data scratch baseline of 12.46 mm using less than 25% of available annotations.Across backbones, transferred initialization outperforms random initialization after partial unfreezing.
- Architectural ablation: The Neural ODE adapter lowers error to 9.47 mm, while combining it with RetNet reaches 8.01 mm; RetNet alone changes the baseline only from 11.69 mm to 11.65 mm.The ablation supports complementary contributions from nonlinear topological alignment and projection.
5 Robustness and Generalization in Real-World Contexts
The real-world evaluation examines noisy detector inputs, unseen domains, and downstream gesture recognition. TransHands shows competitive zero-shot egocentric lifting and transferable frozen representations, while its practical accuracy remains bounded by 2D detection quality.
- Evaluation scope: The robustness study evaluates domain shift, noisy 2D detector inputs, and downstream gesture recognition under privacy-preserving pose-only conditions.The experiments compare cross-site generalization and semantic usefulness of learned representations.
- Zero-shot generalization: On unseen AssemblyHands with MediaPipe keypoints, MotionBERT reaches 92.97 mm MPJPE and 23.82 mm PA-MPJPE without target-domain fine-tuning.This zero-shot lifting result is competitive with V-HPOT on root-aligned error and improves on the reported ArcticNet-SF MPJPE.
- Downstream gesture recognition: TransHands reaches about 88% accuracy on both Jester and EgoGesture using frozen lifting representations.The result indicates transferability of high-level semantic dynamics across static third-person and dynamic egocentric domains.
- Downstream comparisons: On EgoGesture, the approximately 88% result exceeds VGG16+LSTM at 74.7% and C3D at 86.4%, while RGB-Depth ensembles reach 92.2%.Full-RGB models on Jester exceed 95% by using contextual cues at higher computational cost.
- Limitations and future work: The evaluation relies mostly on ground-truth 2D keypoints, so real-world accuracy remains bounded by the 2D detector.Integrating 2D estimation and 3D uplifting through TransHands is left for future analysis.
6 Conclusion
TransHands repurposes body-motion encoders for 3D hand pose lifting by separating topological adaptation from temporal modeling. Across multiple encoders, body-derived priors support data-efficient, robust hand pose estimation and potential reuse across articulated domains.
- TransHands repurposes body-motion encoders for 3D hand pose lifting from monocular 2D inputs with minimal architectural change.
- Decoupling topological adaptation from temporal modeling enables cross-topology transfer.
- Body-derived priors provide strong inductive biases for data-efficient and robust hand pose lifting across multiple encoders.
- The framework highlights motion representation reuse across articulated domains and its relevance for downstream applications in real-world contexts.