PaperlayerView exact revision on arXiv

Source-linked AI summary

One Hand to Rule Them All: Canonical Representations for Unified Dexterous Manipulation

Zhenyu Wei, Yunchao Yao, Mingyu Ding

arXiv:2602.16712v2cs.RO

TL;DR

Dexterous manipulation policies often do not generalize across hands with different morphologies and kinematic layouts. This paper introduces a parameterized canonical representation with unified morphology and action spaces, and demonstrates latent structure, behavioral preservation, and zero-shot cross-embodiment grasping on unseen hands.

  • Problem

    Existing dexterous manipulation methods are generally tailored to specific hand designs, limiting generalization and reuse across embodiments.

  • Method

    The paper maps heterogeneous hands into a parameterized canonical representation and URDF with shared morphology conditioning and action spaces.

  • Results

    The framework preserves grasp and manipulation behavior while enabling cross-embodiment policies that generalize zero-shot to unseen hand morphologies.

  • Takeaways & Limitations

    The canonical representation provides a practical foundation for scalable cross-embodiment dexterous manipulation and policy sharing.

  • Takeaways & Limitations

    Some hands contain structural deviations that canonical assumptions cannot capture exactly, producing small geometric or kinematic discrepancies.

Abstract

from arXiv · show

Dexterous manipulation policies today largely assume fixed hand designs, severely restricting their generalization to new embodiments with varied kinematic and structural layouts. To overcome this limitation, we introduce a parameterized canonical representation that unifies a broad spectrum of dexterous hand architectures. It comprises a unified parameter space and a canonical URDF format, offering three key advantages. 1) The parameter space captures essential morphological and kinematic variations for effective conditioning in learning algorithms. 2) A structured latent manifold can be learned over our space, where interpolations between embodiments yield smooth and physically meaningful morphology transitions. 3) The canonical URDF standardizes the action space while preserving dynamic and functional properties of the original URDFs, enabling efficient and reliable cross-embodiment policy learning. We validate these advantages through extensive analysis and experiments, including grasp policy replay, VAE latent encoding, and cross-embodiment zero-shot transfer. Specifically, we train a VAE on the unified representation to obtain a compact, semantically rich latent embedding, and develop a grasping policy conditioned on the canonical representation that generalizes across dexterous hands. We demonstrate, through simulation and real-world tasks on unseen morphologies (e.g., 81.9% zero-shot success rate on 3-finger LEAP Hand), that our framework unifies both the representational and action spaces of structurally diverse hands, providing a scalable foundation for cross-hand learning toward universal dexterous manipulation. Project Page: https://zhenyuwei2003.github.io/OHRA/

I. INTRODUCTION

The paper addresses embodiment-specific dexterous manipulation by introducing a canonical representation that unifies morphology, kinematics, and action spaces across hands. Experiments evaluate its latent structure, behavioral fidelity, and cross-hand policy transfer.

  • Motivation: Existing dexterous manipulation methods are generally tailored to specific hands, limiting policy generalization and reuse across embodiments.Variations in morphology, degrees of freedom, and kinematic layouts make policies difficult to transfer and heterogeneous datasets difficult to exploit.
  • Method: The framework introduces a parameterized canonical representation that makes diverse hand structures suitable for learning-based models.The representation combines a unified parameter space with a canonical URDF and standardized action space.
  • Validation: A VAE trained on canonical morphology parameters produces a structured latent manifold supporting smooth, physically meaningful interpolation between embodiments.The experiments use latent encoding to assess whether the unified parameter space captures meaningful hand variation.
  • Validation: Canonical URDFs preserve original hand behavior sufficiently for grasp replay and in-hand reorientation while enabling a shared action interface.The evaluation compares canonical and original models across multiple manipulation settings.
  • Results: Cross-embodiment grasp policies transfer across distinct hands and generalize zero-shot to unseen morphologies in simulation and real-world experiments.The approach is evaluated first across three hands and then on nearly one hundred LEAP Hand variants without additional fine-tuning.

A. Motivation

The motivation is to replace heterogeneous, difficult-to-learn URDF descriptions with a compact canonical hand model. The design captures shared human-inspired kinematics while enforcing consistent geometry, parameters, and coordinate conventions.

  • Motivation: A task-agnostic policy needs both hand morphology as input and a consistent action space across embodiments.These requirements allow one policy to operate across hands with different structures and degrees of freedom.
  • Motivation: URDFs encode geometry and motion capabilities but are heterogeneous, tree-structured, and difficult to use directly in learning pipelines.Their variation across platforms includes inconsistent coordinate frames and representations.
  • Canonical Representation: The canonical URDF automatically converts compact hand parameters into a standardized model while preserving structural and functional consistency.This supports morphology conditioning and execution in a shared action space.
  • Kinematic Analysis: The canonical design supports up to five fingers and 22 DoF using capsule links to preserve essential kinematic relationships while reducing geometric complexity.The topology is based on recurring human-inspired organization across representative dexterous hands.
  • Coordinate Conventions: Canonical coordinate conventions align palm, finger, and joint axes to make kinematics consistent and interpretable across hands.The convention addresses global-orientation and local-joint-frame inconsistencies in native URDFs.
  • Parameter Definition: The representation contains 82 parameters, with a 173-parameter template available for specialized hand designs.These parameters consolidate salient geometric and kinematic variation for learning and transfer.

D. URDF Parsing and Generation

The framework automatically parses arbitrary hand URDFs into canonical parameters and generates standardized URDFs and action mappings. These representations support latent modeling, behavioral fidelity tests, and cross-embodiment grasp prediction.

  • Parsing and Generation: An automated parsing and generation framework extracts canonical parameters from hand URDFs and reconstructs standardized morphologies and kinematics.URDF generation uses conditional templates to adapt models to different hand structures.
  • Unified Action Space: Canonical hands share a fixed 22-DoF control and observation structure, with inactive joints represented as dummy variables.Bidirectional joint mappings preserve indexing and sign conventions between original and canonical vectors.
  • Latent Representation: A VAE maps hand morphology parameters into a 16-dimensional latent space that captures geometric and kinematic variation.The latent space is trained on synthetic configurations sampled within physically feasible parameter ranges.
  • Physical Fidelity: Physical fidelity is evaluated by comparing reinforcement-learning agents trained on original and canonical URDFs for in-hand object rotation.The comparison uses Shadow Hand and LEAP Hand models in IsaacGym.
  • Grasp Policy: The grasp model predicts wrist translation and hand joint configuration in two stages within the canonical representation.A diffusion generator predicts translation conditioned on grasp features and wrist rotation, followed by an MLP predicting joint configuration.
  • Grasp Policy: Grasp features combine object point-cloud information with a frozen VAE hand embedding to condition predictions across morphologies.The training objective combines diffusion-based translation prediction with deterministic joint regression using Smooth-L1 loss.

D. LEAP Hand Zero-Shot Generalization

The framework represents diverse LEAP Hand morphologies in a shared canonical space and evaluates latent continuity, cross-hand policy learning, and zero-shot generalization. These experiments target unseen hand designs without requiring additional fine-tuning.

  • Zero-Shot Dataset: 256 LEAP Hand variants are generated by modifying canonical morphology parameters, enabling scalable construction of diverse hand designs.Variants encode the number of links on each finger, and canonical parameters support efficient instantiation.
  • Evaluation Scope: The experiments assess latent-space continuity, canonical-model fidelity, unified grasp-policy generalization, and zero-shot capability, including physical deployment.The evaluation spans morphology representation, motion preservation, cross-hand policies, and simulation-to-real performance.
  • Latent Morphology Space: The VAE produces smooth morphology transitions across interpolated hand models, varying geometry, finger arrangement, and degrees of freedom continuously.Interpolations are decoded into canonical URDF parameters and generate physically meaningful intermediate structures.
  • Latent Morphology Space: The learned morphology representation preserves structural relationships that support morphology-conditioned grasping and cross-hand policy transfer.This connects the VAE’s continuous embedding to downstream manipulation tasks.

B. Canonical Hand Fidelity

The canonical URDF is evaluated as a shared action space that preserves hand functionality while supporting efficient grasp generation and cross-embodiment transfer. Results cover policy replay, unified training, zero-shot variants, and morphology-conditioned behavior.

  • Canonical Hand Fidelity: Canonical and original URDF policies achieve closely matched success rates when replayed in both directions, indicating preserved action semantics.Policies trained in either representation are mapped into the other for bidirectional comparison.
  • Cross-Embodiment Training: The unified model consistently outperforms embodiment-specific models, showing effective policy sharing across hands with distinct kinematics.Shared action-space training allows hands to benefit from heterogeneous embodiment data.
  • Zero-Shot Generalization: Zero-shot models achieve performance on unseen LEAP Hand variants comparable to seen-hand performance, demonstrating transfer without target-variant training data.Additional tests include variants substantially different from the training set.
  • Zero-Shot Generalization: Zero-shot models outperform variant-specific models on leap_0312, leap_2203, and leap_3103, while two-finger variants remain more challenging.The lower performance on leap_0303 and leap_3030 is attributed to their gripper-like morphology and differing grasp patterns.
  • Morphology Conditioning: Incorrect hand conditions substantially reduce grasp success, especially in the zero-shot setting, confirming the role of morphology conditioning.The effect is smaller when the model has seen the original LEAP Hand parameters during training.
  • Morphology Conditioning: Gradients remain low for absent fingers and drop for the removed index finger, indicating focus on functional fingers during grasping.The gradient visualization links canonical parameters to morphology-dependent policy behavior.

E. Real-World Experiment

The canonical grasping policies are deployed on a Franka Research 3 arm across diverse objects and LEAP Hand variants. Trained and zero-shot models both achieve strong real-world performance, with zero-shot results close to trained-model results.

  • Results: Zero-shot models achieve success rates close to trained models in real-world grasping, demonstrating strong generalization to unseen hand variants.The results also support the physical fidelity of the canonical representation and reliable simulation-to-real transfer.
  • Results: The canonical representation preserves essential dynamics and physical fidelity sufficiently for real-world grasping across multiple hand variants.This conclusion is drawn from both trained and zero-shot deployment results.

APPENDIX

The canonical representation compresses diverse hand geometry and kinematics into a standardized parameterization with explicit structural assumptions. Its canonical URDF supports up to five fingers and 22 DoF while retaining essential motion characteristics.

  • Parameterization: The canonical representation contains 82 parameters that capture salient morphological and kinematic variations while removing redundant URDF details.The parameterization is derived from a canonical URDF design and is intended to remain compact and interpretable.
  • Morphological Parameters: Morphology is represented through palm radius, finger radius, and finger-length parameters, with shared non-thumb proportions and separate thumb lengths.Shared parameters reduce dimensionality while preserving realistic proportions and anthropomorphic structure.
  • Kinematic Parameters: Kinematic parameters encode finger-base offsets, thumb orientation, variable thumb axes, joint limits, and canonical joint structure.Most non-thumb rotational offsets and joint axes are fixed to reduce parameter count.
  • Modeling Assumptions: The representation assumes shared finger diameters, identical non-thumb link lengths, constrained finger planes, and fixed local joint orientations or axes.These assumptions capture dominant structural regularities while enabling a compact standardized representation.

2) Extended Canonical Parameterization:

The base canonical parameterization cannot exactly represent some structurally unusual hands, so an extended 173-parameter formulation increases fidelity and expressiveness. An automated parsing pipeline extracts morphology and kinematics from original URDFs for canonical conversion.

  • Extended canonical parameterization: Some embodiments, including Allegro and LEAP Hands, deviate from canonical assumptions and may incur small geometric or kinematic discrepancies.Examples include a different non-thumb joint axis in Allegro and swapped proximal joints in LEAP.
  • Extended canonical parameterization: The extended parameterization uses 173 parameters to exactly encode observed hand designs with substantially reduced approximation error.It is intended for higher-fidelity modeling of unconventional embodiments.
  • Extended canonical parameterization: Per-finger radii and link lengths replace shared geometric parameters, capturing finer variation across fingers.This expands the representation to accommodate future hands with more diverse proportions.
  • Extended canonical parameterization: Additional joint origins and rotation axes explicitly represent broader mounting configurations and kinematic couplings.The expansion covers twelve joints, including selected thumb, little-finger, and other finger joints.
  • Automatic conversion pipeline: The framework parses original URDFs and generates canonical URDFs through a bidirectional conversion pipeline.Parsing requires a joint mapping and canonical palm-root transform; remaining quantities are inferred automatically, with occasional inspection or adjustment.

2) Canonical URDF Generation:

Canonical URDF generation reconstructs consistent hand models from parameterized templates while accommodating structural variation. The resulting fixed-length representation combines continuous morphology with discrete joint-axis and activation information for learning.

  • Canonical URDF generation: A Jinja2-based parameterized template reconstructs full URDFs from canonical parameter values.Template placeholders are populated with the canonical parameters.
  • Canonical URDF generation: Conditional template logic generates valid URDFs for hands with varying numbers of fingers, links, and joint configurations.Joints are instantiated when their limits differ, while optional elements appear when corresponding parameters are present.
  • Canonical URDF generation: The parsing and generation components provide bidirectional conversion between diverse hand models and the unified canonical representation.This supports analysis of existing designs and synthesis of new embodiments.
  • Canonical representation: Each sampled hand is serialized into a fixed-length vector containing continuous geometry, frame parameters, joint-axis encodings, and joint-activation indicators.The representation preserves both continuous morphological variation and discrete structural information.
  • Canonical representation: The VAE uses type-specific reconstruction losses for continuous parameters, categorical joint axes, and binary joint activations.The overall objective combines these reconstruction terms with KL regularization.

D. In-hand Reorientation

In-hand reorientation policies are trained with PPO using recurrent MLP architectures and evaluated with reward terms that balance rotation, stability, motion, torque, and work. The setup uses randomized valid grasp configurations for consistent evaluation.

  • D. In-hand Reorientation: The policy combines a 256-unit GRU with an MLP backbone of hidden dimensions [512, 256, 128].The GRU output is concatenated with the original observation before the MLP.
  • D. In-hand Reorientation: PPO trains the in-hand rotation policies with shared hyperparameter settings for original and canonical robot hands.The experiments use separate policies for the compared hand representations.
  • D. In-hand Reorientation: The reward combines z-axis rotation with penalties for cube velocity, joint displacement, torque, and excessive work.The cube-fall penalty is triggered when its height falls below a threshold.
  • D. In-hand Reorientation: Episodes begin from stable cube grasps with randomly perturbed joint angles to diversify feasible initial configurations.The procedure generates canonical stable grasps before applying perturbations.
  • D. In-hand Reorientation: Each robot hand contributes 10,000 valid grasp configurations for training resets and evaluation sampling.Validity requires close fingertip proximity, at least two contacting fingers, and sufficient cube height.

3) Generate initial grasp configuration:

The grasp evaluation uses a diffusion model and a force-closure test under external perturbations. Success is determined by whether object displacement remains below a fixed threshold after six directional force trials.

  • Generate initial grasp configuration: The grasp model is an MLP diffusion network with hidden sizes 512 and 256 and a 64-dimensional diffusion-step embedding.Training uses 1000 diffusion timesteps with sample prediction.
  • Generate initial grasp configuration: Grasp success is evaluated by applying external forces along six orthogonal directions for 1 second each.The controller executes each predicted grasp on the canonical hand before perturbation testing.
  • Generate initial grasp configuration: A grasp succeeds when object displacement remains below 2 cm after all perturbations.The criterion is force-closure based and evaluated in Isaac Gym.

F. LEAP Hand Zero-Shot Generalization

The LEAP Hand experiment extends the canonical URDF to better represent its distinctive kinematic structure while preserving the general learning framework. Variants are generated programmatically and evaluated with filtered grasp datasets and real-world experiments.

  • Extended canonical URDF: The extended canonical URDF relocates non-thumb abduction/adduction joints to the second link, reducing geometric discrepancy with the original LEAP Hand.The design uses the extended parameter set from Appendix A2.
  • Extended canonical URDF: The LEAP-specific joint-placement extension is used only in this experiment because adding such parameters generally would reduce sharing across dexterous hands.The authors identify abduction/adduction placement as the primary structural deviation from the canonical design.
  • Extended canonical URDF: The targeted extension does not alter the underlying learning framework and supports incorporating hand-specific kinematic features without compromising the canonical representation’s generality.
  • Variant generation: Different LEAP Hand morphologies are generated programmatically by setting selected link and joint parameters to zero and varying the number of finger links.Representative variants are visualized in Fig. 13 using Viser.
  • Grasp-data generation: Grasp candidates are generated for each variant from specified fingertip links and active joints, then filtered in Isaac Gym using the main experiments’ physical validity criteria.Candidates are generated over four independent rounds, and valid-grasp counts are summarized in Table XIV.
Loading 2602.16712v2…