Source-linked AI summary
GOTT: Object-centric Dexterous Manipulation with a Reusable Cross-Embodiment Primitive
Yulin Liu, Lai Wei, Yen-Jen Wang, Akash Sharma, Pieter Abbeel, Henrik I. Christensen, Haozhi Qi
TL;DR
Dexterous hands lack reusable low-level skills for translating manipulation intent into contact-rich behavior across tasks and embodiments. GOTT pairs an object trajectory with a reach specification and uses a shared contact-acquisition primitive before tracking object motion. Experiments demonstrate reuse across objects, embodiments, tasks, workspaces, and high-level input sources.
Problem
Dexterous hands lack reusable general-purpose skills for translating high-level intent into contact-rich behavior across tasks and embodiments.
Source: §1 · p. 2Method
Results
Simulation and real-world experiments demonstrate reuse across objects, embodiments, tasks, workspaces, and high-level input sources.
Source: §5 · p. 15Takeaways & Limitations
A shared contact-acquisition primitive can refine approximate reach specifications into stable contact while separating local contact control from arm-specific reaching and downstream tracking.
Source: §5 · p. 15The approach assumes an approximately fixed hand-object transform after grasping, limiting its stated scope for precise in-hand tasks such as unscrewing.
Source: §5 · p. 15
Abstract
from arXiv · showhide
Foundation models and large-scale human data provide rich sources of manipulation intent, but translating this intent into multi-fingered robot behavior remains difficult. Dexterous hands still lack a reusable low-level primitive that reliably establishes contact across tasks and embodiments. We propose GOTT, a reach-acquire-move framework built around a single cross-embodiment contact-acquisition primitive. Given a robot-agnostic object trajectory and a reach specification, GOTT first brings the hand near a task-relevant contact region. The shared closed-loop primitive then establishes stable contact from this approximate initialization, and a pose-conditioned controller tracks the desired object motion. Reach specifications may come from future-aware planning, external models, or human demonstrations, while the primitive and tracking backend remain unchanged. Simulation and real-world experiments show that GOTT is able to establish robust contact across diverse objects, arm-hand platforms, and seen and unseen hand morphologies. It also consistently improves end-to-end task success over open-loop grasp execution.
1. Introduction
Dexterous hands lack reusable low-level skills for translating high-level manipulation intent into contact-rich behavior across tasks and embodiments. GOTT separates reaching, contact acquisition, and object-motion tracking, using a shared contact primitive with flexible reach specifications.
1. Introduction
- Dexterous hands lack general-purpose low-level skills that coordinate high-dimensional actions, establish complex contacts, and adapt across hand morphologies.Task-specific policies require retraining for new behaviors, while retargeting often lacks perceptual feedback for contact-rich execution.Source: §1 · p. 2
- Object trajectories describe desired motion but not where or how to contact an object, and stable grasps may still be functionally unsuitable.Prior approaches supply pre-grasps or hand references, or learn contact acquisition jointly with execution, but do not expose it as a reusable skill across sources and embodiments.Source: §1 · p. 2
- GOTT represents intent with an object trajectory and reach specification, then uses reach, acquire, and move stages to establish contact and track motion.Reach specifications can come from affordance-aware planning or human trajectories, while the shared closed-loop primitive and pose-conditioned controller remain unchanged.Source: §1 · p. 2
- The evaluation spans diverse objects, hands, tasks, unseen morphologies, and high-level input sources, and compares the decomposition with end-to-end trajectory learning.The experiments also examine robustness to workspace shifts, reuse across robot arms, and ablations of observation, reward, and planning choices.Source: §1 · p. 2
2. Related Work
Prior work connects high-level intent to robot actions through reusable local policies, trajectory transfer, multi-reference tracking, or grasping methods. GOTT distinguishes itself by using a flexible reach specification and a reusable contact-acquisition primitive across trajectories, objects, tasks, and embodiments.
2. Related Work
- High-level interfaces include language, visual constraints, code, and generated videos; GOTT instead pairs object trajectories with flexible reach specifications and reusable contact acquisition.Related approaches include ManipGen, SPOT, VLM-Scaffold, and LUCID, with VLM-Scaffold relying on task-specific residual RL.Source: §2 · p. 3
- Trajectory-transfer methods can reproduce a demonstration through optimization, sampling, or learning, but are often tied to a specific trajectory or embodiment.Examples include Human2Sim2Robot, ManipTrans, REGRIND, and SPIDER.Source: §2 · p. 3
- Multi-reference policies track object or hand trajectories, while some methods jointly learn contact acquisition and motion execution; GOTT separates reach, acquisition, and tracking.The decomposition makes the contact primitive reusable across trajectories, objects, and embodiments.Source: §2 · p. 3
- Dexterous grasping and cross-hand transfer methods primarily focus on grasp acquisition, whereas GOTT refines an approximate task-relevant reach before object-trajectory tracking.The same contact-acquisition primitive is reused across objects, tasks, trajectories, and hand embodiments.Source: §2 · p. 3
3. Dexterous Manipulation via General Object Trajectory Tracking
GOTT executes object trajectories through a reach-acquire-move pipeline, with a reusable cross-embodiment primitive establishing contact from an approximate, task-relevant reach. Reach specifications can be planned or supplied by external models and demonstrations, while the contact and tracking backend remains shared.
3. Dexterous Manipulation via General Object Trajectory Tracking
- GOTT takes a desired object trajectory and reach specification; its future-aware planner can construct the reach from object-side information, while external models or demonstrations can supply it instead.The contact-acquisition primitive and tracking backend remain unchanged across these sources.Source: §3 · p. 4
3.1. Reach-Acquire-Move Decomposition
- Reach follows an approach to an approximate initial configuration, acquire establishes stable contact through closed-loop feedback, and move tracks the object trajectory.This separates high-level contact selection, collision-free approach, and long-horizon reachability from local contact-rich control, so the input need not specify an exactly executable final grasp.Source: §3.1 · p. 4
3.2. General-Purpose Contact Acquisition Primitive
- The contact primitive learns stable acquisition from randomized initial configurations using local hand-object observations, with training randomization and a proprioception-based contact predictor supporting deployment.Training randomizes object properties and robot dynamics; acquisition ends when predicted thumb and other-finger contacts meet the specified threshold for consecutive timesteps.Source: Figure 2 · p. 5, §3.2 · p. 5
- Padding and binary masks map hand-specific observations and actions into canonical keypoint and joint slots, allowing one policy to control hands with different joint counts and kinematics.The canonical representation uses 21 keypoints and 26 joint slots, then remaps active outputs to each robot’s native joints.Source: §3.2 · p. 5
3.3. Reach Specifications from Future-Aware Planning
- Future-aware planning identifies task-relevant contact regions using VLM-predicted keypoints, then evaluates candidate reaches for feasible approaches and arm motion over the full object trajectory.It discards candidates with infeasible approaches, joint-limit violations, or failed IK, and selects the valid candidate with least cumulative joint motion.Source: §3.3 · p. 6
3.4. Reach Specifications from Human Guidance and External Models
- Human demonstrations or external models can provide contact regions, initial configurations, or retargeted approaches, replacing corresponding planning steps while retaining the shared execution backend.Richer guidance may omit robot-specific reachability or collision constraints, so future-aware planning remains the default for arm-hand execution.Source: §3.4 · pp. 6–7
3.5. Execution from High-Level Specifications
- At deployment, the robot follows the specified approach, switches to local acquisition near the object, and proceeds to tracking after predicted stable contact.Acquisition compensates for pose error, geometry mismatch, and contact uncertainty; during tracking, the hand-object transform is approximated as fixed and observed pose error corrects the end-effector target.Source: §3.5 · pp. 7–8
4. Experiments
Experiments assess GOTT’s contact-acquisition reliability, transfer across hands and workspaces, end-to-end task success, support for varied high-level specifications, and design choices. Results cover simulation, real-world platforms, unseen hand morphologies, and human demonstrations.
4.1.1. Simulation grasping evaluation
- Trained on 16 YCB objects, GOTT consistently outperforms BODex and DRO across three hands and generalizes to 9,588 DexGraspNet assets.The comparison uses a tabletop protocol with gravity enabled from the start, unlike prior protocols that apply gravity after contact.Source: §4.1.1 · p. 9
4.1.2. Real-world grasping experiments
- GOTT achieves 90%, 90%, and 96% lifting success on Franka-Sharpa, Franka-XHand, and Tianji-XHand, respectively.The evaluation uses the same checkpoint across platforms, with 50 rollouts per platform.Source: Figure 5 · p. 8, §4.1.2 · p. 9
4.2.2. Zero-shot transfer to unseen Allegro variants
- On unseen Allegro variants, the three-hand policy averages 72.1% success, matching per-variant oracles at 72.2% and exceeding Allegro-only training at 17.2%.The three-hand policy is evaluated zero-shot; the variants are created by removing finger links.Source: Table 2 · p. 10, §4.2.2 · p. 10
4.2.1. Cross-hand evaluation and representation comparison
- The unified per-link policy achieves 78.7% average success versus 51.7% for CrossDex, and joint training improves over specialist policies.Specialists are trained separately for each hand; unified policies train jointly on XHand, Sharpa, and Allegro.Source: §4.2.1 · p. 10
4.3. Does the Primitive Improve End-to-End Task Success?
- GOTT achieves 80.0% average task success versus 35.0% for both open-loop baselines across four real-world tasks.The baselines are sensitive to planning errors; GOTT adjusts the wrist and fingers to recover from imperfect initializations.Source: §4.3 · p. 11, Table 3 · p. 12
4.4. Does Decomposition Improve Robustness to Workspace Variations?
- Below a 0.4 m table-height offset, GOTT achieves 0.914 success versus SimToolReal’s 0.002; near its training configuration, SimToolReal leads 0.983 to 0.792.The comparison supports separating workspace-specific reachability from reusable contact control; GOTT’s failures mainly involve IK singularities.Source: Figure 8 · p. 12, §4.4 · p. 12
4.5. Does It Support High-Level Specifications from Different Sources?
- GOTT accepts object trajectories from diverse sources and unseen human demonstrations without task-specific tuning; on 499 demonstrations, it reaches 69.7% in MuJoCo versus replay’s 12.6%.GOTT also achieves 71.5% in IsaacLab and 72.9% in ManiSkill; Figure 10 contrasts replay failures with stable contact and trajectory execution.Source: §4.5.2 · pp. 12–13, §4.5 · p. 12, Figure 10 · p. 13
4.6. Ablations and Analysis
- Ablations show that observation and reward choices matter, while affordance-conditioned selection improves functional correctness and overall task success.One-timestep history lowers average success by 15.7 points, and removing either contact or lift reward drops it below 2%; affordance conditioning raises func. to 65–85% and tSR to 50–65%.Source: Table 4 · p. 13, §4.6.1 · p. 14, §4.6.2 · p. 14
5. Conclusion
GOTT pairs flexible reach specifications with a shared cross-embodiment contact-acquisition primitive, and experiments demonstrate reuse across objects, embodiments, tasks, workspaces, and input sources. The appendix details its reward, shared hand representation, observations, actions, resets, randomization, and deployment contact detection.
5. Conclusion
- GOTT’s shared contact-acquisition primitive establishes stable contact from approximate reach specifications, decoupling local contact control from arm-specific reaching and downstream tracking.Experiments demonstrate reuse across objects, embodiments, tasks, workspaces, and high-level input sources.Source: §5 · p. 15
A.1. Reward Function Details
- The acquisition reward combines distance, contact, lifting, and object-stability terms, with object-stability weights annealed from 2.0 to 10.0 over 10^5 steps.Distance, contact, and lifting weights are not annealed; object-stability bonuses discourage sliding or rotation during contact acquisition.Source: §A.1 · p. 21
A.2. Cross-Embodiment Unified Policy Representation
- A canonical 26-slot joint layout and 21-keypoint link topology map different hands into shared policy representations, with unused dimensions masked.The wrist is controlled separately through a shared 6-DoF floating-base interface.Source: §A.2 · p. 22
A.3. Observations
- The policy observation combines five-frame proprioceptive history, hand-object displacement vectors, and joint and link masks, without explicitly receiving object pose.The hand-object vectors represent each canonical link’s displacement to its nearest object-surface point.Source: §A.3 · p. 23
A.4. Policy Action
- The 32-dimensional action comprises a 6D end-effector delta and 26 canonical hand-joint deltas, applied as residual changes to prior targets.End-effector deltas are scaled to ±0.008 m per step for translation and ±0.052 rad per step for rotation.Source: §A.4 · p. 23
A.5. Reset and Initialization
- Training resets sample robot-object pairs and valid resting-pose/hand-configuration pairs, then apply shared planar translations and yaw rotations.The same transformation is applied to the object and hand, preserving their relative geometry.Source: §A.5 · p. 24
A.6. Domain Randomization
- Physics parameters are resampled at each environment reset to encourage robustness and sim-to-real transfer.Mass, scale, and actuator gains are scaled relative to nominal values; friction and restitution are sampled directly.Source: §A.6 · p. 24, Table 9 · p. 24
A.7. Contact Prediction Details
- A separate contact predictor trained on frozen-policy rollouts signals when acquisition should switch to tracking.Deployment switches after thumb and at least one non-thumb finger are predicted in contact for five consecutive control steps.Source: §A.7 · pp. 24–25
A.8. Additional Training Details
The appendix describes training assets and a future-aware planner that chooses task-relevant, feasible initializations before contact acquisition. The planner uses VLM-selected contact regions and filters candidate approaches against downstream trajectory feasibility.
A.8. Additional Training Details
- Training uses 16 YCB objects with 2–7 stable resting poses each and 4,000 successful initial-configuration/reference-grasp pairs per object-pose pair.Policies are trained for 2,000 epochs on one NVIDIA L40S GPU, taking 24 hours and approximately 491.5 million environment steps.Source: §A.8 · pp. 25–26
B. Future-Aware Planning Details
- The planner uses four rendered object views and VLM annotations of functional parts to select a task-relevant holding region.Back-projected grasp points inconsistent with the designated part are discarded; the contact region is the union of 3 cm surface neighborhoods around remaining points.Source: B. Future-Aware Planning Details · p. 26
- Candidate hand poses sampled within the selected region are filtered for collisions and ill-posed configurations, retaining 10 valid initializations by default.These candidates are approximate starting configurations for contact acquisition, not final grasps.Source: B. Future-Aware Planning Details · p. 26
- For each candidate, the planner checks a collision-free approach and solves IK along the desired object trajectory, discarding infeasible candidates and selecting the one with least arm motion.The selection assumes a rigid hand-object attachment while computing the hand trajectory along the object motion.Source: B. Future-Aware Planning Details · pp. 26–27
- The example drill-use plan selects the pistol grip as the task-relevant holding region and restricts candidate generation to that region.The prompt distinguishes an object’s holding region from its working end.Source: B. Future-Aware Planning Details · p. 27
C. Real-World Grasping Experiment Details
Across the three tested arm-hand platforms, Tianji-XHand has higher reported overall real-world grasp success than either Franka platform. The table covers per-object results over seen and unseen objects.
C. Real-World Grasping Experiment Details
- 48/50 (96%) for Tianji-XHand exceeds 45/50 (90%) for each Franka platform in overall real-world grasp success.The authors attribute Tianji’s higher success to its larger reachable workspace relative to their Franka setup.Source: C. Real-World Grasping Experiment… · p. 27, Table 11 · p. 28
- Table 11 reports per-object grasp success across embodiments over five trials per object, including five seen and five unseen objects.The caption identifies objects O1–O5 as seen during training and O6–O10 as unseen.Source: Table 11 · p. 28
D. Real-World Full-Pipeline Experiment Visualizations
The qualitative comparisons identify three recurring failure modes for DRO and BODex: unsuitable contact regions, object displacement during approach, and loose cage-like grasps. GOTT’s local closed-loop acquisition addresses these cases by adjusting the hand, reacting to displacement, and establishing opposing finger-thumb contact.
D. Real-World Full-Pipeline Experiment Visualizations
- GOTT adjusts wrist and fingers locally, reacts to object displacement, and establishes opposing finger-thumb contact in contrast to three recurring baseline failure modes.The failures shown are unintended contact regions, object displacement during approach, and cage-like grasps without firm opposing contact.Source: D. Real-World Full-Pipeline Experiment… · p. 28
E. Simulation Grasping Evaluation Protocol and Baseline Details
BODex and DRO success rates in the main paper are lower than in their original publications, prompting controlled protocol comparisons. BODex’s original protocol evaluates floating objects, whereas the paper’s protocol includes a tabletop and collision constraints.
E. Simulation Grasping Evaluation Protocol and Baseline Details
- The protocol comparison explains why the reported BODex and DRO grasp success rates are lower than those in their original publications and motivates a controlled reproduction.Source: E. Simulation Grasping Evaluation… · p. 28
E.1. Protocol Comparison with BODex
- BODex’s original protocol synthesizes grasps without a table and tests floating objects under gravity in six directions; the paper’s protocol starts with tabletop objects under gravity.BODex requires deviations to stay within 5 cm and 15° for at least 3 s; both seed-pose initialization and execution in the tabletop protocol must avoid hand-table collisions.Source: §E.1 · p. 29
E.2. Reproducing BODex under Its Evaluation Protocol
The BODex reproduction isolates generation-time table constraints by holding the evaluation protocol fixed. Adding those constraints reduces grasp success, while using the paper’s original configuration brings the floating-generation result closer to the published value.
E.2. Reproducing BODex under Its Evaluation Protocol
- The comparison synthesizes 20 initial poses per object under floating and tabletop-aware generation, then evaluates both with BODex’s original protocol.Source: §E.2 · p. 29
- 70.7% to 57.1%: tabletop-aware generation lowers BODex success under the same original evaluation protocol.The change is attributed to a harder generation problem: the initial-pose distribution changes and bilevel optimization becomes more constrained.Source: §E.2 · p. 29, Table 12 · p. 30
- With the paper’s configuration of 30 g object mass and k_p=2, floating-generation evaluation reaches 77.2%, versus 79.9% reported in [7].The residual gap is attributed to likely seed variance; the released main-branch defaults use 100 g mass and k_p=5.Source: §E.2 · pp. 29–30
E.3. Evaluation-Phase Comparison
GOTT outperforms BODex under both tabletop and floating evaluation, and its reported tabletop success rates exceed BODex’s results across embodiments. The section also compares DRO protocols and variants and reports runtime measurements.
E.3. Evaluation-Phase Comparison
- GOTT’s tabletop success rates are 0.74 (XHand), 0.75 (Sharpa), and 0.87 (Allegro), above BODex’s results for every embodiment even under BODex’s original protocol.Figure 14 compares the same BODex grasps under the original and tabletop protocols; the estimates cluster near the diagonal.Source: §E.3 · p. 30
E.4. Protocol Comparison with DRO
- DRO’s perturbation test applies forces corresponding to 0.5 m/s², while the tabletop protocol requires lifting against sustained gravity, roughly 20× that force, and accounts for table collisions.Source: §E.4 · p. 31
E.5. DRO Baseline Variants
- DRO’s released checkpoint is the strongest tested variant overall; on Allegro, it reaches 43.2% versus GOTT’s 87.1% under the same tabletop protocol.Fine-tuned or retrained variants underperform the released checkpoint on average across hands; the authors attribute this to their smaller, tabletop-constrained 16-object training set.Source: §E.5 · p. 31, Table 13 · p. 31
F.1. Runtime Breakdown
- Online execution supports a 10 Hz control loop, with FoundationPose at 45 Hz and contact-acquisition and prediction networks taking 0.752 ms and 0.085 ms per step.One-time initialization totals about 29.6 s.Source: §F.1 · p. 31
F.2. Contact Predictor Evaluation
The contact predictor is evaluated on frozen contact-acquisition rollouts against simulator grasp state. Its switch accuracy is high across hands and thresholds, and its proprioceptive design is robust to the tested observation noise.
F.2. Contact Predictor Evaluation
- The evaluation compares each step’s is_grasping signal with simulator ground truth and counts a switch as correct when tracking starts while the hand holds the object.The test uses 2,560 episodes of 60 steps per hand across the 16 training YCB objects, on unseen rollouts of the frozen primitive.Source: §F.2 · pp. 31–32
- Per-episode switch accuracy is 0.953–0.960, with median lag at most one step; accuracy stays 0.94–0.98 for thresholds 0.4–0.8 and degrades only at thresholds ≤0.3.At threshold 0.5, per-step false-positive rates are 0.066–0.125 and false-negative rates are 0.036–0.047 across three hands.Source: §F.2 · p. 32
- Under 0.5 rad joint noise, the proprioceptive predictor loses at most 0.01 accuracy, while jointly trained heads lose 0.35 on average.The in-policy head loses 0.16 on average under object-position noise up to 5 cm; the proprioceptive predictor is unaffected by object-pose noise by construction.Source: Figure 15 · p. 32, §F.2 · p. 33