PaperlayerView exact revision on arXiv

Source-linked AI summary

CARD: Contact-Anchored Retargeting for Dexterous Manipulation

Cunxi Dai, Botao He, Francois R Hogan, Wei Pan, Rao Fu, Wen-Han Hsieh, Ruoshi Liu, Jitendra Malik, Guanya Shi, Akash Sharma

arXiv:2610.36539v1cs.RO

TL;DR

Existing retargeting often maps kinematics while omitting contact dynamics, despite contact being important for manipulation. CARD preserves demonstrated contact force targets across demonstration sources without requiring an object model or simulator. It reports higher contact fidelity than kinematic and object-aware baselines, and improved strict task success when integrated into physics-based retargeting.

  • Problem

    Existing retargeting methods primarily map kinematics and often require object reconstruction or simulation, while manipulation also depends on contact dynamics.

    Source: §1 · p. 2

  • Method

    CARD retargets demonstrations by optimizing robot configurations against contact force targets and unifies video, glove, and motion-capture inputs in a shared hand–contact representation.

    Source: §1 · p. 2

  • Results

    Across video, glove, motion-capture, and GRAB data, CARD maintains contact on 43–78% of contact frames, versus under 10% for kinematic methods and 22.5% for object-aware methods.

    Source: §1 · p. 2, §4.1 · p. 6

  • Takeaways & Limitations

    The results establish contact preservation as a scalable interface for transferring human dexterity to robots.

    Source: §5 · p. 8

    Retargeting quality is bounded by the input demonstration, and open-loop replay cannot recover from a large object displacement.

    Source: §5 · p. 8

Abstract

from arXiv · show

Humans are dexterous, but traditional data collection paradigms such as teleoperation, kinesthetic teaching, and wearable exoskeletons force humans to act through the robot embodiment, bottlenecking their dexterity. Retargeting can avoid this. However, existing methods purely translate kinematics and ignore the real-world contact dynamics essential for task success. We present CARD: Contact-Anchored Retargeting for Dexterous Manipulation, a simulator and object-free framework that retargets human demonstrations to robot actions by explicitly preserving contact force targets. CARD unifies human data sources including video, motion capture, and tactile gloves via a shared hand--contact representation and generates explicit contact force targets in addition to kinematics. To maintain these contact targets during hardware open-loop replay, CARD also introduces a tactile tracking controller. For video which does not contain an explicit tactile source, we present Vid-HACO, our video-to-contact model that sets a new state of the art. Across our video, glove, motion-capture data and the GRAB dataset, CARD maintains contact on $43$--$78\%$ of contact frames, while kinematic and object-aware retargeting methods maintain contact on $<10\%$ and $22.5\%$ of the frames respectively. Further, integrating our formulation into physics-based retargeting improves strict task success from 12\% to 32\% on GRAB dataset. Finally, we show that our tactile tracker is critical to reproduce demonstrated forces during hardware replay. These results establish that contact preservation via CARD is a scalable interface for transferring human dexterity to robots.

1. Introduction

CARD retargets human demonstrations by preserving contact force targets without requiring an object model or simulator. It unifies several demonstration sources and pairs retargeting with tactile tracking for hardware replay.

Contact-Anchored Retargeting

  • CARD optimizes robot configurations against per-link contact targets—center of pressure, push direction, and force—without object geometry, a simulator, or an object pose estimate.Rigid, articulated, and deformable objects need no special handling.Source: §1 · p. 2

Unified Hand–Contact Representation

  • A shared hand–contact representation connects video via Vid-HACO, tactile gloves, and marker-based motion capture through a MANO mesh sequence and contact force targets.Contact targets are bound to the human hand surface, so retargeting to a new robot requires a semantic map from hand regions to robot bodies.Source: §1 · p. 2

Tactile Tracking Controller

  • On hardware, an integral admittance loop over joint impedance tracks demonstrated contact forces during open-loop replay.The replay uses no object pose estimate or learned policy; fingertip force feedback is the only feedback.Source: §1 · p. 2

Contributions

  • CARD reports cross-embodiment retargeting through a semantic map on four robot hands, from a 22-DoF dexterous hand to a parallel-jaw gripper.Its contributions also include a fast GPU sampling optimizer for contact reach, friction-cone alignment, force weighting, and contact stability.Source: §1 · p. 2

Results

  • Across video, glove, and motion-capture sources, CARD matches demonstrated contact on 43–78% of contact frames, versus under 10% for kinematic retargeting and 22.5% for object-aware methods.The tactile tracker also raises hardware replay success from 4 to 25 of 30 trials over position-only replay.Source: §1 · p. 2

2. Related Work

Prior retargeting work spans kinematic mapping, contact transfer, simulation-based dynamics, human-data pipelines, and tactile control. CARD's related-work discussion distinguishes its use of demonstrated contact force targets without an object model or simulator.

Kinematic retargeting

  • DexPilot and related systems [56] retarget hand configurations, while TopoRetarget [76] and Morphometric Imitation [62] extend geometric matching but still rely on an object reference.The authors report they could not compare against the latter methods because code was unavailable at submission.Source: §2 · pp. 2–3

Contact-based retargeting

  • Contact-transfer methods [32] [33] [84] [66] and force-based methods [2] [78] preserve contact structure or forces, but require an object model, simulator, or learned tracker.CARD instead optimizes contact locations, push directions, and forces obtained directly in the real world.Source: §2 · p. 3

Dynamic retargeting

  • Dynamic retargeting methods [9] [14] [34] [39] [40] [42] [43] [82] [90] capture contact dynamics in simulation but need an object model and task-specific simulator.Human-data pipelines, tactile controllers, and vision-based contact estimation are also discussed as related lines of work.Source: §2 · p. 3

Dexterous hand controllers

  • Existing tactile controllers regulate policy- or heuristic-chosen forces, whereas CARD outputs tactile targets from a demonstration for open-loop replay.The discussion also notes that concurrent contact-preserving learning pipelines anchor on a reconstructed object.Source: §2 · p. 3

3. Method

CARD turns diverse human demonstrations into robot trajectories that preserve contact force targets as well as hand motion. Its pipeline combines a shared hand–contact representation, contact-anchored retargeting, tactile force tracking during hardware replay, and semantic mapping across embodiments.

3. Method

  • CARD produces a robot trajectory that reproduces demonstrated contact force targets as well as kinematics, without requiring a physics simulator or object model as input.The optimized hand pose includes wrist position, wrist rotation, and joint angles.Source: §3 · p. 3

3.1. Unified Hand–Contact Representation

  • Each source becomes a MANO mesh sequence, shared semantic hand regions, and at most one contact force target per link and frame.A target specifies the center of pressure, push direction, and nonnegative force magnitude.Source: §3.1 · p. 3

3.2. Vid-HACO: Contact from Monocular Video

  • Vid-HACO predicts contact probabilities for MANO vertices and a frame-level contact indicator from video, using temporal context to distinguish approach, grasp, and release.It is trained on 1.55 million frames from 12 datasets, with contact labels based on a 10 mm distance threshold.Source: §3.2 · p. 3

3.3. Contact-Anchored Retargeting

  • Contact-anchored retargeting balances pad reach to force targets with friction-cone alignment, while style, smoothness, and collision terms guide unconstrained motion and feasible configurations.The contact objective weights targets by applied force and penalizes contact migration across the pad surface.Source: §3.3 · pp. 4–5

3.4. Replay: Tactile Tracking Controller

  • During replay, fingertip force feedback adjusts joint targets through an admittance loop, while feedforward torque applies desired force through the joint impedance controller.The controller also uses a contact state machine to freeze the integrator on sensor dropouts and arm feedforward.Source: §3.4 · p. 5

3.5. Cross-Embodiment Transfer

  • Transfer to another embodiment requires a semantic map; regions map one-to-one on five-fingered hands or many-to-one on hands with fewer fingers.For simultaneous many-to-one anchors, the targets are merged into a force-weighted centroid.Source: §3.5 · pp. 5–6

4. Experiments

CARD is evaluated for contact fidelity, hardware replay, cross-embodiment transfer, video contact estimation, and physics-based retargeting. The experiments compare contact fidelity with kinematic and object-aware baselines and test whether tactile tracking preserves forces during replay.

4.2. Open-loop replay on hardware

  • With tactile tracking, identical CARD trajectories succeed in 8/10 hammer lifts, 10/10 cup lifts, and 7/10 mustard dispenses, versus 1/10, 3/10, and 0/10 with position-only replay.Without tracking, the hammer and cup slip during lifting, while the bottle lifts without being squeezed.Source: §4.2 · p. 6

4.1. Contact fidelity against kinematic retargeting

  • CARD establishes contact on 61% of video, 78% of glove and motion-capture, and 43% of GRAB anchor-frames, versus 3–10% for kinematic baselines and 22.5% for ReGrind on GRAB.On GRAB, CARD also produces the least object penetration at 7.96 mm; its median gap is 1.4–3.6 mm versus 10–23 mm for kinematic baselines.Source: §4.1 · p. 6, Table 1 · p. 7

4.4. Contact estimation from video

  • Vid-HACO reaches 67.2 vertex F1 on held-out HRDexDB, exceeding released HACO by 3.9 points and matched-data HACO-R by 4.6 F1 and 17.7 AP.Frame F1 is 95.3, compared with 75.9 for both baselines, which predict contact on every frame.Source: Table 2 · p. 7, §4.4 · p. 7

4.3. Cross-embodiment transfer

  • CARD transfers to Wuji Hand 2, Unitree Dex3-1, and a parallel-jaw gripper by changing only the semantic map; Unitree actions are scaled 1.2×, while Wuji actions need no scaling.The Wuji Hand 2’s compliant inner surfaces conform to the object; the Unitree Dex3-1 has no tactile sensing.Source: §4.3 · p. 7, Figure 8 · p. 8

4.5. CARD with physically plausible retargeting

  • CARD-physics raises strict task success from 12% to 32% on GRAB and from 40% to 52% on motion-capture trajectories.The variant adds CARD initialization, a contact objective, and extracted contact force targets to SPIDER.Source: Table 3 · p. 7, §4.5 · pp. 7–8

5. Conclusion

CARD preserves task-critical contacts and forces across human demonstration sources and robot embodiments, and its tactile tracking controller supports open-loop hardware replay. The authors identify input demonstration quality and large object displacement during open-loop replay as boundaries.

5. Conclusion

  • CARD is a simulator- and object-free framework for preserving task-critical contacts and forces across demonstration sources and robot embodiments.The authors describe contact preservation as a scalable interface for transferring human dexterity to robots.Source: §5 · p. 8
  • Combined with tactile tracking, CARD enables open-loop hardware replay across rigid, articulated, and deformable objects.Source: §5 · p. 8
  • Retargeting quality is bounded by the input demonstration, and open-loop replay cannot recover from a large object displacement.The authors propose using CARD-generated data to train closed-loop sensorimotor policies.Source: §5 · p. 8

A. Unifying Vision, Glove, and Motion-Capture Demonstrations

The appendix describes how video, tactile-glove, and motion-capture inputs become shared per-link contact force targets, including force synthesis for sources without force measurements. It also reports Vid-HACO training-data and implementation details.

A. Unifying Vision, Glove, and Motion-Capture Demonstrations

  • Video, tactile gloves, and motion capture are converted into per-frame, per-link contact targets using contacting MANO vertices and source-specific weights.Glove pressure supplies weights for glove and motion-capture inputs, video uses a binary contact mask, and GRAB uses hand–object proximity.Source: A. Unifying Vision, Glove, and… · p. 13
  • The pipeline derives contact location from a smoothed weighted centroid, direction from outward surface normals, and raw force magnitude from total contact weight.For video and GRAB, which do not measure force, whole-clip grasp balance estimates force corrections; the pipeline also removes short contact blinks and synchronizes contact onsets.Source: A. Unifying Vision, Glove, and… · p. 13

B.1. Training Data and Implementation

  • Vid-HACO training uses 1.55M frames, 71% from HOT3D and ARCTIC allocentric views; its encoder, decoder, and two prediction heads total 411M parameters.Labels use the right-hand MANO mesh, and the probability threshold is 0.52.Source: §B.1 · p. 13

B.2. Evaluation Details and False Contacts

The appendix explains the video-contact evaluation and false-contact behavior, then details the closed-form geometric initializer and its relationship to the later solve.

B.2. Evaluation Details and False Contacts

  • HRDexDB measures generalization on 434 held-out egocentric videos of 94 objects, while the validation split measures in-distribution accuracy; all models use identical crops and threshold 0.52.Vertex precision, recall, and F1 are pooled over vertices and frames; frame F1 tests whether any vertex is in contact.Source: §B.2 · p. 13
  • Both HACO models over-predict contact on HRDexDB, losing 11 precision points for 3–5 recall points; Vid-HACO reaches frame F1 scores of 98.3 and 95.3 where the baselines score the always-in-contact value.Hallucinated contacts during a reach can pull robot fingers toward empty space.Source: §B.2 · p. 14

C. Geometric initialization

  • The initializer computes robot pose from the human skeleton alone in closed form once per frame, and fingertip matching seats the longer robot fingers behind the human palm.The robot fingers are about 40 mm longer, placing the palm 25–36 mm behind the human palm on video clips.Source: C. Geometric initialization · p. 14
  • For index, middle, ring, and pinky flexion hinges, the initializer copies human bend exactly by matching adjacent bone directions; abduction uses signed spread in the palm plane.The thumb instead uses an affine function of human thumb geometry because the chains and base abduction ranges differ.Source: C. Geometric initialization · p. 14
  • The deterministic initializer serves as half of the sampler’s warm start, while the task-space style reward handles unconstrained fingers; tested alternatives worsen contact metrics.The initializer costs under a millisecond per frame.Source: C. Geometric initialization · p. 15

D. Additional CARD implementation details

CARD’s implementation uses semantic maps to transfer hand-region contacts across robot embodiments, while a shared optimization and sampling setup supports the demonstration sources.

Cross-embodiment semantic maps

Pseudo-object

  • Video and glove clips use the convex hull of accumulated wrist-frame contact points as a pseudo-object collision field; tracked motion-capture clips use the tracked mesh.For GRAB, the pseudo-object supplies the collision field while the ground-truth mesh is used only for evaluation.Source: D. Additional CARD implementation… · p. 15

Sampler

  • The sampler retains the lowest-cost 5% of candidates and updates its mean using standardized elite costs, while its per-DoF noise anneals over K iterations.One solver configuration serves video, glove, and motion-capture sources.Source: D. Additional CARD implementation… · p. 15

Solver settings

E. Ablations

The ablations show how initialization, contact terms, and penetration barriers affect retargeting, while the tactile controller uses force feedback to maintain contact during replay.

E. Ablations

  • Without the initializer, the sampler still finds the contact-force target on half the frames, but worst misses grow to 3 cm and penetration to 6 mm; the earlier initializer lands 13 points below the current one.The barrier costs 5.5 points of contact while keeping deepest incursion at 4 mm rather than 6 mm.Source: E. Ablations · p. 17
  • The friction-cone term raises in-cone rates from 53–58% to 80–82%, while contact rates fall from 69.1% without the cone to 64.7% with it.The authors accept this trade because contact must also carry the demonstrated push.Source: E. Ablations · p. 17
  • A correspondence blend of αf=0.15 raises contact from 58.4 to 64.7 with the cone and from 62.7 to 69.1 without it; αf=1 over-constrains the pad.The style group reduces contact on anchored links from 64.7% to 59.9% but shapes free fingers.Source: E. Ablations · p. 17

F. Tactile tracking controller

  • The tactile controller low-pass filters force error at 10 Hz and applies a rate-limited admittance integrator with a proportional term to deflect finger flexion joints toward closure.It freezes integration during a contact-loss wait, then relaxes curl toward the retargeted trajectory after declared loss.Source: F. Tactile tracking controller · p. 17

G. Metric definitions

Contact fidelity varies with the source’s contact signal quality, and the evaluation metrics distinguish established contact, correct-region contact, force-direction alignment, gap, and penetration.

G. Metric definitions

  • Contact rate is the fraction of anchor-frames with contact inside or within 2 mm of the assigned link surface, reached by the palmar pad.Correct-region rate additionally requires the nearest palmar pad to belong to the demonstrated finger; in-cone rate requires the contact normal within the signed 45° cone around the demonstrated push.Source: G. Metric definitions · p. 18
  • Contact gap measures distance from a target to the assigned link surface, while penetration measures the deepest hand-surface incursion into the object field.Penetration is reported only for GRAB and motion-capture clips with tracked object meshes; video and glove clips have no object ground truth.Source: G. Metric definitions · pp. 18–19

H. Contact fidelity by demonstration source

  • 89.1% contact on glove, 76.1% on motion capture, 61.4% on video, and 43.1% on GRAB; kinematic baselines remain at 1–10% across sources.The reported source groupings reflect measured, inferred, proximity-thresholded, and mesh-derived contact signals, respectively.Source: H. Contact fidelity by demonstration… · p. 19

I. CARD–physics

CARD-physics replaces SPIDER’s fingertip-IK reference and contact term while retaining its physics stage; it improves task success on motion-capture and GRAB runs.

I. CARD–physics

  • CARD-physics keeps SPIDER’s physics-stage budget but initializes and tracks the CARD trajectory and substitutes CARD’s contact objective for SPIDER’s fingertip contact term.The comparison retains SPIDER’s object track and its resampling, padding, and smoothing of the kinematic stage.Source: §I · p. 19
  • 51% vs. 39% task success on motion-capture runs and 43% vs. 10% on GRAB runs, for CARD-physics versus SPIDER.Median peak object errors are 106 vs. 138 mm on motion capture and 120 vs. 274 mm on GRAB.Source: §I · p. 20
  • The rollouts show the CARD reference enclosing the object at demonstrated contact onset, so the sampler only has to squeeze rather than find the grasp from scratch.SPIDER’s IK reference forms thumb–finger opposition only after the object has started moving.Source: §I · p. 20

Related papers

Loading 2610.36539v1…