PaperlayerView exact revision on arXiv

Source-linked AI summary

SPIDER: Scalable Physics-Informed Dexterous Retargeting

Chaoyi Pan, Changhao Wang, Haozhi Qi, Zixi Liu, Homanga Bharadhwaj, Akash Sharma, Tingfan Wu, Guanya Shi, Jitendra Malik, Francois Hogan

arXiv:2511.09484v2cs.ROcs.CV

TL;DR

Robot-specific demonstrations are costly to collect, and human motion lacks the dynamics and contact information needed for direct transfer across embodiments. SPIDER retargets human demonstrations using physics-based sampling and virtual contact guidance; it improves success rates by 18% compared to the baseline and is an order of magnitude faster than RL-based methods.

  • Problem

    Robot-specific whole-body manipulation data is expensive to acquire, while human motion lacks the dynamics and contact information needed for direct transfer across embodiments.

    Source: §1 · pp. 1–2

  • Method

    SPIDER uses human demonstrations for task guidance and physics-based sampling with gradually relaxed virtual contact guidance to refine robot trajectories.

    Source: §1 · p. 2

  • Results

    SPIDER improves success rate by 18% compared to the baseline and increases retargeting speed by an order of magnitude compared to RL-based methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The pipeline generates robot-feasible datasets from human data, including 2.4M frames across five robotic hands and 103 objects.

    Source: §1 · p. 3

    Performance depends on reconstructed 3D human-object interaction data quality; noisy meshes and motion can yield degraded trajectories.

    Source: §6 · p. 13

Abstract

from arXiv · show

Learning dexterous and agile policy for humanoid and dexterous hand control requires large-scale demonstrations, but collecting robot-specific data is prohibitively expensive. In contrast, abundant human motion data is readily available from motion capture, videos, and virtual reality, which could help address the data scarcity problem. However, due to the embodiment gap and missing dynamic information like force and torque, these demonstrations cannot be directly executed on robots. To bridge this gap, we propose Scalable Physics-Informed DExterous Retargeting (SPIDER), a physics-based retargeting framework to transform and augment kinematic-only human demonstrations to dynamically feasible robot trajectories at scale. Our key insight is that human demonstrations should provide global task structure and objective, while large-scale physics-based sampling with curriculum-style virtual contact guidance should refine trajectories to ensure dynamical feasibility and correct contact sequences. SPIDER scales across diverse 9 humanoid/dexterous hand embodiments and 6 datasets, improving success rates by 18% compared to standard sampling, while being 10X faster than reinforcement learning (RL) baselines, and enabling the generation of a 2.4M frames dynamic-feasible robot dataset for policy learning. As a universal physics-based retargeting method, SPIDER can work with diverse quality data and generate diverse and high-quality data to enable efficient policy learning with methods like RL.

1 Introduction

Robot-specific demonstrations are costly, while abundant human motion cannot transfer directly because of embodiment differences and missing dynamics and contact information. SPIDER combines human task guidance with physics-based sampling and virtual contact guidance to generate feasible robot trajectories at scale.

1 Introduction

  • Robot-specific whole-body manipulation data is costly to collect, whereas human motion is abundant in motion-capture, video, and VR sources.Recent vision methods can reconstruct human and hand motion, object meshes, and trajectories from video.Source: §1 · p. 1
  • The embodiment gap—mismatches in morphology, dynamics, and actuation—makes direct transfer infeasible, while human datasets often lack force and contact data.IK can be efficient but dynamically infeasible; RL requires costly trajectory-specific training and reward engineering; teleoperation is labor-intensive and embodiment-specific.Source: §1 · p. 2
  • SPIDER combines human-demonstration guidance with physics-based sampling and virtual contact guidance to produce dynamically feasible robot trajectories with correct contacts.The method uses human motion for global task structure and simulation sampling to refine trajectories.Source: §1 · p. 2
  • The framework spans six datasets, nine robot morphologies, and two task domains: dexterous hands and humanoids.Its contact-aware scheme improves success rate by 18% compared to the baseline and increases retargeting speed by an order of magnitude compared to RL-based methods.Source: §1 · p. 2
  • SPIDER’s pipeline generates robot-feasible datasets from human data, including 262 episodes, 800 hours, and 2.4M frames across five robotic hands and 103 objects.The authors release the full data-generation pipeline.Source: §1 · p. 3
  • The framework is also extended to robustify trajectories for real-robot deployment, augment demonstrations across environments and objects, and support RL policy learning.The pipeline converts human-object interaction trajectories into robot-object trajectories and supports multiple embodiments and task domains.Source: Figure 1 · p. 2, §1 · p. 3

2 Physics-based Retargeting with Sampling

SPIDER formulates retargeting as optimizing robot controls toward a human-derived reference under dynamics, then uses annealed simulation sampling to find feasible trajectories. Virtual contact guidance addresses contact ambiguity, while robustification and physics-aware augmentation extend the approach to uncertain dynamics and varied scenarios.

2.1 Physics-based Retargeting Problem

  • SPIDER optimizes robot controls to minimize distance from a human-derived reference trajectory and control effort, subject to the robot’s state-transition dynamics.The reference state sequence comes from human demonstrations, and the objective weights state and control inputs.Source: §2.1 · p. 3

2.2 Sampling for Physics-based Retargeting

  • Annealed sampling shifts search from broad exploration to targeted refinement, with early stopping when further sampling yields little improvement.Its schedule controls sampling covariance across optimization iterations and the prediction horizon.Source: §2.2 · p. 4

2.3 Virtual Contact Guidance

  • Sampling can reproduce an object trajectory yet select an alternative, implausible contact mode instead of the human-preferred one.For example, a stick can be held between different finger pairs while still completing the task.Source: §2.3 · pp. 4–5

Virtual Contact Guidance.

  • Virtual contact guidance maintains relative positions between intended object-hand contact pairs, enlarging the sampling basin and biasing optimization toward the target configuration.A contact filter disables constraints for contacts that are too brief or whose contact point drifts too far.Source: Virtual Contact Guidance · p. 5

2.4 Trajectory Robustification

  • Trajectory robustification minimizes worst-case cost over bounded dynamics variations, including contact margin, friction, and object mass.Candidate controls are evaluated across a mini-batch of sampled dynamics parameters.Source: §2.4 · p. 6

2.5 Physics-based Data Augmentation

  • Starting from one human demonstration, SPIDER can generate diverse physically feasible actions for downstream training.It can vary object geometry, size, mass, terrain, or external forces to produce adapted behaviors.Source: §2.5 · p. 6, Figure 4 · p. 6
  • Under external pulling forces of 120 N and 240 N, the retargeted robot trajectory resists by leaning forward and moving more slowly.The reported example uses a heavy object with gravity of 320 N.Source: §2.5 · pp. 6–7

3 Performance Evaluation

The evaluation tests SPIDER across varied datasets and robot embodiments, then compares sampling variants and retargeted trajectories on dexterous and humanoid tasks. Results cover tracking, generation speed, and physical execution.

3.1 Experimental Setup

  • The setup spans three dexterous datasets, three humanoid datasets, five hand embodiments, and four humanoid embodiments, with evaluation metrics tailored to object motion or locomotion.Inspire and Ability use simulated 12-DoF configurations without the joint constraints of their real-world versions.Source: §3.1 · p. 7, Figure 5 · p. 7

3.2 Ablation Study

  • Adding virtual contact guidance to annealed-kernel sampling raises success rates by approximately 18% across robot–dataset combinations, despite running at 2.5 Hz versus 3.0 Hz without guidance.The full method combines annealing with virtual contact guidance; the annealed-kernel-only version is faster.Source: §3.2 · p. 8, Table 1 · p. 8

3.3 Dexterous Manipulation Retargeting Results

  • Full-dataset retargeting success varies by dataset and hand: GigaHands pick-and-place motions are more amenable than Oakink pre-grasps, while higher-DoF hands are generally easier to retarget.The evaluation spans five robotic hands and the full dexterous datasets.Source: §3.3 · p. 8, Table 2 · p. 9
  • On Oakink, SPIDER is competitive with policy-learning baselines, but it lags on more complex ARCTIC manipulations while generating trajectories an order of magnitude faster.The comparison evaluates generated trajectories rather than policies; ManipTrans results use its released dataset because its specific evaluation trajectories were not reported.Source: §3.3 · p. 9, Table 3 · p. 10
  • On a physical Franka Panda with an Allegro hand, retargeted trajectories execute successfully on four precise manipulation tasks without additional adaptation beyond robustification.The tasks include rotating a light bulb, manipulating a spoon, playing guitar, and unplugging a charger.Source: §3.3 · p. 9, Figure 7 · p. 10

3.4 Humanoid Whole-Body Control Retargeting Results

  • SPIDER achieves the lowest tracking errors and highest FPS versus GMR across all humanoid datasets.The comparison covers joint, end-effector position and orientation, and object tracking errors.Source: Table 4 · p. 10, §3.4 · p. 10

4 Applications

SPIDER is applied to noisy single-camera demonstrations and used to prepare humanoid references for reinforcement-learning policy training. The examples show physical execution from camera-derived data and improved policy training with retargeted references.

4.1 Retargeting from Single RGB Camera

  • Physics grounding corrects noise, artifacts, and penetrations in single-camera demonstrations, producing trajectories executed on physical robots in cup-pouring and bolt-installation tasks.The input pipeline reconstructs 3D meshes, estimates hand pose, and tracks object pose from a single RGB camera.Source: §4.1 · p. 11, Figure 9 · p. 11

4.2 Retargeting for RL Policy Training

  • On OMOMO, a policy trained with SPIDER-retargeted motion converges faster and tracks the object better than one trained from human motion missing object contact.With the original motion, the robot fails to grasp the object while still achieving its body-tracking goal.Source: §4.2 · pp. 11–12, Figure 10 · p. 12

5 Related Work

Prior work transfers human demonstrations through kinematic mappings or human-centric policies adapted with in-domain data. Retargeting methods vary in their trade-offs between efficiency, physical fidelity, and scalability.

5.1 Learning Manipulation from Human Data

5.2 Retargeting from Human Data

  • Kinematic methods are efficient but physics-inconsistent; learning-based methods can infer fast mappings but need extensive pretraining and may fail on novel motions or embodiments.Optimization and RL can produce physically plausible motion, but often require detailed contact data, strong priors, or substantial computation.Source: §5.2 · pp. 12–13
  • SPIDER aims to combine RL's generality with optimization-based efficiency as a scalable human-to-robot transfer alternative.Existing methods trade off efficiency against physical fidelity.Source: §5.2 · p. 13

5.3 Sampling-based Optimization for Robot Control

  • Sampling-based optimization can handle non-convex, non-smooth control problems, but contact guidance helps preserve intended contact sequences amid solution ambiguity.SPIDER uses sampling to optimize control sequences directly rather than updating a policy network.Source: §5.3 · p. 13

6 Conclusion and Future Work

SPIDER generates robot demonstrations from human data through flexible, efficient physics-based retargeting, with performance competitive with state-of-the-art methods. Its performance depends on the quality of reconstructed 3D human-object interaction data.

6 Conclusion and Future Work

  • SPIDER achieves competitive performance against state-of-the-art methods while running an order of magnitude faster.Its performance depends on reconstructed 3D human-object interaction quality; noisy meshes and motion can degrade trajectories.Source: §6 · p. 13
  • Noisy reconstructed meshes or motion can yield degraded trajectories, limiting performance with lower-quality 3D human-object interaction data.Source: §6 · p. 13
  • SPIDER enables large-scale robot demonstration generation from human data.Source: §6 · p. 13
  • The authors identify behavior cloning as a promising application for unlocking generalizable dexterous manipulation policies.Source: §6 · p. 13

A.1 Preprocessing

Preprocessing represents each hand with MANO-derived fingertip and wrist keypoints, maps them to robot joints with inverse kinematics, then resamples and filters trajectories.

A.1 Preprocessing

  • Each hand is represented by 21D keypoints comprising 3D fingertip positions and a 6D wrist pose from MANO.Source: §A.1 · p. 18
  • An inverse-kinematics solver maps the keypoints to robot-specific joint positions by minimizing an IK objective.The supplied passage gives the objective only partially as LIK = Pn F.Source: §A.1 · p. 18
  • Trajectories are resampled at 50 Hz and low-pass filtered at 10 Hz.Source: §A.1 · p. 18

A.2 Hyperparameters and Setup

Experiments use a fixed sampling configuration, repeat each experiment across five seeds, and run across multiple simulators with specified physics and control rates.

A.2 Hyperparameters and Setup

  • Each experiment is repeated five times with different seeds.Source: §A.2 · p. 18
  • The setup uses horizon H = 1.2 s, 1024 particles, β1 = 0.85, β2 = 0.9, 16 iterations, and annealing ηt = η0 · 1.1t with η0 = 0.01.Source: §A.2 · p. 18
  • The method supports MuJoCo Warp, MJX, Genesis, and Isaac Gym; simulations run at 100 Hz physics and 50 Hz control in MuJoCo Warp or Genesis.Most datasets use MuJoCo Warp, while ARCTIC uses Genesis.Source: §A.2 · p. 18
  • Ablations and speed tests use RTX 4090 GPUs, while dataset generation uses H100 GPUs.Source: §A.2 · p. 18

A.3 Retargeting for RL Policy Training

The RL policy-training details specify PPO, tracking and action-rate rewards, removal of contact-related rewards and penalties, and error-based termination.

A.3 Retargeting for RL Policy Training

  • The RL policy uses PPO, with an implementation ported from Weng et al. (2025).Source: §A.3 · p. 18
  • Rewards track humanoid pelvis, hand, and leg motion and object motion, while penalizing action rate to avoid jittering.Source: §A.3 · p. 18
  • Contact reward, self-collision penalty, and the auxiliary contact reward are removed to evaluate reference-motion retargeting quality.Source: §A.3 · p. 18
  • Training terminates when average body tracking error and average object tracking error are each greater than 10 cm.Source: §A.3 · p. 18
Loading 2511.09484v2…