PiHand: Latent Hand-Object Interaction Priors for Physics-Based Policy Learning

1 INTRODUCTION

• Interaction generation remains challenging because subtle contact dynamics, varied object geometries, and target trajectories must jointly support natural, physically correct grasps.Kinematic approaches can exhibit physical implausibilities, while simulation-based generation must adapt grasp and contact configurations.Source: §1 · pp. 1–2
• Mocap trackers rely on dense references, while hand-only latent priors leave policies to discover contacts in an unstructured space.Action-space exploration also requires engineered rewards and curricula, is sample-inefficient, and can yield unnatural motion.Source: §1 · p. 2
• PiHand distills simulated mocap-tracking experts into an object- and contact-aware latent prior, then uses residual RL exploration to solve new tasks.Its three-stage pipeline first trains expert policies, then distills them, and finally explores the latent space.Source: §1 · p. 2
• 14.0 and 16.9 percentage points higher SR on GRAB and DexYCB, respectively, for unseen objects and trajectories using an object-tracking reward.The prior also achieves 2.0× and 4.3× the strongest-baseline SR on object catching and bimanual handover, respectively.Source: §1 · p. 2
• The paper presents a data-driven interaction prior for efficient RL and reports physically accurate tracking experts and generalization to unseen objects, trajectories, and tasks.The contributions include GRAB tracking experts and transfer to rare bimanual handovers and unseen object-catching tasks.Source: §1 · p. 2

Human and interaction motion imitation

• General motion tracking scales to diverse whole-body behaviors and hardware transfer, while camera- and teleoperation-driven systems imitate motion from visual or teleoperated inputs.The cited examples include PHC, HOVER, ExBody2, GMT, KungfuBot, H2O, TWIST, and VideoMimic.Source: §2 · pp. 2–3
• Human-motion systems extend imitation to interaction using contact-aware objectives, reusable skills, or teacher–student distillation; later loco-manipulation methods add residual refinement or generative priors.The cited examples include PhysHOI, SkillMimic, SkillMimic-V2, InterMimic, MaskedManipulator, ResMimic, DreamControl, VisualMimic, and CoorDex.Source: §2 · p. 3
• Hand–object tracking methods learn control from human references, while demonstration expansion and video-based pipelines broaden available coverage.The paper positions PiHand as supporting novel hand–object motions beyond reference tracking.Source: §2 · p. 3

Interaction motion generation

• Prior interaction-generation systems include task-specific policies and reusable motion priors, while PiHand replaces PULSE’s human-only representation with a hand–object prior modeling geometry, trajectory, and contact dynamics.The paper identifies PULSE as its closest precursor.Source: §2 · p. 3
• Object-manipulation systems use pretrained humanoid priors, goal-conditioned synthesis, masked motion inpainting, or task tokenization, but many still rely largely on human-only priors.The paper contrasts these with interaction priors and kinematic methods whose physical plausibility is limited without simulation.Source: §2 · p. 3

3 METHOD

• The method uses three stages: train per-object experts, distill their behavior into an interaction prior, and explore the prior to generate interactions and solve novel tasks.The overview describes expert tracking in simulation followed by behavior-cloning distillation into a VAE-like student.Source: §3 · p. 3, Figure 2 · p. 4

3.1 STAGE I: PHYSICS-GROUNDED HAND–OBJECT INTERACTION TRACKING

• Stage I trains simulated policies to track mocap interactions, producing physically valid data because mocap records kinematics but not contact forces or dynamics.The policy integrates residual corrections to a reference pose over time, smoothing control and constraining deviations to joint limits.Source: §3.1 · p. 4
• Stage I tracking combines object-surface, contact, and hand-keypoint losses; the contact penalty targets missed reference contacts because mocap contact maps are noisy.Per-object experts use separate policies for single- and bimanual interactions, early termination, and hard-mining of difficult sequences.Source: §3.1 · p. 5

3.2 STAGE II: INTERACTION PRIOR DISTILLATION

• Stage II distills privileged expert behavior into a goal-conditioned VAE whose prior uses future object trajectories, while its encoder also receives reference hand features.Behavior cloning matches decoder PD targets to expert targets, and KL divergence aligns the partial-observation prior with the privileged encoder.Source: §3.2 · pp. 5–6

3.3 STAGE III: INTERACTION GENERATION VIA BOUNDED LATENT EXPLORATION

• Stage III uses PPO to explore residuals around the frozen prior’s mean, bounding latent changes to preserve proximity to the learned interaction space.The prior’s latent space is described as improving sample efficiency and suppressing unnatural motion across manipulation tasks.Source: §3.3 · p. 6

4 EXPERIMENTS

• Evaluation uses 100 simulated rollouts per sequence and reports completion, full-sequence success, object pose errors, and imitation MPJPE.The evaluation includes GRAB and DexYCB, with the interaction prior trained on GRAB.Source: §4 · p. 6

4.1 HAND–OBJECT INTERACTION IMITATION

• Per-sequence experts outperform ManipTrans on all metrics; two per-object experts scale to full GRAB and still exceed per-sequence ManipTrans in success.The scalable full-dataset experts have slightly lower success than the per-sequence experts.Source: §4.1 · p. 6, Table 1 · p. 7

4.2 TABLETOP MANIPULATION

• 14.0 and 16.9 percentage points higher success rate on GRAB and DexYCB, respectively, while using 1/10 of the training time to reach peak baseline performance.The interaction prior is trained only on GRAB; the tabletop policies are evaluated on unseen objects and trajectories and also reduce object pose errors.Source: Figure 3 · p. 7, Table 2 · p. 8, §4.2 · p. 8

4.3 GENERALIZATION TO NEW TASKS

• 2.0× and 4.3× the strongest baseline’s success rate on object catching and bimanual handover, respectively.Catching uses thrown objects; handover requires moving an object from the left hand to the right under workspace constraints.Source: Table 3 · p. 8, §4.3 · pp. 8–9

4.4 ABLATIONS

• 30.8% success without exploration; action-space residuals reach 49.6%, while latent exploration more than doubles frozen-prior performance.Trajectory augmentation adds about six points. B=5 retains prior-distributed latents and reference-like hand use; the policy also retains the highest success rate under tested perturbations.Source: Table 4 · p. 9, §4.4 · p. 9

5 CONCLUSION

• PiHand’s interaction prior achieves 14.0 and 16.9 percentage points higher success rates than the strongest baselines on unseen GRAB and DexYCB objects, respectively.The policies learn roughly ten times faster; the prior also transfers to catching and handover, reaching 2.0× and 4.3× the strongest-baseline success rates, respectively.Source: §5 · pp. 9–10
• The framework trains mocap-tracking experts, distills them into a conditional VAE interaction prior, then explores its latent space to solve novel tasks.The prior jointly models hand motion, object geometry, and contact dynamics, conditioned on target object trajectories.Source: §5 · p. 9
• Tracking experts fall short of 100% success on full GRAB, with failures concentrated in extremely dynamic motions and heavily noisy mocap sequences.The framework focuses on rigid objects and hands only, and experiments use a single MANO-based hand model in simulation.Source: §5 · p. 10

A.1 SHARED DETAILS

• All stages use future object-trajectory frames, while Stage II uses a shorter look-ahead to reduce latent-prior overfitting.The shared setup disables within-hand rigid-body collisions and uses approximate finger-cylinder self-penetration checks for termination.Source: §A.1 · p. 16

A.2 STAGE I: EXPERT TRAINING

• Stage I combines keypoint tracking that shifts from wrist-local to object-centric weighting near contact with failure-proportional sequence sampling.The weighting uses time until the next reference-mocap contact; hard-mining samples difficult sequences more frequently.Source: §A.2 · p. 16

A.3 STAGE II: DISTILLATION

• The actor is a 3-layer MLP and critic a 2-layer MLP; removing velocities, using future poses relative to current state, and shorter look-ahead improved held-out direct-prior-rollout success from 23.7% to 30.8% versus a full-observation prior.The encoder, prior, and decoder share a 64-dimensional latent space, and the encoder mean is parameterized as a residual on the prior mean.Source: §A.2 · p. 16, §A.3 · p. 16

A.4 STAGE III: LATENT EXPLORATION

• Stage III trains an MLP exploration policy with bounded latent residuals and, for tabletop manipulation, only an object surface-sample reward.Episodes terminate when the object deviates more than 10 cm from its reference position.Source: §A.4 · p. 17

B.1 DATASETS AND PREPROCESSING

• GRAB provides the primary demonstrations; preprocessing trims interactions, mirrors sequences, and yields 4,854 training sequences averaging 317.4 frames at 60 Hz.The experiments use 50 objects, excluding the doorknob, and evaluate on a cross-object split.Source: §B.1 · p. 17
• DexYCB evaluation follows unseen-object setting S3, training on 15 objects and holding out five objects across 64 evaluation sequences.The 250 right-handed source sequences cover 20 objects; the processed sequences average 196.4 frames at 60 Hz.Source: §B.1 · pp. 17–18

B.2 SIMULATION SETUP AND MANO INTEGRATION

• The simulated hand uses a single mean-shape MANO template with collision geometry segmented from its mesh, preserving direct correspondence to the visual hand representation.Stage I learns to compensate for morphology differences across the 10 GRAB subjects; simulated objects use density 500 kg/m3 and friction coefficient 0.7.Source: §B.2 · p. 18

B.3 TRAINING TIMES

• Training takes up to 6 hours per Stage I expert, 24 hours for distillation, and up to 36 hours for tabletop policies.Catching and handover policies are trained for 20 hours.Source: §B.3 · p. 18

C.1 ACTION SPACE EXPLORATION; C.2 HAND-ONLY PRIOR

• The action-space baseline explores 102-dimensional PD-target residuals without a learned demonstration prior, while the hand-only baseline explores a proprioceptive motion-only latent space.The latter follows the motion-only prior paradigm used by Omnigrasp (Luo et al., 2024a).Source: §C.1 · p. 19, §C.2 · p. 19

C.2.1 HAND-ONLY IMITATION EXPERTS

• Hand-only imitation experts track GRAB hand kinematics while ignoring objects, with keypoint-distance reward and early termination at self-penetration or average keypoint error above 2.5 cm.The dataset is split into four subsets, with one expert trained per subset.Source: §C.2.1 · p. 19

C.2.2 DISTILLATION INTO A HAND-ONLY LATENT SPACE

• The hand-only latent space removes object information from the conditional VAE inputs, conditioning its prior and decoder only on the simulated hand state.The encoder additionally receives future reference hand kinematics during training.Source: §C.2.2 · p. 19

C.2.3 LATENT EXPLORATION

• The hand-only exploration baseline adds residuals to a motion-only prior, restricting exploration to human-like motions while leaving the decoder without explicit interaction awareness.The exploration policy still receives object-aware observations and target trajectories.Source: §C.2.3 · pp. 19–20

C.3 ADDITIONAL REWARDS AND CURRICULUM FOR TABLETOP MANIPULATION

• PiHand trains tabletop manipulation with only the object surface-sample reward and no curriculum, whereas baselines needed added pre-grasp rewards and a three-level curriculum.Without those additions, neither baseline learned meaningful behavior within six hours; the pre-grasp reward is used only during training, not as an inference input.Source: Table 7 · p. 20, §C.3 · p. 20

D.1 COMPARISON WITH MANIPTRANS

• PiHand reaches ManipTrans’s peak success rate after 175 epochs and ultimately converges to higher success rate and spatial precision.Both methods were trained for 1,000 epochs and evaluated with a strict 5 cm object-translation threshold.Source: §D.1 · p. 20, Table 8 · p. 21, Figure 6 · p. 21

D.2 TABLETOP MANIPULATION

• On held-out GRAB objects, PiHand’s tabletop-manipulation gains over baselines are consistent across seeds, with by far the lowest variance.Table 9 reports mean and standard deviation over three random seeds; SR and TC use a 10 cm threshold.Source: §D.2 · p. 21, Table 9 · p. 21

D.3 NOVEL TASKS: COMMON SETUP

• The shared setup specifies tasks through object-reference trajectories and optimizes tracking rewards with a small number of task-specific terms.Neither task uses a physical table; the hands float in free space.Source: §D.3 · p. 21

D.4 CATCHING

• Catching references use simulator-integrated ballistic arcs through randomized intercept points, followed by at least 60 frames holding the object there.The evaluation uses 20 deterministic episodes per object under an equal wall-clock training budget.Source: §D.4 · p. 22
• The residual wrist-speed penalties and reduced fingertip-proximity reward counter early one-hand riding during the pre-intercept flight.The behavior arose under tracking reward alone because the prior data keeps a hand near the object and proximity reward applies throughout flight.Source: §D.4 · p. 22
• On held-out objects, bimanual catching succeeds in 87.0% of episodes versus 43.0% for action-space exploration; the hand-only prior almost never succeeds.The right-hand-only variant reaches 78.0%, but is a different, harder task.Source: §D.4 · p. 22, Table 10 · p. 22

D.5 HANDOVER

• Handover begins from a left-hand grasp and moves the object across the workspace; wrist zones prevent either hand from carrying it alone from start to goal.Five training objects without left-hand grasp frames are excluded.Source: §D.5 · p. 23
• Held-out handover evaluation uses only the 160 of 1,442 starting frames that keep the object within 5 cm of its initial position for 30 frames without policy action.All methods receive identical start states; the screening separates handover from initially establishing a stable grasp.Source: §D.5 · p. 23, Table 11 · p. 23

• All experiments in this section evaluate Tabletop Manipulation on GRAB.Source: E ROBUSTNESS AND ABLATIONS · p. 23

E.1 ROBUSTNESS TO PHYSICAL PERTURBATIONS

• At four times nominal mass, PiHand loses 11.7% of its success rate, versus 67.2% for the hand-only prior and 52.8% for action-space exploration.Changing friction affects all methods similarly; higher friction makes grasping easier for every method, and PiHand is also most robust to action noise.Source: §E.1 · p. 24, Table 12 · p. 24

E.2 SENSITIVITY TO THE TERMINATION THRESHOLD

• From 10 cm to 5 cm termination thresholds, PiHand's success rate drops 10.5 points, versus 13.4 for the hand-only prior and 21.9 for action-space exploration.The gap to the strongest baseline grows from 14.0 to 16.9 points.Source: §E.2 · p. 24, Table 13 · p. 24

E.3 STAGE III ABLATIONS

• Stage III ablations compare direct prior rollout, action-space residuals, and training without randomized trajectories against the full policy.The two trained variants use the same observations, reward, and PPO settings; Table 14 reports the held-out-object comparison.Source: §E.3 · p. 25, Table 14 · p. 25

E.4 RESIDUAL BOUND

• With B=5, the policy retains most of the unbounded success rate while keeping hand usage close to the human reference; B≥10 brings latent drift and near-exclusive bimanual grasps.Success rises as the bound is relaxed, but improves only marginally beyond B=10.Source: §E.4 · p. 25