ProxiDex: Learning Dynamics-Guided Proximity Policy for Dexterous Manipulation

1 Introduction

• Multi-fingered manipulation requires coordinated arm motion and fingertip interactions, yet occlusion and tactile calibration burdens make local interaction cues difficult to acquire reliably.These cues describe how contact forms, transfers, and is maintained during manipulation.Source: §1 · p. 2
• Existing imitation policies often omit explicit hand-object proximity and its temporal evolution, leaving them brittle under occlusion, tracking noise, and contact uncertainty.Even when contact signals are included, they are often auxiliary inputs rather than action-conditioned interaction states.Source: §1 · p. 2
• ProxiDex reconstructs interaction point clouds and converts distances into hardware-agnostic proximity feedback for VR teleoperation without tactile instrumentation.The framework learns forward-inverse latent dynamics linking action-driven observation transitions with proximity variations.Source: §1 · p. 2
• ProxiDex adaptively reweights proximity tokens across manipulation phases and uses dynamics-consistency-guided inference to improve robustness under occlusion and tracking failures.Its contributions combine proximity-feedback data collection, action-conditioned latent dynamics, and trajectory-adaptive policy learning.Source: §1 · p. 2

Demonstration data acquisition for dexterous manipulation

• Existing data-collection systems trade sensing accuracy against deployment cost, while vision-based teleoperation can miss local proximity under hand-object occlusion.The paper describes its VR co-teleoperation pipeline as recovering interaction structure through geometric proximity for scalable, interaction-aware collection.Source: §2 · pp. 2–3

Imitation learning for dexterous manipulation

• Imitation policies based on 2D observations, point clouds, or reconstructed geometry remain limited in capturing consistent interaction states and their temporal evolution.These limitations motivate an explicit proximity-based interaction representation.Source: §2 · p. 3

Dynamics-aware representation learning

• Prior dynamics approaches predict future states or geometry, but the paper identifies limited explicit modeling of bidirectional geometric-proximity evolution under actions.ProxiDex models interaction as this bidirectional evolution to reason explicitly about contact dynamics.Source: §2 · p. 3

3 Problem Formulation

• Imitation learning trains a policy to match expert behavior by mapping encoded 3D point-cloud observations to an action distribution.The trajectories contain observation and expert-action sequences, and the encoder maps observations to latent representations.Source: §3 · p. 3
• A latent dynamics model predicts a future observation latent from the current latent and a sequence of future expert actions, using feature reconstruction loss.This provides auxiliary supervision for the policy’s awareness of future physical evolution.Source: §3 · p. 3
• Observation-latent prediction alone can miss occluded fingertip regions and prioritize salient object motion over fine-grained distance and contact-state changes.The paper therefore reconstructs interaction geometry and introduces forward dynamics with an inverse differential module to anticipate proximity evolution and support reliable actions under perturbation.Source: §3 · p. 3

4.1 Interaction-Aware Data Acquisition

• Demonstrations combine interaction point clouds, region-level proximity, phase labels, robot state, and expert actions; the point clouds are reconstructed from hand–object geometry.VR teleoperation records robot states, target-object poses, and expert actions; local distances between hand and object surfaces produce region-level proximity cues.Source: §4.1 · p. 4

4.2 Latent Interaction Dynamics Modeling

• A forward model predicts future observation latents from actions, while an inverse module decodes proximity latents from latent changes and aligns them with proximity representations.The learned observation encoder captures action-driven transitions and proximity changes; a trajectory head also predicts interaction-phase probability.Source: §4.2 · p. 5

4.3 Trajectory-Adaptive Proximity Policy

• The policy combines globally available observation features with causally masked proximity tokens, strengthening the proximity branch during interaction phases and suppressing it during motion.A gate conditioned on interaction-phase probability regulates the proximity branch during policy learning.Source: §4.3 · p. 5

4.4 Dynamics-Consistency Guided Policy Inference

• During online inference, a forward dynamics model checks predicted against observed latent states; when deviation exceeds a threshold, autoregressive predictions sustain action generation until feedback is reliable again.Actions are generated and executed in a receding-horizon manner.Source: §4.4 · pp. 5–6

5.1 Simulation Experiments

• 83.9% overall success rate leads the simulation benchmark, exceeding DP3, ManiFlow, AFRO, and CordViP by 12.1, 9.7, 7.0, and 2.9 percentage points, respectively.ProxiDex also leads each of the Adroit, DexArt, and customized IsaacLab task groups.Source: §5.1 · p. 7
• Across six representative Adroit and DexArt tasks, ablations show that removing proximity cues, interaction point clouds, FDM, IDM, or closed-loop checking reduces success.Drops are especially clear on contact-sensitive Bucket and Pen when interaction geometry is removed.Source: §5.1 · p. 7, Figure 4 · p. 7
• Under 2-s and 4-s Pen occlusions, full ProxiDex achieves 80% and 72% recovery; accumulated prediction errors reduce recovery to 35% under 6-s occlusion.Removing FDM causes the largest drop, while without IDM recovery is 71% and 59% at 2 s and 4 s.Source: §5.1 · p. 7, Figure 5 · p. 7

5.2 Real-World Experiments

• 72.6% overall average success leads the real-robot benchmark, with ProxiDex best or tied-best in nine task settings.It outperforms CordViP, AFRO, ManiFlow, and DP3 by 9.7, 19.1, 22.9, and 32.6 percentage points, respectively.Source: §5.2 · pp. 7–8, Table 2 · p. 8
• ProxiDex succeeds in 85/120 unseen-object trials and 67/90 perturbation trials, compared with CordViP’s 77/120 and 56/90, respectively.Visible-scene point-cloud methods DP3 and ManiFlow degrade more clearly in Sweep under hand occlusion and object pose changes.Source: §5.2 · p. 8
• The full ProxiDex improves average success by 7.4 and 4.5 percentage points over its U-Net and standard Transformer variants, respectively.Interaction-phase probability adaptively adjusts attention between proximity and point-cloud tokens; Figure 7 visualizes averaged attention weights across three tabletop tasks.Source: §5.2 · pp. 8–9, Figure 7 · p. 8
• In two contact-rich tasks, ProxiDex maintains stable interaction representations and completes in-hand manipulation under short-term visual occlusion, leading both tasks.Interaction point clouds provide local geometry, while dynamics prediction preserves interaction-state continuity during unreliable visual feedback.Source: §5.2 · p. 9, Figure 8 · p. 9
• Proximity achieves competitive task performance against P1L fingertip tactile arrays, while capturing hand–object distance and spatial-distribution changes around contact.The reported role is geometric cueing for pre-grasp adjustment and contact establishment, not force feedback.Source: §5.2 · p. 9

• ProxiDex constructs region-level proximity representations from reconstructed hand–object interaction point clouds and learns action-conditioned latent interaction dynamics.These dynamics guide trajectory-adaptive policy learning and closed-loop inference.Source: §6 · p. 9
• The combined geometry and dynamics support stable policy execution under short-term unreliable visual feedback, sustained contact, and complex object interactions.Source: §6 · p. 9
• Simulation and real-world experiments report consistent performance across standard, unseen-object, disturbance, visual-occlusion, and contact-rich settings.Source: §6 · p. 9

• Accurate object reconstruction and real-time pose tracking may be unreliable for small components, articulated objects, or high-precision manipulation.Source: §7 · p. 9
• Dynamics takeover is mainly effective for short-term perception failures, while prolonged occlusion can accumulate autoregressive prediction errors and reduce control reliability.Source: §7 · p. 9
• The geometry-driven proximity representation cannot measure real contact forces or precise fingertip force distributions.Source: §7 · p. 9

A.1 VR Teleoperation and Action Retargeting

• Hand retargeting normalizes 21 VR keypoints into a canonical hand frame and fits robot joints to target finger-geometry flows with noise robustness and temporal smoothness.A DexPilot-style grasp prior projection with threshold hysteresis improves fingertip closure while reducing jitter.Source: §A.1 · p. 13
• OpenXR hand keypoints and wrist motion support hand–arm teleoperation, while reconstructed interaction point clouds return to the headset as optional proximity feedback.The loop transmits hand-joint commands at approximately 30 Hz, detects object pose at around 10 Hz, and updates interaction point clouds at about 23 Hz.Source: Figure 10 · p. 14, §A.1 · p. 14

A.2 Object Reconstruction and Metric-Scale Calibration

• The object-reconstruction pipeline completes invisible regions with a SAM3D mesh, then calibrates its scale against an RGB-D point cloud before distance calculations.The median of sorted bounding-box axis ratios provides the uniform scale coefficient and reduces sensitivity to a noisy dimension.Source: §A.2 · p. 14

A.3 Geometric Proximity Representation

• Proximity cues derive from hand–object point-cloud distances and aggregate by physical hand region; trajectory-stage labels provide a coarse phase indicator for dynamics learning.The representation preserves relative hand–object geometry while reducing background-point and self-occlusion influence.Source: §A.3 · pp. 15–16

B Training and Inference Details of ProxiDex

• ProxiDex trains a proximity-aware dynamics representation first, then freezes it while training a diffusion policy that adaptively modulates proximity conditioning.The design emphasizes global geometry during approach and local proximity during dexterous manipulation.Source: B Training and Inference Details of… · p. 16

B.1 Latent Interaction Dynamics Architecture

• The forward dynamics model predicts future observation features from current latents and actions, while the inverse differential module maps latent changes to proximity latents aligned with VAE targets.A trajectory-stage head predicts interaction probability, completing a physically meaningful interaction representation for downstream control.Source: §B.1 · pp. 16–17

B.2 Trajectory-Adaptive Proximity Policy Learning

• The diffusion policy uses observation latents for global approach and strengthens its proximity branch as predicted interaction probability rises.The gating scheme regulates local proximity information while retaining observation latents as stable global context.Source: §B.2 · pp. 17–18

B.3 Dynamics-Consistency Guided Policy Inference

• At inference, the forward model compares predicted and encoded next-state latents; above a threshold, ProxiDex temporarily uses autoregressive predicted latents until observations realign.The system treats large deviations as potentially caused by occlusion, pose drift, or short-term tracking failure.Source: §B.3 · p. 19

C.1 Simulation Setup and Task Description

• Evaluation spans ten simulation tasks across Adroit, DexArt, and custom IsaacLab settings, covering dexterous, articulated-object, and self-designed manipulation.The custom tasks include box transport, drill alignment, and fingertip pinching; methods share the demonstration split and evaluation protocol.Source: §C.1 · p. 20

C.2 Simulation Implementation Details

• 83.9% ten-task average success rate exceeds DP3 by 12.1, ManiFlow by 9.7, AFRO by 7.0, and CordViP by 2.9 percentage points.The improvement is consistent across Adroit, DexArt, and self-designed tasks.Source: §C.2 · p. 20, Table 6 · p. 21
• Qualitative examples contrast baseline failures with ProxiDex’s finger-closure adjustment, local drill alignment under occlusion, and suitable fingertip regions for pinching.The comparisons cover Box, Driller, and Banana tasks.Source: Figure 12 · p. 21, §C.2 · p. 21

D.1 Teleoperation System Comparison

• 75% of 20 participants preferred ProxiDex, which reduced average task completion time to 26.6 ± 5 s compared with the other two teleoperation systems.Participants trained for 10 minutes and collected 30 demonstrations under each setting.Source: §D.1 · p. 21, Figure 13 · p. 22

D.2 Ablation Study

• Removing interaction point clouds lowers average success from 77.6% to 70.1%, while removing proximity cues lowers it to 70.6%.On Bucket, these removals reduce success by 15.1 and 13.4 points; on Pen, removing interaction point clouds reduces it by 8.4 points.Source: §D.2 · p. 22
• Removing FDM or IDM lowers average success to 73.6% or 75.3%, while U-Net and standard Transformer backbones reduce success by about 3.1 and 2.8 points.The tested dynamics horizon Hdyn = 2 achieves 77.6% average success; the authors use it to balance local contact sensitivity and prediction stability.Source: §D.2 · pp. 22–23, Figure 14 · p. 23, Table 8 · p. 23

D.3 Data Scalability Evaluation

• ProxiDex reaches approximately 11.5%, 40.0%, 64.0%, 77.5%, 80.0%, and 81.8% average success with 5, 10, 20, 30, 50, and 100 demonstrations, respectively.Thirty demonstrations recover about 94.7% of the 100-demonstration performance; ProxiDex has the highest average success across the tested data scales.Source: §D.3 · p. 24, Figure 15 · p. 24

D.4 Attention Visualization in Simulation

• Point-cloud attention is more active during early motion and pose adjustment, while proximity attention modulates near-field interactions after contact becomes likely.Trajectory probability typically rises before contact or during early contact establishment, and the two attention branches adaptively balance global geometry and local refinement.Source: §D.4 · pp. 24–25, Figure 16 · p. 24

D.5 Dream-Mode Robustness Analysis

• A 2-s visual occlusion causes about 1 cm of transient deviation, whereas a 5-s occlusion leads to roughly 5 cm of deviation, arm drift, and stalling.The reported reliable open-loop prediction horizon is approximately 3.5 s.Source: §D.5 · p. 25, Figure 17 · p. 25

D.6 Pose-Tracking Analysis under Motion Speed

• Pose and point-cloud errors remain about 8 mm and 6 mm up to 0.8 m/s, then approach 50 mm at 1.2 m/s.The authors identify moderate-speed contact-rich manipulation as the better-suited setting because proximity estimation relies on accurate relative geometry.Source: §D.6 · p. 25, Figure 18 · p. 25

E.1 Experimental Setup and Task Description

• The real-world setup combines a RealMan RM75B arm, RGB-D tracking, and dexterous hands across whole-hand grasping, fingertip pinching, sweeping, and sustained in-hand tasks.Fifty demonstrations per task vary object positions, orientations, and some object types; reconstructed proximity responses concentrate near active contact regions in Pick, Pinch, and Sweep.Source: §E.1 · p. 26, Figure 19 · p. 26

E.2 Evaluation Protocol

• Evaluation covers standard execution, unseen objects with changed geometry or appearance, and perturbations including clutter, visual occlusion, and mild pose changes.Success criteria include contact stability and pose consistency during execution, not just final object location.Source: §E.2 · pp. 26–27

E.3 Failure Case Analysis

• Geometry-driven proximity does not fully capture fingertip force distribution; finger misalignment and pose jitter can destabilize contact during fine manipulation.The reported examples include asymmetric finger closure and cup shaking during detachment and transfer, with partial occlusion also perturbing pose estimates and proximity feedback.Source: §E.3 · p. 27, Figure 20 · p. 27
• Future work proposes aligning higher-precision glove or tactile signals with geometric proximity to improve force awareness and finger coordination while retaining low-cost, cross-platform advantages.Source: §E.3 · p. 27
• Simulation and real-world point-cloud visualizations show fine interaction geometry and proximity responses concentrated near active hand–object contact regions.The real-world reconstructed sequences are described as closely matching camera observations.Source: Figure 21 · p. 28, Figure 22 · p. 29