PaperlayerView exact revision on arXiv

Source-linked AI summary

Dex4D: Task-Agnostic Point Track Policy for Sim-to-Real Dexterous Manipulation

Yuxuan Kuang, Sungjae Park, Katerina Fragkiadaki, Shubham Tulsiani

arXiv:2602.15828v1cs.ROcs.CVcs.LG

TL;DR

Dexterous manipulation lacks scalable, diverse data, and simulation-based generalist training also demands substantial task-specific engineering. Dex4D trains a task-agnostic pose-to-pose policy in simulation and guides it with object-centric point tracks from generated videos. The authors report improvements across simulation and real-robot experiments, along with generalization to novel objects and scenes.

  • Problem

    Real-world dexterous manipulation data are expensive and difficult to scale, while simulation-based generalist policies require substantial task-specific environment design and reward shaping.

    Source: §I · p. 2

  • Method

    Dex4D trains a task-agnostic Anypose-to-Anypose policy in simulation and uses object-centric point tracks extracted from generated videos as goals for real-world deployment.

    Source: §I · p. 2

  • Results

    Experiments in simulation and on real robots show improvements in success rate, task progress, and robustness, with generalization to novel objects, poses, scenes, backgrounds, and trajectories.

    Source: §I · p. 2

  • Takeaways & Limitations

    Dex4D’s task-agnostic policy and point-track interface support zero-shot transfer to real-world tasks without real-robot finetuning.

    Source: §I · p. 2

    The Anypose-to-Anypose formulation is currently limited to single-object manipulation.

    Source: §VI · p. 8

Abstract

from arXiv · show

Learning generalist policies capable of accomplishing a plethora of everyday tasks remains an open challenge in dexterous manipulation. In particular, collecting large-scale manipulation data via real-world teleoperation is expensive and difficult to scale. While learning in simulation provides a feasible alternative, designing multiple task-specific environments and rewards for training is similarly challenging. We propose Dex4D, a framework that instead leverages simulation for learning task-agnostic dexterous skills that can be flexibly recomposed to perform diverse real-world manipulation tasks. Specifically, Dex4D learns a domain-agnostic 3D point track conditioned policy capable of manipulating any object to any desired pose. We train this 'Anypose-to-Anypose' policy in simulation across thousands of objects with diverse pose configurations, covering a broad space of robot-object interactions that can be composed at test time. At deployment, this policy can be zero-shot transferred to real-world tasks without finetuning, simply by prompting it with desired object-centric point tracks extracted from generated videos. During execution, Dex4D uses online point tracking for closed-loop perception and control. Extensive experiments in simulation and on real robots show that our method enables zero-shot deployment for diverse dexterous manipulation tasks and yields consistent improvements over prior baselines. Furthermore, we demonstrate strong generalization to novel objects, scene layouts, backgrounds, and trajectories, highlighting the robustness and scalability of the proposed framework.

I. INTRODUCTION

Dexterous manipulation is difficult to scale through real-world data collection, while simulation-based generalist policies require substantial task-specific engineering. Dex4D instead learns task-agnostic pose-to-pose skills in simulation and uses object-centric point tracks from generated videos to guide deployment; the authors report improvements and generalization across simulation and real robots.

I. INTRODUCTION

  • Real-world teleoperation is expensive, difficult to instrument, and limited in coverage, while dexterous hand control makes large-scale collection slow and error-prone.Simulation offers a scalable alternative, but training generalist policies there entails substantial environment design, task specification, reward shaping, and pipeline tuning.Source: §I · pp. 1–2
  • Dex4D learns an Anypose-to-Anypose policy for moving objects from arbitrary current poses to target poses, using Paired Point Encoding to represent correspondences between current and target object points.The formulation is task-agnostic and is proposed as an alternative to language-conditioned, task-specific policies.Source: §I · p. 2
  • For deployment, Dex4D extracts object-centric point tracks from generated task videos and uses them as policy goals, with online point tracking for closed-loop perception and control.The authors report zero-shot transfer to real-world tasks without real-robot finetuning.Source: §I · p. 2
  • Experiments in simulation and on real robots show improvements in success rate, task progress, and robustness, with generalization to unseen objects, poses, scenes, backgrounds, and trajectories.The paper presents this as evidence of the framework’s scalability and robustness.Source: §I · p. 2

A. Generalizable Dexterous Manipulation

  • Prior dexterous manipulation approaches described here rely on grasp optimization, in-domain imitation data, or reinforcement learning that often lacks autonomous high-level task planning.Dex4D combines video generation and 4D reconstruction for high-level planning with a task-agnostic policy trained via sim-to-real RL.Source: A. Generalizable Dexterous Manipulation · p. 2

B. Video-Based Robot Learning

  • Video- and flow-based planners have used pose estimation or heuristic retargeting, but the cited approaches face embodiment gaps, limited closed-loop feedback, or requirements for object meshes or clean point tracks.Dex4D’s student receives proprioception, the last action, and masked paired points, and jointly predicts actions and future robot states.Source: B. Video-Based Robot Learning · p. 2, Figure 2 · p. 3

C. 3D Policy Learning

Dex4D learns pose-to-pose dexterous manipulation as a task-agnostic goal-conditioned policy, using paired object points to represent current and target poses. The framework is trained in simulation across many objects and uses a representation designed to preserve point correspondence for policy learning.

C. 3D Policy Learning

  • Dex4D extends 3D policy learning by conditioning on current and target object points, enabling task-agnostic learning without language instructions.Source: C. 3D Policy Learning · p. 3

A. Anypose-to-Anypose

  • AP2AP transforms an object from an arbitrary initial pose to an arbitrary target pose without task-specific structure, predefined grasps, or motion primitives.It treats pose transformation itself as the learning objective, rather than decomposing manipulation into grasp generation, pose estimation, and planning.Source: A. Anypose-to-Anypose · p. 3
  • Trained entirely in simulation on 3,200 objects, AP2AP generalizes zero-shot to unseen objects and real-world downstream tasks without task-specific tuning.Training uses diverse pose configurations and extensive domain randomization.Source: A. Anypose-to-Anypose · p. 3

B. Goal Representation via Paired Point Encoding

  • Paired Point Encoding concatenates corresponding current and target points, preserving their correspondence while remaining permutation-invariant for policy learning.The paired points are 6-dimensional, with 3D current and target coordinates; the paper attributes better policy-learning performance to this representation.Source: B. Goal Representation via Paired… · pp. 3–4, Figure 3 · p. 4

C. Teacher-Student Policy Learning

Dex4D trains a teacher policy with privileged simulation information, then distills it into a student that operates with partial observations. Training also uses a curriculum and shaped rewards based on object points and hand-object interactions.

C. Teacher-Student Policy Learning

  • A teacher policy learned with visual RL is distilled into a student using DAgger, which receives proprioception, the last action, and partially observed points.The teacher additionally uses privileged states and points sampled across the whole object.Source: C. Teacher-Student Policy Learning · p. 4
  • The teacher uses PPO with privileged states and fully observed object geometry, encoding paired current and target points with PointNet.Its state includes joint angles and velocities, the last action, and privileged information such as joint torques and fingertip-to-object distances.Source: C. Teacher-Student Policy Learning · p. 4
  • A three-stage curriculum progresses from one object category to all 3,200 objects while tightening speed, reset, initialization, and control conditions.Domain randomization adds observation and action noise, PD-gain variation, hand-object friction, and external force disturbances.Source: C. Teacher-Student Policy Learning · p. 4
  • The shaped reward uses object-point matching alongside hand-object affinity and penalties for exaggerated motions.Its components cover current-target point distances, finger-object and hand-object distances, success, finger curl, table collisions, and action magnitude.Source: C. Teacher-Student Policy Learning · p. 4

1) RL Teacher Policy Learning:

Dex4D distills a teacher into a student action world model and conditions deployment on object-centric point tracks. Generated videos are converted into metric 3D tracks that specify target trajectories for the policy.

2) Student Action World Model Learning:

  • The student action world model is distilled from the teacher with DAgger and jointly learns action prediction and robot joint dynamics.The joint formulation improves action learning and supports safer, more controllable deployment for high-DoF, dynamic hand-arm systems.Source: 2) Student Action World Model Learning · pp. 4–5
  • The student takes proprioception, the last action, and masked paired points, then predicts a robot action and the next joint angles and velocities.The inputs are tokenized with MLPs and a PointNet-style encoder, then processed by self-attention layers.Source: 2) Student Action World Model Learning · p. 5
  • Random plane-height masking removes points to emulate hand occlusion and improve robustness across camera viewpoints under partial observation.Target points are masked consistently with their correspondences.Source: 2) Student Action World Model Learning · p. 5

IV. GENERALIZABLE DEXTEROUS MANIPULATION FROM POINT TRACK POLICY

  • Dex4D conditions AP2AP on desired 3D point tracks, which can come from generated videos or one-shot human demonstrations.The tracks specify the desired object configuration over time for closed-loop execution.Source: §IV · p. 5, A. From Generated Video to… · p. 5

A. From Generated Video to Object-Centric Point Tracks

  • The video-to-track pipeline generates future frames, tracks segmented object points in 2D, and lifts them into metric 3D using relative depth calibrated against the initial depth observation.The resulting object-centric tracks provide target trajectories for the AP2AP policy; relative-depth calibration is reported to produce smoother, more stable metric depths and cleaner tracks than approaches relying on fully metric depth estimation and explicit calibration.Source: A. From Generated Video to… · p. 5

B. Closed-Loop Perception and Control

Dex4D executes target point tracks with online visual tracking and repeated policy updates. In the reported implementation, the system uses a single-view RGBD camera and a 22-DoF hand-arm platform.

B. Closed-Loop Perception and Control

  • At execution, online tracking follows object points from the generated video, and RGBD observations backproject the tracked points into 3D for the policy.The student receives current and target 3D points together with proprioception and the last action, then outputs robot-control actions.Source: B. Closed-Loop Perception and Control · p. 5
  • When the average distance between visible corresponding points falls below a threshold, the system advances to the next target points and repeats until the final targets are reached.The distance is computed over the visible points at each timestep.Source: B. Closed-Loop Perception and Control · p. 5

A. Implementation Details

  • The real-world setup uses a 22-DoF xArm6-and-LEAP hand with a single-view RealSense D435 RGBD camera and Apriltag calibration.The policy uses modular video generation, relative-depth estimation, segmentation, and 2D tracking components; the simulation table reports Success Rate and Task Progress.Source: A. Implementation Details · pp. 5–6, Table I · p. 6

B. Simulation Experiments

The simulation suite evaluates Dex4D across six dexterous tasks using multiple baselines and waypoint-based trials.

1) Tasks:

  • The suite covers Apple2Plate, Pour, Hammer, StackCup, RotateBox, and Sponge2Bowl, spanning grasping, arm movement, reorientation, and spatial reasoning.Objects are drawn from UniDexGrasp and the unseen YCB dataset.Source: 1) Tasks · p. 6

2) Baselines:

  • The baseline comparison includes NovaFlow and NovaFlow-CL, adapting NovaFlow with dexterous grasping and adding closed-loop feedback and real-time IK in NovaFlow-CL.NovaFlow otherwise uses open-loop motion planning based on estimated transformations between current and target object points.Source: 2) Baselines · p. 6

3) Evaluation Protocols and Metrics:

  • Trials use random initial object poses and measure Success Rate by final-waypoint achievement and Task Progress by the average number of waypoints achieved.Each task has 100 trials; goal poses are predefined waypoints, updated after achievement, with independent Gaussian noise added to current and target points.Source: 3) Evaluation Protocols and Metrics · p. 6

4) Results and Analysis:

Dex4D outperforms the simulation baselines and the reported ablations. The analysis attributes gains to closed-loop feedback, paired point encoding, and student-policy architecture choices.

4) Results and Analysis:

  • Dex4D outperforms all simulation baselines by a large margin.The section reports results in Table I but supplies no cell values here.Source: 4) Results and Analysis · p. 6
  • +16.3% in SR and +10.4% in TP versus NovaFlow-CL; closed-loop feedback improves NovaFlow by +9.2% in SR and 16% in TP.The paper attributes the NovaFlow-CL gap to policy state traversal during RL and DAgger, and describes failure causes including insufficient hand reactivity, point occlusion, and pose-estimation errors.Source: 4) Results and Analysis · pp. 6–7

1) Importance of Paired Point Encoding:

  • Paired Point Encoding improves both student and teacher performance; MLP encoding reduces student SR to 5.7%, while separate PointNet encoders lose point correspondence.In teacher training, the method also outperforms both encoding variants.Source: 1) Importance of Paired Point Encoding · p. 7, Figure 4 · p. 7

2) Ablation on Policy Architecture:

  • The student policy outperforms variants without self-attention or world modeling; self-attention captures relations across input tokens, and world modeling also improves performance.The paper relates the latter improvement to synergistic effects between policy learning and world modeling.Source: 2) Ablation on Policy Architecture · p. 7

D. Real-World Experiments

Dex4D is evaluated on four real-world tasks using objects unseen during training and without real robot demonstrations. It outperforms the baseline in success rate, shows robustness to varied real-world conditions, and has limitations in tracking and single-object scope.

D. Real-World Experiments

  • 22.5% performance gain in SR over the baseline on four real-world tasks with unseen objects and no real robot demonstrations.The baseline is NovaFlow-CL.Source: D. Real-World Experiments · p. 7, Table III · p. 8
  • The baseline drops objects and misdirects post-grasp movement, while Dex4D can adjust or regrasp and remains robust with fewer than 10 visible points.The comparison attributes baseline weaknesses to absent hand feedback and sensitivity to noisy, sparse points.Source: D. Real-World Experiments · p. 8, Figure 6 · p. 8
  • Despite training only in simulated single-object scenarios, Dex4D generalizes to unseen object types and poses, backgrounds, camera views, trajectories, and external disturbances.The reported generalization tests are in the real world.Source: D. Real-World Experiments · p. 8
  • Real-world tracking can lose objects during significant movement, similar nearby textures, or rotations that block initially tracked points.The authors identify tracking loss as the major cause of failures; firm-grasp pushes can also knock objects over.Source: D. Real-World Experiments · p. 8

VI. CONCLUSIONS, LIMITATIONS, AND FUTURE WORKS

  • Dex4D decouples recognition from control by using video generation and 4D reconstruction for object-centric point-track planning and a task-agnostic policy for low-level control.The conclusion also identifies Paired Point Encoding and a transformer-based action world model as components of the framework.Source: §VI · p. 8
  • The current AP2AP formulation is limited to single-object manipulation, and the authors identify articulated objects as a future extension.They also note absent human grasp priors and raise tactile sensing and improved online tracking as future directions.Source: §VI · pp. 8–9

APPENDIX

The appendix documents visualization tools, benchmark tasks, and the video-to-point-track process used to specify manipulation goals.

A. Videos and Visualizations

B. Task Specifications

  • Simulation evaluation covers Apple2Plate, Pour, Hammer, StackCup, RotateBox, and Sponge2Bowl, while real-world evaluation covers LiftToy, Broccoli2Plate, Meat2Bowl, and Pour.Source: B. Task Specifications · p. 13

C. Video Generation and Point Track Extraction

  • Video generation uses Wan2.6 with first-frame and language-prompt conditioning, and relative depth estimation yields more spatiotemporally consistent tracks with fewer floater points than metric depth estimation.The videos are generated at 720P, 30 FPS, and five seconds; prompts use Chinese with native prompt enhancement.Source: C. Video Generation and Point Track… · p. 13

D. RL State Space

The appendix describes the RL teacher’s state inputs, three-stage curriculum, and stage-dependent reward shaping.

D. RL State Space

  • The RL teacher state includes robot joint and fingertip signals, hand and object states, the goal pose, a 64-dimensional PointNet object feature, and fingertip-to-object distances.Source: D. RL State Space · p. 13

E. RL Curriculum Learning

F. Reward Function

  • Reward shaping varies across training stages, and the appendix defines finger, object, and palm positions and height alongside contact, success, and stay-success conditions.Source: Table V · p. 13, F. Reward Function · p. 13

G. Domain Randomization and External Perturbation

The appendix describes simulation perturbations, point masking for partial observations, and additional policy hyperparameters.

G. Domain Randomization and External Perturbation

  • In simulation, object pushes occur every four seconds with linear xy and angular velocities sampled from U(−0.2, 0.2), alongside physics-parameter randomization.The domain-randomization setup is listed in Table VI.Source: G. Domain Randomization and External… · p. 13, Table VI · p. 14
  • Fig. 8 reports smoother, more spatiotemporally consistent depth results with fewer floater points for relative depth estimation than metric depth estimation.Source: Figure 8 · p. 14

H. Paired Point Masking

  • Paired point masking simulates single-view occlusion by masking one side of the object and then randomly masking most points above a sampled height.The remaining points also receive Gaussian noise; the procedure is intended to improve generalization to varying camera views, fewer points, and noisy real-world input.Source: H. Paired Point Masking · p. 14

I. Additional Hyperparameters

Loading 2602.15828v1…