Source-linked AI summary
Pre-training Visual Dexterity in Simulation
Sarthak Kamat, Adam Rashid, Satvik Sharma, Aseem Doriwala, Chelsea Finn, Phillip Isola, C. Karen Liu
TL;DR
Dexterous robot hands lack scalable, on-embodiment pre-training data, limiting data-efficient policy learning. SPD collects virtual-reality teleoperation data in simulation and uses it to pre-train policies, improving real-world fine-tuning across five bimanual dexterous tasks versus training from scratch.
Problem
Dexterous manipulation lacks scalable, high-coverage teleoperation data because multi-fingered hardware is costly, while human-hand data creates embodiment and pose-supervision gaps.
Method
SPD uses aligned VR and real-world teleoperation systems to collect on-embodiment simulation demonstrations and pre-train a diffusion transformer for real-world dexterous manipulation.
Results
SPD improves real-world fine-tuning over training from scratch across five bimanual dexterous tasks, with history conditioning enabling shorter, more reactive action chunks.
Takeaways & Limitations
Simulation teleoperation is a viable pre-training source for real-world dexterous manipulation, particularly when visuomotor history supports reactive short action chunks.
Takeaways & Limitations
SPD depends on well-tuned simulation physics and currently evaluates on objects similar to those seen in simulation, leaving broader generalization uncertain.
Abstract
from arXiv · showhide
Large-scale pre-training has made robot policy fine-tuning increasingly data-efficient, but this progress has largely been driven by datasets and embodiments built around simple parallel-jaw grippers. Dexterous, multi-fingered hands remain comparatively data-starved because real teleoperation is costly to scale, while human hand video is off-embodiment and requires lossy pose estimation and retargeting. We introduce Simulation Pre-training for Dexterity (SPD), a pre-training framework for dexterous manipulation that uses data entirely collected in simulation. In SPD, humans manipulate virtual objects inside a VR headset, enabling on-embodiment trajectories and robot-free collection. With the help of five operators, we collect 75 hours of multi-task dexterous manipulation over one week, and use it to pre-train a causal transformer on a sequence modeling objective. We study the benefits of simulation pre-training on real-world tasks by fine-tuning on 1-2 hours of physical demonstrations on a 56-DoF bimanual dexterous setup. We find that our approach outperforms training behavior cloning policies from scratch, showing that simulation teleoperation is a viable pre-training source for real-world dexterous manipulation. We perform ablation studies, measuring the benefits of history conditioning and short action chunks for reactive control.
1 Introduction
Simulation Pre-training for Dexterity (SPD) addresses the data scarcity of multi-fingered dexterous manipulation by collecting aligned teleoperation data in physics simulation and adapting it to real-world tasks. With 75 hours of simulated demonstrations and 1–2 hours of robot fine-tuning data, simulation pre-training outperforms training from scratch across five real-world tasks.
- Motivation: Dexterous manipulation requires precise, low-latency control under high contact and partial observability, while behavior-cloning performance is limited by demonstration quantity and coverage.
- Data challenge: Teleoperation avoids train-test divergence, but multi-fingered hands constrain throughput because physical systems are costly and fragile, while handheld interfaces and human video introduce embodiment challenges.
- Framework: SPD combines spd-vr for simulated pre-training collection with spd-teleop for real-world post-training data, aligning camera placements, dexterous hands, and 6-DoF arms.
- Data collection: 75 hours of long-horizon, multi-task demonstrations were collected by five operators over one week across six scenes and hundreds of object instances.
- Results: Across five real-world tasks, simulation pre-training followed by 1–2 hours of robot fine-tuning outperformed training from scratch.
- Resources: The authors release the spd-75h dataset, spd-vr software, and six curated scenes with tuned contact parameters.
2 Related Work
Related work spans multi-task robot-policy pre-training, off-robot human data collection, and simulation-based dexterous manipulation. These directions broaden supervision and enable large-scale learning while reducing dependence on continuously operating physical robots.
- Pre-training for robot policies: Multi-task imitation learning uses large robot and ALOHA-style teleoperation datasets to expose policies to broad task and scene diversity.This line of work supports increasingly general manipulation policies.
- Pre-training for robot policies: Transformer-based policies and vision-language-action models extend robot-policy pre-training through sequence modeling and action prediction from pre-trained vision-language backbones.Pre-trained policies can provide useful priors for fine-tuning across new tasks, embodiments, modalities, and action spaces.
- Learning from humans: Off-robot supervision reduces the need for continuously operating a physical robot through portable interfaces, human video, and internet-scale visual data.These approaches address the expense of scaling real-world robot teleoperation while preserving action labels or learning useful representations.
- Simulation for dexterous manipulation: Simulation enables large-scale dexterous-manipulation training for high-dimensional, contact-rich hands and supports sim-to-real systems using reinforcement learning with domain randomization.Dexterous simulation benchmarks and training suites have expanded the scope of dexterous learning.
3 Method
The method pre-trains a bimanual dexterous manipulation policy from large-scale VR simulation data, then adapts it to real-world tasks with few demonstrations. It combines randomized, open-ended teleoperation data with a history-conditioned diffusion transformer that predicts action chunks from visual and proprioceptive context.
- 3 Method: SPD pre-trains a policy for contact-rich visuomotor control with two dexterous multi-fingered hands, targeting adaptation to real tasks using few demonstrations.The methodology covers VR teleoperation, real-world teleoperation, and policy learning.
- VR teleoperation: VR teleoperation directly controls the target bimanual robot in MuJoCo, using 60 Hz headset hand tracking, inverse kinematics, and physically simulated hand-object contacts.The simulator steps at 480 Hz, while detected wrist poses and fingertip positions drive the simulated robot arms and hands.
- VR teleoperation: 75 hours of data from approximately 2,000 episodes were collected by five operators across six randomized, open-ended simulation scenes over one week.The scenes include spelling blocks, dishes, mugs, bottles, cups, and Jenga bricks; prompts, assets, physical properties, and object positions are randomized on reset.
- Policy learning: The policy uses a diffusion transformer conditioned on visuomotor history to denoise action chunks, omitting language conditioning because the data lacks dense language annotations and broad scene coverage.Interleaved proprioception, action, visual, and noised future-action tokens are trained with a flow-matching velocity-prediction objective.
- Policy learning: 256-timestep sequences are processed with causal parallel denoising of action chunks, amortizing the cost of long-context sequence modeling.Each sequence is recorded at 30 Hz, and the causal mask permits simultaneous denoising of all chunks.
4 Experimental Results
SPD is evaluated on five real-world bimanual dexterous tasks after fine-tuning with 1–2 hours of demonstrations per task, outperforming same-architecture behavior cloning trained from scratch. Ablations show that history conditioning enables shorter action chunks to improve reactivity without sacrificing temporal coherence.
- Experimental setup: SPD is fine-tuned with 1–2 hours of real-world demonstrations per task and evaluated on five tasks involving objects similar, but not identical, to pre-training objects.The tasks are plate racking, mug hanging, jenga playing, cup stacking, and bottles in bin.
- Evaluation protocol: The comparison uses the same architecture trained from scratch on each task’s real-world demonstrations, with 20 trials per checkpoint measuring the fraction reaching each progress level.Figure 5 also reports average task progress with standard-error bars.
- Simulation pre-training results: Across all five tasks, SPD reaches nearly every manipulation stage more often and achieves higher average task progress than from-scratch behavior cloning.SPD checkpoints also start and converge to lower flow-matching training losses, matching the observed real-world performance.
- History and action chunking: The ablation sweeps history window sizes w ∈{1, 32} and action chunk sizes c ∈{8, 32} for both pre-trained and from-scratch variants.All variants share the architecture, fine-tuning data, and training time, and are evaluated on all five tasks.
- History and action chunking: With a single-frame window, reducing the action chunk from c = 32 to c = 8 collapses performance, producing visibly shaky and less temporally coherent behavior.With a 32-step history window, c = 8 becomes the strongest variant in both training regimes by combining contextual temporal consistency with reactivity.
5 Conclusion
SPD provides a simulation-only, on-embodiment pre-training framework for dexterous manipulation that improves real-world fine-tuning over training from scratch. The conclusion also identifies simulation fidelity and data diversity as limitations, with mixed-source pre-training and greater scale as future directions.
- Conclusion: SPD collects scalable, action-labeled, on-embodiment demonstrations in simulation without physical robot hardware during pre-training.The framework uses simulation pre-training for dexterous manipulation.
- Conclusion: SPD pre-training improves real-world fine-tuning on five bimanual dexterous tasks compared to training from scratch.The conclusion reports this improvement across the evaluated tasks.
- Limitations: SPD depends on simulation scenes with sufficiently accurate physics, and limited scene and object diversity may constrain transfer and evaluation generality.Mismatches in object masses, friction, or contact responses can encode strategies that transfer poorly; evaluated objects are similar to those seen in simulation.
- Future work: Future work includes combining simulation teleoperation with real-world teleoperation and egocentric human video, while scaling scenes, objects, and pre-training hours.These directions aim to study mixed-corpus benefits and generalization from simulation data itself.
A Appendix · A.1 spd-vr: Simulation Teleoperation System
spd-vr streams MuJoCo simulation to a Meta Quest 3, mapping tracked operator hands onto the simulated robot while computation remains on a host workstation. Task-registry episodes support randomized, exactly reconstructable collection with foot-pedal recording controls and contact-free checkpoints.
- A.1 spd-vr: Simulation Teleoperation System: spd-vr streams a MuJoCo simulation to a Meta Quest 3 headset and maps the operator’s tracked hands onto the simulated robot.The headset renders the scene and reports hand poses through a WebXR client.
- A.1 spd-vr: Simulation Teleoperation System: All computation runs on a host workstation, while the headset operates only as a WebXR client for rendering and hand-pose reporting.The division keeps simulation computation off the headset.
- A.1 spd-vr: Simulation Teleoperation System: Scene bodies are streamed to the client as binary meshes at 60 Hz, and hand poses are sent back over a USB tether.The passage specifies bidirectional streaming between the workstation and headset.
- A Appendix: The simulation steps at 480 Hz using the implicitfast integrator and elliptic friction cones.The supplied passage also mentions a no-slip setting, but the text is truncated after that phrase.
- A.1 spd-vr: Simulation Teleoperation System: A task registry defines each task’s natural-language prompt, target duration, and reset function for randomized episode initialization.Resets randomize asset selection, object placement, and physical properties.
- A Appendix: Every sampled value is logged, allowing episodes to be reconstructed exactly after randomized resets.The logged values cover the reset-time randomization described in the passage.
- A.1 spd-vr: Simulation Teleoperation System: Operators control recording with a three-button foot pedal for checkpointing, pausing, and reverting or skipping.Checkpoints are rejected during hand-object contact, so reverting restores a contact-free state.
A.2 spd-75h Dataset
The spd-75h dataset contains 1,930 teleoperated simulation episodes totaling approximately 75 hours, collected across six scenes on the target bimanual dexterous embodiment.
- spd-75h Dataset: 1,930 teleoperated simulation episodes comprise approximately 75 hours of data collected across six scenes on the target bimanual dexterous embodiment.Table 2 reports per-task episode counts and durations grouped by scene; durations are measured in minutes at 30 Hz, with spelling variants merged and tasks under ten episodes omitted.
A.3 spd-teleop: Real-World Teleoperation System
spd-teleop implements real-world dexterous teleoperation as a process-based ZeroMQ stack, converting VR wrist and glove finger tracking into robot arm and hand commands at high control rates.
- System architecture: A process-per-component architecture communicates over ZeroMQ publish–subscribe on local IPC sockets, with JSON headers and raw array bytes.The leader streams at 60 Hz, arm and hand followers servo at 120 Hz, and cameras publish at 30 fps.
- Wrists: Quest WebXR controllers stream wrist poses at 60 Hz, while pedal anchoring enables relative end-effector motion solved by differential IK.Wrist targets use four QP iterations per tick with posture, joint-limit, and velocity tasks, plus exponential smoothing.
- Fingers: Manus gloves track 25 keypoints per hand, mapping calibrated fingertip positions into five robot-hand targets that drive mocap-based IK.The resulting 22 joint angles form the hand command.
- Robot control: Arm joint positions are controlled with per-joint PD gains and MuJoCo inverse-dynamics gravity compensation, while hands use vendor SDK position control.Motors communicate over CAN at 1 Mbit/s with internal servo threads at 250 Hz.
A.4 Real-World Evaluation
Real-world evaluation uses 20 randomized-placement trials per task, scored with task-specific rubrics. Reported task progress is the achieved score normalized by each task’s maximum achievable score.
- Evaluation protocol: 20 trials per task evaluate each checkpoint from randomized initial object placements.The evaluation protocol uses repeated trials under varied starting configurations.
- Evaluation protocol: Each trial is scored against the corresponding per-task rubric in Table 3.The rubric defines the maximum achievable score for each evaluation task.
- Evaluation metric: Reported task progress equals the achieved score normalized by the task’s maximum.This normalization makes progress relative to each task’s maximum achievable score.
A.5 Model Architecture · A.6 Training Hyperparameters
The policy is a 222M-parameter diffusion transformer that processes interleaved proprioceptive, action, and visual tokens to denoise future 8-step action chunks. Simulation pre-training uses a defined hyperparameter set, while real-world fine-tuning changes only selected settings and inherits the rest.
- A.5 Model Architecture: The 222M-parameter diffusion transformer consumes 256-timestep multimodal sequences at 30 Hz and denoises future 8-step action chunks.Each timestep includes normalized 56-D proprioception and previous-action tokens; every eighth timestep adds four pooled tokens per camera and a noised action chunk.
- A.5 Model Architecture: Three-camera images are encoded with frozen DINOv3 ViT-B/16 features, then pooled into four tokens per camera and frame using learnable cross-attention queries.The pooled tokens re-attend to raw patch features through camera-specific cross-attention blocks interleaved every two trunk blocks.
- A.5 Model Architecture: The trunk is an 8-block, 768-hidden-size causal transformer with 12 attention heads, 4× MLP expansion, rotary temporal embeddings, and a 32-timestep sliding window.The action-denoising expert has its own 58M-parameter weights, while observation tokens share the base trunk.
- A.5 Model Architecture: Flow matching trains the model to predict velocity v = x1 − x0 from independently noised action chunks, using flow-time and within-chunk position embeddings.All chunks are denoised in parallel under the causal mask, with independent per-chunk flow times.
- A.5 Model Architecture: At deployment, a rolling KV cache matches the 32-timestep training window; chunk boundaries use 10 Euler steps to integrate the flow ODE and emit 8 actions.Current observation tokens are appended at each control tick, and the next action chunk is emitted on chunk boundaries.
- A.6 Training Hyperparameters: Simulation pre-training uses the hyperparameters listed in Table 4, while real-world fine-tuning changes only the settings listed in Table 5.The supplied passages identify the tables but do not provide their individual hyperparameter values.
- A.6 Training Hyperparameters: All real-world fine-tuning hyperparameters not listed as changed are inherited from simulation pre-training, and dataset size is reported per task.Table 5 is explicitly limited to settings that differ from pre-training.