Source-linked AI summary
DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Reality
Ankur Handa, Arthur Allshire, Viktor Makoviychuk, Aleksei Petrenko, Ritvik Singh, Jingzhou Liu, Denys Makoviichuk, Karl Van Wyk, Alexander Zhurkevich, Balakumar Sundaralingam, Yashraj Narang, Jean-Francois Lafleche, Dieter Fox, Gavriel State
TL;DR
Dexterous manipulation policies learned in simulation are difficult to transfer to real robots because of the sim-to-real gap. DeXtreme trains a robust manipulation policy and simulation-trained pose estimator using an affordable Allegro Hand, vision-only sensing, and diverse simulation randomisations. Its vision-based state estimation outperforms prior vision-based results and compares favorably with marker-based estimation, while the authors report remaining sim-to-real limitations.
Problem
High-DoF dexterous manipulation is difficult to control, and transferring simulation-trained policies to real robots remains challenging because of the sim-to-real gap.
Method
DeXtreme trains a dexterous in-hand reorientation policy and pose estimator entirely in simulation using an Allegro Hand, vision-only sensing, and randomized simulated conditions.
Results
The vision-based state estimation outperforms prior vision-based results and is comparable to prior marker-based results despite different hardware.
Takeaways & Limitations
Affordable hardware, off-the-shelf cameras, and GPU-based simulation can support reported sim-to-real dexterous manipulation results.
Takeaways & Limitations
The real-world policy remains below simulation performance, and the authors identify unresolved pose-estimation, data-diversity, hardware, and physics-calibration issues.
Abstract
from arXiv · showhide
Recent work has demonstrated the ability of deep reinforcement learning (RL) algorithms to learn complex robotic behaviours in simulation, including in the domain of multi-fingered manipulation. However, such models can be challenging to transfer to the real world due to the gap between simulation and reality. In this paper, we present our techniques to train a) a policy that can perform robust dexterous manipulation on an anthropomorphic robot hand and b) a robust pose estimator suitable for providing reliable real-time information on the state of the object being manipulated. Our policies are trained to adapt to a wide range of conditions in simulation. Consequently, our vision-based policies significantly outperform the best vision policies in the literature on the same reorientation task and are competitive with policies that are given privileged state information via motion capture systems. Our work reaffirms the possibilities of sim-to-real transfer for dexterous manipulation in diverse kinds of hardware and simulator setups, and in our case, with the Allegro Hand and Isaac Gym GPU-based simulation. Furthermore, it opens up possibilities for researchers to achieve such results with commonly-available, affordable robot hands and cameras. Videos of the resulting policy and supplementary information, including experiments and demos, can be found at https://dextreme.org/
NVIDIA
The paper’s authors are primarily affiliated with NVIDIA, with additional affiliations at the University of Toronto and the University of Southern California. The paper also states that author contributions are listed at the end.
- All authors are with NVIDIA except Denys Makoviichuk, who is with Snap.
- Arthur Allshire, Jingzhou Liu, and Ritvik Singh also work with the University of Toronto.
- Aleksei Petrenko is also affiliated with the University of Southern California.
- Author contributions are listed at the end of the paper.
1 Introduction
The introduction motivates accessible dexterous manipulation by highlighting the difficulty of controlling high-DoF hands and reproducing prior sim-to-real successes. DeXtreme addresses this with affordable hardware, vision-based estimation, simulation-trained policies, and reproducible pipelines that achieve strong reported comparisons.
- Motivation: High-DoF multi-fingered hands enable stable grasps and diverse object interactions, but remain difficult to control.
- Motivation: Reproducing and extending prior deep-RL sim-to-real manipulation results has remained challenging because of complex training architectures and hardware requirements.
- Approach: DeXtreme uses an affordable Allegro Hand, three off-the-shelf RGB cameras, and GPU-based Isaac Gym to simplify dexterous manipulation development and deployment.
- Contributions: The system demonstrates learning-based dexterous in-hand manipulation with low-cost hardware, vision-only sensing, diverse pose targets, and cheaper compute.
- Contributions: The pose estimator is trained entirely in simulation and is designed to remain robust under heavy occlusions and varied robotic settings.
- Results: The vision-based state estimator outperforms the prior work’s best vision-based results and is comparable to its marker-based results, despite differing hardware.
- Contributions: The authors plan to release the vision and RL pipelines to support reproducibility and broader access to the manipulation system.
2 Method
The method learns recurrent dexterous reorientation policies in simulation for an anthropomorphic hand, using vision-based cube tracking and privileged critic observations. Isaac Gym enables parallel simulation, while vectorised automatic domain randomisation broadens training conditions to improve robustness for sim-to-real transfer.
- 2.1 Task: The task repeatedly reorients a cube toward randomly sampled targets in SO(3), counting consecutive successes while preventing drops or prolonged stuck configurations.A new target is sampled when orientation error falls within 0.4 radians; consecutive successes become harder as the hand must retain the cube.
- 2.1 Task: The hardware uses an Allegro Hand and three calibrated RGB cameras to estimate cube pose relative to the palm without markers or depth images.The cube carries coloured and lettered stickers so the vision system can distinguish faces and track its pose.
- 2.3 Policy Learning with RL: PPO learns an LSTM policy mapping partial observations to actions, while the critic receives additional state information unavailable to the actor.The policy is recurrent because the task is formulated as a partially observable Markov decision process.
- 2.5 Simulation: Training uses GPU-based Isaac Gym to simulate thousands of robots in parallel, reducing dependence on large CPU resources despite its different contact model from MuJoCo.The reward combines weighted terms based on rotational distance, object and goal positions, actions, target changes, and joint velocity.
- 2.6 Domain Randomisation: Simulation randomises physics, actions, and observations, including delays, Gaussian noise, and structured adversarial perturbations to address the sim-to-real gap.Mass and scale remain randomised only within fixed ranges because of current API limitations, although the resulting policies achieved high consecutive successes in real-world rollouts.
- 2.6 Domain Randomisation: Vectorised automatic domain randomisation adjusts parameter ranges to remain as broad as possible while maintaining policy performance above thresholds.The implementation evaluates boundary conditions in a fraction of environments and tightens or expands ranges accordingly.
3 Results
The experiments evaluate simulation training, real-world deployment, robustness, and extended goal holding for cube reorientation. ADR policies perform best in reality, while vision-based policies outperform prior vision baselines and approach privileged-state performance, despite remaining sim-to-real limitations.
- 3.1 Training in Simulation: Manual domain randomisation reached an average of 35 consecutive successes in simulation after approximately 24 hours.All curves in Figure 7 are from simulation experiments.
- 3.1 Training in Simulation: ADR policies achieved the best real-world performance after 2.5 days of training on 8 NVIDIA A40s.The paper contrasts this with prior training durations of 2 weeks to months for block reorientation.
- 3.1 Training in Simulation: 32 hours of manual domain-randomisation training generated 700K frames/sec across 8 NVIDIA A40s before convergence.At dt = 1/60, this corresponds to approximately 42 years of real-world experience.
- 3.2 Real-World Policy Performance: The real-world pipeline used a simulation-trained pose estimator to provide cube pose input to the policy, with policy and pose estimation sharing one machine.Allegro control and live Omniverse visualisation ran on separate machines communicating via ROS messages.
- 3.2 Real-World Policy Performance: Real-world policies showed limited inter-day performance degradation across trials collected under differing dynamics, temperature, and lighting conditions.The evaluation used 10 trials per policy and multiple days to measure variability.
- 3.2 Real-World Policy Performance: ADR policies outperformed manually tuned non-ADR policies and avoided the stuck behaviours observed with non-ADR training.The authors attribute this to increased diversity in training data and report higher average consecutive successes for ADR.
- 3.2 Real-World Policy Performance: The best vision-based policy produced approximately 2.5× higher peak and 1.5× higher mean consecutive successes than prior vision policies.It remained below the average successes of OpenAI et al.’s ADR (XXL) policy using motion-capture state information.
- 3.3 Quirks, Problems, and Surprises: The pose estimator remained robust in real-world scenarios beyond Allegro Hand manipulation, while ADR policies sometimes occluded cube markings by caging the object.These configurations made pose estimation unreliable, although the policy absorbed errors and performed better overall.
4 Related work
Related work frames dexterous in-hand manipulation as a difficult high-degree-of-freedom control problem. Prior analytical controllers struggle with dynamic contact changes, while pose estimation faces severe occlusion and motion-blur challenges.
- Classical methods: Analytical model-based controllers work well for no-slip contacts but struggle with dynamic tasks involving repeated making and breaking of contacts.These limitations motivate learning-based approaches for contact-rich manipulation.
- Reinforcement learning: Deep reinforcement learning, fast simulation, and domain randomisation have enabled robust learned behaviours in high-degree-of-freedom robots.The paper identifies prior in-hand cube reorientation work as a central inspiration.
- Pose estimation: Contact-rich dexterous manipulation makes pose estimation difficult because large occlusions and motion blur exclude many off-the-shelf estimators.Pose estimation is otherwise widely studied for robotic manipulation.
5 Limitations
The paper reports persistent limitations in sim-to-real transfer and in extending its simulation-based recipe to tasks that are difficult to simulate or evaluate. Pose-estimation mismatch and uncertain sources of real-world performance gaps remain open issues.
- Sim-to-real gap: The manual-DR policy averaged 35 consecutive successes in simulation but about 15 in the real world.ADR policies performed better but still failed to reach simulation-level average successes.
- Sim-to-real gap: The source of the remaining real-world performance gap is unclear, with possible contributions from pose estimation, training-data diversity, or a malfunctioning thumb.The best ADR npd was around -0.2, leaving room for further investigation.
- Pose estimation: A pose-estimation sim-to-real gap produced occasional interpenetrations when real states were replayed in simulation with physics enabled.This prevented straightforward calibration of cube physics parameters that might have improved transfer.
- Scope boundary: Successful transfer depends on simulating the task and defining a clear reward function for policy training.The paper presents these as key conditions in the context of this task.
- Scope boundary: Randomising interpretable simulator parameters and using ADR as a curriculum were crucial to transfer in this work.Examples include friction, damping, and stiffness.
- Scope boundary: Many real-world tasks are hard to simulate, and successful execution may be difficult to evaluate even when simulation and rewards are available.Cooking is given as an example where success evaluation is not straightforward.
7 Contributions
The project involved separate development of the simulation environment, domain-randomisation and ADR systems, vision pipeline, real-robot experiments, deployment infrastructure, and paper and video materials.
- Simulation and RL: The Allegro Hand Isaac Gym environment and simulation domain randomisations were developed to support sim-to-real transfer.Vectorised ADR and RL policy training were also implemented across the project.
- Vision pipeline: The vision contribution included synthetic-data rendering, augmentation, pose-estimation training, and real-to-sim analysis.The rendering pipeline was generalised and extended for vision-model development.
- Deployment and experiments: The team conducted real-world experiments and built the policy, vision-inference, live-visualisation, and Allegro Hand infrastructure.These activities covered both deployment software and experimental hardware management.
- Dissemination: The paper, videos, editing, project leadership, and supporting resources were distributed across multiple contributors.Several contributors drafted, edited, reviewed, and supported the project materials.
A.1 Compute Budget Comparisons
This section compares compute budgets for the block reorientation task with previous work and reports estimated AWS costs for equivalent setups.
- Compute budgets are compared for the block reorientation task against previous work.
- $14,280 is the estimated cost for an OpenAI-equivalent setup using one p3.16xlarge and one c6i.4xlarge instance.The estimate uses 8×V100 GPUs, 16 CPU cores, and 50 hours.
- $553.80 and $977.28 are the estimated compute costs for the authors’ manual DR and ADR experiments, respectively.These estimates use a g5.48xlarge instance with eight NVIDIA A10G GPUs.
A.2 Hardware Comparisons
This section compares the authors’ hardware setup with those used in two OpenAI studies, noting that one prior experiment used marker-based systems rather than RGB cameras.
- The hardware setup is compared against the setups used in OpenAI et al. [1] and OpenAI et al. [8].The comparison notes that the block reorientation experiment in [8] used marker-based systems rather than RGB cameras.
A.3 PPO Hyperparameters
This section lists the hyperparameters used to train the policy with PPO.
- Various hyperparameters used to train the policy with PPO are provided.
A.4 Isaac Gym Simulation Parameters
This section documents Isaac Gym simulation settings, the KUKA arm’s default rest configuration, and the DeXtreme model card. It also records intended use and the simulated emphasis of quantitative evaluation.
- Simulation settings for the Allegro Hand environment are provided.
- The default rest configuration of the KUKA arm joints is specified in degrees.
- DeXtreme’s model card follows the framework presented by Mitchell et al. [45].The stated primary use is research on pose estimation and control policies conditioned on pose.
- Most quantitative evaluations are conducted only in simulated settings because no real-world dataset provides ground truth for this use case.The evaluation targets challenging conditions including occlusions, blur, and lighting changes.