Source-linked AI summary
Learning Dexterous In-Hand Manipulation
OpenAI, Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, Wojciech Zaremba
TL;DR
Dexterous in-hand manipulation remains difficult for autonomous robots, especially across diverse environments and with complex end-effectors. The paper uses reinforcement learning in randomized simulation to train vision-based policies for a physical Shadow Dexterous Hand, achieving dexterous behaviors and sim-to-real transfer without human demonstrations. The results support applying contemporary deep RL to complex real-world robotics problems.
Problem
Autonomous robots remain limited in performing dexterous manipulation across diverse environments and using complex end-effectors.
Method
The authors train memory-augmented reinforcement-learning policies in extensively randomized simulations and deploy them with vision-based object-pose sensing on a physical Shadow Dexterous Hand.
Results
The learned policies transfer to the physical hand and naturally exhibit human-like behaviors including finger gaiting, multi-finger coordination, and controlled use of gravity, without demonstrations or reward priors.
Takeaways & Limitations
The results demonstrate that contemporary deep reinforcement learning can achieve complex dexterous manipulation on a physical five-fingered hand beyond existing non-learning-based approaches.
Takeaways & Limitations
Hardware breakages made experiments difficult, and repairs often changed aspects of the system, so some results were collected at different times.
Abstract
from arXiv · showhide
We use reinforcement learning (RL) to learn dexterous in-hand manipulation policies which can perform vision-based object reorientation on a physical Shadow Dexterous Hand. The training is performed in a simulated environment in which we randomize many of the physical properties of the system like friction coefficients and an object's appearance. Our policies transfer to the physical robot despite being trained entirely in simulation. Our method does not rely on any human demonstrations, but many behaviors found in human manipulation emerge naturally, including finger gaiting, multi-finger coordination, and the controlled use of gravity. Our results were obtained using the same distributed RL system that was used to train OpenAI Five. We also include a video of our results: https://youtu.be/jwSbzNHGflM
1 Introduction
Dexterous in-hand manipulation remains difficult for autonomous robots, especially with complex five-fingered hands. This work trains transferable control policies in simulation and deploys them on a physical robot, producing vision-based and human-like manipulation behaviors.
- Dexterous manipulation is fundamental for humans but remains challenging for autonomous robots, which typically operate in constrained settings with task-specific end-effectors.
- Five-fingered robotic hands remain difficult to control: simulated methods often do not transfer, while physical-only training is slow and produces limited behaviors.The Shadow Dexterous Hand has five fingers and 24 degrees of freedom.
- The work trains in-hand manipulation policies and deploys them on a physical robot, including vision-based object-pose sensing.
- The policy naturally discovers tripod, prismatic, and tip-pinch grasps, along with finger gaiting, multi-finger coordination, and controlled use of gravity.The work does not use human demonstrations or encode these behaviors in the reward function.
- Transfer is attributed to extensive simulation randomization, calibration, memory-augmented control, and large-scale distributed reinforcement learning despite simulation–reality differences.
2 Task and System Overview
The task is vision-based in-hand reorientation on a physical five-fingered hand, with a calibrated MuJoCo simulation used for training. The simulation remains an imperfect approximation of the real system, creating a reality gap for transfer.
- The task repeatedly reorients a block or octagonal prism from the palm toward newly provided target configurations until the object is dropped.
- The physical platform is a 24-DoF Shadow Dexterous Hand tracked with PhaseSpace fingertip markers and either object markers or three Basler RGB cameras.Vision-based pose estimation is intended for operation beyond a controlled laboratory environment.
- The control system updates low-level relative-position targets at roughly 12 Hz while the low-level controller runs at roughly 1 kHz.
- The simulation uses MuJoCo for physics and Unity for vision rendering, with the hand model calibrated to better match the physical system.
- The simulator remains a rough approximation because it models joint torques and rigid contacts rather than tendon actuation and deformable contacts, producing a reality gap.
3 Transferable Simulations
The approach replaces one imperfect simulator with a distribution of randomized environments and carefully chosen observations. Randomized physics, sensing, unmodeled effects, and visual appearance are used to improve transfer and generalization.
- A distribution of randomized simulations addresses the dilemma between costly physical training and poor transfer from a single inaccurate simulator.
- The policy observes fingertip positions and object pose, while omitting built-in sensors whose state-dependent noise is difficult to model accurately.
- The method randomizes simulated environments to train policies and vision models that generalize to reality.
- Observation noise includes episode-level correlated noise and timestep-level uncorrelated Gaussian noise.
- Physics randomizations sample parameters such as friction at episode start and center many values on calibrated real-system estimates.
- Unmodeled effects are approximated through motor backlash, action delays and noise, temporary marker loss, and marker occlusion.
- Visual randomizations vary camera properties, lighting, hand and object pose, materials, and textures; Figure 4 shows three simultaneously fed camera views.
4 Learning Control Policies From State
The control policy uses recurrent PPO with asymmetric actor-critic training, discrete relative joint-angle actions, and reward shaping for reorientation and object retention. Training scales across distributed workers and GPUs using randomized simulations.
- Policy architecture: The policy is a recurrent neural network with an LSTM and an intermediate ReLU layer, allowing adaptation to episode-persistent environment properties.Interaction can reveal properties such as object weight or finger speed.
- Policy architecture: PPO trains separate policy and value networks, using asymmetric actor-critic observations because the value network is needed only during training.
- Actions and rewards: Actions are desired joint-angle changes relative to current angles, with each coordinate discretized into 11 bins.The authors report that discrete action spaces worked better than continuous ones in their experiments.
- Actions and rewards: The reward is the decrease in orientation error, with +5 for achieving a goal and −20 for dropping the object.
- Distributed training: Training uses PPO across 384 worker machines and an eight-GPU optimizer, with Redis communication and MPI gradient averaging.Workers generate experience from randomized simulations, while optimizer threads stage minibatches on GPUs.
5 State Estimation from Vision
The vision system replaces laboratory motion-capture object tracking with pose estimation from synthetic-data-trained RGB-camera inputs. It predicts object position and orientation and feeds those estimates into the manipulation policy.
- Motivation: Vision-based pose estimation addresses the laboratory limitation of motion capture, which requires markers placed on each object.
- Model and execution: Three RGB cameras with differing viewpoints provide images for a convolutional network that predicts the object’s position and orientation.
- Model and execution: During physical execution, the predicted object pose is supplied to the control policy, which produces the next action.
- Model architecture: Each camera’s feature stack uses two convolutional layers, max-pooling, 4 ResNet blocks, and spatial softmax with shared weights across cameras.The resulting representations are flattened, concatenated, and passed to a fully connected network with ReLU activations.
- Training: The network is trained only on synthetic data by minimizing mean squared error on randomized and pose-augmented rendered images.Training uses one million simulator-gathered states, with two GPUs for rendering and one GPU for network training.
6 Results
The policies transfer dexterous manipulation from simulation to a physical robot, while naturally discovering human-like strategies without demonstrations or reward priors. Performance depends on randomization, memory, and scale, but reality gaps, object geometry, and hardware failures remain limiting factors.
- 6.1 Qualitative Results: The policy naturally discovers human-like grasps and dexterous strategies, including finger gaiting, multi-finger coordination, gravity use, and coordinated translational and torsional forces.These behaviors were not directly incentivized through demonstrations or reward-function priors.
- 6.1 Qualitative Results: The policy adapts grasp strategies to the hand’s morphology, using the little finger for precision grasps because it has an extra degree of freedom.This differs from the typical human use of the index and middle fingers for precision grasps.
- 6.2 Quantitative Results: Simulation-to-real transfer narrows but does not eliminate the reality gap: block rotation reaches a median of 50 successes in simulation versus 13 on the physical setup.The authors report the same overall trend across tasks, with real-system performance below simulation performance.
- 6.2 Quantitative Results: Policies transferred successfully to octagonal prisms, although performance remained below block results in both simulation and reality, indicating a need for further tuning and randomization.The policy was fine-tuned using the same randomized environment distribution originally designed for blocks.
- 6.4 Effect of Memory in Policies: Memory improves randomized-task performance and physical transfer, with LSTM state predicting whether a block is larger or smaller than average in 80% of cases after five seconds.Feed-forward policies often get stuck on the physical robot, while memory may support adaptation to environment properties.
7 Related Work
Prior dexterous-manipulation work spans diverse strategies, but simulation-only methods often lack physical-robot transfer, while direct robot learning is constrained by limited trials. Domain randomization offers a simulation-based alternative but traditionally assumes access to real data in related transfer methods.
- Dexterous manipulation research has explored rolling, sliding, finger gaiting, tracking, pushing, re-grasping, pivoting, tilting, tumbling, tapping, and multi-point manipulation.
- Closed-loop approaches use sensor feedback such as tactile sensing, but still require reasonable robot kinematic and dynamic models.
- Several methods generate or learn complex in-hand manipulation skills in simulation, but do not evaluate transfer to a real robot.
- Direct robot learning avoids system modeling but is limited to a handful of trials and therefore generally targets simple manipulation tasks.
- Domain adaptation, progressive networks, and inverse-dynamics methods assume real data, whereas domain randomization adapts policies during simulation training.
8 Conclusion
The work demonstrates that reinforcement-learning skills trained in simulation can reach high dexterity on a physical five-fingered hand. Transfer is supported by randomized simulation, distributed training, memory, and model-compatible sensing.
- Simulation-trained reinforcement-learning policies achieve an unprecedented level of dexterity on a physical five-fingered hand.
- Transfer is attributed to extensive simulator randomizations, large-scale distributed training, memory-augmented policies, and sensing modalities that can be modeled in simulation.
- The results indicate that contemporary deep reinforcement learning can address complex real-world robotics problems beyond existing non-learning-based approaches.
Appendices
The appendices define the reinforcement-learning framework, describe the Shadow hand and sensing hardware, and document control, calibration, and simulator implementation details. They also summarize empirical timing behavior relevant to deployment.
- A.1 Reinforcement Learning (RL): Reinforcement learning models an agent interacting with states, actions, rewards, transitions, and a discount factor to maximize expected return.
- A.2 Generalized Advantage Estimator (GAE): Generalized Advantage Estimation combines multi-step returns, with λ controlling the estimator's bias-variance tradeoff.
- A.3 Proximal Policy Optimization (PPO): PPO alternates collecting transition batches with minibatch stochastic-gradient optimization of policy and value networks, using clipped policy changes.
- B.1 ShadowRobot Dexterous Hand: The Shadow hand has 24 degrees of freedom, 40 tendons, and 20 motors, with 16 independently controllable degrees of freedom and four coupled joint pairs.
- B.2–B.3 Sensing: Finger tips are tracked with PhaseSpace, while three RGB cameras estimate object pose and resolve monocular pose ambiguities.
- B.4 Control: The deployed controller queries sensors every 80 ms, runs neural-network inference, and sends relative actuator-position actions to a separate real-time low-level controller.
- B.4 Control: Reducing the action interval to 40 ms increased training time without noticeably improving real-world performance.
- B.5 Joint Sensor Calibration: Joint-angle calibration minimizes fingertip reprojection error between PhaseSpace measurements and estimates from Hall sensors.
C.1 Deterministic Environment
The deterministic environment defines randomized object-reorientation tasks, observations, rewards, actions, timing, and simulated disturbances. Randomization targets physical parameters, sensing, appearance, actuation, delays, and unmodeled dynamics to narrow the simulation-to-reality gap.
- C.1 Deterministic Environment: The simulator uses MuJoCo, represents the system with a 60-dimensional state, and samples randomized object initialization on the palm.
- C.1 Deterministic Environment: Goals are desired object orientations represented as quaternions, with a new goal generated after achieving the current one within 0.4 rad.
- C.1 Deterministic Environment: Rewards measure progress toward the desired orientation, add 5 for achieving a goal, and penalize object drops by 20.
- C.1 Deterministic Environment: Actions specify relative hand-joint angles, discretize each coordinate into 11 bins, and use 20 action dimensions.
- C.1 Deterministic Environment: Episodes terminate after 50 consecutive goals, failure to achieve a goal within 8 seconds, or object drop.
- C.2 Randomizations: Randomization covers physical parameters, observation noise, marker loss and occlusion, action noise and delay, timing, backlash, random forces, and visual appearance.
- C.2 Randomizations: The simulator applies random object forces with episode-level probability sampled log-uniformly from 0.1% to 10%.
- C.2 Randomizations: Visual randomization changes object and robot materials, textures, colors, lighting, and camera characteristics.
C.3 MuJoCo Model Calibration
The MuJoCo model is calibrated against real robot trajectories so randomized simulation parameters remain physically reasonable. The section also specifies the learning-network normalization and training configuration.
- MuJoCo Model Calibration: Simulation parameters are randomized, but their distribution means are calibrated to reasonable physical values using robot trajectories.Calibration minimizes simulated-versus-real trajectory error before training.
- MuJoCo Model Calibration: Calibration trajectories measure joint-limit behavior and dynamic finger responses through inward, outward, and oscillatory motions.The recorded trajectory across all fingers lasts a few minutes.
- MuJoCo Model Calibration: The optimization replays recorded actions open loop in simulation and matches joint angles after 1 second, retaining only changes improving error by more than 0.1%.Parameters are adjusted using iterative coordinate descent.
- MuJoCo Model Calibration: 264 parameter values are optimized across joint properties and actuator properties, including damping, friction, stiffness, gains, force ranges, and backlash.Coordinate descent adjusts parameters until trajectory-matching error is minimized.
- MuJoCo Model Calibration: Observations are normalized with running means and standard deviations, clipped within 5 standard deviations, and paired with normalized advantages and value targets.The policy and value networks use the architecture depicted in Figure 12.
D.2 Vision Model
The vision model is trained with randomized object poses and data augmentation. The supplied passages identify the associated training and architecture hyperparameter tables.
- D.2 Vision Model: The vision-model training hyperparameters are listed in Table 11.
- D.2 Vision Model: The vision-model architecture hyperparameters are listed in Table 12.
- D.2 Vision Model: Vision training uses data augmentation that preserves, rotates, or jitters object pose with probabilities of 20%, 40%, and 40%, respectively.Jitter adds independent Gaussian noise to position and rotation.