Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning

Yuanpei Chen; Tianhao Wu; Shengjie Wang; Xidong Feng; Jiechuang Jiang; Stephen Marcus McAleer; Yiran Geng; Hao Dong; Zongqing Lu; Song-Chun Zhu; Yaodong Yang

Source-linked AI summary

Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning

Yuanpei Chen, Tianhao Wu, Shengjie Wang, Xidong Feng, Jiechuang Jiang, Stephen Marcus McAleer, Yiran Geng, Hao Dong, Zongqing Lu, Song-Chun Zhu, Yaodong Yang

arXiv:2206.08686v2Jun 17, 2022cs.ROcs.AIcs.LGcs.MA

TL;DR

Human-level dexterous manipulation remains difficult for RL because of high degrees of freedom and heterogeneous-agent cooperation. The paper introduces Bi-DexHands, a diverse and cognitively grounded bimanual benchmark, and finds strong single-task performance but poor multi-task and few-shot learning.

Problem
Human-level dexterous manipulation is difficult for RL because hands have high degrees of freedom and require cooperation among heterogeneous agents.
Method
Bi-DexHands is an Isaac Gym simulator and benchmark with two dexterous hands, diverse tasks, thousands of objects, and multiple RL settings.
Results
PPO-type on-policy algorithms master simple manipulation skills up to the level of 48-month infants, while complex multi-task and few-shot learning remains largely unsuccessful.
Takeaways & Limitations
Bi-DexHands provides a shared platform for evaluating bimanual cooperation, offline learning, and task generalization across dexterous manipulation tasks.
Takeaways & Limitations
The benchmark does not support deformable-object manipulation and primarily uses state-based observations, limiting direct sim-to-real transfer.

Abstract

from arXiv · show

Achieving human-level dexterity is an important open problem in robotics. However, tasks of dexterous hand manipulation, even at the baby level, are challenging to solve through reinforcement learning (RL). The difficulty lies in the high degrees of freedom and the required cooperation among heterogeneous agents (e.g., joints of fingers). In this study, we propose the Bimanual Dexterous Hands Benchmark (Bi-DexHands), a simulator that involves two dexterous hands with tens of bimanual manipulation tasks and thousands of target objects. Specifically, tasks in Bi-DexHands are designed to match different levels of human motor skills according to cognitive science literature. We built Bi-DexHands in the Issac Gym; this enables highly efficient RL training, reaching 30,000+ FPS by only one single NVIDIA RTX 3090. We provide a comprehensive benchmark for popular RL algorithms under different settings; this includes Single-agent/Multi-agent RL, Offline RL, Multi-task RL, and Meta RL. Our results show that the PPO type of on-policy algorithms can master simple manipulation tasks that are equivalent up to 48-month human babies (e.g., catching a flying object, opening a bottle), while multi-agent RL can further help to master manipulations that require skilled bimanual cooperation (e.g., lifting a pot, stacking blocks). Despite the success on each single task, when it comes to acquiring multiple manipulation skills, existing RL algorithms fail to work in most of the multi-task and the few-shot learning settings, which calls for more substantial development from the RL community. Our project is open sourced at https://github.com/PKU-MARL/DexterousHands.

1 Introduction

Bi-DexHands addresses the difficulty of learning human-level dexterous manipulation by providing a diverse bimanual RL benchmark grounded in infant motor-skill development. It supports efficient simulation and evaluation across several RL settings.

Dexterous manipulation remains difficult because it requires combining complex skills in unstructured, contact-rich environments and generalizing across tasks.Prior RL successes largely concern simpler behaviors such as throwing, sliding, poking, pivoting, and pushing.
Bi-DexHands provides a bimanual dexterous-manipulation benchmark built in Isaac Gym with diverse tasks and objects.The benchmark is designed to help robots acquire human-like dexterity through reinforcement learning.
The benchmark covers single-agent RL, multi-agent RL, offline RL, multi-task RL, and Meta RL.These settings target bimanual collaboration, learning from demonstrations, and task generalization.
Bi-DexHands links dexterous tasks to human motor skills at different ages using principles from the Fine Motor Subtest.
30,000+ mean FPS is achieved on one NVIDIA RTX 3090 by running 2,048 environments in parallel.

2 Related Work

Existing robotic systems handle some repetitive or simple manipulation behaviors, but unstructured, contact-rich, and bimanual dexterous manipulation remains challenging. Bi-DexHands is proposed as a benchmark for studying these challenges.

Robots perform well in repetitive, familiar environments but struggle with unstructured scenes and contact-rich manipulation.Existing capabilities include behaviors such as grasping, pushing, and throwing.
Bimanual manipulation offers additional dexterity through the use of a second hand, but remains an underdeveloped research area.
The paper proposes a bimanual dexterous manipulation benchmark to help address these robotic manipulation challenges.
Dexterous manipulation is difficult because five-finger hands combine high-dimensional actuation with contact-rich dynamics.

3 Formulations & Algorithms

Bi-DexHands uses two Shadow Hands and formulates manipulation learning across single-agent, multi-agent, offline, multi-task, and Meta RL settings. Its formulations represent cooperation, offline learning, and task generalization within a common benchmark.

Two 24-DoF Shadow Hands are used to create a platform for human-level dexterous manipulation.Each hand is actuated by 20 agonist-antagonist tendon pairs, while four joints remain underactuated.
The simulator runs a 1k Hz low-level controller and produces relative joint-position actions from the RL policy at 30 Hz.Some tasks also allow the policy to control the hand base within a restricted space.
The benchmark considers five formulations: single-agent RL, multi-agent RL, offline RL, multi-task RL, and Meta RL.
RL/MARL: RL and MARL scenarios are formulated as decentralized partially observable Markov decision processes with agents receiving observations and producing actions.The agents jointly transition the environment and optimize cumulative rewards.
Offline RL: Offline RL learns only from previously collected transitions, with out-of-distribution actions identified as a fundamental challenge.The benchmark implements BCQ, TD3+BC, and IQL.
Multi-task RL / Meta RL: Multi-task RL trains one policy across tasks using task information, whereas Meta RL learns common features for rapid adaptation to new and unseen tasks.The benchmark implements multi-task PPO, TRPO, SAC, and Meta-RL task structures.

4 Bimanual dexterous manipulation benchmark

Bi-DexHands combines simulated hands, diverse objects, cognitively grounded tasks, and multi-task or Meta-RL categories to study bimanual dexterity and generalization. Tasks span developmental skill levels and vary in goals, task identities, and test conditions.

Bi-DexHands consists of datasets, tasks, and learning algorithms for two Shadow Hands performing diverse human-like skills.Example skills include reaching, throwing, catching, picking, and placing.
The datasets generate varied scenes using two Shadow Hands and everyday objects from the YCB and SAPIEN datasets.SAPIEN additionally provides articulated objects with motion annotations.
Design of tasks: Tasks are mapped to infant ages through the Fine Motor Subtest, with difficulty increasing as the corresponding developmental age increases.This mapping is used to relate agent performance to approximate levels of infant motor ability.
Design of Multi-task/Meta RL: Multi-task and Meta-RL categories include ML1, MT1, ML4, MT4, ML20, and MT20, with each task offering object variation.
Design of Multi-task/Meta RL: ML1 uses few-shot adaptation without goal information, while MT1 generalizes within a fixed set using provided goal information.
Design of Multi-task/Meta RL: MT4 and MT20 train one policy across 4 and 20 tasks respectively, using a one-hot task ID to identify tasks.
Design of Multi-task/Meta RL: ML4 and ML20 adapt to new test tasks after meta-training on 3 and 15 tasks, respectively.ML20 is described as the most difficult challenge because its tasks differ substantially.

5 Benchmarking reinforcement learning algorithms

Bi-DexHands benchmarks reinforcement-learning formulations across human-skill-oriented bimanual tasks, emphasizing efficient simulation, algorithm comparisons, offline learning, and cross-task generalization. Results show strong single-task performance for PPO, limited offline and multi-task generalization, and increasing RL difficulty for tasks mapped to older human ages.

RL/MARL results: PPO performs well on most tasks, while multi-agent methods become relatively more competitive as tasks require greater two-hand cooperation.The experiments use two agents for MARL, with each hand represented as an agent.
Offline RL results: BCQ and TD3+BC substantially improve over behavior cloning on offline tasks, whereas IQL improves performance in only several datasets.The larger action and state spaces increase the severity of out-of-distribution action problems.
Generalization ability: Multi-task PPO performs poorly, and ProMP shows only a tiny improvement over random policy across the evaluated generalization settings.The benchmark identifies cross-task generalization in bimanual dexterous manipulation as an open challenge.
Generalization ability: RL difficulty generally increases with the human age associated with the task after MT20 results are normalized and ordered by increasing age.This relationship supports the relevance of the benchmark’s age-based task design.

6 Conclusion and Future Work

Bi-DexHands benchmarks bimanual dexterous manipulation with human-development-inspired tasks and efficient parallel simulation. Simple skills are learnable, but complex and multi-skill manipulation remains challenging, especially for multi-task and Meta RL.

Bi-DexHands provides more than twenty human-development-inspired tasks and a large variety of objects for bimanual dexterity learning.
In multi-task settings, RL performance follows a trend associated with increasing human ages of task difficulty.
Thousands of environments can run in parallel in Isaac Gym, improving RL sampling efficiency.
RL and MARL achieve strong performance on tasks requiring simple manipulation skills, while complex manipulations remain challenging.
Multi-task and Meta RL results are unsatisfactory when agents must master multiple manipulation skills.
The authors report that current RL reaches the level of 48-month-old infants in bimanual dexterous manipulation.
Bi-DexHands lacks deformable-object tasks and primarily uses state-based observations, limiting direct sim-to-real transfer.
Future directions include learning from demonstration, deformable-object simulation, improved multi-task algorithms, and sim-to-real transfer.

A Task Specifications

The Shadow Hand task specifications define joint ranges and physical parameters for a human-mimetic dexterous hand. The hand uses coupled finger joints and wrist mobility to support manipulation.

The Shadow Hand has 24 degrees of freedom, with five thumb degrees of freedom and four joints per other finger.
Distal finger joints are coupled so the middle-joint angle is always greater than or equal to the distal-joint angle.
An extra little-finger joint permits rotation toward the thumb, while two wrist joints allow 360-degree hand rotation.
Stiffness, damping, friction, and armature are adjustable physical parameters for each Shadow Hand joint in Isaac Gym.

A.2 Detailed components of tasks

Bi-DexHands defines high-dimensional bimanual manipulation environments with varied objects, observations, actions, and goal-based rewards. Tasks include handovers, catches, simultaneous throws, and pot lifting.

Bi-DexHands supports single- and multi-agent control in high-dimensional continuous spaces, reaching up to 400 state dimensions and 40 action dimensions.
Tasks use YCB objects for catching and Sapien objects for other categories, with replacement flexibility depending on pose or shape requirements.
Task observations combine left-hand, right-hand, object, and target state values, with a point-cloud interface available for future study.
Goal-based rewards increase as object translation and rotation errors decrease, balancing both terms with α.
Multi-agent observations depend on each agent’s associated hand while retaining object and target information.
Hand Over requires one hand to throw a falling object so it remains airborne before the other hand receives it.
Some tasks extend handover difficulty through movable hand bases, larger observation and action spaces, or simultaneous two-object throws.
The pot-lifting task requires two hands to grasp a handle and lift the pot to a designated position.

A.3 Offline Data Collection

The benchmark collects offline datasets from PPO, random, replay, and mixed expert policies, normalizes task scores, and evaluates diverse single- and multi-agent RL algorithms. It also describes policy-optimization objectives and update mechanisms.

Offline Data Collection: The medium offline dataset comes from an early-stopped PPO policy, while random and replay datasets use random or training-generated experience.
Offline Data Collection: The medium-expert dataset contains 2 × 10^6 samples formed by equally mixing expert-policy and medium-policy data.
Offline Data Collection: Offline scores are normalized to 0–100 using random and expert returns.
Policy Optimization Algorithms: TRPO uses a surrogate objective and constrained updates, with conjugate-gradient optimization followed by line search.
Policy Optimization Algorithms: PPO constrains policy-ratio updates by clipping the ratio to [1 − ε, 1 + ε].
Actor-Critic Algorithms: DDPG uses target actor and critic networks with soft updates for off-policy continuous-action learning.
Actor-Critic Algorithms: TD3 reduces overestimation bias through clipped double Q-learning, target-policy smoothing, and delayed updates.
Multi-Agent Algorithms: HATRPO and HAPPO support heterogeneous agents without requiring homogeneous agents or decomposable joint value functions.

B.4.1 Multi-task PPO/SAC/TRPO

The paper extends PPO, SAC, and TRPO to multi-task settings and describes MAML- and ProMP-based meta-learning approaches. Multi-task SAC uses task-specific entropy coefficients, while meta-learning adapts policies from limited experience.

Multi-task RL: Multi-task PPO, SAC, and TRPO retain their original algorithms, with Multi-task SAC adding separate entropy coefficients for each task.These disentangled alphas are intended to improve exploration across tasks.
Meta RL: MAML adapts an agent’s policy to new tasks using only a small amount of testing-phase experience.Tasks are modeled as MDPs whose aspects may vary across a task distribution, and K rollouts support adaptation.
Meta RL: The MAML objective defines each task’s loss from rewards accumulated over an MDP horizon H.The policy maps state x_t to action a_t, and task-specific rollouts provide the adaptation data.
ProMP: ProMP combines PPO with MAML and controls the statistical distance between pre-adaptation and adapted policies.Its PPO-based objective permits using a single trajectory for multiple gradient-update steps.

C Performance discussion of PPO and SAC

The performance discussion attributes PPO’s success and SAC’s broad failure in bimanual dexterous manipulation to how large-scale parallel simulation improves sampling efficiency. Experiments vary the number of humanoid environments to examine this explanation.

Observed performance: SAC fails on almost all bimanual dexterous manipulation tasks, whereas PPO achieves good performance in the reported experiments.The authors describe SAC’s behavior as anomalous and investigate the difference experimentally.
Sampling-efficiency explanation: 2048 parallel environments are proposed as the reason PPO benefits more from Isaac Gym’s sampling efficiency than SAC.The authors speculate that on-policy RL gains more than off-policy RL from increased sampling efficiency.
Experimental test: The humanoid comparison evaluates SAC and PPO with 8, 16, 32, 64, 128, 256, 512, 1024, and 2048 environments.The results are presented in Figure 6.
Pipeline context: The multi-task/meta-learning pipeline trains all tasks simultaneously rather than sampling only a subset of tasks for training.This design is intended to exploit Isaac Gym’s large-scale parallel simulation.

D.1 High performance multi-task/meta RL pipeline using Isaac Gym

The proposed Isaac Gym pipeline loads all tasks into one environment instance and uses them jointly for sampling and policy updates. This design targets efficient multi-task and meta-learning with GPU-parallel simulation.

Pipeline design: Isaac Gym performs simulation calculations on the GPU, enabling many environments to run in parallel without frequent CPU–GPU data transfers.The resulting sampling-efficiency improvement is described as especially helpful for on-policy, multi-task, and meta RL.
Pipeline design: Because Isaac Gym permits only one environment instance per GPU, the pipeline loads all tasks into that single instance at initialization.All tasks then contribute to data sampling and policy updates at the same time.
Implementation overview: Figure 7 provides details of the multi-task and meta-learning settings.The figure accompanies the pipeline description.
Motivation: The implementation is presented as an improvement over simulators relying on Python parallel programs for multi-task and meta-learning training.The authors hope this facilitates research in those settings.

D.2 Detail implementation of MT1, ML1, MT4, ML4, MT20, and ML20

The benchmark organizes multi-task and meta-learning categories from individual manipulation tasks with increasing breadth, from pose variations to all 20 tasks. Visual observations are explored separately but remain constrained by camera parallelism and lower throughput.

Category design: MT and ML categories share task groupings, but ML separates meta-training and meta-testing tasks while MT trains on all tasks together.MT1/ML1, MT4/ML4, and MT20/ML20 use corresponding task sets under different training and testing arrangements.
MT1 and ML1: MT1 and ML1 vary only the goal pose of Catch Underarm, using left, right, original, and backward translations.ML1 trains on left, right, and in-position variants before adapting to backward translation.
MT4 and ML4: MT4 and ML4 contain Catch Underarm, Hand Over, Catch Abreast, and Two Catch Underarm, which require relatively similar throwing and catching skills.The handover task’s base is not fixed to maintain environment consistency.
MT20 and ML20: MT20 and ML20 include all 20 designed tasks and are described as the most challenging and meaningful categories because of their broad skill span.The implementation scales rewards to reduce order-of-magnitude differences between tasks.
Visual observations: Bi-DexHands uses state-based observations, while visual-input RL is considered important for sim-to-real transfer but is not used as a baseline.The authors identify camera parallelism and visual-training difficulty as constraints.
Visual observations: Point-cloud input performs worse than full-state input under the same episode and environment count, while reaching some performance.The point-cloud setup uses PointNet features.
Visual observations: Point-cloud RL reaches only 200+ FPS versus 30000+ FPS for full-state input on one RTX 3090 and supports at most 256 environments.The authors attribute this limitation to Isaac Gym’s weak camera parallelism.