PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations

Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, Sergey Levine

arXiv:1709.10087v2cs.LGcs.AIcs.RO

TL;DR

Dexterous hands are difficult to control because of their high dimensionality and complex contacts, while model-free DRL has not scaled to such manipulation and remains sample-inefficient. The paper evaluates model-free DRL on simulated 24-DoF five-fingered-hand tasks and augments policy learning with human demonstrations. Demonstrations reduce training to a few robot hours while producing more natural and robust policies.

  • Problem

    High-dimensional dexterous manipulation lacks demonstrated model-free DRL solutions, and model-free DRL requires too many physical samples for practical deployment.

  • Method

    The paper uses model-free DRL on simulated 24-DoF five-fingered-hand tasks, with behavior-cloning pretraining and augmented-loss policy-gradient fine-tuning from human demonstrations.

  • Results

    Demonstrations reduce sample complexity to the equivalent of a few robot hours and yield more human-like, substantially more robust policies.

  • Takeaways & Limitations

    Model-free DRL augmented with demonstrations is presented as a viable approach for real-world learning of dexterous manipulation skills.

Abstract

from arXiv · show

Dexterous multi-fingered hands are extremely versatile and provide a generic way to perform a multitude of tasks in human-centric environments. However, effectively controlling them remains challenging due to their high dimensionality and large number of potential contacts. Deep reinforcement learning (DRL) provides a model-agnostic approach to control complex dynamical systems, but has not been shown to scale to high-dimensional dexterous manipulation. Furthermore, deployment of DRL on physical systems remains challenging due to sample inefficiency. Consequently, the success of DRL in robotics has thus far been limited to simpler manipulators and tasks. In this work, we show that model-free DRL can effectively scale up to complex manipulation tasks with a high-dimensional 24-DoF hand, and solve them from scratch in simulated experiments. Furthermore, with the use of a small number of human demonstrations, the sample complexity can be significantly reduced, which enables learning with sample sizes equivalent to a few hours of robot experience. The use of demonstrations result in policies that exhibit very natural movements and, surprisingly, are also substantially more robust.

I. INTRODUCTION

Dexterous manipulation combines versatile, contact-rich tasks with high-dimensional control challenges that have limited prior model-free DRL results. This work introduces simulated tasks and shows that demonstrations improve training efficiency, motion naturalness, and robustness.

  • Multi-fingered hands support contact-rich tasks but involve high-dimensional observations and actions, discontinuous contacts, and under-actuation.
  • Prior model-free methods had been limited to simpler behaviors with 2–3-finger hands or whole-arm manipulators.
  • The authors propose four simulated dexterous manipulation tasks spanning grasping, object relocation, in-hand manipulation, and tool use.
  • Existing RL algorithms solve the tasks but require significant reward shaping and exhibit poor sample complexity, idiosyncratic strategies, and weak robustness.
  • A small number of VR human demonstrations, used for behavior-cloning pretraining and augmented-loss policy-gradient fine-tuning, enables training equivalent to a few robot hours.
  • Demonstration-trained policies produce more human-like movements and greater robustness to environmental variation.

II. RELATED WORK

Related work spans model-based trajectory optimization, model-free reinforcement learning, imitation learning, and methods combining reinforcement learning with demonstrations. The paper positions its approach as model-free DRL for complex anthropomorphic-hand tasks while addressing real-world sample demands.

  • Model-based trajectory optimization: Model-based trajectory optimization can achieve dexterous behaviors but depends on accurate dynamics models and state estimates that are difficult for contact-rich real-world manipulation.
  • Model-free reinforcement learning: Model-free RL avoids dynamics models by optimizing policies directly, but typically requires many real-world samples.
  • Combining RL with demonstrations: The paper extends model-free DRL toward anthropomorphic hands by reducing learning sample complexity with a small number of human demonstrations.
  • Imitation learning: Imitation learning uses successful demonstrations, but behavior cloning suffers from distribution drift and cannot exceed the demonstrator without a task-performance objective.
  • Combining RL with demonstrations: The paper combines imitation learning for bootstrapping and exploration with RL fine-tuning against the actual task objective.
  • Combining RL with demonstrations: Prior demonstration-based methods include trajectory-centric DMPs, Q-function pretraining, rule-based shaping, and replay-buffer integration such as DDPGfD.

III. DEXTEROUS MANIPULATION TASKS

The paper introduces four dexterous-manipulation task classes designed to capture technical challenges representative of real-world robot assistance. These include object relocation, in-hand manipulation, environmental-prop manipulation, and tool use.

  • Task suite: The task suite contains object relocation, in-hand manipulation, environmental-prop manipulation, and tool-use tasks.Together, these classes cover challenges including exploration, changing contacts, hidden subgoals, under-actuation, and forceful coordination.
  • Object relocation: Object relocation requires picking up an object and moving it to a randomized target, making exploration difficult without shaped rewards or demonstrations.Success is defined by placing the object within an epsilon-ball of the target.
  • In-hand manipulation: In-hand manipulation repositions objects such as pens while navigating joint-limit constraints and frequently changing contacts in a high-DoF hand.The pen target is randomized across configurations, and success requires matching its orientation within tolerance.
  • Environmental props: Environmental-prop manipulation involves multi-step tasks with hidden subgoals, such as undoing a latch before opening a door, on a narrow constrained manifold.The door task includes dry friction, a closing bias torque, randomized position, and completion at the door stopper.
  • Tool use: Tool-use tasks require coordinated grasp reconfiguration and force application rather than merely relocating a tool.In the hammer task, nail position is randomized and dry friction can absorb up to 15N; success requires the nail to be fully inside the board.

B. Experimental setup

The experimental setup uses a simulated 24-DoF anthropomorphic ADROIT hand in the MuJoCo physics simulator, with contact and hardware properties modeled for physical realism.

  • ADROIT hand: The simulated ADROIT hand is a 24-DoF anthropomorphic platform for dynamic and dexterous manipulation.The first, middle, and ring fingers have 4 DoF, the little finger and thumb have 5 DoF, and the wrist has 2 DoF.
  • ADROIT hand: Each ADROIT degree of freedom uses position control and includes a joint-angle sensor.
  • Simulator: The simulator is MuJoCo, chosen for stable contact dynamics in contact-rich hand-manipulation tasks.The setup models hardware kinematics, dynamics, and sensing, along with joint dry friction and frictional hand-object contacts.

C. Demonstrations

The paper collects a small number of high-fidelity demonstrations in virtual reality and incorporates them into reinforcement learning to reduce sample complexity and aid exploration.

  • Demonstration collection: Virtual-reality collection makes accurate demonstrations practical for complex dexterous systems where kinesthetic teaching is impractical.The setup records finger motion with CyberGlove III and tracks the hand base with an HTC Vive tracker while providing stereoscopic visualization.
  • Integration with RL: The learning procedure combines reinforcement learning and imitation learning, using a few expert demonstrations to reduce sample complexity and help exploration.
  • RL formulation: The control problem is modeled as an MDP with state, action, reward, transition, initial-state, and discount-factor components.Model-free RL requires sampling access to transition dynamics rather than knowledge of the transition function.
  • Demonstration data: Demonstrations are represented as a dataset used to guide reinforcement learning and significantly reduce its sample complexity.
  • RL formulation: The objective is to optimize a parameterized stochastic policy for expected cumulative reward.

B. Natural Policy Gradient

The paper uses Natural Policy Gradient, a model-free policy-gradient method that directly optimizes policy parameters and preconditions the gradient with the inverse Fisher information matrix.

  • Policy-gradient method: Policy-gradient methods directly optimize policy parameters to maximize the objective through local search such as gradient ascent.
  • NPG procedure: Natural Policy Gradient first computes the vanilla REINFORCE policy gradient.
  • NPG procedure: NPG preconditions the policy gradient using the inverse Fisher information matrix.
  • NPG procedure: NPG then applies a normalized gradient-ascent update with a step-size parameter.The implementation is based on the cited procedure of Rajeswaran et al.

C. Augmenting RL with demonstrations

Pure RL faces severe shaping, sample-efficiency, and behavior-quality problems on these dexterous tasks, motivating demonstrations as an auxiliary source of guidance.

  • RL solves the considered tasks only with careful, laborious task reward shaping.
  • Some tasks require approximately 100 hours of samples, making pure-RL training impractical for physical hardware.
  • Pure-RL behaviors are unnatural, noisy, and less robust to environmental variations.
  • Demonstrations guide exploration, reduce sample complexity, and help produce robust, natural-looking behaviors.
  • The proposed demonstration-augmented policy gradient method incorporates demonstrations into policy gradients after behavior-cloning pretraining.

2) RL fine-tuning with augmented loss:

The augmented-loss method combines reinforcement learning with demonstration guidance, then gradually reduces the demonstration contribution as policy performance approaches the demonstrations.

  • Behavior cloning cannot reliably learn all behaviors in the correct sequence when demonstrations are limited.The hammering example requires reaching, grasping, and hammering in sequence.
  • The augmented gradient combines on-policy policy-gradient terms with a weighted demonstration term.The policy data are represented by ρπ, while demonstration actions are weighted by w(s, a).
  • When w(s, a)=0, the method recovers policy gradients; with sufficiently large constant weighting, it reduces to behavior cloning.
  • The heuristic demonstration weight decays with iteration through λ0λ1^k, reducing auxiliary bias late in training.The decay reflects the premise that demonstrations are initially at least as good as the policy, but should not bias a comparable policy.
  • The experiments use λ0 = 0.1 and λ1 = 0.95, and performance is reported as insensitive to these hyperparameters.

V. RESULTS AND DISCUSSION

The evaluation tests whether RL can learn high-dimensional dexterous skills and whether demonstrations improve learning speed, robustness, reward sparsity, and movement quality. Shaped-reward NPG succeeds, while pure-RL behaviors remain unnatural and sample-inefficient.

  • The experiments compare RL algorithms and reward conditions, then evaluate demonstrations for faster learning, robustness, and sparse-reward performance.
  • The evaluation asks whether RL handles high-dimensional dexterous tasks and whether its policies are robust, safe, elegant, nimble, and human-like.
  • A. Reinforcement Learning from Scratch: With carefully shaped rewards, NPG achieves high task success, whereas DDPG fails to learn successful policies despite substantial hyperparameter tuning.
  • A. Reinforcement Learning from Scratch: Pure-RL policies often produce unnatural, erratic movements and are too sample-inefficient for physical-hardware training.Examples include socially unacceptable finger positions, unnatural hammer grasps, and unnatural wrist use for unlatching a door.
  • A. Reinforcement Learning from Scratch: Sparse task-completion rewards are primarily ineffective, while reward shaping helps NPG but DDPG still struggles to find success.
  • A. Reinforcement Learning from Scratch: The authors attribute these artifacts and brittleness to the difficulty of encoding robust human solution strategies in mathematical reward functions.

B. Reinforcement Learning with Demonstrations

DAPG incorporates demonstrations into policy-gradient reinforcement learning and substantially improves learning speed, robustness, and behavioral quality on dexterous manipulation tasks. Compared with DDPGfD and learning from scratch, DAPG succeeds in complex tasks within practical robot-training timescales and generalizes better across environment variations.

  • Evaluation setting: The evaluation uses sparse task-completion rewards because demonstrations are available.This setting tests whether demonstrations can support learning without relying on shaped rewards.
  • Algorithm comparison: DAPG convincingly outperforms DDPGfD across all evaluated tasks before DDPGfD shows progress.The comparison uses demonstrations to bootstrap learning, including replay-buffer demonstrations and other regularizations in DDPGfD.
  • Sample efficiency: DAPG trains policies for complex dexterous tasks in under a few robot hours and is almost 30 times faster than learning from scratch on object relocation.Table I defines robot hours and RL iterations needed to achieve a 90% success rate.
  • Robustness and behavior: DAPG policies are significantly more robust than policies trained with shaped rewards and produce human-like motions.The authors attribute these properties to human priors captured from demonstrations.
  • Robustness and behavior: DAPG succeeds on an ensemble of environments varying the manipulated object's size and mass, whereas RL from scratch with shaped rewards cannot learn a robust policy in a comparable timeframe.Training on the environment distribution further improves robustness for DAPG.

VI. CONCLUSION

The paper develops 24-DoF five-fingered-hand tasks and DAPG, then compares model-free reinforcement learning from scratch with demonstration-based methods. Demonstrations produce more human-like and robust policies, while DAPG reaches the evaluated tasks in under five hours and can be up to 30x more sample efficient than shaped-reward learning from scratch.

  • Contributions: The work introduces everyday-life manipulation tasks requiring control of a 24-DoF five-fingered hand and proposes DAPG for incorporating demonstrations into policy-gradient methods.The evaluation compares DDPG and NPG with DDPGfD and DAPG.
  • Findings: NPG solves the tasks only after significant manual reward shaping, while shaped-reward policies remain non-robust and produce idiosyncratic, unnatural motions.These findings concern model-free reinforcement learning from scratch.
  • Findings: DAPG policies exhibit more human-like motion and are substantially more robust after incorporating human demonstrations.The conclusion links these behavioral properties to the demonstration-based approach.
  • Sample efficiency: DAPG is up to 30x more sample efficient than reinforcement learning from scratch with shaped rewards and trains the evaluated tasks in under 5 hours.The authors describe this training time as likely practical for real systems.
  • Scope and future work: Success on real hardware remains to be demonstrated, although the authors view the simulation results and sample efficiency as progress toward practical real-world learning.The stated future direction is learning policies on real hardware while further reducing sample complexity.
Loading 1709.10087v2…