PaperlayerView exact revision on arXiv

Source-linked AI summary

Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost

Henry Zhu, Abhishek Gupta, Aravind Rajeswaran, Sergey Levine, Vikash Kumar

arXiv:1810.06045v1cs.AIcs.RO

TL;DR

Dexterous multi-fingered hands are difficult to control because of their high-dimensional configurations and complex contact interactions. The paper uses model-free deep RL to learn contact-rich manipulation directly in the real world on low-cost hands, without task-specific models or simulation. Across diverse tasks, training takes a few hours, and human demonstrations substantially reduce learning time.

  • Problem

    High-dimensional configurations and complex intermittent contacts make autonomous control of versatile multi-fingered hands challenging.

  • Method

    The paper trains general-purpose neural network policies with model-free deep RL directly on low-cost multi-fingered hands in the real world, using optional human demonstrations.

  • Results

    Model-free on-policy RL solves valve turning, box flipping, and door opening in 7, 4, and 16 hours respectively, while demonstrations significantly reduce learning time.

  • Takeaways & Limitations

    Direct real-world model-free deep RL with demonstration-driven acceleration is a viable approach for learning diverse manipulation skills with low-cost multi-fingered hands.

Abstract

from arXiv · show

Dexterous multi-fingered robotic hands can perform a wide range of manipulation skills, making them an appealing component for general-purpose robotic manipulators. However, such hands pose a major challenge for autonomous control, due to the high dimensionality of their configuration space and complex intermittent contact interactions. In this work, we propose deep reinforcement learning (deep RL) as a scalable solution for learning complex, contact rich behaviors with multi-fingered hands. Deep RL provides an end-to-end approach to directly map sensor readings to actions, without the need for task specific models or policy classes. We show that contact-rich manipulation behavior with multi-fingered hands can be learned by directly training with model-free deep RL algorithms in the real world, with minimal additional assumption and without the aid of simulation. We learn a variety of complex behaviors on two different low-cost hardware platforms. We show that each task can be learned entirely from scratch, and further study how the learning process can be further accelerated by using a small number of human demonstrations to bootstrap learning. Our experiments demonstrate that complex multi-fingered manipulation skills can be learned in the real world in about 4-7 hours for most tasks, and that demonstrations can decrease this to 2-3 hours, indicating that direct deep RL training in the real world is a viable and practical alternative to simulation and model-based control. \url{https://sites.google.com/view/deeprl-handmanipulation}

I. INTRODUCTION

Multi-fingered hands offer versatility for general-purpose robotics but are difficult to control because of high-dimensional configurations and complex contact interactions. The paper studies model-free deep RL as a direct, scalable way to learn diverse dexterous skills on low-cost hardware in the real world, with demonstrations accelerating training.

  • Multi-fingered hands are needed for diverse tasks in human-centric environments, but their high-dimensional configurations and complex finger-object contacts make control challenging.
  • Model-free RL can optimize controllers end-to-end without explicit modeling or system identification, reducing engineering effort and adapting controllers to the physical environment.
  • The study trains general-purpose neural network policies directly on low-cost multi-fingered hands for valve rotation, box flipping, and door opening.
  • 4-7 hours: successful controllers for most tasks can be trained directly in the real world, while a small number of human demonstrations further accelerates learning.
  • The experiments cover contact-rich manipulation with rigid and deformable objects and demonstrate finger gaits, including combined hand and arm control, without a model.
  • Demonstration-augmented policy-gradient methods extend prior demonstration-based acceleration to real-world dexterous manipulation with significant learning-time benefits.

III. HARDWARE SETUP

The setup uses two low-cost multi-fingered platforms and evaluates them on valve rotation, box flipping, and door opening tasks modeled with reinforcement learning.

  • Hardware platforms: Two platforms are used: the 9-DoF Dynamixel claw costs under $2,500, while the 16-DoF Allegro hand costs about $15,000.The Allegro hand is anthropomorphic and supports payloads up to 5 kg; the Dclaw can run up to 24 hours without intervention or hardware damage.
  • Experimental tasks: The evaluation considers valve rotation, vertical box flipping, and door opening with challenging contact patterns and coordination.The tasks are inspired by everyday hand manipulations.
  • Valve rotation: Valve rotation uses hand joint angles, valve rotation, distance to the goal angle, and the previous action as state, with hand joint angles as actions.A trajectory succeeds when |dθ| < 20° for at least 20% of the trajectory.
  • Vertical box flipping: Box flipping rotates a freely spinning rectangular box from 0° to 180° while fingers learn alternating coordinated pushing motions.The state includes hand joints, box angle, angular distance to the goal, and the previous action; actions are hand joint angles.
  • Door opening: Door opening requires the arm and hand to approach the door, grip its flexible handle, and pull backward.The task has more degrees of freedom because the arm participates, and success is defined by reaching dθ > 30° at any point.

A. Dynamixel Driven State Estimation

Dynamixel servo motors provide both object-state measurement and automatic resetting for the manipulation experiments.

  • Sensing and reset: Servo motors measure object angle or position while resetting the valve, box, or door to its original position.The same mechanism supports both state estimation and repeatable task resets.

V. ALGORITHMS

The algorithms use model-free on-policy reinforcement learning with neural-network policies, including natural policy gradients and truncated natural policy gradient updates. The reported tasks require 4–16 hours of learning.

  • 4 hours for box flipping, 7 hours for valve turning, and 16 hours for door opening were required using model-free on-policy reinforcement learning.
  • Reinforcement learning seeks a policy that selects actions in each state to maximize expected cumulative rewards.
  • The policy is parameterized by a neural network and optimized with gradient ascent on the expected-return objective.
  • Natural policy gradients precondition the REINFORCE gradient with the inverse Fisher Information Matrix to stabilize optimization and enable faster convergence.
  • The implementation uses truncated natural policy gradient, a variant of natural policy gradient.

B. Demonstration Augmented Policy Gradient

Demonstration Augmented Policy Gradient accelerates reinforcement learning by incorporating human demonstrations into policy training. It first uses behavior cloning, then fine-tunes with an augmented, Fisher-preconditioned policy gradient.

  • Slow environment interaction in natural policy gradient motivates using prior knowledge to accelerate learning.
  • DAPG pre-trains the policy with behavior cloning on demonstrations before fine-tuning it with an augmented policy gradient.
  • The augmented gradient is subsequently preconditioned with the Fisher Information Matrix for the policy update.
  • The augmented gradient includes a demonstration term that encourages actions resembling expert actions on expert-visited states.
  • The paper applies DAPG to accelerate deep reinforcement learning on real hardware for challenging manipulation problems.

VI. EXPERIMENTAL RESULTS AND ANALYSIS

The experiments evaluate whether model-free deep reinforcement learning can learn dexterous manipulation directly in the real world across hardware and physical setups. They also examine demonstrations and the effects of reward-function and actuation-space choices.

  • The experiments test whether model-free deep RL can practically learn diverse dexterous manipulation behaviors directly in the real world.
  • The study evaluates whether model-free deep RL learns across different hardware platforms and physical setups.
  • The experiments assess whether a small number of kinesthetic-teaching demonstrations can accelerate learning.
  • The analysis examines how reward-function and actuation-space design choices affect learning.

A. Model-Free Deep RL

Model-free deep RL learned coherent multi-fingered manipulation skills directly on hardware, with valve rotation and box flipping learned more smoothly than door opening.

  • A. Model-Free Deep RL: The DClaw learned valve rotation through alternating finger movements, box flipping through staged pushing motions, and door opening by grasping and pulling the handle.The door-opening policy moved toward the door, grasped its deformable handle, and pulled it open.
  • A. Model-Free Deep RL: 7.4 hours for valve rotation, 4.1 hours for box flipping, and 15.6 hours for door opening reached 100% average success rates.Training completion was measured when the deterministic policy achieved 100% success over 10 evaluation rollouts.
  • A. Model-Free Deep RL: Valve rotation and box flipping improved monotonically on continuous rewards, while door opening was harder because reward appeared only after the door opened.Sparse reward produced large learning spikes because the agent had to consistently pull the door open before receiving reward.
  • A. Model-Free Deep RL: The learned policies developed coordinated finger gaits that moved quickly to rotate objects and then moved out of the way.These coordination patterns were observed in the learned manipulation behaviors.

B. Learning on Different Hardware and Different Materials

The same model-free RL approach transferred across a different hand morphology and deformable materials, while demonstrations substantially accelerated learning and produced gentler behavior.

  • B. Learning on Different Hardware and Different Materials: The 16-DoF Allegro hand learned valve rotation in a time comparable to the DClaw using the same deep RL algorithm.The result indicates morphology affected learning rate but did not prevent eventual task learning.
  • B. Learning on Different Hardware and Different Materials: Model-free RL learned to rotate a deformable foam valve by focusing manipulation near its more rigid center.The foam valve was challenging because contact dynamics were difficult to simulate and the hand could deform it in multiple directions.
  • B. Learning on Different Hardware and Different Materials: 20 kinesthetic demonstrations were collected for each task to guide exploration and accelerate policy learning.The demonstrations were slow and suboptimal but still provided useful guidance.
  • B. Learning on Different Hardware and Different Materials: DAPG reduced learning time by 2x compared with training from scratch and produced behaviors that were more gentle and legible to humans.Training times were measured using the defined success metrics across tasks.
  • B. Learning on Different Hardware and Different Materials: DAPG was more robust to changes in valve initial position but became less robust as observation and action noise increased.A wider initial-position demonstration set still learned well, though it was less effective than demonstrations from the same environment configuration.

D. Performance with Simulated Training

Simulation-to-real transfer required task-specific modeling and parameter randomization, while the authors identify several practical limitations of this strategy.

  • D. Performance with Simulated Training: Simulation-to-real transfer requires modeling the task in a simulator and manually choosing parameters to randomize.These requirements add task-specific design effort before training and transfer.
  • D. Performance with Simulated Training: Randomizing PID parameters and friction was crucial for effective simulation-to-real transfer.The transfer strategies were evaluated for valve turning with DClaw.
  • D. Performance with Simulated Training: Randomization can produce overly conservative policies, depends critically on which parameters are randomized, and increases training time and computational resources.The authors describe these as limitations of randomization-based transfer.
  • D. Performance with Simulated Training: Accurate simulation is difficult to construct for tasks involving deformable objects such as foam valves or soft door handles.Each new task must be modeled manually, requiring substantial time and expertise.

E. Design Choices

The design analysis found that position control offered the lowest vibrations and best learning performance, likely because low-level PID stabilization reduced the learning burden.

  • E. Design Choices: Position control was selected because it induced the fewest vibrations and was easiest to learn with.The actuation choice also affected hardware smoothness and sustainability.
  • E. Design Choices: Learning became harder when controlling higher-order derivatives, and position control was easier to learn than torque control.The analysis compared position, torque, and higher-order derivative control schemes.
  • E. Design Choices: Vibration was measured using the sum of magnitudes of the highest Fourier coefficients from joint-angle trajectories.The comparison used random trajectories induced by each control scheme.
  • E. Design Choices: Position control achieved lower vibration and better performance than the compared control schemes.The authors attribute this likely to stabilizing low-level PID control, which reduces the learning algorithm’s load.

2) Impact of Reward Function:

The reward function materially affects learning progress in simulation. Rewards without a control cost learn most effectively, while adding a control penalty slows optimization despite producing smoother operation; real-world training without that cost still produced safe behaviors.

  • Impact of Reward Function: Simulation compared three reward functions to guide the choice of reward for real-world training.The comparison evaluated learning progress under different reward formulations.
  • Impact of Reward Function: r1 and r2 produced the most effective learning, whereas r3 with a control cost learned more slowly.The control cost encourages smoother operation but reduces exploration by penalizing control.
  • Impact of Reward Function: Training without a control cost still produced safe behaviors in real-world experiments.
  • Impact of Reward Function: Reward design is part of a broader real-world model-free RL approach demonstrated for complex dexterous manipulation behaviors.The paper positions the method as practical and general for high-dimensional multi-fingered hands.
Loading 1810.06045v1…