Source-linked AI summary
Learning Dexterous Manipulation for a Soft Robotic Hand from Human Demonstration
Abhishek Gupta, Clemens Eppner, Sergey Levine, Pieter Abbeel
TL;DR
Dexterous soft hands are difficult to control because they lack precise sensing and actuation, while conventional multi-fingered hands are often expensive and fragile. The paper learns from object-centric human demonstrations using reinforcement learning, selecting feasible demonstrations and extending guided policy search to combine controllers into a generalizable policy. On the RBO Hand 2, the approach learned valve turning, abacus manipulation, and grasping skills.
Problem
Multi-fingered hands are difficult to control, and the RBO Hand 2 lacks precise sensing and actuation needed for standard control and conventional demonstrations.
Method
The method uses object-centric demonstrations, reinforcement learning, feasible-demonstration selection, and an extended guided policy search framework to learn generalizable policies.
Results
The approach learned dexterous manipulation skills on the RBO Hand 2, including valve turning, abacus manipulation, and grasping.
Takeaways & Limitations
The method can acquire complex behaviors on a soft robot despite limited sensing and challenging actuation mechanisms.
Takeaways & Limitations
Future work proposes replacing motion capture with computer vision and extending policies to pressure and tactile sensing.
Abstract
from arXiv · showhide
Dexterous multi-fingered hands can accomplish fine manipulation behaviors that are infeasible with simple robotic grippers. However, sophisticated multi-fingered hands are often expensive and fragile. Low-cost soft hands offer an appealing alternative to more conventional devices, but present considerable challenges in sensing and actuation, making them difficult to apply to more complex manipulation tasks. In this paper, we describe an approach to learning from demonstration that can be used to train soft robotic hands to perform dexterous manipulation tasks. Our method uses object-centric demonstrations, where a human demonstrates the desired motion of manipulated objects with their own hands, and the robot autonomously learns to imitate these demonstrations using reinforcement learning. We propose a novel algorithm that allows us to blend and select a subset of the most feasible demonstrations to learn to imitate on the hardware, which we use with an extension of the guided policy search framework to use multiple demonstrations to learn generalizable neural network policies. We demonstrate our approach on the RBO Hand 2, with learned motor skills for turning a valve, manipulating an abacus, and grasping.
I. INTRODUCTION
The paper addresses dexterous manipulation with the inexpensive, compliant RBO Hand 2, whose limited sensing and actuation make conventional teaching and control difficult. It introduces object-centric demonstrations and an algorithm that selects feasible demonstrations while learning a generalizable policy.
- Multi-fingered hands are difficult to control because of complex dynamics, non-prehensile manipulation, under-actuation, and delicate mechanical designs.
- The RBO Hand 2 is actuated by inflating or deflating air chambers, without accurate kinematic finger control or precise sensing.
- Kinesthetic teaching, visual storage, and direct teleoperation are difficult because the hand cannot be manually moved, lacks position sensing, and mismatches human hand degrees of freedom.
- Object-centric demonstrations record manipulated-object trajectories, offering an intuitive way to specify tasks without hand-specific information.
- The proposed algorithm softly assigns demonstrations to controllers and optimizes those controllers with reinforcement learning, selecting demonstrations feasible for each initial state.
- A single nonlinear neural-network policy is trained from multiple controllers using an extension of guided policy search to generalize across initial states.
II. RELATED WORK
Prior manipulation methods use planning, reinforcement learning, or learning from demonstration, but often rely on detailed models, simplifying assumptions, or demonstrations that may not transfer across morphologies. This work combines demonstration structure with reinforcement learning for soft-hand manipulation.
- Planning-based methods generate manipulation trajectories using detailed hand-object models, contact constraints, grasp stability, or simplifying assumptions.
- Soft-hand modeling and simulation can be computationally expensive because accurate behavior may require finite-element models.
- Reinforcement-learning approaches have learned manipulation policies through dynamic motor primitives, value-function approximation, or neural-network policy search.
- Related in-hand manipulation work includes tactile-feedback policies and rotation learning, whereas this paper targets global policies that generalize local solutions.
- Learning from demonstration has reproduced gestures and dual-arm manipulation, while combining demonstrations with reinforcement learning can refine solutions or constrain search.
III. ALGORITHM OVERVIEW
The algorithm learns multiple local controllers from object-centric demonstrations, assigns demonstrations softly to controllers, optimizes controller behavior, and trains a neural policy to generalize their capabilities.
- Object-centric demonstrations capture only manipulated-object motion, while reinforcement learning trains policies to imitate those trajectories.
- The learning problem jointly selects demonstrations for controllers and trains each controller to follow its assigned demonstration.
- The algorithm alternates soft correspondence assignment, optimal control using the correspondences, and supervised learning within BADMM-based guided policy search.
- Controllers are learned from different initial conditions so each imitates the most closely achievable demonstration from its starting state.
- States and actions are represented as x_t and u_t, and controller distributions combine action policies with system dynamics over trajectories.
- The objective matches controller distributions to demonstrations only over object-centric state elements, represented by trajectories of tracked object markers.
- Demonstration trajectories are modeled as a mixture whose components are multivariate Gaussians with means equal to recorded object trajectories.
- The controller-distribution objective is a KL-divergence minimization optimized through a variational upper bound and alternating updates.
1) Weight assignment phase:
The weight-assignment phase alternates closed-form updates of correspondence variables until convergence, assigning demonstrations probabilistically to controllers.
- The objective is convex in the correspondence variables a and b, enabling alternating optimization with one variable fixed at a time.
- The algorithm alternates updates for a and b until convergence to compute the optimal correspondence parameters.
2) Controller optimization phase:
After correspondence weights are fixed, the optimization separates across controller distributions, with time-varying weights assigned independently at each time step.
- Fixed demonstration-controller correspondences make D_KL(a||b) independent of p, so the remaining optimization recovers the optimal controller distribution.
- The optimization is factorized independently for each controller distribution p_j(τ).
- Because controllers are time varying, weight assignment is performed independently at every time step.
V. CONTROLLER OPTIMIZATION WITH AN LFD OBJECTIVE
Controller optimization uses trajectory-centric reinforcement learning with locally fitted dynamics, maximizing demonstration imitation while constraining updates to remain near the previous controller.
- Trajectory-centric reinforcement learning learns time-varying linear Gaussian controllers from iteratively refitted local linear dynamics models.The method is chosen for systems with unknown dynamics, such as soft hands, and assumes Gaussian noise.
- The maximum-entropy objective minimizes expected l2 distance between object trajectories and demonstrations, weighted by normalized correspondence weights.Object trajectories are recovered from markers, while locally fitted dynamics replace analytical linearizations for controller optimization.
- The controller parameters K_jt, k_jt, and C_jt are optimized with LQG after fitting time-varying local dynamics from real-system samples.
- A constrained update limits controller changes because fitted local dynamics are valid only near the previous controller.Without this bound, the new controller may visit states where the fitted model is inaccurate and optimization may diverge.
VI. SUPERVISED LEARNING USING GPS
The method extends guided policy search to choose a single neural policy that generalizes across unseen initial states by matching multiple local controllers.
- Multiple local controllers leave controller selection ambiguous at unseen initial states, motivating a unified policy π_θ.
- The RBO Hand 2 uses seven actuators, with three forming the palm and thumb and air chambers that may be coupled or separately actuated.
- The extended GPS framework combines controllers into one nonlinear neural network policy for varying conditions.
- The neural network parameters are learned by regressing states to controller actions using samples generated on the physical system.Supervised learning alone is insufficient when the learned policy's state distribution differs from the controllers' distribution.
VII. RBO HAND 2 AND SYSTEM SETUP
The RBO Hand 2 is an inexpensive pneumatic soft hand whose actuator and object-marker sensing setup supports experiments across three manipulation tasks.
- The hand has four single-actuator fingers and a thumb with three independent pneumatic actuators.Its thumb achieves seven of eight configurations in the Kapandji test.
- Valve opening durations control inflation or deflation, with pulse width modulation converting discrete valve actions into continuous signals at 5 Hz.
- Experiments cover turning a valve, pushing beads on an abacus, and grasping a bottle from a table.
- The state combines seven pressure readings and derivatives, object-marker positions and velocities, and task-dependent robot-arm joint angles.LED markers are placed on the object rather than the hand.
VIII. EXPERIMENTS
The experiments evaluate the learning approach on valve turning, abacus manipulation, and grasping with the RBO Hand 2. Results test whether the method supports fine manipulation, selects achievable demonstrations, and generalizes across initial states.
- The evaluation covers valve turning, pushing beads on an abacus, and grasping tasks with the RBO Hand 2.
- Valve task: Valve rotation requires coordinating multiple fingers while accommodating different wrist positions along the lever.
- Valve task: Three human demonstrations captured clockwise and anticlockwise valve rotations, but not every demonstration was achievable from every training position.
- Valve task: Our method produced the most robust valve policy across four hand positions, nearly matching the oracle while baselines failed at least one position.
- Valve task: The learned policy used different finger strategies at different valve positions, whereas single-demonstration and open-loop baselines failed to generalize those behaviors.
B. Pushing the beads of an abacus
The abacus experiment measures whether policies move selected beads accurately across positions, while the grasping experiment tests learned finger control during a delayed-reward lifting task.
- B. Pushing the beads of an abacus: The abacus task requires moving particular beads while leaving others stationary through precise individual-finger motions.
- B. Pushing the beads of an abacus: Abacus evaluations recorded each bead’s movement over ten samples at three positions, comparing it with the demonstrated target movement.
- B. Pushing the beads of an abacus: Our method moved beads closer to target positions than single-demonstration and hand-designed baselines at all test positions, matching the oracle.
- B. Pushing the beads of an abacus: The learned policy selected discretely different fingers according to abacus position, unlike open-loop baselines and a controller trained at one position.
- Grasping: The grasping task uses reinforcement learning to coordinate the fingers with a scripted arm motion and retain a deodorant bottle during lifting.
- Grasping: The learned grasping policy matched the hand-designed baseline, with both policies succeeding on all 10 test samples.
2) Results and Discussion:
The paper concludes that object-centric demonstrations and reinforcement learning can train a soft hand for varied dexterous skills. It also identifies sensing and tactile-policy extensions as future work.
- The learned policy grasped the bottle successfully on all 10 samples, matching the hand-designed policy.
- Future work includes replacing motion capture with computer-vision tracking and conditioning policies on finger pressure or tactile sensors.
- The approach learns dexterous manipulation skills on the RBO Hand 2 from object-centric demonstrations, including valve turning, abacus bead movement, and grasping.