PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Contact-Rich Manipulation Skills with Guided Policy Search

Sergey Levine, Nolan Wagener, Pieter Abbeel

arXiv:1501.05611v2cs.RO

TL;DR

The paper tackles the limited expressiveness of low-dimensional policy representations for autonomous manipulation learning. It combines trajectory-centric linear-Gaussian controllers with guided policy search to train general nonlinear policies, adding sampling and step-size improvements for real robots. The method learns robust controllers for several complex contact-rich tasks after only minutes of interaction.

  • Problem

    Existing robot motion-skill learning methods often depend on compact, low-dimensional policy representations, limiting behavioral expressiveness and generality.

  • Method

    The method iteratively learns time-varying linear-Gaussian controllers and unifies them through guided policy search into a general nonlinear policy, using practical sampling and training-set improvements.

  • Results

    The method learned robust controllers for complex manipulation tasks, including toy-airplane assembly, tight-fitting block and ring placement, shoe-tree insertion, and bottle-cap screwing.

  • Takeaways & Limitations

    General-purpose policy representations can support fast learning of contact-rich manipulation skills and nonlinear policies that generalize to larger changes such as new target locations.

Abstract

from arXiv · show

Autonomous learning of object manipulation skills can enable robots to acquire rich behavioral repertoires that scale to the variety of objects found in the real world. However, current motion skill learning methods typically restrict the behavior to a compact, low-dimensional representation, limiting its expressiveness and generality. In this paper, we extend a recently developed policy search method \cite{la-lnnpg-14} and use it to learn a range of dynamic manipulation behaviors with highly general policy representations, without using known models or example demonstrations. Our approach learns a set of trajectories for the desired motion skill by using iteratively refitted time-varying linear models, and then unifies these trajectories into a single control policy that can generalize to new situations. To enable this method to run on a real robot, we introduce several improvements that reduce the sample count and automate parameter selection. We show that our method can acquire fast, fluent behaviors after only minutes of interaction time, and can learn robust controllers for complex tasks, including putting together a toy airplane, stacking tight-fitting lego blocks, placing wooden rings onto tight-fitting pegs, inserting a shoe tree into a shoe, and screwing bottle caps onto bottles.

I. INTRODUCTION

The paper addresses the difficulty of learning contact-rich manipulation when policy search depends on restrictive low-dimensional representations. It combines guided policy search with general-purpose policies and practical sampling improvements, demonstrating several real-robot tasks.

  • Contact-rich manipulation can defeat standard kinematic methods because tight-fitting, force-dependent tasks require very high precision or fail altogether.
  • Policy-search applications commonly rely on compact, low-dimensional policy classes, making the choice of an effective representation a significant challenge.
  • Guided policy search trains multiple sample-efficient linear-Gaussian controllers and unifies them into a single nonlinear policy that can generalize across conditions.
  • Adaptive sampling, step-size adjustment, synthetic policy-training samples, and a general cost-function framework make the method practical for robotic deployment.
  • Experiments cover assembling a toy airplane, stacking tight-fitting Lego blocks, placing rings on pegs, inserting a shoe tree, and screwing bottle caps.

III. OVERVIEW

The method separates fast trajectory-centric optimization from nonlinear policy learning. Time-varying linear-Gaussian controllers are optimized under locally fitted dynamics, then used by guided policy search to train flexible policies.

  • The objective is to find policy parameters θ for a policy πθ(ut|xt) that minimizes expected trajectory cost.
  • The method trains time-varying linear-Gaussian controllers and combines them through guided policy search into nonlinear policies with arbitrary parameterizations.
  • Linear-Gaussian controllers optimize trajectories with stabilizing feedback, while the final nonlinear policy is reserved for generalization to new situations.
  • Optimizing linear-Gaussian controllers exploits their special structure for rapid learning, while supervised matching reduces the difficulty of training high-dimensional nonlinear policies.
  • Both controller types are highly general: linear-Gaussian controllers can represent any trajectory and use enriched state variables for feedback.
  • The algorithm fits time-varying linear-Gaussian dynamics from samples and solves a constrained LQG problem so successive trajectory distributions remain nearby.

A. KL-Divergence Constrained LQG

The controller update solves an LQG problem under a modified cost that penalizes divergence from the previous trajectory distribution. A dual variable enforces the divergence constraint during optimization.

  • The LQG solution is obtained by backward dynamic programming, computing quadratic Q-functions and value functions from the trajectory end.
  • The optimal feedback controller is expressed using derivatives and Hessians of the cost with respect to state-action variables.
  • The linear-Gaussian controller is also the optimal solution to a maximum-entropy objective combining expected cost with trajectory entropy.
  • The update constrains the new trajectory distribution to remain close to the previous one, preserving the validity of the fitted local dynamics model.
  • Assuming shared dynamics simplifies the constrained update into an LQG problem with modified cost, while dual gradient descent adjusts the constraint multiplier.
  • The sample complexity of fitting linear dynamics scales with system dimensionality, potentially causing long training times for high-dimensional robots.

B. Background Dynamics Distributions

The method reduces dynamics-estimation demands by pooling samples across nearby times and prior iterations into a Gaussian-mixture prior. Adaptive sampling and step sizes reduce training requirements further.

  • Dynamics at nearby time steps and prior iterations are treated as correlated, allowing additional samples to form a prior for current linear-dynamics fitting.
  • A Gaussian mixture model represents piecewise linear dynamics, with each cluster defining a local linear-Gaussian model and its valid state-action region.
  • The GMM prior is constructed from state-action-next-state points gathered across current and previous iterations.
  • Cluster-weighted means and covariances provide a normal-inverse-Wishart prior, which is conditioned on state and action to obtain the dynamics model.
  • Adaptive sample counts and increasing the step size ϵ reduce the number of samples and iterations needed for learning.

C. Adaptive Adjustment of Step Size and Sample Count

The method adapts the KL-divergence step size using modeled and observed cost changes, while adjusting sample counts when dynamics estimates are unreliable. These adaptations reduce manual tuning and can accelerate learning.

  • Adaptive step size: The KL-divergence limit ϵ acts as a step size: larger values can accelerate learning but risk overly large, unproductive updates.The method adapts ϵ in both directions by modeling additional cost from unmodeled dynamics changes.
  • Adaptive step size: The adaptive step-size scheme removed manual step-size tuning and achieved significantly faster overall learning in the experiments.The method explicitly models additional cost caused by unmodeled changes in dynamics.
  • Adaptive step size: The algorithm estimates a new step size ϵ′ from the step size that would have been optimal at the previous iteration.It compares costs under previous and current dynamics and controllers to estimate the needed quantities.
  • Adaptive step size: The step-size model assumes cost improvement is linear in ϵ and additional cost from unmodeled dynamics changes is quadratic.The parameters a and b represent the additional cost and expected linear improvement rate, respectively.
  • Adaptive step size: Under linear dynamics and a local quadratic cost expansion, expected trajectory cost can be evaluated analytically from Gaussian state-action marginals.This also supports the assumed relationship between cost change and KL-divergence.
  • Adaptive sample count: The sample-count heuristic compares expected cost under estimated dynamics with the Monte Carlo average of sampled costs.A discrepancy indicates that too few samples may be available to estimate the dynamics accurately.

V. GENERAL PARAMETERIZED POLICIES

Guided policy search trains a general parameterized policy from optimized trajectories rather than applying reinforcement learning directly to the high-dimensional policy. Alternating trajectory and policy optimization enforces matching state distributions across training conditions.

  • V. GENERAL PARAMETERIZED POLICIES: Simple linear-Gaussian controllers handle only small initial-condition variations, so broader generalization requires a more expressive parameterized policy.The paper trains this policy on multiple initial conditions instead of hard-coding generalization through object positions.
  • V. GENERAL PARAMETERIZED POLICIES: Guided policy search optimizes trajectories first, then uses their samples to train a shared nonlinear policy with potentially high-dimensional parameterization.The final policy is not trained directly with reinforcement learning.
  • V. GENERAL PARAMETERIZED POLICIES: Supervised policy training is paired with trajectory reoptimization so the trajectory and policy state distributions match at convergence.This addresses the mismatch that otherwise prevents supervised learning from being guaranteed to improve expected cost.
  • V. GENERAL PARAMETERIZED POLICIES: The constrained optimization minimizes expected trajectory cost while requiring zero KL-divergence between policy-induced and trajectory state-action distributions at every time step.The constraint is relaxed and optimized using dual gradient descent.
  • V. GENERAL PARAMETERIZED POLICIES: Dual gradient descent alternates optimization of trajectory distributions and policy parameters with subgradient updates to dual variables λ_t.For multiple training conditions, separate trajectories share the same policy.
  • V. GENERAL PARAMETERIZED POLICIES: Algorithm 1 generates robot samples from each linear-Gaussian controller before updating the trajectory distributions and shared policy.The listed procedure is the implementation of guided policy search with unknown dynamics.

B. Augmenting Policy Training with Synthetic Samples

Using very few real-system samples can make nonlinear-policy training overfit to the current trajectories. The method addresses this by augmenting training with synthetic state-action samples generated from trajectory distributions.

  • B. Augmenting Policy Training with Synthetic Samples: Reducing training samples makes the nonlinear policy’s training set very small, causing overfitting to the current samples.This is a consequence of the sample-reduction improvements described earlier.
  • B. Augmenting Policy Training with Synthetic Samples: Synthetic samples are generated by sampling states from trajectory state marginals and computing the linear-Gaussian controller’s mean action at each state.Because the policy trains on state-action pairs, these samples need not be collected from the real system.

VI. DEFINING OBJECTIVES FOR ROBOTIC MANIPULATION

The objectives encode manipulation through target positions for object points, with penalties designed to combine rapid approach and precise final placement. Additional waypoints and task-specific motion terms accommodate obstacles and bottle-cap rotation.

  • Task setting: These objectives target assembly and placement tasks involving objects constrained by frictional contacts rather than free-space positioning.Examples include inserting parts into slots, threading rings onto pegs, and screwing caps onto bottles.
  • Objective specification: The cost specifies desired positions for several points on the manipulated object rather than a demonstration of the robot’s motion.A human positions the object at its desired final location; intermediate waypoints can also be supplied for complex tasks.
  • Task-specific terms: The task objective can include an intermediate waypoint to avoid obstacles and a wrist-spin term to encourage bottle-cap rotation.The shoe-tree task used a waypoint, while bottle-cap tasks encouraged the wrist to spin at a set rate.
  • Penalty design: The position penalty combines squared distance with a concave logarithmic term to encourage both quick approach and precise placement.The squared term brings the object near the target, while the Lorentzian term emphasizes accuracy at the target.
  • Penalty design: The penalty parameters trade off approach and precision, with experiments using w = 1, v = 1, and α = 10^-5.α controls the funnel depth and floor width, while w and v trade off the two penalties.

VII. EXPERIMENTAL RESULTS

Linear-Gaussian controllers were evaluated on contact-rich manipulation tasks, including stacking, assembly, insertion, and screwing. They learned successful behaviors with few real-world samples and showed robustness to perturbations and a changed bottle.

  • Tasks and setup: The experiments covered tight-fitting lego stacking, toy-airplane assembly, shoe-tree insertion, ring placement, and bottle-cap screwing.Lego stacking was tested with fixed, free-standing, and gripper-held conditions.
  • Tasks and setup: The controllers and neural networks directly commanded seven joint torques at 20 Hz using robot, object, target-relative, and previous-torque inputs.Object motion was represented by Cartesian velocities and target-relative position vectors for two or three object points.
  • Learning efficiency: 20-25 samples were typically sufficient to learn a successful controller, with about ten minutes of total learning time and only 3-4 minutes of system interaction.The remaining time involved resetting and computation, neither of which was optimized.
  • Learned strategies: Contact-specific strategies used alignment and friction: the ring controller positioned before pressing, while the airplane controller used peg-slot contact to align insertion.The shoe-tree controller used a user-supplied waypoint but still required careful post-waypoint sliding to avoid snagging.
  • Robustness: Training with larger target perturbations improved robustness at test time for the lego and ring tasks.Controllers were trained with target-position perturbation standard deviations of 0, 1, and 2 cm.
  • Transfer: A controller trained on one pill bottle successfully completed the task on a different pill bottle.This test assessed transfer across bottle instances.
  • Robustness: The learned controllers outperformed a straight-path kinematic baseline by a wide margin under target perturbations.The baseline succeeded for lego only without perturbations, while the rounded peg yielded occasional successes at higher perturbations.

B. Neural Network Controllers

Guided policy search trained nonlinear neural-network policies from multiple trajectory controllers to generalize manipulation across target locations. Training required roughly the same samples per trajectory as the linear-Gaussian method, with additional computation for network optimization.

  • Training setup: Two neural-network policies were trained for lego placement and ring placement across multiple target locations.The lego policy used four corner targets on a rectangular base, while the ring policy used a range of peg locations.
  • Training efficiency: Neural-network training required roughly the same number of samples per trajectory as the standard linear-Gaussian method.The plotted total combines samples across all training trajectories.
  • Training efficiency: Network optimization added about 50 minutes of computation time during training.The paper notes that an optimized implementation could reduce this additional time.
  • Network architecture: The neural networks used two hidden layers with 40 units each, with soft rectifying nonlinearities and linear output connections.The authors found the deeper two-layer architecture necessary to capture the complexity of the manipulation behaviors.
  • Generalization: Adding noise to peg position prevented ring-policy overfitting to training arm configurations and enabled generalization to all test positions.The perturbations helped the policy associate target position with success rather than particular joint-angle configurations.

VIII. DISCUSSION

The method combines rapidly learned linear-Gaussian controllers with guided policy search to train nonlinear policies that are robust to perturbations and generalize to larger task changes. Practical improvements reduce real-world sampling and support training large neural networks from limited interaction data.

  • Linear-Gaussian controllers handle complex contact tasks, provide intricate feedback, and remain robust to small perturbations.
  • Nonlinear neural-network policies trained from multiple controllers generalize to larger task changes, including new target locations.
  • Adaptive step sizes, sample-count adjustment, and synthetic state-action samples make guided policy search practical with few real-world samples.
  • Training repeatedly on a controlled set of targets allows separate controllers to be unified into one policy for arbitrary targets.
  • Guided policy search can use different inputs for the nonlinear policy and linear-Gaussian controllers, supporting joint perception and control.
Loading 1501.05611v2…