PaperlayerView exact revision on arXiv

Source-linked AI summary

Path Integral Guided Policy Search

Yevgen Chebotar, Mrinal Kalakrishnan, Ali Yahya, Adrian Li, Stefan Schaal, Sergey Levine

arXiv:1610.00529v2cs.ROcs.LG

TL;DR

Direct deep reinforcement learning is challenging for real-world robotic tasks because of high sample complexity, while prior GPS local optimization struggles with discontinuous contact dynamics. This paper combines PI2-based local optimization with new-instance on-policy sampling, achieving improved manipulation performance and generalization.

  • Problem

    Direct deep reinforcement learning for real-world robotic tasks has high sample complexity, and prior LQR-based GPS optimization performs poorly with severe contact-related discontinuities.

  • Method

    The method integrates model-free PI2 stochastic policy optimization into GPS and samples new task instances on-policy during each training iteration.

  • Results

    PI2 consistently opened the door after three policy updates, while LQR could not handle its contact nonlinearities; global policy sampling raised door-opening success from 43.3% to 93.3%.

  • Takeaways & Limitations

    The approach supports high-dimensional neural-network policies for vision-based torque control and improves generalization by training on diverse newly sampled task instances.

  • Takeaways & Limitations

    PI2's stochastic local search typically requires human demonstrations to provide the initial behavior structure.

Abstract

from arXiv · show

We present a policy search method for learning complex feedback control policies that map from high-dimensional sensory inputs to motor torques, for manipulation tasks with discontinuous contact dynamics. We build on a prior technique called guided policy search (GPS), which iteratively optimizes a set of local policies for specific instances of a task, and uses these to train a complex, high-dimensional global policy that generalizes across task instances. We extend GPS in the following ways: (1) we propose the use of a model-free local optimizer based on path integral stochastic optimal control (PI2), which enables us to learn local policies for tasks with highly discontinuous contact dynamics; and (2) we enable GPS to train on a new set of task instances in every iteration by using on-policy sampling: this increases the diversity of the instances that the policy is trained on, and is crucial for achieving good generalization. We show that these contributions enable us to learn deep neural network policies that can directly perform torque control from visual input. We validate the method on a challenging door opening task and a pick-and-place task, and we demonstrate that our approach substantially outperforms the prior LQR-based local policy optimizer on these tasks. Furthermore, we show that on-policy sampling significantly increases the generalization ability of these policies.

I. INTRODUCTION

The paper addresses the difficulty of learning high-dimensional visual control policies for manipulation tasks with discontinuous contact dynamics. It combines PI2-based local optimization with on-policy sampling in GPS to improve contact handling and generalization.

  • GPS combines local policies optimized for specific task instances with supervised learning of a high-dimensional global policy.The global policy can use observations rather than the full system state.
  • LQR-based local optimization struggles with discontinuous contact switching in tasks such as door opening and object manipulation.The paper motivates a model-free alternative for these dynamics.
  • The proposed method replaces LQR with PI2 for local policy optimization and samples new task instances at every GPS iteration.These changes target contact discontinuities and increase training-instance diversity.
  • On-policy sampling substantially improves generalization by training the global policy on diverse randomized task instances.The paper evaluates this effect on door opening and pick-and-place tasks.
  • The approach trains deep neural network policies that use visual features directly for torque-control manipulation.The policies are initialized from demonstrations and use automatically learned visual features.

III. BACKGROUND

GPS separates trajectory optimization from supervised learning of a complex policy. Its constrained formulation addresses mismatch between local and global policies, while this paper replaces discontinuity-sensitive LQR optimization with model-free PI2.

  • GPS first optimizes simple local controllers for varied task initial conditions, then trains a complex global policy from their optimized controls.The global policy can contain thousands of parameters and learn from observations.
  • The global policy is trained by supervised learning on samples generated by optimized local policies.This allows trajectory optimization to use full state while the final policy uses observations.
  • MDGPS constrains the relationship between local trajectory distributions and the global policy with KL divergence.The algorithm alternates local-policy optimization with global-policy fitting.
  • Prior GPS methods used KL-constrained LQR with fitted time-varying linear dynamics, but smooth LQR performs poorly under severe discontinuities.This motivates model-free PI2 for local optimization.

B. Policy Improvement with Path Integrals

PI2 improves time-varying linear-Gaussian local policies through model-free stochastic sampling. It assigns higher probability to lower-cost trajectories and updates feedforward controls and covariances accordingly.

  • PI2 is a model-free reinforcement-learning algorithm grounded in stochastic optimal control and statistical estimation.The paper applies it to feedforward commands of time-varying linear-Gaussian controllers.
  • PI2 learns feedforward controls and covariances while keeping feedback gains fixed after initialization.The local policy is initialized, for example, from human demonstrations.
  • Each PI2 iteration samples trajectories on the robot, computes cost-to-go values and probabilities, and shifts the policy toward lower-cost trajectory regions.Lower-cost trajectories receive higher probabilities, with η acting as a soft-max temperature.
  • The policy parameters are updated using a probability-weighted maximum-likelihood estimate of sampled feedforward controls.Covariance updates automatically determine exploration magnitude at each time step.

IV. PATH INTEGRAL GUIDED POLICY SEARCH

The algorithm combines PI2 local optimization with GPS and global policy sampling to train visuomotor policies on randomized task instances. Its neural-network architecture and visual features support learning from raw observations.

  • The method uses PI2 as the local optimizer within GPS and introduces global policy sampling for randomized task instances.The section also covers visual-policy architecture and feature pre-training.

A. PI2 for Guided Policy Search

The method incorporates PI2 into GPS through KL-constrained local policy updates, then trains the global policy using the optimized controls. Temperature selection controls the allowed policy change, and the procedure is summarized as MDGPS with PI2.

  • PI2 is incorporated into GPS using time-varying linear-Gaussian local policies.
  • Each iteration performs noisy policy sampling, PI2 optimization under a KL-divergence constraint, and supervised training of the global policy.
  • The local policy update limits deviation from the old policy by constraining D_KL(p(u_t|x_t)∥p̄(u_t|x_t)) to ϵ.
  • Temperatures η_t are computed independently at each time step by optimizing a dual function based on policy-sample cost-to-go.
  • Replacing the old local policy with the global policy yields the mirror gradient descent formulation of GPS.

B. Global Policy Sampling

Global policy sampling removes the fixed-instance restriction of standard GPS by sampling new task instances each iteration. PI2 optimizes these samples locally, and supervised learning transfers the resulting controls back to the global policy.

  • Standard GPS is constrained to a fixed set of task instances and initial conditions throughout learning.
  • New task instances, such as door poses, are sampled at every iteration using noisy rollouts from the global policy.
  • PI2 performs one local optimization step per sampled instance, with updates constrained against the old global policy.
  • The optimized controls are fed into supervised global-policy learning, while its noise covariance is updated by averaging local-policy covariances.
  • Unlike direct policy-gradient optimization, the method performs trajectory-centric optimization in a lower-dimensional action space before guiding the global policy.

C. Global Policy Initialization

The global policy is initialized with standard GPS and PI2 on fixed task instances before broader sampling. Its visuomotor architecture extracts multiscale image features, pretrains pose representations, and combines visual history with robot state to produce torques.

  • C. Global Policy Initialization: Several standard-GPS iterations with PI2 on fixed task instances initialize the global policy before using broader sampling.
  • D. Learning Visuomotor Policies: The policy processes RGB images through convolutional layers, pooling, multiscale feature recombination, and spatial soft-argmax coordinates.
  • D. Learning Visuomotor Policies: A fully connected layer predicts object and robot pose from feature coordinates to pretrain the vision layers.
  • D. Learning Visuomotor Policies: Current feature points, initial-image feature points, and a 33-dimensional robot state are concatenated before fully connected torque prediction.
  • C. Global Policy Initialization: The experiments compare PI2 and LQR, evaluate global-policy sampling with and without instance resampling, and test design choices in simulation.

A. Experimental setup

The experiments evaluate torque-controlled manipulation from monocular RGB images on randomized door-opening and pick-and-place setups. A controlled single-instance comparison tests PI2 against LQR during adaptation.

  • A. Experimental setup: The robot uses a 7-DoF arm, monocular RGB images, and direct torque commands at 20 Hz.
  • 1) Door opening:: Door opening is evaluated using IMU readings from the handle, with demonstration-derived target readings and smoothness costs.
  • A. Experimental setup: Pick-and-place is evaluated by the final bottle pose relative to a demonstrated upright target, using pose estimation only for cost evaluation.
  • 1) Door opening:: The controlled adaptation experiment displaces the demonstrated door by 5cm and evaluates recovery from a single task instance.
  • 1) Door opening:: After two policy updates, PI2 opened the door in 50% of samples; after three updates, it opened the door consistently, whereas LQR could not handle the discontinuous contact dynamics and cost.

2) Pick-and-place:

The pick-and-place experiments evaluate PI2-based guided policy search for adapting and learning bottle-placement behavior. The method reaches reliable upright placement while PI-GPS achieves lower costs with fewer samples than the compared approaches in simulation.

  • Pick-and-place: The pick-and-place adaptation displaces the bottle by 5cm and rotates it by 30 degrees from its demonstrated position.The local policy uses 200 time steps, with initial noise allowing grasping or partial grasping at the new position in at least 10% of trials.
  • Pick-and-place: 100% success placing the bottle upright at the target is achieved after 8 iterations.The task requires both grasping and stable upright release, and the robot learned to grasp slightly above the bottle’s center of mass.
  • Simulation: PI-GPS achieves lower costs at convergence with fewer samples than the compared methods in the simulated point-mass experiment.PI-GPS-W and REPS tend to become unstable without converging when training complex neural network policies.
  • Simulation: The simulated comparison evaluates PI-GPS, PI-GPS-W, and REPS using 30 trajectory samples per iteration for one task instance.The curves select runs by lowest mean cost at iteration 50 or across all iterations.
  • Simulation: Training nonlinear policies is more stable when examples come from optimized local policies rather than noisy reweighted samples.The observed stability difference disappears for linear policies.

C. Evaluating generalization

The generalization experiments train visuomotor neural-network policies on randomized door and bottle configurations using global policy sampling. New task instances each iteration substantially improve performance on held-out random instances, though training requires reduced learning rates to avoid forgetting earlier instances.

  • Generalization setup: The policies map RGB camera images directly to torque commands for randomized door-opening and pick-and-place tasks.Training uses randomized task instances, including varied door poses and bottle positions.
  • Door opening: The door task varies position by 16cm in x, 8cm in y, and orientation by 60◦ within the training area.Training begins from five demonstrations corresponding to five door poses.
  • Training curves: Figure 7 compares success-rate training curves for door opening and pick-and-place under global policy sampling.Each iteration contains 50 trajectories, with 10 samples for each of five random task instances; dashed lines show local-sampling results.
  • Door opening: Global policy sampling raises door-opening success from 43.3% to 93.3% on 30 random test poses.The comparison is between repeated local sampling on five fixed instances and new random instances at every iteration.
  • Training limitation: Global sampling requires reducing the learning rate from 5 × 10^-3 to 10^-3 to prevent forgetting earlier task instances.Experience replay and reuse of old instances are suggested as ways to mitigate this issue.

2) Pick-and-place:

The pick-and-place policy was trained on increasingly broad bottle-pose regions and improved success on random poses, while requiring slower learning and careful optimization to avoid forgetting.

  • Pick-and-place: Training gradually expanded the bottle-pose region beyond demonstrations because the initial policy generalized poorly far outside that region.The continuation method maintained at least 10% grasp success without excessive added noise.
  • Pick-and-place: 86.7% success was achieved on 30 random bottle poses after global policy sampling, compared with 40% after initialization.The evaluation measured successful placement at the goal position.
  • Pick-and-place: Learning the pick-and-place behavior was slower than door opening and required more iterations.Performance did not improve continuously because later training instances were harder and more widely distributed.
  • Pick-and-place: The final policy reduced gripper-orientation variation and motion speed, using gripper compliance and slower placement to handle bottle grasping.These changes differed from the demonstrated behavior while supporting more careful placement.
  • Pick-and-place: Global policy sampling required an SGD learning rate of 10^-4 to avoid forgetting old task instances.This was lower than the 5 × 10^-3 rate used for local policy sampling.
Loading 1610.00529v2…