PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-End Training of Deep Visuomotor Policies

Sergey Levine, Chelsea Finn, Trevor Darrell, Pieter Abbeel

arXiv:1504.00702v5cs.LGcs.CVcs.RO

TL;DR

Hand-engineered perception and control components limit policy-search applications, motivating the question of whether joint end-to-end training improves visuomotor policies. The paper develops guided policy search for raw-image-to-torque policies and finds improved consistency, generalization, and complex manipulation performance across real-world tasks.

  • Problem

    Policy-search methods often rely on hand-engineered perception and control components, motivating whether jointly training perception and control improves sensorimotor policies.

  • Method

    The method combines guided policy search with a CNN policy, using trajectory-centric reinforcement learning to generate supervision for end-to-end visuomotor learning.

  • Results

    End-to-end visuomotor training improves consistency and generalization across four real-world manipulation tasks compared with separately trained vision and control components.

  • Takeaways & Limitations

    The method enables raw-camera robotic policies to execute complex manipulation skills while optimizing high-dimensional policies through supervised learning.

  • Takeaways & Limitations

    The method does not generalize to dramatically different settings, particularly when visual distractors occlude the manipulated object or alter its silhouette beyond training conditions.

Abstract

from arXiv · show

Policy search methods can allow robots to learn control policies for a wide range of tasks, but practical applications of policy search often require hand-engineered components for perception, state estimation, and low-level control. In this paper, we aim to answer the following question: does training the perception and control systems jointly end-to-end provide better performance than training each component separately? To this end, we develop a method that can be used to learn policies that map raw image observations directly to torques at the robot's motors. The policies are represented by deep convolutional neural networks (CNNs) with 92,000 parameters, and are trained using a partially observed guided policy search method, which transforms policy search into supervised learning, with supervision provided by a simple trajectory-centric reinforcement learning method. We evaluate our method on a range of real-world manipulation tasks that require close coordination between vision and control, such as screwing a cap onto a bottle, and present simulated comparisons to a range of prior policy search methods.

1. Introduction

The introduction frames autonomous robot perception and control as a major challenge and asks whether jointly training perception with control can produce more effective sensorimotor policies. It presents deep-network-based guided policy search and evaluates it on real-world manipulation tasks requiring visual coordination and contact handling.

  • Motivation: Autonomous operation remains challenging because designing perception and control software is difficult even for basic robotic tasks.Policy search methods are presented as a way for robots to learn new behaviors through experience.
  • Research question: The paper asks whether jointly training perception and control produces more effective sensorimotor policies than training them separately.Deep neural networks are used to represent policies that combine perception and control.
  • Method: The learned visuomotor policies directly use camera images to set motor torques on a PR2 robot.The policy input-output mapping is illustrated in Figure 1.
  • Method: The method develops guided policy search for sensorimotor deep learning and introduces a CNN architecture designed for robotic control.Guided policy search converts policy search into supervised learning by iteratively constructing training data through model-free trajectory optimization; the method is formalized as Bregman ADMM.
  • Evaluation: The method is evaluated on four PR2 manipulation tasks requiring localization, visual tracking, and complex contact dynamics.Tasks include inserting a block into a shape-sorting cube, screwing a cap onto a bottle, fitting a toy hammer claw under a nail, and placing a coat hanger on a rack.

2. Related Work

Prior robotics work applied reinforcement learning and policy search across diverse tasks, while deep learning was more established in visual recognition than robotic control. This work addresses visuomotor learning challenges with guided policy search and learned visual feedback without hand-specified features or controllers.

  • Reinforcement learning and policy search: Reinforcement learning and policy search have been applied to robotic games, manipulation, locomotion, and flight.Examples include table tennis, object manipulation, locomotion, and flight.
  • Convolutional neural networks: CNNs have a long history in computer vision and deep learning and have recently achieved excellent results on multiple vision benchmarks.The cited applications include classification and several major vision benchmarks.
  • Challenges in robotic control: Deep learning has been less prevalent in robotic control because backpropagation through dynamics and image formation is often impractical and numerically unstable.Long-range backpropagation can be unstable when the linearization of a suboptimal policy is unstable.
  • Prior sensorimotor learning: Earlier neural-network control used small, simple networks, while later work emphasized carefully designed reinforcement-learning policies and simpler sensorimotor tasks.More recent sensorimotor deep-learning studies addressed simple task-space motions or used unsupervised learning for low-dimensional representations.
  • Guided policy search: Guided policy search addresses complex observations and high-dimensional policy representations by optimizing policies through supervised learning.Its training data can be constructed using trajectory optimization under known dynamics.
  • Visual servoing: Unlike visual servoing, the proposed visuomotor policies are learned from real-world data without hand-specified feature points, feedback controllers, or camera calibration.This allows greater flexibility in how the visual signal is used.

3. Background and Overview

The approach combines supervised policy learning with trajectory-centric reinforcement learning, using a compact CNN architecture and pretraining to enable end-to-end visuomotor control with limited experience.

  • Approach: Guided policy search separates visuomotor policy learning into supervised learning and trajectory learning phases, making each easier than direct policy optimization.The method alternates between these phases to train policies for real robotic platforms.
  • Approach: The policy outputs a Gaussian action distribution whose nonlinear mean is a deep CNN and whose covariance is observation-independent and learned.The policy is represented as πθ(ut|ot) = N(µπ(ot), Σπ(ot)).
  • Approach: Alternating trajectory-centric RL with supervised learning supplies training data from the policy’s own state distribution, reducing compounding errors from long-horizon mistakes.The RL stage adapts to the current policy and provides supervision at states progressively closer to those the policy visits.
  • Policy architecture: The visuomotor architecture has 7 layers and around 92,000 parameters, transforming convolutional features into spatial feature points for feedback control.The spatial feature points are concatenated with robot configuration before fully connected layers produce torques.
  • Pretraining: A pretraining scheme initializes low-level visual processing independently to reduce the experience required for training visuomotor policies.The pretraining is intended to enable effective policies with a relatively small number of iterations.

4. Guided Policy Search with BADMM

Guided policy search reformulates policy search as supervised learning by alternating between trajectory optimization and fitting a CNN policy to guiding distributions. BADMM makes this optimization tractable with linear-Gaussian controllers and learned local dynamics, while retaining convergence guarantees.

  • Guided policy search: Guided policy search generates training trajectories with trajectory-centric reinforcement learning, then trains the final policy to predict actions from observations rather than full states.The final CNN policy can generalize across initial states represented by multiple training trajectories.
  • BADMM optimization: BADMM alternates supervised learning that matches πθ to p(τ) with trajectory optimization that updates p(τ), followed by dual-variable updates.This procedure inherits BADMM’s convergence guarantees; α is a step size and νt is set heuristically.
  • Guiding distributions: The guiding distribution p(τ) is chosen as a mixture of N Gaussian trajectory distributions, one per initial-state sample, yielding linear-Gaussian controllers and dynamics.This representation is efficient when dynamics are deterministic or have Gaussian or small noise, and it is sufficiently noise-tolerant for real physical systems.
  • Trajectory optimization: For unknown dynamics, the method fits local linear-Gaussian dynamics to sampled trajectories and optimizes the controller using these learned models.A separate time-varying dynamics model for each initial state can represent locally linearized continuous deterministic systems and can be learned from few real-world samples.
  • BADMM optimization: BADMM makes trajectory optimization substantially faster and easier to implement than the more complex non-convex forward-backward phase produced by the alternative formulation, especially with many trajectories.The trajectory optimization phase is convex under BADMM.

5. End-to-End Visuomotor Policies

This section presents an end-to-end visuomotor policy that maps monocular RGB images and robot configuration directly to joint torques using a deep CNN with spatial feature coordinates. Guided policy search trains the policy under full-state supervision before optimizing the observation-only network end-to-end.

  • Policy inputs and outputs: The policy runs at 20 Hz, mapping monocular RGB images and 7-DoF arm configurations to joint torques without directly observing the target object’s position.The configuration includes joint angles, end-effector pose, and their velocities.
  • CNN architecture: Three convolutional layers extract hierarchical local image features, with the third layer producing 32 response maps at 109 × 109 resolution.Each convolutional layer is followed by a rectifying nonlinearity.
  • Spatial representation: Spatial softmax and expected-position computations convert each feature map into a 2D spatial coordinate, which is concatenated with robot configuration and processed by two 40-unit rectified layers.The coordinate computation implements a differentiable soft-argmax through a fixed, sparse fully connected layer.
  • Spatial representation: Spatial softmax provides lateral inhibition that suppresses weak erroneous activations, making the policy more robust to distractors and supporting generalization.The resulting spatial coordinates can be manipulated into 3D positions or motor torques by the fully connected layers.
  • Training procedure: Guided policy search optimizes trajectories using the system’s full state while training the final policy only on observations, including unobserved target-object pose during training.This instrumented training setup supports deployment in uncontrolled real-world situations.
  • Training procedure: Training initializes vision layers and trajectory distributions using a fully observed setup, then optimizes motor layers before further end-to-end optimization of the full network.The staged procedure accelerates early training and avoids optimizing the full policy before the small motor-control layers are learned.

6. Experimental Evaluation

The evaluation spans simulated continuous-control tasks and real PR2 manipulation, showing that guided policy search can learn effective controllers with limited interaction and scale to visuomotor policies. Experiments also support the proposed spatial-softmax architecture and reveal both the benefits and limitations of end-to-end visual learning.

  • Simulated policy search: Guided policy search learned more effective simulated controllers with fewer samples than many prior policy search methods across insertion, swimming, walking, and manipulation tasks.The evaluation used a physical simulator and simpler nonvisual policies to compare algorithms across diverse tasks; prior methods struggled with medium-sized neural policies and more than 100 parameters.
  • Simulated policy search: On swimming, the method matched the linear-Gaussian performance while producing a smoother gait and reached comparable distances with many fewer samples than RWR or CEM.RWR reached 0.5m after 4000 samples, whereas CEM reached 2.1m after 3000 samples.
  • Real-robot manipulation: Real-robot trajectory optimization learned a wide variety of PR2 manipulation controllers using around 20-25 samples per controller.Each task took about ten minutes overall, with only 3-4 minutes involving system interaction.
  • Architecture evaluation: The proposed spatial-softmax and expectation architecture substantially improved pose-estimation accuracy over standard convolutional architectures by learning concise feature-point representations.The architecture also benefits from convolutional weight sharing, which helps reduce overfitting on relatively small datasets.
  • End-to-end visuomotor learning: End-to-end policies learn task-relevant visual features and show moderate tolerance to visually separated distractors, but perform poorly under drastic backdrop changes.End-to-end training shifts feature points toward task-relevant objects and away from background objects, differing from pose-prediction initialization.
  • End-to-end visuomotor learning: The feature-point representation remains restrictive because it assumes every learned feature is always present and appears only once in the image.The paper notes that a more flexible architecture retaining a concise feature representation could further improve policy performance.

7. Discussion and Future Work

The paper presents end-to-end learning of raw-camera robotic control policies through a convolutional architecture and guided policy search that separates trajectory optimization from supervised learning. It identifies limited generalization and full-state training requirements as key limitations, and proposes recurrent policies and richer sensory modalities as future directions.

  • Contributions: The method learns robotic control policies from raw monocular-camera input using a convolutional architecture trained end-to-end with guided policy search.Guided policy search decomposes learning into trajectory optimization using full state information and supervised learning using only observations.
  • Limitations: The method shows moderate generalization over scene variations but fails in dramatically different settings, particularly when distractors occlude or alter the manipulated object’s silhouette.The paper suggests CNNs may ultimately learn invariance to irrelevant distractors.
  • Limitations: Full-state training enables linear-Gaussian controllers and efficient guided policy search with few samples, but restricts the tasks the method can address.Some tasks require only consistent object placement, whereas freely moving objects require more extensive instrumentation.
  • Future Work: Future work will investigate recurrent policies that retain past observations to handle extensive occlusions and extend the method to more tasks and sensory modalities.Proposed modalities include haptic input from pressure sensors and auditory input.

Appendix A. Guided Policy Search Algorithm Details

Appendix A describes implementation details of the BADMM-based guided policy search algorithm and the linear-Gaussian controller optimization method.

  • Appendix A. Guided Policy Search Algorithm Details: The appendix focuses on implementation details for the BADMM-based guided policy search algorithm and linear-Gaussian controller optimization.It covers both the guided policy search procedure and controller optimization method.

A.1 BADMM Dual Variables and Weight Adjustment

The method uses BADMM dual-variable updates with a stable step size and an adaptive schedule for KL-divergence penalty weights. Dual-variable expectations are estimated from sampled trajectories and Gaussian policy and trajectory distributions.

  • Weight adjustment: α = 0.1 is used in all experiments because it was more stable than α = 1.0, with ν_t initialized to 0.01.The weights are adjusted using the average and standard deviation of KL-divergence across time steps.
  • Weight adjustment: ν_t is doubled when its time-step KL-divergence exceeds the average and halved when it is at least two standard deviations below average.This schedule aims to keep policy and trajectory agreement roughly uniform across time steps.
  • Dual-variable updates: Dual variables λ_{μt} are updated using expectations over sampled states and closed-form expectations over actions under π_θ and p.The state expectations use the latest batch of sampled trajectories, while action expectations use the means of conditional Gaussian distributions.

A.2 Policy Variance Optimization · A.3 Dynamics Fitting

Policy variance is optimized analytically from the objective, while dynamics fitting uses global, data-driven priors to reduce the sample burden of time-step-specific linear regression. A Gaussian mixture prior further models the piecewise-linear modes of contact-rich articulated systems, with practical safeguards against overfitting.

  • A.2 Policy Variance Optimization: Policy variance optimization isolates the terms of Lθ(θ, p) that depend on the observation-independent Gaussian policy covariance Σπ.The covariance’s observation dependence could be added straightforwardly, but is omitted in the described formulation.
  • A.2 Policy Variance Optimization: Differentiating the covariance-dependent objective and setting its derivative to zero yields an equation for Σπ.The resulting optimization is therefore expressed through a stationary-condition equation.
  • A.2 Policy Variance Optimization: The expectation under π(x_t) is omitted because C_ti does not depend on x_t.This simplification applies directly to the covariance-optimization derivation.
  • A.3 Dynamics Fitting: Dynamics fitting is performed at each iteration using physical-system samples generated by the previous controller π̂(u_t|x_t).The fitted linear-Gaussian model represents p(x_t+1|x_t, u_t).
  • A.3 Dynamics Fitting: Direct linear regression estimates f_x, f_u, and f_c and fits F_t from errors, but its sample complexity scales with the dimensionality of x_t.This can require impractically many samples for high-dimensional robotic systems.
  • A.3 Dynamics Fitting: A global model pools transitions across all time steps and three prior iterations, then serves as a prior for each time-step dynamics fit.The global model need not be an accurate forward model; it only needs to reduce linear-regression sample complexity.
  • A.3 Dynamics Fitting: Recasting regression as Gaussian fitting enables a normal-inverse-Wishart prior before conditioning on [x_t; u_t] to recover linear-Gaussian dynamics.The prior combines empirical dataset statistics with parameters Φ, μ_0, m, and n_0.
  • A.3 Dynamics Fitting: The global model is a Gaussian mixture over [x; u; x′], approximating articulated contact dynamics as piecewise linear with mixture elements representing distinct modes.In practice, the prior used at least 40 samples per mixture element or 20 total mixture elements, whichever was lower; excessive mixtures could overfit early iterations.

A.4 Trajectory Optimization

The constrained trajectory optimization is solved by locally approximating costs and policies quadratically, then applying an LQR backward pass within dual gradient descent. This produces optimal linear feedback or linear-Gaussian controllers for the maximum-entropy objective.

  • Constrained optimization: The constrained trajectory optimization uses an LQR backward pass to optimize the constrained objective.The optimization is formulated within the dual gradient descent procedure.
  • Constrained optimization: The augmented Lagrangian is locally approximated quadratically by expanding ℓ(x_t, u_t) and fitting a linear Gaussian to π_θ(u_t|x_t).The resulting primal optimization can be solved with a standard LQR backward pass.
  • Maximum-entropy solution: The maximum-entropy trajectory problem minimizes E_p(τ)[c̃(τ)] − H(p(τ)), which can be solved using LQR.The transformed cost includes the original cost and a log-density term weighted by η and ν_t.
  • Maximum-entropy solution: The standard LQR backward pass computes feedback terms K_t and k_t from the quadratic cost approximation and linear-Gaussian dynamics.For the maximum-entropy objective, the optimal controller is linear-Gaussian: p(u_t|x_t) = N(K_tx_t + k_t; Q^-1_u,u_t).

Appendix B. Experimental Setup Details

Appendix B provides a detailed summary of the experimental setup used in the paper’s simulated and real-world experiments.

  • The appendix details the experimental setup for both simulated and real-world experiments.

B.1 Simulated Experiment Details

The simulated experiments used MuJoCo with frictional contacts and torque actuation, while linear-Gaussian controllers introduced noise and were initialized near the starting state. Task-specific simulations varied in state/action dimensionality, rollout duration, and physical modeling across peg insertion, octopus arm, swimmer, and walker environments.

  • Simulation setup: All simulations used MuJoCo with frictional contacts and torque motors, without added control or state noise.Noise arose naturally from the linear-Gaussian controllers.
  • Simulation setup: Linear-Gaussian controllers were initialized near the initial state using proportional-derivative linear feedback for all tasks except one unspecified case.The supplied passage truncates the exception.
  • Peg insertion: Peg insertion used 6 state and 2 action dimensions in 2D, 12 state dimensions in 3D, and 800-step rollouts over 8 seconds at 100 Hz.The 3D arm had 3 shoulder, 1 elbow, and 2 wrist degrees of freedom.
  • Peg insertion: The peg-insertion cost used a soft combination of ℓ2 and ℓ1-like penalties to encourage precise target reaching while penalizing distant states more strongly.The weights were wu = 10−6 and wp = 1; the 3D task took longer without the square-root term.
  • Other simulated tasks: The octopus arm modeled six four-sided chambers with simulated muscles, while the swimmer used 3 links, 5 degrees of freedom, 10 state dimensions, 2 torque actions, and 400-step rollouts.The swimmer simulation applied drag to each link and ran for 20 seconds at 20 Hz.
  • Other simulated tasks: The bipedal walker had 9 degrees of freedom, 18 state dimensions, and 6 actions, with 500-step rollouts over 5 seconds at 100 Hz.The supplied passages specify additional walker cost-function details but do not provide the full cost expression.

B.2 Robotic Experiment Details

Robotic experiments used a PR2 controlled at 20 Hz from downsampled RGB images, with one arm executing learned policies and the other varying initial conditions. Tasks varied object poses and grasp angles during training, with task-specific success criteria and controller initialization.

  • Robot and control setup: Experiments used a PR2 at 20 Hz with direct effort control, 240 × 240 × 3 RGB images, and one 7 DoF arm for learned policies.The second arm moved objects to automatically vary initial conditions, and the camera remained fixed in each experiment.
  • Controller initialization: Linear-Gaussian controllers were initialized near the initial state with diagonal noise, inverse-effective-mass covariance, and LQR feedback based on an approximate linear model.The inverse effective mass and approximate model came from manufacturer-provided PR2 information.
  • Task variations: Training varied task configurations: the coat hanger used two grasp angles and three rack distances, while the toy hammer used three grasp angles and five nail positions.Coat-hanger grasp angles differed by about 35° and rack positions by about 10 cm; hammer angles differed by 22.5° across a 45° range.
  • Success criteria: Success required the coat hanger to remain on the rack after release, the trapezoid to be completely inside the cube, and the bottle cap to resist vertical removal.These criteria evaluated the final physical outcome of each manipulation task.
  • Task variations: The shape-sorting cube and bottle-cap tasks each used nine object positions across a 16 cm × 10 cm rectangular region during training.The cube task inserted a red trapezoid into a matching hole, while the bottle-cap task required screwing a cap onto a bottle.
Loading 1504.00702v5…