PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Spatial Autoencoders for Visuomotor Learning

Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, Pieter Abbeel

arXiv:1509.06113v3cs.LGcs.CVcs.RO

TL;DR

Robotic reinforcement learning needs detailed task-relevant state representations, but manually designing them is difficult and learning them from images remains challenging. The paper learns spatial feature points with a deep spatial autoencoder and uses them in trajectory-centric reinforcement learning, demonstrating closed-loop manipulation across several PR2 tasks. The approach learns task-relevant object tracking and manipulation from camera images using tens of real-robot trials.

  • Problem

    Robotic reinforcement learning requires detailed representations of task-relevant objects, while manually designing state spaces and learning suitable visual representations remain difficult.

  • Method

    A deep spatial autoencoder learns image-space feature points, which are combined with robot configuration and used by trajectory-centric reinforcement learning.

  • Results

    The method learned vision-based manipulation skills across pushing, scooping, spatula-based lifting, and rope-hanging tasks, with controllers using visual state information for object manipulation.

  • Takeaways & Limitations

    Spatial feature representations can support compact, continuous-control state spaces for real-world manipulation learned with tens of robot trials.

Abstract

from arXiv · show

Reinforcement learning provides a powerful and flexible framework for automated acquisition of robotic motion skills. However, applying reinforcement learning requires a sufficiently detailed representation of the state, including the configuration of task-relevant objects. We present an approach that automates state-space construction by learning a state representation directly from camera images. Our method uses a deep spatial autoencoder to acquire a set of feature points that describe the environment for the current task, such as the positions of objects, and then learns a motion skill with these feature points using an efficient reinforcement learning method based on local linear models. The resulting controller reacts continuously to the learned feature points, allowing the robot to dynamically manipulate objects in the world with closed-loop control. We demonstrate our method with a PR2 robot on tasks that include pushing a free-standing toy block, picking up a bag of rice using a spatula, and hanging a loop of rope on a hook at various positions. In each task, our method automatically learns to track task-relevant objects and manipulate their configuration with the robot's arm.

I. INTRODUCTION

The paper addresses the difficulty of manually designing state representations for robotic reinforcement learning by learning task-relevant visual features from camera images. It combines spatial feature-point representations with sample-efficient trajectory-centric reinforcement learning for vision-based manipulation.

  • Motivation: Manual enumeration, detection, and encoding of scene objects makes reinforcement learning difficult to apply across complex, unstructured manipulation tasks.The paper seeks to automate task-specific vision and state-space construction with minimal human supervision.
  • Motivation: Learning representations directly from camera images remains difficult because unsupervised methods often require large datasets and arbitrary visual features may be poorly suited to control.
  • Approach: Deep spatial autoencoders learn feature points that encode object configurations while using fewer nonconvolutional parameters for data-efficient representation learning.Their real-valued positional features are intended to be more amenable to control than discrete or sparse features.
  • Control: The learned controllers use visual feature points for closed-loop control of object configurations, while avoiding hand-specified visual features and camera calibration.
  • Approach: The method combines unsupervised deep spatial autoencoders with simple, sample-efficient trajectory-centric reinforcement learning for vision-based manipulation.The same algorithm is demonstrated on pushing, scooping, spatula-based lifting, and rope hanging tasks using PR2 onboard sensors.
  • Scope: Unlike approaches requiring impractical data or instrumented training, this method learns pushing, tossing, and scooping without prior knowledge of scene objects or additional instrumentation.The paper contrasts this capability with prior methods that require mechanisms such as motion capture.

III. PRELIMINARIES

The reinforcement-learning preliminaries formulate skill learning as minimizing expected trajectory cost under learned dynamics and a controller. The method uses iterative trajectory-centric optimization with time-varying linear-Gaussian controllers and constrained policy updates.

  • Trajectory optimization: The objective is to minimize expected total cost over trajectories containing states and motor-torque actions.The trajectory distribution depends on the initial-state distribution, dynamics, and controller.
  • Optimization: Each iteration gathers N = 5 trajectory samples, fits dynamics by linear regression with a GMM prior, and updates the controller under a KL-divergence constraint.
  • Controller: The controller is a time-varying linear-Gaussian policy optimized through a trajectory-centric algorithm.This controller can be viewed as a trajectory with time-varying linear stabilization.
  • Optimization: When costs are quadratic and dynamics are linear-Gaussian, the optimal controller can be obtained with LQR.The implementation fits dynamics to data, making the procedure a variant of iterative LQR.
  • Stability: The KL-divergence constraint limits controller changes so fitted dynamics remain valid in the visited state-space region.

B. Learning Nonlinear Policies with Guided Policy Search

Linear-Gaussian controllers are efficient but limited in expressiveness, so the paper combines them with guided policy search to learn a nonlinear policy for rope hanging across hook positions.

  • Controller limitations: Linear-Gaussian controllers are easy to train but essentially encode trajectory-following controllers and cannot express every control strategy.
  • Guided policy search: Guided policy search uses simpler controllers to generate supervision for training more complex policies such as deep neural networks.
  • Application: For rope hanging, the method trains a separate linear-Gaussian controller for each hook position, then unifies them into one neural-network policy.
  • Generalization: The unified nonlinear policy is designed to generalize across hook positions, including positions not used to train the individual controllers.

IV. ALGORITHM OVERVIEW

The algorithm first explores without vision, then learns a spatial visual representation from collected images, and finally trains a vision-based controller using the combined robot and visual state.

  • IV. ALGORITHM OVERVIEW: The first stage optimizes a controller without vision using robot configuration variables to produce focused task-specific exploration.The initial controller is rarely sufficient for tasks requiring object tracking but supplies images for representation learning.
  • IV. ALGORITHM OVERVIEW: The second stage trains a deep spatial autoencoder on images collected by the initial controller and extracts feature points describing object configurations.
  • IV. ALGORITHM OVERVIEW: The final state concatenates robot configuration with learned visual features so the controller can use both proprioceptive and image-based information.
  • IV. ALGORITHM OVERVIEW: The third stage trains a vision-based controller with the same trajectory-centric reinforcement-learning algorithm, using costs defined over visual features and robot configuration.
  • IV. ALGORITHM OVERVIEW: A suitable learned state representation should be temporally coherent, task-relevant, and compatible with time-varying linear dynamical modeling.The paper targets Cartesian coordinates that move with relevant objects over time.

A. Deep Spatial Autoencoders

The deep spatial autoencoder compresses images into feature-point coordinates that emphasize where relevant visual features are located, then reconstructs a downsampled image from them. These coordinates, their velocities, and the robot configuration form a control-oriented state representation.

  • Representation: The bottleneck encodes feature locations rather than feature values, forcing the representation to capture the spatial organization of objects.This “where” representation is intended to provide coordinates that are useful for robotic control.
  • Control state: The reinforcement-learning state concatenates the learned feature encoding and its time derivatives with the robot’s configuration.The feature derivatives provide visual feature velocities for dynamic tasks.
  • Architecture: The encoder uses convolutional features, spatial softmax, and expectation to extract one image-space point of maximal activation per channel.The decoder reconstructs a downsampled grayscale image from these feature points.
  • Representation: Learned feature points often select task-relevant moving objects while ignoring background features, but exactly one point per channel cannot gracefully represent occlusions or repeated objects.The fixed one-point-per-channel structure is therefore a stated boundary of the representation.
  • Training objective: The autoencoder adds a penalty encouraging feature points to change velocity slowly, making their dynamics more suitable for modeling.The penalty is based on changes in successive feature-point velocities.

B. Filtering and Feature Pruning Using Feature Presence

Feature presence identifies reliable spatial features, enables Kalman filtering through intermittent observations, and supports pruning features that are absent or difficult to predict.

  • Motivation: Lighting and camera-exposure changes can create noisy features that improve reconstruction but are difficult for the controller’s dynamics model to fit.This motivates filtering and selecting features rather than retaining every learned point.
  • Feature presence: Feature presence measures the softmax probability mass concentrated at a feature point and marks it present when the value reaches β = 0.95.For boundary points, presence sums probability over a 3 × 3 neighborhood.
  • Filtering: A second-order Kalman filter receives feature-point observations only when presence exceeds the threshold, producing smooth predictions during occlusion or absence.The filter parameters are fitted using expectation maximization.
  • Feature pruning: Goal-state feature positions are averaged over fifty images collected during two seconds, while features that are not present are pruned from the state.This procedure reduces noise in the target feature representation.
  • Predictiveness: Feature predictiveness is assessed by fitting dynamics models for the selected state and the remaining features, then evaluating how well the selected state predicts omitted features.The measure is defined to assess whether the state captures predictable scene dynamics.
  • Feature pruning: Iterative likelihood-based pruning provides a predictiveness ranking, while feature presence offers a faster approximation for selecting highly predictive features.Low-ranked features were consistently absent or extremely noisy in the reported evaluation.

VI. CONTROL OVER VISUAL FEATURES

The control procedure first explores with a vision-free controller, learns feature points from collected images, and then optimizes a final controller using visual goals for feedback control.

  • Exploration: An exploration controller is first trained using only the robot’s configuration, even though its configuration-based goal is typically inadequate for task completion.Its role is to explore the state space and collect varied images.
  • Visual goal: The target visual state is specified by showing the robot an image of the desired scene configuration and encoding it as target features.The cost depends on distance to those target features and uses a short image sequence to reduce noise.
  • State construction: The collected images train a deep spatial autoencoder, whose selected feature points and velocities are added to the robot’s configuration to form the full state.The full state is then used by the final reinforcement-learning controller.
  • Final control: Algorithm 2 trains the final controller with the full visual state and image-derived cost to perform feedback control on robot configuration and visual features.The procedure separates image collection and representation learning from final controller optimization.

VII. EXPERIMENTAL EVALUATION

The evaluation tests whether learned visual representations support manipulation behaviors requiring objects to be tracked with vision, comparing the proposed controllers against vision-free controllers and other learned representations.

  • Evaluation design: The experiments compare vision-based controllers with controllers that optimize toward a goal end-effector position without using vision.The comparison targets behaviors requiring tracking of objects visible only through vision.

A. Experimental Tasks

The evaluation covers vision-dependent manipulation, rope-hook generalization, prior autoencoder comparisons, and sample efficiency on a PR2 robot.

  • A. Experimental Tasks: The tasks include sliding a lego block, transferring a bag with a spoon, scooping rice with a spatula, and hanging rope on hooks.The rope task uses four training positions and tests three unseen positions.
  • A. Experimental Tasks: Controllers without vision fail the tasks, while the learned state contains information necessary for completing them.Each controller was evaluated over 10 trials using average distance to goal or success rate.
  • A. Experimental Tasks: The vision-based rope policy succeeds at three test positions unseen during training, whereas the policy without vision generally fails.Training positions were spaced 8 cm apart.
  • A. Experimental Tasks: Prior autoencoder architectures struggle with the high-dimensional real-world tasks, and stable control requires both smoothness and feature pruning.A larger baseline achieved lower reconstruction loss but did not reliably integrate visual input into control.
  • A. Experimental Tasks: Each autoencoder uses about 50 trials, and final controller training adds 50–75 trials, totaling roughly 10–15 minutes of robot interaction per controller.The autoencoder data comprise approximately 5000 image frames per task.

VIII. DISCUSSION AND FUTURE WORK

The method learns compact spatial state representations from images for continuous-control manipulation, while the authors identify exploration and sensing boundaries for future work.

  • VIII. DISCUSSION AND FUTURE WORK: Deep spatial autoencoders learn compact feature-point states from high-dimensional images, and trajectory-centric reinforcement learning uses them to learn manipulation skills.The learned coordinates are suited to continuous control, and skills are learned with tens of real-robot trials.
  • VIII. DISCUSSION AND FUTURE WORK: Tasks requiring broad state-space visitation can be difficult to explore without vision before representation learning.The authors propose interleaving representation learning with controller optimization and adding depth or haptic sensing.
  • VIII. DISCUSSION AND FUTURE WORK: The paper demonstrates a step toward learning compact, portable state representations for complex environments directly from raw sensory inputs.The authors point to deformable-object manipulation and navigation in unknown environments as future application areas.

APPENDIX

The appendix specifies the PR2 setup, visual-and-pose cost, task definitions, and rope-hook evaluation protocol.

  • Robotic Experiment Details: Experiments use a PR2 arm controlled at 20 Hz with direct effort control, 240 × 240 RGB images, and 5-second episodes.The camera remains fixed in each experiment.
  • Robotic Experiment Details: The cost function combines visual-feature and gripper-pose errors with quadratic and logarithmic terms weighted by w_l2 = 10^-3, w_log = 1.0, and w_u = 10^-2.The quadratic term encourages progress when far from the target, while the logarithmic term encourages precise reaching.
  • Robotic Experiment Details: The lego task measures the distance from the goal block position to the nearest final corner after pushing the block 30 cm.The nearest corner is used even when some baselines flip the block.
  • Robotic Experiment Details: Bag-transfer and rice-scooping trials are successful when the bag is entirely inside the bowl without extending outside it.The transfer uses a spoon, while the rice task uses a spatula.
  • Robotic Experiment Details: For rope-hook evaluation, four training positions span 24 cm and three midpoint positions are held out for testing.Failed trials with the authors’ approach were often 1–2 mm off, while no-vision failures were typically several centimeters off.

B. Neural Network Architectures for Prior Work Methods

The appendix describes two prior-work neural autoencoder baselines used for comparison with the proposed architecture.

  • B. Neural Network Architectures for Prior Work Methods: The first baseline follows Lange et al. and processes converted 60 × 60 grayscale images through eight encoder and eight decoder layers.Its final convolution uses stride 2.
  • B. Neural Network Architectures for Prior Work Methods: The second baseline directly processes 240 × 240 × 3 images with convolution, batch normalization, ReLU nonlinearities, and max-pooling.Its initial layers use 64 7 × 7 filters, then 32 and 16 filters with 5 × 5 kernels.
Loading 1509.06113v3…