PaperlayerView exact revision on arXiv

Source-linked AI summary

Composable Deep Reinforcement Learning for Robotic Manipulation

Tuomas Haarnoja, Vitchyr Pong, Aurick Zhou, Murtaza Dalal, Pieter Abbeel, Sergey Levine

arXiv:1803.06773v1cs.LGcs.AIcs.ROstat.ML

TL;DR

Real-world robotic manipulation exposes the sample inefficiency of model-free deep reinforcement learning under limited interaction. The paper applies maximum-entropy soft Q-learning with expressive policies and policy composition, finding substantially greater sample efficiency than prior model-free methods and successful composition in simulated and physical tasks.

  • Problem

    Model-free deep reinforcement learning can perform poorly when interaction with the environment is limited, as in many real-world robotic tasks.

  • Method

    The paper uses soft Q-learning to train expressive maximum-entropy policies and composes previously learned policies into compound skills.

  • Results

    Soft Q-learning substantially outperforms prior model-free deep reinforcement learning methods in sample efficiency, while policy composition succeeds in simulated and real-world tasks.

  • Takeaways & Limitations

    Composing maximum-entropy policies provides a framework for initializing compound robotic skills from previously learned building blocks.

Abstract

from arXiv · show

Model-free deep reinforcement learning has been shown to exhibit good performance in domains ranging from video games to simulated robotic manipulation and locomotion. However, model-free methods are known to perform poorly when the interaction time with the environment is limited, as is the case for most real-world robotic tasks. In this paper, we study how maximum entropy policies trained using soft Q-learning can be applied to real-world robotic manipulation. The application of this method to real-world manipulation is facilitated by two important features of soft Q-learning. First, soft Q-learning can learn multimodal exploration strategies by learning policies represented by expressive energy-based models. Second, we show that policies learned with soft Q-learning can be composed to create new policies, and that the optimality of the resulting policy can be bounded in terms of the divergence between the composed policies. This compositionality provides an especially valuable tool for real-world manipulation, where constructing new policies by composing existing skills can provide a large gain in efficiency over training from scratch. Our experimental evaluation demonstrates that soft Q-learning is substantially more sample efficient than prior model-free deep reinforcement learning methods, and that compositionality can be performed for both simulated and real-world tasks.

I. INTRODUCTION

The paper develops soft Q-learning for real-world robotic manipulation, targeting the limited interaction budgets that challenge model-free deep reinforcement learning. It emphasizes expressive maximum-entropy policies, sample-efficient skill learning, and composition of learned policies into compound tasks.

  • Motivation: Model-free reinforcement learning offers flexible behavioral representations but is difficult to apply when real-world robotic interaction is limited.Neural networks can represent complex behaviors, while reinforcement learning provides a framework for sequential decision making.
  • Approach: Soft Q-learning uses maximum-entropy policies to support informed exploration and multimodal behavior through expressive energy-based representations.Higher-return actions receive greater sampling probability while all actions retain non-zero probability.
  • Contributions: The proposed framework learns robotic manipulation skills with expressive neural-network policies and reports greater sample efficiency than DDPG and NAF.The method is evaluated on a real robotic system without demonstrations, simulated experience, or additional supervision.
  • Compositionality: Policy composition combines constituent skills to initialize compound policies that satisfy multiple objectives simultaneously.The paper distinguishes this conjunction-like composition from soft maximization, which selects among alternative objectives.
  • Scope: The work extends soft Q-learning for learning individual manipulation tasks and constituent policies that can be composed into new controllers.The formal setting uses continuous state and action spaces with unknown transition dynamics.

B. Maximum Entropy Reinforcement Learning

Maximum-entropy reinforcement learning augments reward maximization with policy entropy, encouraging stochastic and potentially multimodal behavior. Soft Q-learning optimizes this objective with expressive energy-based policies and soft Bellman updates that converge toward the optimal soft Q-function.

  • Objective: The maximum-entropy objective augments expected cumulative reward with expected policy entropy, favoring stochastic policies.The temperature α controls the trade-off between reward and entropy.
  • Objective: Maximum-entropy policies explore broadly while retaining multiple modes of optimal behavior when several actions are similarly attractive.As α approaches zero, the conventional reward-only objective is recovered.
  • Soft Q-Learning: Soft Q-learning represents the optimal policy as an expressive energy-based model whose negative energy is the Q-function.The method is chosen because it can represent arbitrarily complex action distributions for continuous-action tasks.
  • Soft Q-Learning: Soft Q-learning samples from the intractable energy-based policy using amortized Stein variational descent and a stochastic neural network.The sampling procedure avoids requiring direct evaluation of the policy partition function.
  • Soft Bellman Updates: The soft Bellman backup is a contraction, making the optimal Q-function the fixed point of repeated soft Bellman updates.In practice, a parametric function approximator is trained by minimizing the squared soft Bellman residual.

D. Baselines

The paper contrasts soft Q-learning with deterministic baselines and motivates composing separately learned policies to solve compound robotic tasks efficiently. Its composition analysis combines soft Q-functions and bounds the resulting policy's suboptimality using constituent-policy divergence.

  • Baselines: DDPG and NAF optimize conventional maximum return with deterministic policies, whereas soft Q-learning uses maximum-entropy exploration.DDPG uses an approximate hard maximum, while NAF uses an exact hard maximum and a quadratic action-value form.
  • Baselines: Soft Q-learning explores broadly when action values are similar while avoiding actions that are clearly sub-optimal.This explicitly balances exploration and exploitation.
  • Baselines: NAF's quadratic Q-function and deterministic policy limit its representational power and prevent diverse or multimodal solutions.The quadratic form enables closed-form maximization but constrains the policy class.
  • Compositionality: Policy composition combines constituent skills conjunctively, so the resulting policy can solve multiple tasks simultaneously rather than one task at a time.The paper contrasts this with reward composition that yields a disjunctive alternative between constituent tasks.
  • Compositionality: Adding constituent Q-functions does not generally recover the combined-task Q-function, but the resulting regret is bounded by the divergence between the composed policies.The bound is useful when constituent policies agree on actions or are indifferent to one another's actions.
  • Compositionality: Training individual policies and later combining them avoids directly optimizing every compound reward combination, whose number grows exponentially with the number of tasks.The framework uses compositionality to initialize compound skills from previously learned components.

B. Bounding the Sub-Optimality of Composed Policies

The paper analyzes the suboptimality of a policy formed by additively composing soft Q-functions. Its bounds relate approximation and policy regret to divergence between the optimal constituent policies, especially along states visited by the composed policy.

  • Bound construction: For two optimal policies, the analysis defines QΣ as the average of their soft Q-functions and compares it with the optimal soft Q-function for the averaged reward.The derivation treats α = 1 and notes that extensions to more policies and other α values are straightforward.
  • Bound construction: The additive composition of Q-functions never overestimates the optimal combined Q-function by more than the divergence of the constituent policies.The bound uses the Rényi divergence of order 1/2.
  • Policy-value bound: The value of the composed policy is bounded through soft policy evaluation, with a fixed-point term determined by the preceding Q-function bound.The theorem introduces D∗ as the fixed point used in this value bound.
  • Policy-value bound: The regret bound becomes tighter when constituent policies agree in states visited by the composed policy.The paper describes the bound as likely loose but states that regret decreases as policy divergence decreases.
  • Implications: Deterministic policies have infinite divergence unless identical, whereas highly stochastic maximum-entropy policies have more overlap and are expected to be more composable.This provides the paper's rationale for using soft Q-learning to produce composable policies.

V. EXPERIMENTS

The experiments evaluate soft Q-learning for sample-efficient robotic manipulation and test whether maximum-entropy policy composition can construct compound skills from previously trained components.

  • Experiments: The experiments demonstrate fast and reliable soft Q-learning in simulated and real-world robotic manipulation domains.The evaluation includes a simulated pushing task and real-world manipulation experiments.
  • Experiments: Soft Q-learning achieves substantially better sample complexity than existing model-free deep reinforcement learning methods.The passage presents sample complexity as a central experimental comparison.
  • Experiments: The experiments show that maximum-entropy policy compositionality can provide a practical tool for building compound skills from previously trained components.This tests composition as an alternative to learning each compound skill from scratch.

A. Experimental Setup

The study uses MuJoCo for simulation and a 7-DoF Sawyer manipulator for real-world experiments, with torque actions and proprioceptive, end-effector, object, and force observations.

  • Experimental Setup: Simulated experiments use the MuJoCo physics engine, while real-world experiments use a 7-DoF Sawyer robotic manipulator.The real-world setup applies torque commands at each joint.
  • Experimental Setup: Observations include joint angles, angular velocities, and end-effector position; simulation additionally includes object coordinates.Sawyer experiments also include end-effector forces estimated from actuator currents.
  • Experimental Setup: The Q-function and policies use two-layer neural networks with 100 or 200 units per layer and rectifier linear activations.This parameterization is used for both Q-functions and policies.

B. Composing Policies for Pushing in Simulation

The study composes independently learned pushing policies by adding their Q-functions, producing compound policies without new environmental samples. In simulation, composition is fast and generally effective, while direct offline training on the composed task fails to converge reasonably.

  • B. Composing Policies for Pushing in Simulation: Policies for separate cylinder-pushing goals are combined by adding their Q-functions to form QΣ, from which a compound policy is extracted.The constituent tasks specify target positions along separate axes, enabling combined objectives such as bottom-left, bottom-middle, and bottom-right.
  • B. Composing Policies for Pushing in Simulation: The composed policy satisfies both original pushing goals by reaching their intersection rather than averaging the constituent policies’ final locations.The combined policy reaches the lower intersection of the target lines without additional environmental samples.
  • B. Composing Policies for Pushing in Simulation: SQL and DDPG perform well across the combined simulation tasks, whereas NAF combines some policies but fails on others.NAF’s unimodal Gaussian policies are poorly suited to composition.
  • B. Composing Policies for Pushing in Simulation: Extracting a policy from a composed Q-function takes virtually no time compared with training a Q-function and policy from scratch.The proposed offline extraction uses data collected for the constituent Q-functions, while online SQL requires orders of magnitude more gradient steps and new environmental experience.
  • B. Composing Policies for Pushing in Simulation: Offline SQL trained directly on the composed task using pre-collected constituent-task data fails to converge in reasonable time.This contrasts with direct extraction from the additively composed Q-function.
  • B. Composing Policies for Pushing in Simulation: Compound policies can be obtained quickly by adding constituent Q-functions, and SQL-based policies can be reused to form compound real-world skills.The paper presents composition as a way to build new skills from previously trained components.

2) Lego Block Stacking:

Soft Q-learning supports efficient and robust robotic manipulation, including Lego stacking and composing obstacle avoidance with stacking. These experiments show that combining learned policies can solve compound real-world tasks without retraining from scratch.

  • SQL Reaching: SQL learns Sawyer reaching substantially faster than DDPG and NAF, including when target locations are randomly sampled.The random target is provided by concatenating the desired location to the observation vector.
  • Lego Block Stacking: After two hours, the Lego-stacking policy fully converged and achieved a 100% success rate over 20 trials.The robot first inserted the block after half an hour and successfully recovered from forced configurations on every robustness test.
  • Lego Block Stacking: The stacking policy recovered from configurations far outside normal trajectories and eventually completed the task every time.The authors attribute this robustness to the diverse exploration induced by maximum entropy policies.
  • Composing Policies: The avoidance policy never stacked the blocks, while the stacking policy collided with the obstacle every time and succeeded 30% of the time.These policies were evaluated on the insertion task with the obstacle present.
  • Composing Policies: The combined policy avoided the obstacle and stacked the blocks successfully 100% of the time.The evaluation executed each policy 10 times and counted successful insertions.
  • Composing Policies: Policy compositionality can make compound real-world skills more efficient than retraining policies for every reward combination.The paper frames this as useful where retraining is time-consuming and expensive.

A. Proof of Lemma 1

The proof of Lemma 1 uses induction with soft Bellman backups to bound the composed soft Q-function using policy divergence and an iteratively defined correction term.

  • A. Proof of Lemma 1: The composed soft Q-function is defined from the soft Q-functions of the two constituent optimal policies.The combined reward is formed by averaging the constituent rewards.
  • A. Proof of Lemma 1: The induction begins with Q^(0)=QΣ and C^(0)=0, so the lower-bound inequality holds trivially at k=0.The proof assumes Q^(k) ≥ QΣ − C^(k) for the induction step.
  • A. Proof of Lemma 1: Soft Bellman iteration converges to the optimal soft Q-function from any bounded initialization, allowing the bound to hold at the limit.The proof uses the inductive inequality after repeated backups.
  • A. Proof of Lemma 1: Applying the soft Bellman backup propagates the induction bound while introducing a correction involving Rényi divergence and the maximum future correction.The divergence is of order 1/2.
  • A. Proof of Lemma 1: The upper bound follows analogously by initializing at QΣ and using the positivity of Rényi divergence.The argument defines the corresponding correction recursion and applies the same iterative structure.

B. Proof of Theorem 1

The proof of Theorem 1 bounds the value of the composed policy by combining Lemma 1, Corollary 1, and soft policy evaluation through induction and a limiting argument.

  • B. Proof of Theorem 1: Theorem 1 states a value bound for the composed policy πΣ using a fixed-point error term D*.The theorem introduces D* as the fixed point of a recursion.
  • B. Proof of Theorem 1: The proof uses induction and soft policy evaluation to establish the required inequality for every iteration.The induction is initialized at k=0.
  • B. Proof of Theorem 1: The second inequality follows from Lemma 1 and Corollary 1, then the argument takes the limit as k →∞.The limiting expression involves the value of the composed policy and the fixed-point terms.
Loading 1803.06773v1…