PaperlayerView exact revision on arXiv

Source-linked AI summary

Residual Policy Learning

Tom Silver, Kelsey Allen, Josh Tenenbaum, Leslie Kaelbling

arXiv:1812.06298v2cs.ROcs.LG

TL;DR

Complex robotic manipulation can make reinforcement learning from scratch inefficient or intractable, despite the availability of useful but imperfect controllers. RPL learns a residual on top of hand-designed or model-predictive policies, and experiments show consistent improvement with greater data efficiency than model-free RL alone.

  • Problem

    Long-horizon, sparse-reward manipulation remains difficult for reinforcement learning from scratch, while imperfect controllers still require improvement.

  • Method

    RPL learns a residual with model-free deep reinforcement learning on top of an initial hand-designed or model-predictive controller.

  • Results

    RPL consistently improves initial policies and achieves better data efficiency than learning from scratch across complex manipulation tasks.

  • Takeaways & Limitations

    RPL can improve long-horizon, sparse-reward policies in settings where model-free reinforcement learning fails.

Abstract

from arXiv · show

We present Residual Policy Learning (RPL): a simple method for improving nondifferentiable policies using model-free deep reinforcement learning. RPL thrives in complex robotic manipulation tasks where good but imperfect controllers are available. In these tasks, reinforcement learning from scratch remains data-inefficient or intractable, but learning a residual on top of the initial controller can yield substantial improvements. We study RPL in six challenging MuJoCo tasks involving partial observability, sensor noise, model misspecification, and controller miscalibration. For initial controllers, we consider both hand-designed policies and model-predictive controllers with known or learned transition models. By combining learning with control algorithms, RPL can perform long-horizon, sparse-reward tasks for which reinforcement learning alone fails. Moreover, we find that RPL consistently and substantially improves on the initial controllers. We argue that RPL is a promising approach for combining the complementary strengths of deep reinforcement learning and robotic control, pushing the boundaries of what either can achieve independently. Video and code at https://k-r-allen.github.io/residual-policy-learning/.

I. INTRODUCTION

Residual Policy Learning improves imperfect controllers by learning a residual with deep reinforcement learning, targeting manipulation settings where learning from scratch is inefficient or intractable. The method combines structured control with learned corrections across challenging sources of uncertainty.

  • Motivation: Deep RL can remain intractable for long-horizon, sparse-reward manipulation tasks that hand-designed policies can solve.The hook task illustrates this gap: a hand-designed policy succeeds, while current deep RL methods fail to reach the target.
  • Motivation: Hand-designed policies often fail under substantial changes in object geometry, table structure, and unobserved task variables.In ComplexHook, object shape and bump configurations vary while the state omits information about both.
  • Method: RPL augments an arbitrary initial policy with a learned residual function, producing a policy that adds corrective actions to the original controller.Because the gradient depends only on the residual, policy-gradient methods can train RPL even when the initial policy is nondifferentiable.
  • Method: RPL can treat a strong controller as a corrective-policy base or a weak controller as an exploration hint.The paper studies tasks spanning this spectrum, from nearly perfect initial policies to controllers far from ideal.
  • Experimental claims: Across experiments, RPL substantially improves hand-designed and model-predictive initial policies while requiring less data than learning from scratch.The study includes MPCs with known or learned transition models, including PETS-derived controllers.

III. BACKGROUND

The paper frames RPL within partially observable MDPs and standard actor-critic reinforcement learning. DDPG, experience replay, and HER provide the background machinery for learning continuous-action policies with sparse rewards.

  • MDP formulation: An MDP specifies states, actions, rewards, transition probabilities, and a discount factor, with policies mapping states to actions.The framework also allows partial observability and finite-horizon episodes.
  • Value functions: The action-value function Qπ estimates expected future discounted rewards when following policy π after taking an action.Its Bellman equation relates current reward and future value under the policy.
  • Actor-critic learning: Actor-critic methods jointly learn a parameterized policy and action-value function, training the actor to choose actions favored by the critic.The critic uses a Bellman-derived loss, and this pairing is typically more stable than training the actor alone.
  • Implementation: DDPG is used for RPL experiments because it supports continuous states and actions, while experience replay reduces correlations between consecutive transitions.Stored transitions are randomly sampled from a replay buffer during training.
  • Implementation: HER improves data efficiency for sparse binary-reward goals by relabeling replay transitions with goals derived from episode outcomes.The reward and policy are parameterized by the goal as R(s, a, g) and π(s, g).

IV. RESIDUAL POLICY LEARNING (RPL)

RPL converts an initial controller into a residual-learning problem: a neural residual policy is trained with model-free deep RL while preserving the controller’s action structure. Initialization and critic burn-in are practical safeguards for retaining good initial behavior.

  • Residual policy: RPL learns a residual fθ on top of an initial policy, yielding the final policy πθ(s) = π(s) + fθ(s).The residual supplies action corrections while the initial controller remains part of the deployed policy.
  • Residual MDP: The fixed initial policy induces a residual MDP in which the residual function can be learned as an ordinary policy.The paper parameterizes fθ with a neural network and applies model-free deep RL.
  • Learning procedure: RPL preserves the initial policy’s role by creating a residual policy and then applying standard deep reinforcement learning.The method is described as a simple transformation of the original control problem.
  • Initialization: The residual network is initialized to output zero, so the initial residual policy matches the original controller.This is intended to prevent a good initial policy from becoming worse at initialization.
  • Training stabilization: A critic burn-in period keeps the policy fixed while the action-value estimate improves.This addresses the risk that a poor critic could degrade performance even when the initial policy is perfect.

C. Recurrent RPL for POMDPs

RPL can address partially observable manipulation by using short state histories, and the study evaluates it across six simulated MuJoCo tasks with varied action and reward settings.

  • C. Recurrent RPL for POMDPs: RPL handles POMDPs by augmenting the policy input with a history of previous states.The paper reports that a history length of 1, consisting of the current and previous states, was effective in practice.
  • C. Recurrent RPL for POMDPs: The experiments investigate whether RPL improves initial policies, learns faster than model-free RL, and succeeds where model-free RL is intractable.
  • C. Recurrent RPL for POMDPs: The study evaluates six simulated manipulation tasks implemented in MuJoCo, including Push, PickAndPlace, SlipperyPush, NoisyHook, ComplexHook, and MBRLPusher.
  • C. Recurrent RPL for POMDPs: The first five tasks use end-effector position changes and gripper control, while the sixth directly actuates a 7-DOF arm’s joints.
  • C. Recurrent RPL for POMDPs: The Push environment randomizes object and target positions, uses a 25-dimensional state, and provides sparse binary success rewards.
  • C. Recurrent RPL for POMDPs: SlipperyPush changes the cube’s sliding friction coefficient to 0.18 while keeping the original Push environment’s other task parameters unchanged.

3) PickAndPlace:

The paper studies manipulation settings with altered targets, hooks, observation noise, structured object variation, and unknown table bumps, spanning increasingly difficult control conditions.

  • 3) PickAndPlace:: PickAndPlace randomizes whether the target is on the table or above it, with a 0.5 probability for each case.
  • 3) PickAndPlace:: Hook tasks require the robot to use a newly introduced hook because the gripper cannot initially reach the block.
  • 3) PickAndPlace:: The hook adds 15 state dimensions, producing a total 40-dimensional state space without additional shaping rewards.
  • 3) PickAndPlace:: NoisyHook adds IID diagonal Gaussian observation noise to object positions and rotations and doubles the episode length to 100 frames.
  • 3) PickAndPlace:: ComplexHook varies object mass, friction, and shape using 100 objects while retaining the hook-and-object manipulation objective.
  • 3) PickAndPlace:: ComplexHook also randomizes rigid table bumps whose properties are absent from the state space.
  • 3) PickAndPlace:: MBRLPusher uses a 7-DOF arm to push a cylinder, with a 20-dimensional state, weighted distance-and-action rewards, and a 150-frame horizon.

B. Initial Policies

RPL is paired with several initial controllers and standardized deep-RL training choices, while an expert-exploration baseline tests whether controller guidance alone explains its gains.

  • B. Initial Policies: The initial-policy set includes discrete MPC, reactive controllers, and cached model-based controllers for the manipulation tasks.
  • B. Initial Policies: The study uses DDPG with HER and a three-layer, 256-unit-per-layer actor-critic architecture across experiments.
  • B. Initial Policies: For noisy environments, RPL uses a one-step state history and averages features from the two states because this variant performed better in practice.
  • B. Initial Policies: The baselines include running the initial policy without learning and learning from scratch with DDPG and HER.
  • B. Initial Policies: Expert Explore uses the initial policy only during exploration, separating exploration guidance from residual policy learning.
  • B. Initial Policies: Expert Explore selects expert, random, or learned-policy actions according to ε and α, with the learned policy acting most of the time.

E. Results

Across the evaluated manipulation tasks, RPL improves imperfect initial controllers and generally reaches strong performance more efficiently than learning from scratch, although early training can temporarily degrade performance.

  • E. Results: Results are reported across six tasks with mean and standard deviation over five random seeds, comparing RPL against three baselines.
  • E. Results: RPL reaches perfect performance in Push and SlipperyPush while outperforming learning from scratch before convergence.
  • E. Results: RPL requires an order of magnitude fewer training samples than learning from scratch to reach average success rate 0.9 in Push.
  • E. Results: RPL training can drop in performance early before quickly recovering and surpassing the baselines.
  • E. Results: RPL converges slightly faster than Expert Explore, suggesting that both exploration guidance and residual parameterization contribute to the Push advantage.
  • E. Results: ReactivePush achieves success rate around 0.45 on SlipperyPush because the transferred controller mishandles the five-times-smaller friction setting.
  • E. Results: ReactivePickAndPlace’s artificially increased gains produce oscillations and reduce its success rate to roughly 0.5.

3) ReactivePickAndPlace

Across challenging manipulation settings, RPL improves imperfect initial policies where learning from scratch performs poorly, including sensor noise and structured uncertainty.

  • ReactiveHook: RPL quickly reaches roughly 0.8 success in NoisyHook, while DDPG+HER fails to achieve nontrivial success.Expert Explore only begins reaching nontrivial success after 5 million simulator steps.
  • ReactiveHook: In ComplexHook, randomly scattered bumps and varied object properties each reduce ReactiveHook performance by roughly 20%.The state omits information about these randomized obstacles and object characteristics.
  • ReactiveHook: RPL converges below 0.8 success in ComplexHook despite diverse objects and obstacles absent from the state.The authors characterize the learned policy as conformant across most randomized conditions.
  • CachedPETS: RPL improves substantially on CachedPETS and the original PETS controller, while converging faster than DDPG+HER.PETS performance is averaged over 10 trials in the comparison.

6) CachedPETS

RPL extends model-based control by learning residuals over CachedPETS, improving performance while retaining data-efficiency advantages relative to model-free learning.

  • CachedPETS: RPL improves substantially on both the CachedPETS initial controller and the original PETS controller.The initial controller is derived from a learned transition model.
  • CachedPETS: RPL converges faster than DDPG+HER, indicating that the initial model-based controller benefits learning.The paper emphasizes that no domain knowledge was used to design this initial policy.
  • Discussion: The discussion frames RPL as combining model-based data efficiency with model-free asymptotic performance.The reported evidence supports this interpretation across the presented experiments.
  • CachedPETS: In MBRLPusher, RPL improves on the initial policy, converges faster than DDPG+HER, and outperforms PETS.The dashed curve marks PETS average performance over 10 trials.
  • ComplexHook: On ComplexHook, RPL learns behavior that pulls the object to the target, unlike the original policy's failure on the larger obstacle-rich object.The learned behavior qualitatively resembles lifting the hook and reaching around obstacles at a wider angle.
  • Scope: RPL is intended for continuous-action domains with gradient-based learning methods, especially where good but imperfect initial policies exist.The paper highlights long-horizon, sparse-reward manipulation tasks as a particularly suitable setting.

A. Initial Policies

The experiments use hand-designed reactive policies and model-predictive controllers, including discrete MPC and CachedPETS, as imperfect starting points for RPL.

  • Model-predictive controllers: The discrete MPC Push policy searches action sequences using six cardinal-direction gripper movements and a distance-based heuristic.The heuristic uses object-to-target and gripper-to-push-location distances.
  • Reactive policies: ReactivePush is designed to move an object to a target but loses performance when reduced sliding friction makes the task SlipperyPush.Its manually tuned gain was selected for near-optimal performance on the original Push task.
  • Reactive policies: ReactivePickAndPlace is designed to pick up a cube and bring it to a target location on or above the table.Its policy is specified through ordered conditional statements over the input state.
  • Reactive policies: ReactiveHook grasps and lifts a hook, moves it behind and right of an object, then pushes and pulls the object toward a target.It works nearly perfectly for a cube on a clear table with noise-free observations.
  • Model-predictive controllers: CachedPETS uses an ensemble probabilistic transition model and cross-entropy planning, caching actions for 500 input states because PETS alone is too slow.At test time, the cached controller selects actions using nearby stored states.

B. Model Hyperparameters

Experiments use fixed actor–critic architecture, optimization, replay, rollout, and exploration settings, with task-specific worker configurations and a cache-size choice for CachedPETS.

  • Actor and critic networks have 3 layers with 256 units each and ReLU nonlinearities.
  • Training uses Adam with learning rate 1·10^-3 for both actor and critic, a buffer of 10^6 transitions, and Polyak coefficient 0.95.
  • Optimization runs 50 cycles per epoch, 40 batches per cycle, and 50 test rollouts per epoch, with action L2 norm coefficient 1.0.
  • Exploration and replay use random actions with probability 0.3, Gaussian noise scale 0.2, and HER replay probability 0.8.For hooks, the Gaussian noise scale is 0.1.
  • CachedPETS cache sizes from 1 to 1000 were evaluated over 25 trials each, leading to a selected database size of 500.The plot reports mean and standard deviation on MBRLPusher.
  • Push and PickAndPlace use 19 MPI workers with rollout batch size 2, whereas Hook uses 1 worker and batch size 4; burn-in threshold is β = 1.0.Expert Explore uses ϵ = 0.6 and α = 0.8, selected by a small grid search.
Loading 1812.06298v2…