PaperlayerView exact revision on arXiv

Source-linked AI summary

Residual Reinforcement Learning for Robot Control

Tobias Johannink, Shikhar Bahl, Ashvin Nair, Jianlan Luo, Avinash Kumar, Matthias Loskyll, Juan Aparicio Ojea, Eugen Solowjow, Sergey Levine

arXiv:1812.03201v2cs.ROcs.LG

TL;DR

Manufacturing control becomes difficult when contacts and friction are poorly captured by physical models, while RL can require unsafe exploration and expensive data. This paper combines conventional feedback control with residual RL, and demonstrates successful real-world block assembly, including under randomized orientations. The method still depends on a carefully crafted vision setup, limiting application to novel scenes.

  • Problem

    Contacts and friction are difficult to model accurately for manufacturing control, while standard RL requires potentially unsafe exploration and expensive interaction data.

  • Method

    The method decomposes control into a conventional feedback component for known structure and an RL-learned residual, then superposes their control signals.

  • Results

    Residual RL successfully performs the real-world block assembly task and learns under randomized block orientations where the hand-designed controller fails.

  • Takeaways & Limitations

    Residual RL combines the efficiency of conventional controllers with RL flexibility for manipulation tasks involving friction, contacts, and unstable objects.

  • Takeaways & Limitations

    The method requires a carefully crafted vision setup, limiting application to novel scenes and controller quality when important visual information is lost.

Abstract

from arXiv · show

Conventional feedback control methods can solve various types of robot control problems very efficiently by capturing the structure with explicit models, such as rigid body equations of motion. However, many control problems in modern manufacturing deal with contacts and friction, which are difficult to capture with first-order physical modeling. Hence, applying control design methodologies to these kinds of problems often results in brittle and inaccurate controllers, which have to be manually tuned for deployment. Reinforcement learning (RL) methods have been demonstrated to be capable of learning continuous robot controllers from interactions with the environment, even for problems that include friction and contacts. In this paper, we study how we can solve difficult control problems in the real world by decomposing them into a part that is solved efficiently by conventional feedback control methods, and the residual which is solved with RL. The final control policy is a superposition of both control signals. We demonstrate our approach by training an agent to successfully perform a real-world block assembly task involving contacts and unstable objects.

I. INTRODUCTION

Classical controllers struggle with manufacturing tasks involving contacts, friction, and environmental variability because these effects are difficult to model and require extensive tuning. The paper proposes combining conventional feedback control with residual RL for real-world assembly.

  • I. INTRODUCTION: Classical controllers often follow predefined trajectories with limited adaptation, while designing robust feedback for variable manufacturing tasks requires substantial engineering effort.The paper notes that fine-tuning controller designs can cost as much as the robot hardware.
  • I. INTRODUCTION: Contacts and friction are difficult to characterize accurately, making conventional feedback control brittle for tasks involving environmental interactions.The unknown coupling between robot and object dynamics must capture contact forces, friction, and algebraic constraints.
  • I. INTRODUCTION: Residual RL combines a conventional controller for tractable structure with RL for the residual task involving contacts and external object dynamics.The two control signals are superposed to form the commanded control.
  • I. INTRODUCTION: The approach targets manipulation objectives that control robot states while indirectly controlling underactuated environmental objects through their dynamic coupling.The robot is fully actuated, whereas environmental objects are not directly controlled.
  • I. INTRODUCTION: When only a final object goal is known, conventional error-based feedback may be unsuitable because the objective cannot be expressed as deviation from a desired trajectory.The paper frames reward functions as an alternative way to specify such goals.

B. Interpretation as a Reinforcement Learning Problem

The paper interprets the control problem through the standard Markov decision process and value-based RL framework. RL learns actions that maximize expected returns without modeling unknown coupled dynamics.

  • B. Interpretation as a Reinforcement Learning Problem: RL models sequential control as an agent selecting actions from states, receiving rewards, and maximizing expected return over a finite horizon.The return discounts future rewards using γ, and the objective is defined from the start-state distribution.
  • B. Interpretation as a Reinforcement Learning Problem: Unlike classical control, RL avoids explicitly modeling the unknown coupled dynamics of the robot and manipulated object.It instead searches for actions that maximize rewards without assumptions about system dynamics.
  • B. Interpretation as a Reinforcement Learning Problem: The paper uses value-based RL methods that estimate a state-action value function.These methods support action selection through estimates of expected returns.
  • B. Interpretation as a Reinforcement Learning Problem: The Bellman equation recursively relates value estimates and enables approximate dynamic programming.Its off-policy learnability makes value-based methods sample-efficient for real-world robot learning.

III. METHOD

The method decomposes the reward into robot-state and object-state components, assigning the first to conventional feedback control and the residual to RL.

  • III. METHOD: The conventional controller maximizes reward terms that depend on robot states, while a superposed RL controller maximizes terms that depend on object states.This decomposition separates structured geometric objectives from residual object-manipulation objectives.

A. Residual Reinforcement Learning

Residual RL combines a fixed hand-engineered controller with a learned policy, allowing conventional control to handle known structure while RL learns contact-sensitive corrections.

  • A. Residual Reinforcement Learning: In the assembly task, f(s_m) rewards moving the gripper between standing blocks, while g(s_o) rewards keeping the blocks upright and in place.The reward decomposition assigns geometric robot-state objectives and object-state stability objectives to separate terms.
  • A. Residual Reinforcement Learning: Residual RL uses conventional control to optimize tractable robot-state objectives and RL to learn fine-grained object-state behavior under friction and contacts.The paper identifies this division as the key insight behind improved efficiency and flexibility.
  • A. Residual Reinforcement Learning: The executed action adds the human-designed controller π_H(s_m) to the learned policy π_θ(s_m,s_o), which is optimized for expected task returns.The learned policy supplies the residual control signal.
  • A. Residual Reinforcement Learning: The hand-engineered controller provides stabilizing internal structure, but the coupled dynamics require joint optimization of the combined control policy.The learned controller cannot independently optimize object behavior while the conventional controller enforces zero robot-state error.
  • A. Residual Reinforcement Learning: The algorithm additively combines a fixed controller and a learnable policy during interaction, stores transitions, and updates the policy from replayed samples.Exploration noise is added before executing the combined action.
  • A. Residual Reinforcement Learning: The implementation uses a TD3 variant, selected for stability and sample efficiency, while the residual formulation remains independent of the specific RL algorithm.The authors state that other RL algorithms could also be used.

IV. EXPERIMENTAL SETUP

The experiments evaluate the residual reinforcement learning method on the block assembly task in both simulation and the real world.

  • The evaluation uses the block assembly task in simulation and in the real world.

A. Simulated Environment

The simulated environment uses a Sawyer robot and block-insertion task with position-based control and rewards that account for placement and block orientation.

  • The simulated task uses a seven-degree-of-freedom Sawyer arm with a parallel gripper and two standing blocks separated by an insertion gap.
  • The robot is commanded with a Cartesian-space position controller in the MuJoCo simulation.
  • The reward penalizes distance from the goal and deviations of the two blocks from their upright orientations.
  • The real-world environment uses a compliant joint-space impedance controller designed to be smooth and tolerant of contacts.
  • Real-world observations estimate block position with camera tracking and include only end-effector force in the z-direction.

C. Overview of Experiments

The experiments compare residual RL with pure RL and a hand-engineered controller across sample efficiency, environmental variation, and control noise.

  • The study asks whether a hand-designed controller improves RL performance and sample efficiency while allowing recovery from controller imperfections.
  • It also tests whether residual RL increases tolerance to environmental variation and noisy control systems.
  • The comparison includes only RL without a hand-engineered policy, residual RL, and the hand-engineered controller alone.
  • Simulation plots use 10 random seeds and report a 95% confidence interval for the mean.

B. Effect of Environment Variation

The experiments vary block orientation and control noise to test whether residual RL can handle environmental and controller variability better than a hand-engineered controller.

  • Environment Variation: Block orientations are varied independently in simulation over rotation ranges r ∈ {0, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3}.
  • Environment Variation: In real-world resets, each block is randomly set straight, tilted clockwise, or tilted counterclockwise by ± 20◦.
  • Control Noise: Zero-mean control noise is tested with σ ∈ [0.01, 0.1].
  • Control Noise: Biased control noise is tested with σ = 0.05 and µ ∈ [0, 0.2].
  • Evaluation: The real-world figure records success when blocks remain upright and finish in the correct positions.

D. Sim-to-Real with Residual RL

Residual RL combines conventional control with reinforcement learning to transfer simulated policies to real-world block insertion. Simulation initialization improves learning efficiency, while residual RL handles environment variation and contact-related corrections.

  • D. Sim-to-Real with Residual RL: Residual RL superposes a conventional controller with learned corrective behavior for control tasks involving contacts and external object dynamics.The conventional controller specifies the general trajectory, while RL learns the residual needed for realistic control.
  • D. Sim-to-Real with Residual RL: Residual RL outperforms pure RL and requires fewer samples in both simulation and physical hardware.Pure RL must learn the position-control structure from scratch and explores a wider, potentially unsafe state distribution.
  • D. Sim-to-Real with Residual RL: 15/20 trials succeeded with residual RL under randomized block orientations, compared with 2/20 for the hand-engineered controller.The learned policy develops small corrective nudges that avoid tipping the blocks, a behavior difficult to specify manually.
  • D. Sim-to-Real with Residual RL: 8,000 real-world samples, or about three hours of training, were sufficient to acquire the corrective behavior.The human controller supplies the general trajectory, reducing the behavior that must be learned from environment interaction.
  • D. Sim-to-Real with Residual RL: With randomized block rotations, residual RL maintains constant average performance while the hand-engineered controller degrades as rotation increases.This simulation experiment evaluates orientation changes after every reset.

C. Recovering from Control Noise

Residual RL remains effective under environmental and actuator disturbances, including control noise and bias. Simulation results show stable performance across variations, while hardware training remains feasible under biased actuator noise and simulation initialization accelerates learning.

  • C. Recovering from Control Noise: Residual RL maintains average performance as block rotation and control bias increase, unlike the hand-engineered controller.Simulation evaluates block rotations after resets and biased actuator inputs.
  • C. Recovering from Control Noise: Residual RL reduces the effect of control noise because its final average return is not greatly affected by noise magnitude.The method also compensates for biased controllers as control bias increases.
  • C. Recovering from Control Noise: Hardware training proceeds without significant issues as biased actuator noise increases.The hardware experiments investigate biased actuator noise rather than general control noise.
  • C. Recovering from Control Noise: Simulation policy initialization significantly speeds up both RL and residual RL in the real-world transfer experiment.Residual RL with simulation initialization solves the task in under one thousand interaction timesteps.

VIII. DISCUSSION AND FUTURE WORK

The paper combines conventional feedback control with deep RL to solve manipulation tasks involving friction, contacts, and unstable objects. The method is sample efficient but depends on carefully crafted vision, limiting application to novel scenes.

  • Discussion: The combined controller uses prior feedback-control knowledge with model-free RL to solve complicated manipulation tasks involving friction and contacts.The approach is intended for tasks largely addressed by prior knowledge but requiring additional model-free reasoning.
  • Discussion: The authors report that combining conventional control and RL can circumvent the disadvantages of both approaches and produce a sample-efficient controller coping with contact dynamics.
  • Future Work: The method requires a carefully crafted vision setup to infer real-world positions and angles.The hand-engineered vision system must preserve information such as precise edge locations for effective feedback.
  • Future Work: This vision requirement limits application to novel scenes and can reduce learned-controller quality when important feedback information is lost.The paper suggests incorporating vision input end-to-end into learning as future work.
Loading 1812.03201v2…