End-to-End Robotic Reinforcement Learning without Reward Engineering

Avi Singh; Larry Yang; Kristian Hartikainen; Chelsea Finn; Sergey Levine

Source-linked AI summary

End-to-End Robotic Reinforcement Learning without Reward Engineering

Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, Sergey Levine

arXiv:1904.07854v2Apr 16, 2019cs.LGcs.CVcs.ROstat.ML

TL;DR

Real-world reinforcement learning normally needs manually engineered rewards or instrumentation, limiting end-to-end learning from images. The paper introduces VICE-RAQ, which learns rewards from successful-outcome examples and actively solicited binary labels; it learns several manipulation tasks from images with 1-4 hours of interaction.

Problem
Real-world image-based reinforcement learning requires manually programmed rewards, perception pipelines, or environmental instrumentation to determine task success.
Method
VICE-RAQ learns a classifier-based reward from successful-outcome examples, adversarially mined negatives, and actively queried binary user labels during off-policy reinforcement learning.
Results
The method learns real-world image-based manipulation skills including cloth draping, bookshelf placement, and mug pushing with 1-4 hours of interaction.
Takeaways & Limitations
The approach supports robotic skill learning without user-programmed rewards, demonstrations, or environmental instrumentation.
Takeaways & Limitations
The method requires user labels and around 50 queries per training run, and does not exploit shared structure between tasks.

Abstract

from arXiv · show

The combination of deep neural network models and reinforcement learning algorithms can make it possible to learn policies for robotic behaviors that directly read in raw sensory inputs, such as camera images, effectively subsuming both estimation and control into one model. However, real-world applications of reinforcement learning must specify the goal of the task by means of a manually programmed reward function, which in practice requires either designing the very same perception pipeline that end-to-end reinforcement learning promises to avoid, or else instrumenting the environment with additional sensors to determine if the task has been performed successfully. In this paper, we propose an approach for removing the need for manual engineering of reward specifications by enabling a robot to learn from a modest number of examples of successful outcomes, followed by actively solicited queries, where the robot shows the user a state and asks for a label to determine whether that state represents successful completion of the task. While requesting labels for every single state would amount to asking the user to manually provide the reward signal, our method requires labels for only a tiny fraction of the states seen during training, making it an efficient and practical approach for learning skills without manually engineered rewards. We evaluate our method on real-world robotic manipulation tasks where the observations consist of images viewed by the robot's camera. In our experiments, our method effectively learns to arrange objects, place books, and drape cloth, directly from images and without any manually specified reward functions, and with only 1-4 hours of interaction with the real world.

I. INTRODUCTION

The paper addresses the difficulty of applying end-to-end reinforcement learning to real robots when rewards require manually engineered perception or environmental instrumentation. It proposes learning rewards from outcome examples and active binary queries, enabling image-based manipulation with limited real-world interaction.

I. INTRODUCTION: Inverse reinforcement learning reduces manual reward design but typically burdens users with demonstrations that are difficult to provide.Demonstrations can involve kinesthetic robot movement or teleoperation and may undermine autonomous skill discovery through trial and error.
I. INTRODUCTION: Naïve goal classifiers can be exploited by reinforcement learning agents that reach out-of-distribution observations incorrectly labeled as successful.VICE addresses exploitation by adversarially mining policy-generated negatives, but relies on many initial positive examples and millions of on-policy samples.
I. INTRODUCTION: The proposed framework learns rewards from a small number of successful-outcome examples followed by modest binary queries, without demonstrations or hand-designed reward functions.Queries are solicited as the robot gathers experience, and the framework uses efficient off-policy reinforcement learning.
I. INTRODUCTION: 1-4 hours of real-world interaction sufficed to learn manipulation skills including cloth draping, bookshelf placement, and pushing mugs onto coasters.The experiments use image observations and require minimal user supervision.
I. INTRODUCTION: Manual reward engineering or instrumentation remains a bottleneck for applying robotic reinforcement learning in open-world environments.Reward design may require thermal cameras, motion capture, computer vision systems, or accelerometers to determine task success.

III. PRELIMINARIES

The preliminaries formulate image-based robotic control as reinforcement learning and explain how learned classifiers can provide rewards. Maximum-entropy, off-policy learning is used for stable real-world training, while classifier rewards avoid programming success conditions directly.

III. PRELIMINARIES: The robotic Markov decision process uses image observations as states, end-effector motions as actions, and an unknown transition dynamic.A policy generates state-action trajectories and is optimized for expected total trajectory reward.
III. PRELIMINARIES: Maximum-entropy reinforcement learning augments reward optimization by maximizing the entropy of the resulting policy distribution.The paper uses off-policy soft actor-critic, which stores transitions in a replay buffer and updates the policy and critic from sampled data.
III. PRELIMINARIES: Off-policy maximum-entropy learning is selected for stable, robust real-world reinforcement learning and straightforward integration with VICE.The framework could also be combined with standard reinforcement learning.
III. PRELIMINARIES: A goal classifier predicts success or failure from example states, and its output distribution can be converted into a reinforcement-learning reward.This replaces programmatic identification of object states and success conditions with binary classification of observations.
III. PRELIMINARIES: Classifier-based rewards can be exploited when policies visit states unlike those represented in the training data.Avoiding exploitation with static data would require comprehensive negative examples covering the entire state space, motivating active queries.

IV. REINFORCEMENT LEARNING WITH ACTIVE QUERIES

RAQ learns a classifier-based reward while actively querying labels for selected states encountered during training. This reduces reliance on comprehensive negative examples and mitigates classifier exploitation without labeling every state.

IV. REINFORCEMENT LEARNING WITH ACTIVE QUERIES: RAQ trains classifiers to distinguish goal from non-goal observations and uses them to compute reinforcement-learning rewards.Unlike prior static-dataset approaches, it actively requests binary success labels for selected encountered states.
IV. REINFORCEMENT LEARNING WITH ACTIVE QUERIES: Active queries remove the need for comprehensive negative examples supplied before training.The robot obtains labels for states it encounters rather than requiring the user to enumerate failures in advance.
IV. REINFORCEMENT LEARNING WITH ACTIVE QUERIES: The algorithm chooses which encountered states to query as learning proceeds, rather than labeling every state collected by reinforcement learning.A typical run may collect tens or hundreds of thousands of states, making universal labeling impractical.

A. Active Queries

Active queries focus labeling effort on states most likely to represent successful outcomes, reducing unnecessary user queries while addressing classifier exploitation.

The method selects previously unlabeled states with the highest classifier-assigned probability of success for user labeling.This targets potential positive examples rather than uncertain states broadly.
This selection rule is more selective than maximum entropy querying because negative examples are generally easier to obtain and positive predictions are the source of false positives.
Labels are requested at fixed intervals, with the query frequency chosen from the expected training length and query budget.
Real-world experiments use between 25 to 75 active queries per reinforcement-learning run.

B. Classifier-Based RL with Active Queries

RAQ trains a goal classifier from labeled outcomes, uses its success probabilities as rewards, and actively adds labels during off-policy reinforcement learning.

B. Classifier-Based RL with Active Queries: RAQ initializes a classifier on labeled examples, uses log p_g(y|s) as the reward, and periodically queries the state with the largest predicted success value.
B. Classifier-Based RL with Active Queries: The policy and critic are trained from replay-buffer samples, enabling the active-query framework to use off-policy reinforcement learning.
B. Classifier-Based RL with Active Queries: Each queried state receives a binary success label and is added to the classifier’s dataset before the classifier is updated.
B. Classifier-Based RL with Active Queries: VICE represents task completion with binary event variables in a graphical-model formulation of reinforcement learning.
B. Classifier-Based RL with Active Queries: VICE-RAQ combines active queries with VICE in an off-policy setting to provide a practical method for robotic reinforcement learning.

A. VICE

VICE formulates successful task completion as an event in a graphical model and learns event probabilities through adversarial inverse reinforcement learning.

A. VICE: VICE introduces binary event variables y_t indicating whether the task-related event has occurred at each time step.
A. VICE: The desired policy is formulated as inference conditioned on the event variables being successful throughout the task.
A. VICE: Adversarial inverse reinforcement learning scales the event-probability learning procedure to high-dimensional continuous robotic state spaces.
A. VICE: At convergence, f_ψ(s,a) recovers log p(e = 1|s,a) under the adversarial learning procedure.
A. VICE: The paper extends VICE to off-policy learning and removes the need for user-provided ground-truth action labels.
A. VICE: Because isolated successful-outcome images provide states without actions, the method simplifies the discriminator update while preserving the same fixed point for the action distribution.

C. Off-Policy VICE-RAQ

Off-policy VICE-RAQ reduces VICE’s initial positive-example burden by adding active queries and trains its classifier, policy, and critic from collected experience.

C. Off-Policy VICE-RAQ: Integrating active queries with VICE reduces the initial positive examples required, at the cost of several tens of binary queries.
C. Off-Policy VICE-RAQ: The policy, critic, and learned reward function share the same convolutional neural-network architecture.
C. Off-Policy VICE-RAQ: The method begins with positive examples, collects environment data, and updates the reward classifier, policy, and Q-function iteratively.
C. Off-Policy VICE-RAQ: The algorithm initializes the classifier, policy, critic, and replay buffer before repeatedly collecting transitions and updating the classifier from positive and replay-buffer samples.
C. Off-Policy VICE-RAQ: Policy updates use classifier rewards, while active queries select the state with the highest predicted success probability and add successful states to the positive dataset.

VI. VICE-RAQ FOR IMAGE-BASED MANIPULATION

VICE-RAQ smooths classifier-based rewards for image-based reinforcement learning by using mixup, making the learned reward more suitable for policy optimization.

The reward function fψ(s) uses a convolutional neural network with the same architecture as the policy and critic.
Neural-network classifier rewards can create hard decision boundaries, making reinforcement learning resemble optimization with sparse rewards.
Mixup interpolates states and labels using λ sampled from Beta(α, α), with α controlling the degree of interpolation.
The experiments indicate that mixup produces smoother learned rewards that are more amenable to reinforcement learning.

VII. SIMULATED EXPERIMENTS

Simulated experiments compare classifier-based reward methods across three tasks and show that VICE-RAQ provides the strongest and most consistent performance, especially on harder tasks.

The study compares RAQ, off-policy VICE, VICE-RAQ, naïve classifier rewards, ground-truth sparse rewards, and standard classifier-based rewards across three simulated tasks.
The simulated tasks are Visual Pusher, Visual Door Opening, and Visual Picker, with randomized initial configurations and specified goal states.
Across five random seeds, off-policy VICE and VICE-RAQ solve pushing for every seed, whereas RAQ, naïve classifier, and sparse-reward baselines succeed only in some runs.
On door opening, VICE-RAQ reaches nearly 100% success across seeds, while naïve-classifier and sparse-reward baselines fail for all but one seed.
Figure 5 reports five-run means and individual seeds, showing substantial variance for RAQ and VICE but relatively low late-training variance for VICE-RAQ.
VICE-RAQ strongly outperforms other methods on Visual Picker, while naïve classifier-based and sparse rewards fail for most seeds.

VIII. REAL-WORLD EXPERIMENTS

Real-world experiments test image-based reward learning on pushing, draping, and bookshelf placement. VICE-RAQ succeeds across these tasks with limited interaction, while competing methods are less reliable.

The experiments evaluate RAQ, off-policy VICE, and VICE-RAQ on visual mug pushing, cloth draping, and book insertion without engineered rewards or environmental instrumentation.
Visual Pushing: The Visual Pushing task requires interpreting RGB images while handling randomized mug placement and determining whether the mug is fully inside the coaster.
Visual Bookshelf: Classifiers allow the robot to choose different bookshelf slots based on the starting state, unlike goal images that would always specify one of two slots.
Visual Bookshelf: Visual Bookshelf requires inserting a book into an empty slot from randomized starting positions, with multiple slots allowing different successful outcomes.
The real-world experiments use 80 success examples per task, fixed-interval queries, and approximately 90 minutes to 4 hours of interaction depending on the task.
Visual Pushing: VICE-RAQ achieves a 100% success rate on Visual Pushing, compared with 60% for RAQ; off-policy VICE and the naïve classifier fail.
Visual Draping: All reward-learning methods solve Visual Draping, while the naïve classifier and the robot-state-only baseline fail, indicating that visual cloth deformation matters for success.
Visual Bookshelf: VICE-RAQ solves Visual Bookshelf consistently, whereas RAQ cannot solve it and off-policy VICE succeeds only sporadically.

IX. DISCUSSION AND FUTURE WORK

VICE-RAQ constructs rewards from successful-outcome examples, adversarially mined negatives, and about 50 active user queries rather than hand-programmed reward functions. The method improves substantially over naïve classifier rewards and VICE, but still depends on user labels and does not share structure across tasks.

Method: VICE-RAQ constructs rewards from successful-outcome examples, adversarially mined negatives, and active binary queries about the robot’s outcomes.The user labels only whether queried outcomes are successful, rather than labeling all experience.
Efficiency: About 50 queries are used per experiment, compared with tens of thousands of transitions.
Results: The method improves substantially over naïve classifier rewards and VICE in the reported experiments.
Limitations: User labels remain an additional assumption, and around 50 queries per training run remain non-trivial.The authors suggest better query criteria and classifier uncertainty estimates as ways to reduce this burden.
Future work: VICE-RAQ does not currently benefit from shared structure between tasks, although multi-task classifiers could in principle reduce required queries and examples.
Implications: The approach is positioned as a step toward practical robotic reinforcement learning and lifelong learning directly in real-world environments.The authors connect this direction to acquiring broad and generalizable skill repertoires through interaction.

APPENDIX

The appendix specifies three simulated visual manipulation tasks, their evaluation rollouts, and key training settings. Results are plotted across methods and baselines using five random seeds for each task and method.

Simulated tasks: Visual Pusher moves a mug onto a coaster, with randomized starts over 20cm x 15cm and success within 3cm of the goal.
Simulated tasks: Visual Door requires opening a hooked door by 45 degrees from closed or slightly open initial states in full 3D space.
Simulated tasks: Visual Picker requires lifting a randomized tennis ball and moving it to a fixed location 20 cm above the table.The task controls both the end effector and a parallel-jaw gripper.
Training settings: Soft Actor-Critic uses the open-source implementation’s default hyperparameters, including a 3e-4 Adam learning rate, batch size 256, and discount factor 0.99.
Training settings: Discriminator training uses Adam with learning rate 3e-4 and batch size 256, while sweeping update counts N across simulated methods and tasks.
Evaluation: Figure 10 plots all simulated-task results for methods and baselines, showing five random-seed results for every task–method combination.Figures 11–13 provide example evaluation rollouts for Visual Pusher, Visual Door Opening, and Visual Picker policies learned using VICE-RAQ.