Source-linked AI summary
Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces
Garrett Warnell, Nicholas Waytowich, Vernon Lawhern, Peter Stone
TL;DR
Existing reinforcement-learning methods often require extensive data for high-dimensional tasks, motivating faster learning from real-time human feedback. Deep TAMER extends TAMER with deep neural networks and, after 15 minutes of feedback, trains Bowling agents that outperform human trainers and several deep reinforcement-learning methods.
Problem
High-dimensional state spaces make reinforcement-learning methods data- and time-intensive, while prior TAMER-style human-feedback approaches had not addressed these spaces with deep learning.
Method
Deep TAMER extends the TAMER framework to high-dimensional state spaces by incorporating deep-learning function approximation for real-time, scalar-valued human feedback.
Results
After 15 minutes of real-time human interaction, Deep TAMER agents achieved higher scores than agents trained with state-of-the-art deep reinforcement learning and exceeded their trainers’ performance on Bowling.
Takeaways & Limitations
On pixel-level Bowling, human training with Deep TAMER significantly outperformed human training with TAMER, reaffirming the attractiveness of the TAMER paradigm.
Takeaways & Limitations
The evaluation covered only Bowling and did not include additional hyperparameter searches because large amounts of human interaction data were difficult to obtain.
Abstract
from arXiv · showhide
While recent advances in deep reinforcement learning have allowed autonomous learning agents to succeed at a variety of complex tasks, existing algorithms generally require a lot of training data. One way to increase the speed at which agents are able to learn to perform tasks is by leveraging the input of human trainers. Although such input can take many forms, real-time, scalar-valued feedback is especially useful in situations where it proves difficult or impossible for humans to provide expert demonstrations. Previous approaches have shown the usefulness of human input provided in this fashion (e.g., the TAMER framework), but they have thus far not considered high-dimensional state spaces or employed the use of deep learning. In this paper, we do both: we propose Deep TAMER, an extension of the TAMER framework that leverages the representational power of deep neural networks in order to learn complex tasks in just a short amount of time with a human trainer. We demonstrate Deep TAMER's success by using it and just 15 minutes of human-provided feedback to train an agent that performs better than humans on the Atari game of Bowling - a task that has proven difficult for even state-of-the-art reinforcement learning methods.
Introduction
Deep TAMER extends TAMER with deep learning to use real-time human feedback in high-dimensional state spaces. The paper evaluates this approach on Atari Bowling, where brief human training produces strong performance.
- Motivation: Deep reinforcement learning often requires extensive training data and time, especially when state spaces contain raw image data.High-dimensional inputs require estimating many parameters from scratch in the absence of prior information.
- Research gap: TAMER previously used real-time scalar-valued feedback from human trainers but had only been demonstrated in low-dimensional state spaces.The paper targets this limitation by extending TAMER to higher-dimensional settings.
- Approach: Deep TAMER incorporates deep-learning function approximation so human feedback can shape behavior in high-dimensional state spaces.The framework learns a deep neural network that predicts human feedback and uses that prediction to drive the behavior policy.
- Evaluation: The evaluation focuses on Atari Bowling, a task that has challenged state-of-the-art deep reinforcement learning methods.Those methods reportedly achieved raw scores of 35 to 70 out of a maximum of 270 even after training on the order of days.
- Results: With only a few minutes of interactive training, human trainers produced Bowling agents that outperformed the trainers themselves.The paper reports this result alongside its comparison with state-of-the-art reinforcement learning methods.
Problem Formulation
The paper formulates learning from real-time scalar human feedback as estimating a hidden feedback function in a sequential decision-making setting. The estimate is learned online from temporally weighted experience-feedback pairs and drives myopic action selection.
- Problem Formulation: The agent estimates a hidden human-feedback function H over state-action pairs from observed scalar feedback.The human observes state trajectories and periodically provides feedback, with larger values indicating more positive assessments.
- Problem Formulation: Given the current estimate ˆH, the behavior policy selects the action maximizing predicted human feedback.This is a fixed, myopic action-selection policy.
- Problem Formulation: The learning problem is supervised prediction from experience x = (s, a, ts, te) and feedback observations y = (h, tf).Each feedback signal may correspond to several recent experience records, while some experiences have no feedback.
- Problem Formulation: The loss weights each experience-feedback pair according to the hypothesized temporal correspondence between the feedback and state-action event.The weighting is zero for feedback observed before the state-action pair and negligible when feedback arrives sufficiently later.
- Problem Formulation: The estimate ˆH is learned online by minimizing expected loss over experience-feedback pairs generated during real-time human-agent interaction.This formulation supports learning throughout the interaction rather than only after data collection.
- Problem Formulation: The state encoder forms the fixed front half of ˆH and is pretrained as an autoencoder on states from random-policy simulation.The encoder has 76,035 parameters to learn.
Method
Deep TAMER combines a deep neural reward model with importance-weighted stochastic optimization and replay to learn from sparse human feedback in high-dimensional state spaces. Its CNN is pretrained as an autoencoder, while feedback and experience are reused through a replay buffer.
- Optimization: SGD updates estimate Ĥ from randomly sampled experience-feedback pairs, using gradients of the loss with respect to the model parameters.The iteration index k counts feedback-experience updates, and each sampled pair comes from streams of experience and feedback observations.
- Importance Weights: Importance weights assign feedback to prior state-action intervals according to the assumed delay distribution, while zero-weight pairs are excluded from SGD sampling.The implementation uses a continuous uniform delay distribution over [0.2, 4] seconds, so feedback applies to experience from 0.2 to 4 seconds earlier.
- Deep Reward Model: Deep TAMER models predicted human reward as Ĥ(s, a) = z(f(s), a), combining a CNN encoder with fully connected layers.The network uses a deep convolutional front end followed by fully connected layers that produce action-specific outputs.
- Deep Reward Model: The CNN encoder is pretrained with an autoencoder by minimizing reconstruction error over states, then fixed as the front end of Ĥ.The encoder maps high-dimensional states to lower-dimensional representations before real-time human interaction.
- Feedback Replay Buffer: The feedback replay buffer stores observed feedback and relevant experience so the agent can repeatedly learn from each feedback signal.Deep TAMER updates when new feedback arrives and also samples from the buffer at a fixed rate; experiments use buffer updates every 10 time steps.
- Deep TAMER: Unlike TAMER, Deep TAMER can learn from each state-action pair multiple times, using replay specifically to address sparse feedback.The complete procedure is summarized as Deep TAMER, including autoencoder pretraining, importance-weighted optimization, and feedback replay.
Experiments
Deep TAMER was evaluated on Atari BOWLING with human trainers and comparisons against reinforcement-learning, demonstration-based, and original TAMER baselines. Using limited human feedback, it learned successful play rapidly and often exceeded human performance, while the study also examined credit-assignment choices and acknowledged scope limitations.
- Experimental Setting: Deep TAMER was evaluated on Atari BOWLING, an environment difficult for several state-of-the-art deep reinforcement-learning algorithms.The study used the Arcade Learning Environment implementation included in OpenAI Gym.
- Training Procedure: Nine human trainers trained Deep TAMER agents, with each trainer first playing two complete games and receiving a 10-minute practice session.The experiment then used the same training procedure to compare agents trained with original TAMER.
- Human-Trainer Comparison: After 15 minutes, all Deep TAMER agents met or exceeded their human trainers, and 6 out of 9 agents performed better than their trainers.The individual-trainer analysis reported noisy performance increases, likely related to the stochastic optimization technique.
- Baseline Comparisons: Deep TAMER exceeded the human trainers after 7 minutes and surpassed the expert human after 15 minutes, while Double-DQN, A3C, and original TAMER failed to learn useful policies in the allotted time.Deep TAMER also outperformed the previous best Atari BOWLING method based on learning from human demonstrations.
- Credit Assignment: Uniform [0.28, 4.0] credit assignment yielded better results than Gamma(2.0, 0.28), likely because trainers often waited to observe how many pins fell before giving feedback.The comparison averaged the uniform condition over 9 subjects and the gamma condition over 8 subjects.
- Scope and Limitations: Because human interaction data were difficult to obtain, the study performed no additional hyperparameter search and limited analysis to the Bowling domain.The authors planned to examine additional domains and hyperparameter settings in future work.
Summary
Deep TAMER extends TAMER for real-time human interaction in high-dimensional state spaces by combining a deep neural network reward model with a modified supervised learning procedure. In Atari Bowling, human-trained agents significantly outperformed TAMER-trained agents and surpassed deep reinforcement learning agents and, in most cases, their human trainers after just 15 minutes.
- Deep TAMER extends TAMER to high-dimensional environments using a deep neural network to approximate the human trainer’s reward function.The method also uses a modified supervised learning procedure to estimate the model parameters.
- Deep TAMER agents significantly outperformed agents trained by humans using the original TAMER method on Atari Bowling.
- 15 minutes of real-time human interaction produced Deep TAMER agents that exceeded state-of-the-art deep reinforcement learning agents and, in most cases, human trainers.The comparison involved deep reinforcement learning methods using orders of magnitude more training data.