PaperlayerView exact revision on arXiv

Source-linked AI summary

FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback

Baicen Xiao, Qifan Lu, Bhaskar Ramasubramanian, Andrew Clark, Linda Bushnell, Radha Poovendran

arXiv:2001.06781v1cs.AI

TL;DR

In high-dimensional environments with sparse or delayed rewards, human players can outperform deep RL, while feedback is limited and agents encounter unseen states. FRESH combines human feedback with neural-network generalization and reward shaping. It outperforms state-of-the-art deep RL on Bowling and Skiing, and its Bowling score exceeds the human expert’s by 21.4%.

  • Problem

    In high-dimensional settings with sparse or delayed rewards, humans can obtain higher rewards than deep RL, while limited feedback and unseen test-time states make generalization important.

    Source: §1 · p. 1, §3.2 · p. 3

  • Method

    FRESH uses human judgments on actions and states, neural networks to generalize feedback, and feedback-derived shaping reward alongside environment reward.

    Source: §1 · pp. 1–2, §3.2 · p. 3, §3.3 · p. 4

  • Results

    FRESH outperforms state-of-the-art deep RL in Bowling and Skiing; its average Bowling score of 187 is 21.4% higher than the human expert’s score.

    Source: §6 · p. 8

  • Takeaways & Limitations

    In the evaluated Atari games, qualitative human feedback can be integrated with environment reward to guide deep RL beyond state-of-the-art scores.

    Source: §1 · p. 2, §6 · p. 8

    The operator need not be an expert, but is assumed able to understand gameplay after explanation by an expert.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Reinforcement learning has been successful in training autonomous agents to accomplish goals in complex environments. Although this has been adapted to multiple settings, including robotics and computer games, human players often find it easier to obtain higher rewards in some environments than reinforcement learning algorithms. This is especially true of high-dimensional state spaces where the reward obtained by the agent is sparse or extremely delayed. In this paper, we seek to effectively integrate feedback signals supplied by a human operator with deep reinforcement learning algorithms in high-dimensional state spaces. We call this FRESH (Feedback-based REward SHaping). During training, a human operator is presented with trajectories from a replay buffer and then provides feedback on states and actions in the trajectory. In order to generalize feedback signals provided by the human operator to previously unseen states and actions at test-time, we use a feedback neural network. We use an ensemble of neural networks with a shared network architecture to represent model uncertainty and the confidence of the neural network in its output. The output of the feedback neural network is converted to a shaping reward that is augmented to the reward provided by the environment. We evaluate our approach on the Bowling and Skiing Atari games in the arcade learning environment. Although human experts have been able to achieve high scores in these environments, state-of-the-art deep learning algorithms perform poorly. We observe that FRESH is able to achieve much higher scores than state-of-the-art deep learning algorithms in both environments. FRESH also achieves a 21.4% higher score than a human expert in Bowling and does as well as a human expert in Skiing.

1 INTRODUCTION

FRESH addresses the difficulty of learning in high-dimensional environments by combining human feedback with deep reinforcement learning. It uses neural networks to generalize feedback and estimate confidence, then shapes rewards while retaining the environment reward.

1 INTRODUCTION

  • Humans can obtain higher rewards with fewer actions than RL algorithms, particularly in high-dimensional games with sparse or delayed rewards.Prior work cited by the authors found humans could win where deep RL struggled.Source: §1 · p. 1
  • FRESH trains from human judgments of actions and sometimes states in replay-buffer trajectories, converting qualitative feedback into shaping reward alongside the environment reward.The feedback neural network generalizes the judgments to unseen states and actions.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Unlike Deep-TAMER [45], FRESH retains environment reward and asks operators for qualitative good-or-bad judgments rather than numerical behavior ratings.Deep-TAMER instead used human feedback in a supervised-learning framework without environment reward.Source: §1 · p. 2
  • Because feedback is limited and test-time states may be unseen, FRESH uses neural networks to generalize feedback and confidence thresholds to reject low-confidence outputs.The paper favors bootstrapped ensembles over dropout for lower inference-time complexity.Source: §1 · p. 2

1.1 Contributions

  • On the Bowling and Skiing Atari games, FRESH outperforms state-of-the-art DRL algorithms in both: it averages ∼180 versus a human expert’s ∼150 in Bowling, and performs as well as an expert in Skiing.Source: §1.1 · p. 2

2 REINFORCEMENT LEARNING

The section defines the MDP setting and the quantities used to describe an RL agent’s objective and policy. It also introduces Q-values, advantages, and common exploration strategies.

2 REINFORCEMENT LEARNING

  • An MDP is defined by states, actions, transition probabilities, an initial-state distribution, and rewards for transitions.The transition function encodes the probability of the next state given the current state and action.Source: §2 · p. 2
  • An RL agent seeks a policy that maximizes expected discounted reward over trajectories, without direct access to transition probabilities or rewards.Policies may be deterministic or return probabilities over actions.Source: §2 · p. 2
  • The Q-function values state-action pairs; subtracting a state’s value gives an action’s advantage, and Q-learning updates Q-values.Early exploration strategies include ε-greedy and Boltzmann exploration.Source: §2 · pp. 2–3

3 FEEDBACK-BASED REWARD SHAPING

FRESH addresses how to generalize limited human feedback in high-dimensional states and integrate it with deep reinforcement learning. It trains feedback networks from binary judgments and uses their predictions to shape rewards; in Bowling and Skiing, it performs at least as well as a human expert.

3 FEEDBACK-BASED REWARD SHAPING

  • FRESH performs at least as well as a human expert in Bowling and Skiing.These games are examples where human experts still outperform state-of-the-art deep reinforcement learning algorithms.Source: §3 · p. 3

3.1 Binary-valued Human Feedback

  • Human operators provide binary good/bad feedback on actions and states; action judgments are local, while state judgments reflect rewards obtainable from that state.Operators may also choose “not sure” or skip the rest of a trajectory; those signals are not used to train the agent and can reduce feedback time.Source: §3.1 · p. 3

3.2 Generalizing Human Feedback

  • Feedback neural networks generalize judgments to unseen states and actions, while an ensemble of bootstrapped heads provides prediction uncertainty.The shared network architecture uses separate heads for state and action feedback; prediction spread across heads serves as a confidence proxy.Source: §3.2 · pp. 3–4

3.3 Reward Shaping

  • FRESH converts confident action and state predictions into additional rewards and adds weighted shaping rewards to the environment reward.Action shaping rewards agreement with the predicted best action; state shaping rewards predictions that the next state is good.Source: §3.3 · p. 4

3.4 Algorithm

  • The procedure initializes feedback networks from human-labeled random-play trajectories, then trains a DQN-based value learner with the shaped reward while periodically collecting feedback and updating the networks.Trajectories are sampled from the replay buffer for operator review, and feedback updates use stochastic gradient descent.Source: §3.4 · pp. 4–5

4 EXPERIMENTAL EVALUATION

FRESH is evaluated on Bowling and Skiing against deep reinforcement-learning algorithms and human experts, then examined through Bowling ablations of feedback type, ensemble size, and confidence thresholds. It exceeds human performance in Bowling and matches it in Skiing.

4.1 Game Description and Experiment Setup

  • Bowling delays a strike reward until the next turn, while Skiing provides its gate reward only at game end, making credit assignment difficult in both games.Source: §4.1 · p. 5
  • DQN follows with double Q-learning [42]; FNN shares DQN convolutional layers and uses fully connected heads with batch normalization.Source: §4.1 · p. 5

4.2 FRESH for Bowling and Skiing

  • FRESH uses 500 Bowling and 5,000 Skiing feedback signals, collects feedback every 30 episodes, and updates its FNN when 300 new signals are available.Source: §4.2 · p. 6
  • Each curve averages five runs; comparisons include Double-DQN [42], IMPALA [8], Rainbow [16], a human expert, and the FNN used alone as a policy.Source: §4.2 · p. 6
  • 187 Bowling average (21.4% above a human expert), and −4400 Skiing average (equal to a human expert); FRESH outperforms state-of-the-art deep RL in both games.In Bowling, the highest score exceeds 200; in Skiing, FRESH also outperforms the policy induced by the FNN.Source: Figure 4 · p. 6, Figure 5 · p. 6, §4.2 · p. 6

4.3 Discussion: Bowling

  • Action feedback alone or combined with state feedback reaches super-human Bowling performance, while state-only feedback does not yield high reward.Source: Figure 6 · p. 7, §4.3 · p. 7
  • Ten FNN heads give Bowling’s highest average reward and lowest variability; no ensemble underperforms the expert, while too many heads complicate training and too few weaken filtering.Source: Figure 7 · p. 7, §4.3 · p. 7
  • βa=1.0 and βs=0.02 yield Bowling’s highest reward and lowest variance, retaining about 97% of action outputs and 85% of state outputs.Other tested thresholds degrade performance when they reject or accept too many outputs.Source: Figure 9 · p. 7, §4.3 · pp. 7–8, Figure 10 · p. 7

5 RELATED WORK

Related work explores several ways to use human guidance in reinforcement learning, including reward shaping, preferences, policy feedback, and demonstrations. These approaches differ in the form of feedback and the assumptions they require.

  • TAMER extensions use human feedback to shape learning, while Deep-TAMER trains by supervised learning without the environment reward and DQN-TAMER models observer characteristics.Source: §5 · p. 8
  • Potential-based reward shaping can preserve optimal policies, but its typically pre-specified potential functions have restricted its use to tabular or low-dimensional state spaces.Source: §5 · p. 8
  • Preference-based RL translates human comparisons of trajectories into scalar rewards, but assumes a total order over trajectories.Source: §5 · p. 8
  • COACH interprets human feedback as an unbiased advantage estimate, and Deep-COACH extends this approach to large state spaces.Source: §5 · p. 8
  • Policy shaping treats feedback as action-optimality labels used to infer a human’s belief about the optimal policy; extensions address attention and uncertain feedback.Source: §5 · p. 8
  • Imitation approaches use demonstrations or expert-provided actions to guide learning, with CHAT and HG-DAGGER modeling uncertainty in demonstrations or predicted performance.Source: §5 · p. 8

6 CONCLUSION

FRESH combines a feedback neural network and an ensemble to shape deep RL with human feedback, and is evaluated on Bowling and Skiing. It outperforms state-of-the-art deep learning algorithms in both games, exceeding a human expert’s Bowling score and matching the expert in Skiing.

  • FRESH averaged 187 in Bowling, 21.4% higher than a human expert, and −4400 in Skiing, equal to a human expert.Source: §6 · p. 8
  • FRESH uses a feedback neural network to generalize human feedback and an ensemble of neural networks to represent confidence in its output.Source: §6 · p. 8
  • FRESH performed much better than state-of-the-art deep learning algorithms in Bowling and Skiing.Source: §6 · p. 8

Related papers

Loading 2001.06781v1…